Detection Health: Knowing Which of Your Rules Would Actually Fire

A broken rule and a working rule look identical

Here's the question almost no security team can answer: what percentage of your detections work right now?

The reason it's hard is that detections are never finished. Teams identify threats and write rules to catch them, but what ships has many unstable, live dependencies:

• log schema
• source category
• lookup tables
• parsers
• SIEM configuration

All of those are living, breathing things that keep changing, frequently without giving anyone a heads up. For example, a vendor can start shipping logs with different naming at any time.

When a rule breaks that way, it doesn't throw an error. It either sits there failing in silence, or floods your queue with noise that makes it hard to sort the meaningful from the meaningless.

Neither of those behaviors even seems inherently "broken", and our default behavior as SecOps teams is to assume they aren't. Could a rule with no recent alerts just be a healthy rule with no relevant threats against it? In the same vein, could a noisy rule just be one that needs to be tuned?

Real-world data from Spectrum's Delta research team tells us that 29.2% of detection rules are broken on average, with some environments approaching 50% broken detections. That's probably a much bigger problem than you even realized.

The cost of broken detections

Unfortunately, the cost of broken detections is greater than an annoyance. Silent failures are the misses nobody learns about until an incident, and they are the expensive kind. On the other hand, noise buries the alerts that matter and burns the team down triaging a rule that was broken the whole time.

Then, once you do catch one, working out why is its own project. Root-causing a single rule means understanding data models, lookup tables, parsers, log schemas, and SIEM configuration. That janitorial work consumes SecOps teams, and the backlog never empties.

Detection-as-code won't save you

Plenty of teams have detection-as-code with tests in CI, and they should keep it. But those tests run at commit time, against assumptions about your data. They confirm the rule is valid, the logic parses, and the change merges cleanly.

Every failure described above happens months later, in production, where nothing is running tests. Your pipeline reports green while the field your rule depends on sits empty.

Finding those breaks means reasoning about live data against what each rule actually needs. That takes a brain, not a pipeline.

How it works

Spectrum treats every detection as a chain of dependencies, and checks the whole chain against your live environment.

• Reverse-engineer the rule. An agent reads each detection and works out the threat behavior it's trying to catch, then builds its dependency list: the behavior, the fields the rule needs, the values those fields have to carry, and the lookups it calls.
• Check the logic against the intent. If the detection logic doesn't match the behavior the rule is supposed to catch, that's a logic issue, and it comes back with a recommended fix.
• Validate every dependency against live data. Spectrum pulls your log schema catalog, confirms the source category is still delivering events, confirms each required field exists, is populated, and carries the expected type, and confirms external lookups are present and reachable.
• Run the rule over the last 30 days. Hits get checked as true positives for the intended behavior. No hits sends the rule through dependency validation, which is what separates a dormant rule from a dead one.

Every rule comes back with one of three verdicts:

• Healthy (active). The rule fired in the last 30 days and its hits validate as true positives for the behavior it was written to catch.
• Healthy (dormant). The rule hasn't fired, but every dependency it needs validates successfully. If the behavior occurred, this rule would trigger. You can leave it alone.
• Unhealthy. The rule would not trigger even if the behavior occurred, and Spectrum identifies the reasons.

Every unhealthy rule has a specific root cause, drawn from a documented taxonomy spanning detection logic, log source availability, schema and field mapping, and external dependencies. Here are real examples from customer environments, all found in the last two weeks:

• A threat intel feed's API key expired. The indicator list drained to empty, and the rule evaluated to false, cleanly, every run since.
• A schema change turned a single-value field into an array. The rule's exact match is comparing a value against a list, so it stopped matching.
• A field that used to arrive as a number started arriving as a string. Every greater-than comparison in the rule stopped evaluating correctly.
• A vendor began hashing usernames to meet a privacy requirement. The rule still keyed on that field, which carried a hash instead of a name.

Fixing rules

Once an unhealthy rule has been identified, Spectrum uses its research and environment context to propose a fix customized to your environment.

Here's an example of a real broken rule that Spectrum has analyzed, and suggested a fix for.

Our health-check agents are built to be experts in every SIEM we support, so the checks account for the nuances that actually break rules in practice: Splunk data models, lookup tables, parsers, log schemas, and SIEM configuration for whatever SIEM(s) you are using.

Every verdict carries the dependency list it was checked against, the check that failed, and the reasoning behind it, so your engineers can confirm a finding instead of taking it on faith. As always, humans make the final call on any suggested fix.

Impact

The first thing customers get with Detection Health is an answerable question. "What percentage of our detections work?" stops being a guess and becomes a real, measurable number, with research behind why they don't work.

The second is trust: today, analysts, managers, and leaders alike have an inherent feeling that their detections are broken. Security teams want to know they're right, when they tell their boss or board that they're covered.

The third is capacity. Root-causing a broken rule is expert, per-SIEM work, and it's a janitorial load that consumes detection teams and keeps their backlog full. Handing that work to agents puts your engineers back on detections that don't exist yet.

Getting started

Want to know what share of your detections would fire tonight? Connect Spectrum and find out.