Introducing DefendBench: Measuring the SecOps Work That Matters

There's no shortage of benchmarks that measure how well AI models can find vulnerabilities, write exploits and attack systems. Far fewer measure how well they can detect and respond to those same attacks. DefendBench is built to fill that gap: it grades agents on the defensive work security teams do every day.

The results

The best model passes 31% of DefendBench's 100 tasks.

Figure 1. Share of DefendBench tasks each model and agent harness passes, or that score against average API cost per task. Every pair gets the same tasks, environment and grader.

The 100 tasks span five disciplines: threat research, detection authoring, detection tuning, detection health and maintenance, and SecOps data engineering. Two more, threat hunting and investigation, are coming soon.

Explore the interactive leaderboard →

Why we built DefendBench

DefendBench measures what the average SecOps professional does on a Tuesday morning.

Every new generation of AI security agents gets evaluated on benchmarks. Most of them measure whether an agent can answer cybersecurity questions, solve CTF challenges, classify ATT&CK techniques, or reconstruct a historical intrusion.

Those benchmarks are useful. They measure knowledge, reasoning, and technical problem solving. They don't answer the question we actually care about.

The question: Can an agent independently produce a production-worthy security outcome in an unfamiliar enterprise environment?

That question became the foundation for DefendBench.

Measuring work, not questions

A security operations program is rarely limited by an analyst's ability to answer trivia or solve isolated puzzles. The day-to-day work is operational.

Teams spend their time understanding new threats, deciding whether those threats are relevant to their organization, evaluating existing coverage, finding telemetry gaps, writing and tuning detections, validating them against realistic data, cutting false positives, and keeping the whole SecOps program healthy.

Those activities determine whether an organization becomes more secure. They're also the work organizations actually pay security teams to do.

Real security operations work spans many analytical and engineering disciplines at once. Detecting and responding to threats in an enterprise means understanding how the technology under attack actually works, from software-defined networking to endpoint operating systems to SaaS and cloud-native infrastructure. It means knowing the tradecraft of the attack techniques themselves, and all the ways they can be observed in telemetry. It means knowing the quirks of logging configurations and of the detection tools and products, and what each one can and can't do. And it takes statistics and data science to come up with ways to catch those behaviors in mountains of data.

The problem: Existing benchmarks are insufficient to capture the real work that security operations professionals do.

Existing cyber-agent benchmarks mostly evaluate whether an agent can answer questions, solve CTFs, or reconstruct bounded historical incidents. Each one tests a real skill, but none tests the whole job, and none puts the agent in an environment that looks like a real enterprise.

DefendBench asks something different.

What DefendBench measures: Can an agent do economically valuable SecOps work: understand an ambiguous customer objective, reason over a realistic enterprise environment, make expert decisions, and produce validated operational outcomes?

Instead of optimizing for the number of questions, DefendBench is built around tasks. Each one starts with a request a security team might really receive:

• What is our coverage for AI agents being abused?
• We just onboarded a new telemetry source. What threats can we now detect?
• TeamPCP launched a new campaign yesterday. Is it relevant to us, and if so, what do we need to do?
• Our firewall logs stopped parsing after last week's upgrade. Find what broke and fix it without losing events.

These are open-ended engineering problems. There is rarely a single correct answer, and success depends on making a sequence of good decisions under uncertainty.

The benchmark is also broad. We created 100 tasks for DefendBench v1 across five disciplines, and each one tests something different:

• Threat research: finding new and emerging threats, and breaking them down into behaviors that can be observed and detected.
• Detection authoring: finding the right logs, validating the fields, and writing rules that work in that specific SIEM.
• Detection tuning: cutting false positives without losing the true ones.
• Detection health and maintenance: finding rules that have silently stopped working, and figuring out why.
• SecOps data engineering: configuring pipeline tools, fixing broken parsers, and filtering and routing data so the SIEM gets what detections need.

Two more disciplines are coming soon:

• Threat hunting: finding active threats in the environment that existing detections missed.
• Investigation: taking an alert to a verdict, with scope and a timeline.

The tasks run on Splunk, Microsoft Sentinel, Elastic, Google SecOps, CrowdStrike Falcon Next-Gen SIEM and Databricks, and we deliberately spread them across more than 150 log types: AWS, GCP and Azure cloud logs, endpoint products, SaaS applications, identity providers, and even custom application logs.

Knowledge is becoming a commodity. Whether an AI system can produce a plausible answer is no longer the hard part. The hard part is whether it can repeatedly make expert decisions that improve a real security program. That's the standard DefendBench is designed to measure, because the goal isn't to build an AI system that looks senior. It's to build one security teams can rely on.

Realistic environments produce realistic evaluations

Security operations at enterprise scale is messy, and every organization's environment is different. Existing detections overlap, and some threats are impossible to detect with the data available. The reality is: tradeoffs between coverage and operational cost are unavoidable.

The key is capturing the idiosyncrasies and minutiae of each security tool and scenario. For example, every SIEM is wildly different, and I'm not just talking about the query language. In Splunk, one detection can depend on a CIM data model, a KV store lookup, and a stack of macros that hide half the logic. In Google SecOps, the same idea might be a composite rule built on YARA-L and UEBA functions. Sentinel, Elastic, CrowdStrike Next-Gen SIEM and Databricks each bring their own query language, rule schema and quirks. An agent that writes a perfect detection in one SIEM has to relearn the job in the next.

We took inspiration from Hex's DataBench, which tests analytics agents inside Shorelane, a fictional company with its own data and history. Ours is a fictional enterprise built for security work.

So we built simulated enterprise environments across Splunk, Microsoft Sentinel, Elastic, Google SecOps, CrowdStrike Falcon Next-Gen SIEM and Databricks, and the simulations go deep. We simulate each SIEM's entire API surface: search and query endpoints, rule management, lookups, saved searches, and the errors and edge cases the real product returns.

Each environment holds the artifacts a real deployment builds up over years: detection rules that are healthy, broken or overlapping, macros, data models, lookup tables and KV stores, reference lists, configuration settings, and index permissions. Behind all of it sits realistic telemetry and user behavior. When an agent calls the API, it gets what the production tool would return, quirks included.

Figure 2. Enterprise state (telemetry, detection rules and configuration) is replayed into each simulated product, and the agent works through that product's full API, the same way it would in production.

Under the hood, each simulation combines generative models with deterministic code. We've found that people are notoriously bad at designing environment conditions in a vacuum. They bring their own biases and leave room for assumptions that don't hold up in the real world. That's why every environment condition and fact we use comes from a real environment. Generative AI is excellent at helping us design scenarios and convert real ones into replayable form, while service mocks written in code simulate each product's actual API surface.

The hardest part is internal consistency. The captured state has to reflect the world accurately, down to how every entity, log and configuration relates to the others. A user who appears in an alert has to exist in the identity logs, own the device in the endpoint data, and hold the role the audit trail says they were granted.

That's also why we don't simulate an entire enterprise. We capture the state the expert's investigation actually touched, every log, entity and configuration along the path from question to answer, and simulate that slice faithfully and consistently with everything around it. There's rarely only one right way to reach an answer, though. One analyst starts from the identity logs, another from the endpoint data, a third from the rule library. So we also map the other reasonable routes an expert could take, and give those paths state too. An agent that reaches the correct answer by a different, sound route finds real data along the way instead of a dead end, and gets graded on its conclusion, not on whether it copied our steps.

Figure 3. We simulate the relevant state along the agent's path, a fraction of the enterprise, rather than the whole thing.

Why simulate instead of paying for and running real cyber range infrastructure? Real SIEMs are slow: one search over a week of data can take minutes, and a single task may need dozens of them. They are also never in the same state twice. Data ages out, rules get edited, and a log source that was flowing yesterday stops today, so the same task can have a different correct answer from one week to the next. Even injecting synthetic logs into a real SIEM continuously carries overhead in pipelines, licenses and drift. The highest-return option was to build a way to capture an environment's state and replay it indefinitely.

The result is a simulated training ground. Every run starts from exactly the same state, so results stay comparable across models and over time. Our own agents can run thousands of experiments in it by themselves, optimizing for cost, latency, design patterns and new architectures.

That's why we designed the tasks the way we did. Capture the real state, replay it faithfully, and grade the outcome against criteria an expert would sign off on. Nothing about that recipe is specific to one kind of work, so it generalizes to any cybersecurity task you want to measure and improve.

This is just the beginning. We believe the future of scaling expertise lies in designing and building environments that reflect economically valuable knowledge work, and we'll be sharing much more about ours here. How we built these simulations deserves a full write-up of its own, and we'll publish one in a separate post soon.

Methodology

When we design a task, the most important thing to capture is what makes it difficult and ambiguous. Expertise shows most clearly when information is incomplete. Anyone can follow a runbook when the data is clean and the question is precise. An expert is the person who still makes the right call when the logs are partial, the request is vague, and two plausible answers point in opposite directions. A task with the ambiguity sanded off measures knowledge, not expertise.

A seasoned practitioner's work product is the result of many small expert decisions chained together. We modeled the benchmark on the approach Harvey took for legal work, where agents operate over virtual data rooms and are graded against explicit criteria.

• Hand-built tasks. Our Research team designs every task to reflect economically valuable SecOps work our customers run into every day.
• Deliberate red herrings. Distractors are planted based on situations we've encountered in real environments: a rule that looks broken but isn't, a log source that looks relevant but doesn't carry the field you need.
• Criteria, not vibes. Each task carries a list of assertions that must all be true for the task to pass. Verifiers check them, some deterministically and some with an agent as judge.
• Same tasks, same grader. Every model runs the same tasks and is scored by the same verifiers. A pass means the output is correct.

The key ingredient is human expertise. For every task, an expert works the problem by hand in a real environment and writes the ground-truth answer themselves. Along the way, they record evidence for every fact, condition and observation that led to their conclusion: each query they ran and what it returned, each raw event they sampled, each piece of vendor documentation they checked. That evidence trail becomes the blueprint for the simulated environment. The simulation contains the conditions the expert actually encountered, along with the other reasonable routes an expert could have taken, and every agent is graded against conclusions a person proved.

Every task is designed on purpose

Most of the work in DefendBench goes into task design. Every edge case and scenario we run into in a real environment gets captured, anonymized, rebuilt inside our simulated enterprise, and replayed through our simulations, which we run as an environment as a service. A benchmark is only as hard as the reality it copies, so we try to copy all of it. Each discipline is hard in its own way.

Threat research. Take a request like "What's my coverage for TeamPCP?" A complete answer has to capture every distinct campaign attributed to the group and every procedure used in each one. Then it has to work out which procedures can be detected with the same logs and which need different ones, where procedures overlap, and whether there is real evidence the group used each procedure at all. Then comes observability: can each procedure actually be seen in osquery, in Windows Security events, in Sysmon and in CrowdStrike FDR logs, or only in some of them? Emerging threats add a clock. When a new attack breaks, the question is how quickly an agent can determine every technique used, completely and accurately, and how quickly it can find sources on the web that are both credible and reliable, rather than repeating the first vendor blog or social post it finds. We define great threat research as capturing every procedure, accurately and completely, along with how each one is observed.

Threat modeling and attack path mapping. Some tasks start from the environment instead of a threat: given these systems, identities and trust relationships, how would an attacker get from an initial foothold to the most valuable assets, and which of those steps would the current detections actually see? That means reasoning over identity and access relationships, network reachability and cloud permissions at the same time, then mapping each step on the path to the telemetry that would record it.

Detection health. Deciding whether a rule is healthy is hard because a rule depends on far more than its own query. The log source has to keep flowing, the parser has to populate the fields, lookups and watchlists have to hold what the rule assumes, the schedule has to cover the lookback, and other rules may feed it. Any of those can break silently, and a rule that returns nothing may be broken or may be perfectly healthy.

Detection tuning. Tuning is hard because the benign behavior you need to remove often looks almost exactly like the attack you need to keep. Getting both precision and recall right means applying real statistics, such as baselines, rarity and per-entity thresholds, within the limits of a particular SIEM and its query engine, which decide what can even be computed in a scheduled search.

SecOps data engineering. Before anyone can detect anything, the right data has to arrive in the right shape. Much of that work now happens in data pipeline tools like Cribl and Bindplane, which sit between log sources and the SIEM. These tasks ask the agent to configure and repair that layer: fix a parser that broke after a vendor changed its log format, normalize a new source to the schema downstream rules expect, filter or sample a high-volume source to cut ingest cost without dropping the events detections depend on, and route each stream to the SIEM, the data lake or cold storage. A filter that saves money but silently removes the field a critical rule needs is a failed task.

Fast enough to iterate

Because we no longer depend on a live SIEM, compute is essentially the only bottleneck. Tasks that used to take hours or days now run in minutes. We can run far more experiments than before and compare new harnesses, design patterns, architectures, tools and models side by side.

Example tasks

Anyone can list task names. What makes a benchmark trustworthy is what sits behind each one. Here are two in full: a detection authoring task and a detection health task. Each one combines conditions we've seen in real environments.

Task 01 · Detection authoring: Tampering with source control

Difficulty: Hard

The request

Create detections for the following two use cases. Our GitHub audit logs are in Splunk, so use those for the GitHub side.

1. Unauthorized modification or deletion of an internal source artifact. A commit, tag, release or CI workflow file is modified, replaced, re-pointed or deleted in an approved repository outside the expected CI/CD process, or by an unexpected user or service account. Compromised source can silently inject malicious code into every pipeline that builds from it.

2. Anomalous access or deployment using source control or CI/CD service accounts. A service account or token is used from an unexpected IP, geography or time, or does unusual things such as accessing other repositories or deployment systems outside its normal build job.

The environment

The company keeps code in three places, the way large enterprises end up after a few acquisitions: GitHub Enterprise Cloud for most repositories, a self-managed GitLab inherited from an acquired company, and Azure DevOps for one business unit. Its security data is split across two SIEMs as well. Splunk holds the GitHub, endpoint and GitLab request data; Microsoft Sentinel, run by the business unit on Azure, holds the Azure DevOps and GitLab audit data. Together they contain eight sources that all look like source control data, and they see very different things.

How it's graded

Nine criteria, each graded pass or fail. Every one must pass for the task to pass.

1. C-01 · Inventories every source by its contents. Pass if the agent examines all eight sources and identifies the legacy GitHub connector as dead and the GitHub Actions source as CI job output, not audit data. Fail if it skips any of the eight sources, or treats either trap as a usable detection source.
2. C-02 · Reports the named GitHub source as blind to pushes. Pass if the agent states that the GitHub audit stream in Splunk doesn't record which ref or files a push changed, so tag moves and workflow edits can't be reliably detected there. Using branch-protection overrides, rejected updates or workflow-run events as partial signals is fine. Fail if it offers an audit-stream rule as coverage for tag moves or workflow-file changes.
3. C-03 · Finds the webhooks and scopes them as partial. Pass if the agent uses the GitHub webhook events for tag and workflow detection and states they cover only about 40 of roughly 600 repositories. Fail if it misses the webhooks, or presents them as full coverage.
4. C-04 · Detects workflow-file changes. Pass if the agent detects changes to CI workflow files from the webhooks' changed-file data, excluding dependency bots by authenticated identity. Fail if workflow-file changes aren't covered, or bots are excluded by display name.
5. C-05 · Scopes GitLab correctly. Pass if the agent uses GitLab request logs in Splunk for who pushed and from where, states they don't show which ref or files changed and miss pushes over SSH, and uses GitLab audit events in Sentinel for protection and release changes. Fail if it claims the request logs show which ref or files changed.
6. C-06 · Queries the right index and table. Pass if every rule reads its source from where that data actually lives: in Splunk, GitHub webhook events from index=scm_github sourcetype=github:webhook and GitLab request logs from index=scm_gitlab sourcetype=gitlab:requests; in Sentinel, Azure DevOps audit events from the ADOAudit_CL table and GitLab audit events from GitLabAudit_CL. Fail if any rule queries a guessed or wrong location, such as the dead legacy connector's index, the Actions job-log sourcetype, or a table that exists only in the other SIEM.
7. C-07 · Doesn't build on build output or client-side commands. Pass if no detection relies on CI job-log text or endpoint command lines as evidence of a server-side change. Fail if any rule does.
8. C-08 · Calls GitHub token coverage partial. Pass if the agent states GitHub tokens can be baselined on the hashed token, token type, user agent and country but not IP or network, because IP disclosure is off and bot and app actors never get IPs, builds GitLab per-account baselines on IP, user agent and time, and states that Azure DevOps doesn't audit token use. Fail if it proposes IP or network baselining for GitHub tokens.
9. C-09 · Recommends the logging fixes. Pass if the agent recommends an organization-level webhook (or a GitHub App) for push and release events covering every repository, tag rulesets and immutable releases for release tags, IP disclosure on GitHub, push telemetry for GitLab (group webhooks with the three-ref limit per push raised; their payloads list changed files for only the newest 20 commits of a push, so larger pushes need a follow-up API lookup) and Azure DevOps (a Code pushed service hook), and retiring the dead connector. Fail if any of these is missing.

Task 02 · Detection health: Determine if this detection rule is healthy or broken

Difficulty: Extreme

The request

Determine if this detection rule is healthy or broken, and if it's broken, fix it.

Gain code execution on ADFS server via remote WMI execution. A Microsoft Sentinel scheduled rule. It first finds ADFS servers by looking for the ADFS service-host process, then alerts when remote WMI activity reaches one of them: wmiprvse spawning rundll32 in a session opened by a network logon, or WMI event-subscription activity (Sysmon events 19 to 21). It reads Windows Security events, directly and as forwarded events, and Sysmon.

The environment

A single Microsoft Sentinel workspace with Windows Security events from a handful of servers, Microsoft Defender for Endpoint on thousands of devices, and Entra ID sign-in logs. The rule is enabled, runs daily, and hasn't fired in the past year. Nothing about it looks wrong from the rule list.

How it's graded

Six criteria, each graded pass or fail. Every one must pass for the task to pass.

1. C-01 · Verdict is Broken. Pass if the agent's verdict is Broken. Fail if it's Healthy or Warning, including Healthy because the rule never fired, or Warning because the Windows Security branch looks structurally sound.
2. C-02 · Finds the empty ProcessName. Pass if the agent states that ProcessName is empty on Windows Security process-creation events and the executable path is in NewProcessName. Fail if it treats ProcessName as correct.
3. C-03 · Finds that Sysmon is absent. Pass if the agent states that no Sysmon events are ingested into the workspace. Fail if it claims Sysmon data is present, or names a table that holds it.
4. C-04 · Moves the rule to Defender. Pass if the fixed rule reads Defender for both detection paths: process events joined to Defender logon events for the remote process path, and Defender WMI device events (subscription bindings or remote WMI operations) for the WMI path. A repaired Windows Security branch may stay alongside. Fail if the fix stays on Windows Security only, joins Defender process events to Windows Security logons, or offers nothing beyond onboarding Sysmon.
5. C-05 · Calls the rule fixable. Pass if the agent assesses the rule as fixable by rewriting the query. Fail if it calls the rule not fixable, for example because the Sysmon branch can't be repaired or because the fix can't be verified without a live ADFS event.
6. C-06 · Reads the zero correctly. Pass if the agent states that no ADFS server shows up anywhere in the environment, so a repaired rule that returns nothing is expected. Fail if it treats the empty result as a failed fix, or claims ADFS hosts exist.

What comes next

Knowing security isn't the same as doing security. DefendBench grades agents on the second: the coverage answers, rule fixes, and tuning changes a security team would actually ship. Today's best model passes fewer than a third of those tasks. That gap is what we're working to close.

It's helpful for our research team to keep DefendBench private, so the tasks don't end up in model training data and the benchmark stays an honest signal for our own development. That said, we plan to open-source the simulated enterprise environment behind it. A realistic public environment will help everyone building AI agents for security operations, a hard and important problem, and we're excited to share it.

We'll keep adding tasks and models. The DefendBench leaderboard has the latest results, broken out by discipline.

Want to see how an agent performs on your own environment? Connect Spectrum.