Overview
This research investigates the performance of language model security agents, moving beyond traditional success rate metrics to incorporate cost considerations. The study focuses on evaluating these agents in both offensive and defensive cybersecurity scenarios, specifically using Cybench challenges for offensive tasks and Splunk BOTS v1 investigation challenges for defensive operations. A key aspect of the evaluation is the decomposition of agent performance based on inference spend and tool spend, comparing models at fixed cost levels rather than solely reporting best-case success.
Research Context
Current evaluations of security agents often prioritize peak offensive capability, with an emphasis on vulnerability discovery, exploit development, penetration testing, and Capture The Flag (CTF) completion. These measurements typically operate under generous inference budgets. While acknowledged as useful, the source highlights that such evaluations are incomplete in an operational security context. In real-world security operations, every reasoning step, tool call, telemetry query, and enrichment request consumes a budget. This economic reality underpins the study's focus on cost-aware evaluation.
Approach
The study implemented a cost-success evaluation framework for language model security agents. This framework assesses performance against two distinct types of security challenges:
- Offensive Tasks: Utilized Cybench challenges, which are framed as CTF-like scenarios.
- Defensive Tasks: Employed Splunk BOTS v1 investigation challenges, designed to simulate Security Operations Center (SOC) investigations.
Instead of focusing exclusively on maximum success rates, the methodology involved:
- Comparing models at predefined fixed cost levels.
- Decomposing performance metrics to distinguish between inference spend (the cost associated with the model's reasoning processes) and tool spend (the cost associated with external tools, telemetry queries, and data enrichment requests).
The research presents its results through an interactive website, https://evals.frontier.security.
Findings
The evaluation revealed distinct scaling regimes for red-team (offensive) and blue-team (defensive) tasks:
- Offensive CTF Performance: Performance in offensive CTF scenarios demonstrably improves with an increase in test-time compute. The study found that scaled open-weight models could achieve performance levels comparable to frontier proprietary systems while maintaining cost competitiveness.
- Defensive SOC Investigation: Success in defensive SOC investigation tasks did not scale in the same manner as offensive tasks. Instead, defensive success was observed to be more heavily dependent on disciplined tool use, effective telemetry navigation, and selective data enrichment, rather than solely on the magnitude of the raw reasoning budget provided to the agent.
Why This Matters
The researchers argue that security-agent benchmarks should incorporate measures of economic efficiency and operational fit in addition to traditional task success metrics. Cost-aware evaluations, particularly those aligned with SOC-native operational constraints, can offer a more practical understanding of which security models are currently useful and where defensive agents require further development.