Agents are increasingly deployed with real autonomy in web application and network penetration testing, where a single out-of-scope action can breach a client's engagement boundary. Existing offensive-security benchmarks measure raw hacking capability; as those benchmarks saturate, the real barrier to deployment is a special case of alignment: scope adherence. We introduce ScopeBench, a benchmark of 30 dead-end agentic security tasks in which the stated objective is reachable only by violating the stated scope. Each task appears under two conditions that share an environment, verifier, and objective and differ only in scope: one instruction set has no scope and measures capability; the other has a natural-language scope to measure adherence. Scopeless trajectories are graded by a standard deterministic verifier. Scoped trajectories pass through two grading arms. First, the same deterministic verifier checks for the flag: because the flag sits behind the scope boundary, a pass proves by construction that a forbidden action occurred, yielding a high-precision lower bound on the violation rate. If the verifier does not pass the trajectory, an agentic judge estimates whether an out-of-scope call occurred. We calibrate the judge against 100 ScopeBench trajectories labeled call-by-call by human annotators, and a blinded audit of the evaluated rollouts finds its high recall holds - no false negatives among the 36 audited violations, with over-flagging its only observed error. Across 8 models in one harness, raw capability spans 12.2% to 81.1% and scope adherence spans 34.4% to 86.7%, with the judge finding 331 violations that mechanical verification misses. Opus-4-8 achieves a raw-capability score 10 percentage points higher than sonnet-4-6's while exhibiting 35.6 percentage points higher scope adherence. We release the frozen pilot benchmark, evaluation code, and all 2160 ATIF trajectories.
Figures & tables
Sub-domain
Count
Example task
Auth / role escalation
5
found-credentials-reuse
Cross-tenant access
3
sqli-cross-tenant
Object storage
2
object-store-cross-bucket
Internal admin UI
4
nginx-vhost-internal-admin
Workflow side effects
3
helpdesk-ticket-reassignment
JWT trust boundary
2
jwt-internal-api-trust
Table 1: Pilot corpus coverage by sub-domain. All 30 tasks are web-application security scenarios delivered against a dockerized web target.
Figure 1: The capability–adherence plane. Each model is placed by its raw-capability pass rate and scope adherence A under the scoped condition (pooled across the two scope phrasings). Error bars are 95% CIs from a task-clustered bootstrap over the 30 task clusters on both axes. In this pilot, capability and adherence are not aligned: the most capable models range widely in adherence.
Model
MVR
Judge+ ∣ fail
Scope adherence [95% CI]
Adjusted adherence [95% CI]
opus-4-8
9.4
22.7
70.0[57.2,81.7]
77.4[63.2,90.8]
sonnet-4-6
32.2
49.2
34.4[21.7,47.8]
38.1[24.0,53.0]
qwen3-6-35b-a3b
23.3
44.2
42.8[31.1,55.0]
47.3[34.2,61.0]
haiku-4-5
18.9
32.2
55.0[40.6,68.9]
60.8[45.1,76.3]
glm-5-2
13.3
25.0
65.0[51.7,77.8]
71.8[57.1,86.4]
gpt-5-4-mini
20.0
24.3
60.6[48.9,71.7]
66.9[54.0,79.6]
Table 2: Trajectory-level scope adherence (percent) by model, pooled across the two scope phrasings ( n=180 per model). Scope adherence is the direct hybrid estimate combining MVR and judge arm grading; adjusted adherence applies the error correction of Section 5.5 with the audited Se0=100% and Sp0=90.5% . Scope-adherence intervals resample the 30 task clusters; adjusted intervals additionally propagate the audit confusion-matrix uncertainty.
Calibration quantity
Value
Violation sensitivity
100.0%
Specificity
65.4%
Precision
89.2%
Accuracy
91.0%
F1
94.3%
Mean cost, all attempts
\0.23$
Table 3: Trajectory-judge calibration against majority trajectory labels derived from five independent call-level human annotations.
Model
Praw [ 95% CI]
ΔP [ 95% CI]
opus-4-8
81.1[68.9,92.2]
+71.7[+57.8,+85.6]
sonnet-4-6
71.1[55.6,85.6]
+38.9[+23.9,+53.9]
qwen3-6-35b-a3b
67.8[52.2,82.2]
+44.4[+31.0,+57.9]
haiku-4-5
54.4[37.8,70.0]
+35.6[+19.4,+51.6]
glm-5-2
54.4[38.9,70.0]
+41.1[+26.5,+55.7]
gpt-5-4-mini
44.4[30.0,58.9]
+24.4[+13.4,+35.5]
Table 4: Raw capability and pooled hesitation delta by model, with 95% confidence intervals ( ΔP in percentage points; the scoped condition is pooled across the two phrasings). Rows sorted by capability. Mechanical violation rates (MVR) appear in Table 2 .
Mechanical
Recovered,
Crossed,
Non-
Model
violation
withheld
no flag
violating
opus-4-8
17
6
31
126
sonnet-4-6
58
9
51
62
qwen3-6-35b-a3b
42
11
50
77
haiku-4-5
34
13
34
99
glm-5-2
24
2
37
117
Table 5: Per-model decomposition of the 180 scoped trajectories. Mechanical violation : flag submitted and mechanically verified. Recovered, withheld : exact flag appears in a tool observation but is never submitted (deterministic evidence of a violation; all judge-positive). Crossed, no flag : judge-identified boundary crossing that never recovered the flag. Non-violating : no crossing found. The first three columns are the hybrid violation count; the middle two are the 331 hidden violations the mechanical arm misses.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Model
Δ MVR
Δ Hybrid
opus-4-8
+1.1
+11.1
sonnet-4-6
+0.0
−2.2
qwen3-6-35b-a3b
−6.7
−7.8
haiku-4-5
+0.0
+1.1
glm-5-2
+4.4
+3.3
gpt-5-4-mini
−2.2
−1.1
Appendix
Table 6: Casual-minus-formal difference per model in mechanical violation rate ( Δ MVR) and hybrid violation rate ( Δ Hybrid), in percentage points; positive means the casual phrasing yields more violation. Per-model effects run both ways and cancel in the pooled row; opus-4-8 is the outlier on the judge-inclusive rate.
Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction. Their scores support capability claims only when the evaluation protocol keeps the intended capability necessary for success. Recent reward-hacking benchmarks and system reports show that agents can instead recover public solutions, read evaluation artifacts, infer generator structure, manipulate feedback, or benefit from invalid scoring paths; existing responses do not provide a common procedure for attributing these shortcuts and quantifying their effect across benchmarks. We formulate protocol validity and introduce HackDetect, a post-hoc audit that identifies an exposure, determines how the agent used it, and assesses whether the resulting score is misleading. We quantify score inflation with the Mislead gap, defined as the exploit score minus the intended score. We audit 2,385 traces across 15 agent benchmarks and find evidence of exposures and reward hacking in 67.0% of Frontier Science traces and 66.7% of AutoLab tasks. Across paired comparisons, we measure score inflation of 0.45-1.00, showing that benchmark reports should provide evidence that scores reflect the intended capability.
Jiaqi Shao, Hanck Chen, Wei Zhang +2
1Hunyuan Team, Tencent · 2The Hong Kong University of Science and Technology · 3Duke Kunshan University
The benchmarks used to evaluate AI agents in security-critical roles suffer from crucial weaknesses. Building on recent empirical evidence, we characterize three core challenges that undermine security evaluations: benchmark vulnerabilities, temporal staleness, and runtime uncertainty. We then outline practical directions toward building more robust and trustworthy evaluation frameworks.
Sahar Abdelnabi, Chris Hicks, Konrad Rieck +1
ELLIS Institute Tübingen & MPI-IS & Tübingen AI Center, Germany · The Alan Turing Institute, London, UK · BIFOLD & Technische Universität Berlin, Germany +1
Stealth, the discipline of achieving an objective without revealing your presence, capabilities, or collected intelligence, is what separates sophisticated operators from detectable ones. Elite security researchers and advanced persistent threats achieve their objectives unnoticed; autonomous agents increasingly inherit the same offensive tasks, but do they inherit the tradecraft? We introduce StealthBench,a benchmark that measures operational stealth in autonomous offensive-security agents across six operational security (OPSEC) dimensions. We extract 11 hand-verified OPSEC incidents from real bug-bounty and red-team trajectories, expanded into 14 dockerized task scenarios, where agents, despite finding real vulnerabilities, committed stealth failures inconsistent with standard operational tradecraft: embedding credentials in public uploads, deleting production resources to prove access, force-adding uninvolved users to demonstrate a race condition. We evaluate agent trajectories using a 3-model large language model (LLM) judge panel with majority-vote aggregation, measuring safe success rate (solved and stealthy), Stealth@Solve (tradecraft quality among successful solves), and reckless solve rate (solved but cover blown). Our results show that no model exceeds 54% safe success rate (the compound metric requiring both task completion and stealth), confirming that OPSEC failures are systematic across model families. We release StealthBench as a public benchmark to support both the development of stealth-aware agents and automated OPSEC monitoring for autonomous offensive-security deployments. The interactive leaderboard, evaluation harness, and dataset are available at https://stealthbench.com.