Scalable Weak-to-Strong Verifier Hardening for Agent Environments

Chong Zeng; Ziqian Zhong; Kexun Zhang; Ivan Bercovich; Ivgeni Segal; Yuzhen Mao; Shashwat Saxena; Aditi Raghunathan

cs.CROct 11, 2026

Scalable Weak-to-Strong Verifier Hardening for Agent Environments

Authors: Chong Zeng, Ziqian Zhong, Kexun Zhang, Ivan Bercovich, Ivgeni Segal, Yuzhen Mao, Shashwat Saxena, Aditi Raghunathan

Organizations: Stanford University · Carnegie Mellon University · Fewshot Corp · Independent Researcher

Abstract

Agent environments are essential for large language model (LLM) development. They use programmatic verifiers to turn complex outcomes into rewards. Verifiers can be attacked into awarding full credit to cheating agents without completing the intended task, causing inaccurate evaluation and misalignment in training. As agent environments scale in numbers and difficulty, so must the repairing of exploitable verifiers. To this end, we ask whether today's model with scaled compute can automatically harden agent environments to prevent potentially stronger future models from reward hacking. We introduce a scalable, fully automatic approach to weak-to-strong verifier hardening. It first sets up an isolated verifier sandbox that agents cannot directly modify, then uses a hacker-fixer loop to find and repair remaining verifier vulnerabilities. We apply this loop to Terminal Bench, KernelBench, and SETA, substantially reducing attack success across these agent environments. The iterative design scales with the hardening budget, producing progressively stronger defenses as the number of loop iterations increases and outperforming the static, non-iterative baseline. An earlier version of our hacker-fixer loop has been adopted by the official Terminal Bench repository as a way to audit exploits in its environments. Project page: https://verifier-hardening.github.io/.

Figures & tables

Panel of Figure 1 — Figure 1: Scalable weak-to-strong verifier hardening. Fixed weaker models (e.g. GPT-5.6-Luna) can progressively harden verifiers against stronger agents (e.g. GPT-5.6-Sol). As the proposed hardening loop runs for more iterations, its advantage over the ablative baselines becomes more apparent: (a) attack success rate (ASR) from stronger agents decreases significantly, making the benchmarks more robust to reward hacking; (b) benign solve rates remain high indicating that legitimate solutions still pass the hardened verifiers. Moreover, as shown by (c) , we observe significantly reduced ASR by the hardening loop across various benchmarks and strong LLMs. All rates are per-trial (%).

Figure 2: Our hardening workflow. Automatic separation and a solver precheck establish the initial grading boundary. A verifier-aware hacker and a fixer then alternate search and repair, with patch review, legitimate-solution replay, and a fresh blind solver constraining accepted updates. Rejected candidates generate feedback while leaving the accepted environment state unchanged.

Evaluator	State	Benign $\uparrow$	Unhinted ASR $\downarrow$	Hinted ASR $\downarrow$	Verifier-aware ASR $\downarrow$
GPT-5.6-Sol	Before	97.1 / 90.5	68.6 / 43.5	99.0 / 88.2	98.0 / 82.4
	AutoSep	98.0 / 91.2	31.4 / 19.9	4.9 / 3.9	96.1 / 89.9
	After	99.0 / 92.5	12.7 / 7.8	3.9 / 2.3	61.8 / 49.3
Opus 5	Before	96.1 / 93.1	87.3 / 69.9	100.0 / 95.4	100.0 / 99.3
	AutoSep	99.0 / 94.4	47.1 / 30.1	27.5 / 18.3	99.0 / 96.1
	After	98.0 / 91.2	17.6 / 12.1	6.9 / 5.2	71.6 / 63.1

Table 1: Terminal Bench environments across hardening states. Benign solve rates and ASRs at Before, AutoSep, and After. All three evaluators use the same 102-task pool. Benign rates use rewards only; ASRs use Sol judgments.

Strategy	Benign $\uparrow$	Unhinted ASR $\downarrow$	Hinted ASR $\downarrow$	Verifier-aware ASR $\downarrow$
Original (shared)	96.1 / 93.1	87.3 / 69.9	100.0 / 95.4	100.0 / 99.3
AutoSep (iter 0)	99.0 / 94.4	47.1 / 30.1	27.5 / 18.3	99.0 / 96.1
Static-15, gated	98.0 / 93.8	38.2 / 24.5	22.5 / 15.4	87.3 / 83.3
Static-15, ungated	88.2 / 83.0	27.5 / 18.6	12.7 / 8.8	78.4 / 72.5
Full (final)	98.0 / 91.2	17.6 / 12.1	6.9 / 5.2	71.6 / 63.1

Table 2: Offline static versus online iterative repair on Terminal Bench . Per-task / per-trial benign solve rate (

\uparrow

) and judge-confirmed ASRs (

\downarrow

) on the 102-task cohort. Original is the pre-separation reference. Static-15 builds an offline attack bank against the initial AutoSep environment before one repair; Full interleaves online attack search with repair.

State	Benign $\uparrow$	Unhinted ASR $\downarrow$	Hinted ASR $\downarrow$	Verifier-aware ASR $\downarrow$
Original (shared)	96.1 / 93.1	87.3 / 69.9	100.0 / 95.4	100.0 / 99.3
0 (separated)	99.0 / 94.4	47.1 / 30.1	27.5 / 18.3	99.0 / 96.1
2	99.0 / 92.8	28.4 / 18.3	14.7 / 10.5	84.3 / 76.8
5	98.0 / 91.8	19.6 / 13.1	9.8 / 7.2	79.4 / 72.9
10	97.1 / 91.5	18.6 / 12.7	9.8 / 7.2	74.5 / 67.0
15 (final)	98.0 / 91.2	17.6 / 12.1	6.9 / 5.2	71.6 / 63.1

Table 3: Full-method performance on Terminal Bench . Per-task / per-trial success (benign

\uparrow

, ASR

\downarrow

) on the fixed 102-task pool.

Evaluator	State	Benign $\uparrow$	Unhinted ASR $\downarrow$	Hinted ASR $\downarrow$	Verifier-aware ASR $\downarrow$
GPT-5.6-Sol	Before	100.0 / 95.6	100.0 / 85.5	98.1 / 88.7	100.0 / 95.6
	AutoSep	100.0 / 98.7	0.0 / 0.0	0.0 / 0.0	100.0 / 96.9
	After	100.0 / 96.9	0.0 / 0.0	0.0 / 0.0	90.6 / 62.9
Opus 5	Before	100.0 / 98.7	98.1 / 93.1	90.6 / 67.9	100.0 / 98.1
	AutoSep	100.0 / 100.0	7.5 / 3.1	11.3 / 3.8	98.1 / 92.5
	After	100.0 / 98.1	0.0 / 0.0	0.0 / 0.0	88.7 / 81.1

Table 4: KernelBench environments across hardening states. Benign solve rates and judge-confirmed ASRs at Before, AutoSep, and After for three evaluators on 53 Level-1 tasks. AutoSep includes out-of-process timing isolation.

Evaluator	State	Benign $\uparrow$	Unhinted ASR $\downarrow$	Hinted ASR $\downarrow$	Verifier-aware ASR $\downarrow$
GPT-5.6-Sol	Before	96.1 / 94.2	89.0 / 69.0	99.4 / 93.1	96.1 / 80.6
	AutoSep	93.5 / 91.2	63.2 / 46.9	4.5 / 3.0	92.3 / 86.5
	After	96.1 / 90.3	31.0 / 20.2	2.6 / 1.9	82.6 / 66.2
Opus 5	Before	96.8 / 95.3	99.4 / 90.1	100.0 / 98.7	100.0 / 98.9
	AutoSep	98.1 / 96.3	76.8 / 62.8	54.2 / 39.1	98.1 / 95.1
	After	94.8 / 90.3	50.3 / 35.1	23.9 / 15.9	86.5 / 74.2

Table 5: SETA environments across hardening states. All three evaluators use 155 tasks, with three attempts per state and arm. Benign rates use rewards only; ASRs use Sol judgments.

Group	Configuration	Benign $\uparrow$	Unhinted ASR $\downarrow$	Hinted ASR $\downarrow$	Verifier-aware ASR $\downarrow$
	Original (shared)	96.1 / 93.1	87.3 / 69.9	100.0 / 95.4	100.0 / 99.3
	Full	98.0 / 91.2	17.6 / 12.1	6.9 / 5.2	71.6 / 63.1
Hardening capability	w/o verifier access	99.0 / 93.8	45.1 / 25.8	27.5 / 15.7	96.1 / 91.2
	Fixed output spec.	99.0 / 93.8	37.3 / 24.2	22.5 / 14.1	96.1 / 86.6
	w/o separation	96.1 / 91.2	53.9 / 33.3	94.1 / 82.0	97.1 / 92.8
Benign validity	w/o corpus replay	93.1 / 87.6	19.6 / 9.5	7.8 / 4.9	77.5 / 65.7

Table 6: Component ablations on Terminal Bench . Opus 5 per-task / per-trial results on the 102-task cohort (benign

\uparrow

, ASR

\downarrow

). Original is the pre-separation reference; the other rows report final outcomes.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Three example exploits on SETA task 1219 (headless multi-resolution virtual display).

Parameter	Terminal Bench / SETA	KernelBench
Model	GPT-5.6-Luna	GPT-5.6-Terra
Maximum iterations per hardening pass ( $K$ )	15	15
Attack attempts per iteration ( $B_{h}$ )	3	3
Additional fixer retries ( $F$ ) / maximum candidates	3 / 4	3 / 4
Attack reward threshold ( $\tau_{h}$ )	1.0	10.0
In-loop solver reward threshold ( $\tau_{s}$ )	1.0	0.5

Appendix

Table 7: Hardening models, budgets, and hyperparameters. Reward thresholds apply within the hardening loop.

Benchmark	Benign reward	Attack reward
Terminal Bench / SETA	$\geq 1$	$\geq 1$
KernelBench	$\geq 0.95$	$\geq 10$

Appendix

Table 8: Reward thresholds for held-out evaluation.

State	Full	w/o verifier access	Fixed output spec.	w/o separation
Pristine	87.3 / 69.9	87.3 / 69.9	87.3 / 69.9	87.3 / 69.9
Iter 0	47.1 / 30.1	47.1 / 30.1	47.1 / 30.1	87.3 / 69.9
Iter 2	28.4 / 18.3	44.1 / 26.8	44.1 / 29.1	78.4 / 60.8
Iter 5	19.6 / 13.1	43.1 / 24.2	41.2 / 27.5	65.7 / 48.7
Iter 10	18.6 / 12.7	43.1 / 24.2	40.2 / 24.8	57.8 / 38.9
Iter 15	17.6 / 12.1	45.1 / 25.8	37.3 / 24.2	53.9 / 33.3

Appendix

Table 9: Unhinted ASR across hardening capability ablations. Rates are per-task / per-trial (%), with Sol judgments.

State	Full	w/o corpus replay	w/o validation
Pristine	96.1 / 93.1	96.1 / 93.1	96.1 / 93.1
Iter 0	99.0 / 94.4	99.0 / 94.4	99.0 / 94.4
Iter 2	99.0 / 92.8	95.1 / 88.6	79.4 / 74.8
Iter 5	98.0 / 91.8	94.1 / 87.9	72.5 / 70.9
Iter 10	97.1 / 91.5	91.2 / 86.9	72.5 / 70.9
Iter 15	98.0 / 91.2	93.1 / 87.6	73.5 / 71.2

Appendix

Table 10: Benign solve rates across validity ablations. Rates are per-task / per-trial (%).

Benchmark	Configuration	Tasks	Total	Mean/task
Terminal Bench	Full	105	152.59	1.45
SETA	Full	155	255.66	1.65
KernelBench	Full	53	234.49	4.42
Terminal Bench	w/o verifier access	105	56.40	0.54
	Fixed output spec.	105	104.80	1.00
	w/o separation	105	134.12	1.28

Appendix

Table 11: Estimated hardening API costs (USD). Counts refer to unique launched tasks; mean cost is the total divided by this count.

Role	Inputs and conditional additions
Hacker	Original instruction; attack goal; current verifier source and output specification; within-round failure feedback; shared notes on KernelBench .
Fixer	Original instruction; attack evidence; current environment; boundary-repair instructions; reference and selected corpus examples; defense guidance; previous rejection evidence; shared notes on KernelBench .
Precheck solver	Original instruction plus reference-solution hint.
Fresh blind solver	Original instruction, without reference or verifier-source hints.
Diagnostic solver	Original instruction plus the failed candidate’s verifier source; result used only for failure feedback.
Patch reviewer	Original instruction, source context, and before/after patch evidence.

Appendix

Table 12: Prompt inputs by role.

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 517 papers

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops
Jun 8, 2026Ziqian Zhong, Ivgeni Segal, Ivan Bercovich +3Reward Hacking AI Agent Security Benchmarks
Carnegie Mellon University · Fewshot Corp · Independent Researcher
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Oct 1, 2026Caiqi Zhang, Rujun Han, Zifeng Wang +4LLM Agent Verification Long-Horizon LLM Agents
University of Cambridge · Google Cloud AI Research
VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
Oct 19, 2026Jianing Yin, Ziyi Chen, Hao Wang +4LLM Agent Verification AI Agent Benchmarks
University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley +2

Jun 8, 2026J/K move · Enter open · S save

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

Ziqian Zhong, Ivgeni Segal, Ivan Bercovich, Shashwat Saxena+2

Carnegie Mellon University · Fewshot Corp · Independent Researcher

Reward Hacking AI Agent Security Benchmarks AI Agent Benchmarks Automated Red Teaming

Open paper AskPDF arXiv

Agent benchmarks score submissions with outcome verifiers that are typically hand-written and brittle, leaving them open to reward hacking. We audit 1,968 tasks across five terminal-agent benchmarks and find 323 (16%) hackable by frontier models given only the task description. This corrupts both leaderboard rankings and RL training signal, yet the standard response is manual and reactive. We introduce the hacker-fixer loop, a method for building exploit-resistant verifiers without per-task manual patching. The loop alternates three LLM agents: a hacker tries to pass the verifier without solving the task, a fixer patches the verifier to reject each discovered exploit, and a solver confirms the patched verifier still admits legitimate solutions. The loop iterates: each patch reshapes what the verifier rewards, surfacing the next exploit. We further add verifier access, and let patches transfer across tasks, to broaden the exploits the loop discovers. On KernelBench, the loop drives the attack success rate from 62% to 0% on a held-out corpus of publicly reported exploits. We also find that weaker agents in the loop can defend against much stronger hackers: Gemini 3 Flash's loop drives the stronger Gemini 3.1 Pro and Claude Opus 4.7's attack success rate from 76% and 61% to 0% on KernelBench, and Gemini 3.1 Pro's from 39% to 17% on Terminal Bench across 77 tasks. We release Terminal Wrench (323 hackable environments, 3,632 hack trajectories) as a snapshot of the current attack surface, our patched verifiers, the exploits the loop discovered, and our implementation as a basis for future work.

Abstract

AI Agent Security

Explore similar work

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents