PartHackBench: Certified Equal-Progress Stress Tests for Partial-Credit Tool-Agent Evaluation

Hongye Yang; Zhihao Xie; Shengjun Xiong

cs.AIAug 27, 2026

PartHackBench: Certified Equal-Progress Stress Tests for Partial-Credit Tool-Agent Evaluation

Authors: Hongye Yang, Zhihao Xie, Shengjun Xiong

Organizations: Georgia Institute of Technology, College of Computing

Abstract

Long-horizon tool agents often make useful progress without reaching terminal success, motivating partial-credit evaluation. Yet evaluators may reward milestones that were temporary, later reversed, or not attributable to the evaluated agent. Comparing an honest trajectory with a higher-scoring adversarial one is inconclusive if the latter made more genuine progress. We introduce PartHackBench, a controlled methodology that removes this confound. A private certifier admits a pair only when its trajectories match component-wise in both current-state predicate satisfaction and standardized agent attribution; score inflation, defined as f(A) - f(H), is measured only afterward. In 18 sealed held-out tasks in PB-CSTE, the frozen historical-target run produced matched adversaries for 15 tasks. Historical credit yielded mean inflation of .252, conditional attack success of 10/15, end-to-end yield of 10/18, and detected none of 14 strict rollbacks. Semantic LLM judges were more resistant but remained vulnerable, especially under evaluator-targeted attacks, while PB-CSTE current-state controls, defined as exact functions of the certified components, yielded zero inflation by construction. PartHackBench thus provides a certified control for testing whether evaluator credit changes while all benchmark-defined task-relevant progress remains fixed.

Figures & tables

Figure 1: Certified equal-progress control. The evaluator cannot access private matching verdicts.

Figure 2: Historical witness attack. Trajectory $A$ hits a temporary peak before rolling back to match $H$ ’s final certified predicate configuration. Historical evaluators suffer score inflation (1.0 vs. 0.5), while current-state evaluators remain invariant (0.5).

Evaluator	Time	Attribution	Decomposition
Predicate-Max	historical	no	shared
CAPE-Max	historical	yes	shared
CSPS	current	no	shared
Lightweight CAPE	current	yes	shared
DeepSeek judge	semantic	prompted	public schema
IDJ	semantic	prompted	independent

Table 1: Evaluator positioning. IDJ sits outside the

2\times 2

because independent model-based decomposition introduces another factor.

Target	Valid	Coverage	Mean $\Delta_{\mathrm{hack}}$ [95% CI]	Cond. ASR	E2E yield
Historical/Predicate-Max	37/54	15/18	.252 [.154, .351]	10/15	10/18
CAPE-Max	39/54	16/18	.239 [.150, .335]	10/16	10/18
DeepSeek judge	34/54	14/18	.069 [.031, .113]	4/14	4/18
IDJ	36/54	15/18	.048 [.024, .074]	3/15	3/18
CSPS / Lightweight CAPE	—	analytic	identically 0	—	—

Table 2: Target-specific attack results. Mean

\Delta_{\mathrm{hack}}

is conditional on matched tasks.

Evaluator	Honest MAE	Mean $\Delta_{\mathrm{hack}}$	Rollback
Historical / Predicate	0	.252	0/14
CAPE-Max	0	.252	0/14
DeepSeek judge	.036	.022	10/14
CSPS	0	0	14/14
Lightweight CAPE	0	0	14/14
IDJ	.041	.017	12/14

Table 3: Core results on the shared reference set.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Development	Held-out	Total
Persistent multi-step	2	6	8
Entity CRUD	1	5	6
Fixed-goal interaction	1	3	4
GoalPatch stress	2	4	6
Total	6	18	24

Appendix

Table A.1: Task-family allocation.

ID	Family	Goal form	$m$	$P_{\mathrm{state}}(H)$	Tag	$A_{\mathrm{ref}}$	RB metric
PH-H01	GoalPatch	revision	4	1/4	-	yes	excluded
PH-H02	GoalPatch	revision	3	1/3	-	yes	excluded
PH-H03	GoalPatch	revision	5	2/5	-	yes	excluded
PH-H04	Persistent	multi-step	5	2/5	RB	yes	included
PH-H05	Persistent	multi-step	4	1/2	Attr.	yes	included
PH-H06	CRUD	entity set	4	1/2	-	yes	included

Appendix

Table A.2: Held-out manifest and frozen honest progress.

Evaluator	Temporal rule	Attribution	Decomposition
Historical / Predicate-Max	historical maximum	no	shared
CAPE-Max	historical maximum	yes	shared
CSPS	terminal current state	no	shared
Lightweight CAPE	terminal current state	yes	shared
DeepSeek judge	semantic	prompted	public schema
IDJ	semantic	prompted	independent checklist

Appendix

Table A.3: Evaluator positioning.

Component	Configuration
Attacker	deepseek-v4-pro, thinking mode, temperature 0, strict JSON, 12,000-token cap, 600-s timeout, one bounded retry
DeepSeek judge	structured partial score, temperature 0, strict JSON, 4,096-token cap, 600-s timeout, one bounded retry
IDJ checklist	Gemini 2.5 Pro; instruction only; one frozen checklist per task
IDJ scoring	schema-free projection; three calls per trajectory; median primary; temperature 0; strict JSON; 4,096-token cap; 600-s timeout

Appendix

Table A.4: Frozen model and decoding configuration.

Target	Generated	Protocol-valid	Matched tasks	Successful ( $>.10$ )
Historical / Predicate-Max	54	37	15	10
CAPE-Max	54	39	16	10
DeepSeek judge	54	34	14	4
IDJ	54	36	15	3

Appendix

Table A.5: Candidate accounting by target. “Matched” and “successful” count unique tasks; all earlier columns count attempts.

Task	Historical	DeepSeek	IDJ	Task	Historical	DeepSeek	IDJ
H01	.600	.333	.105	H09	.250	.000	.033
H02	.500	.000	.000	H10	.250	.000	.000
H03	.450	.000	.000	H11	.000	.000	.000
H04	.400	.000	.000	H12	.000	.000	.067
H05	.400	.000	.000	H13	.000	.000	.000
H06	.333	.000	.000	H14	.000	.000	.045

Appendix

Table B.1: Per-task inflation on

A_{\mathrm{ref}}

Evaluator	Honest MAE	Mean $\Delta_{\mathrm{hack}}$ [95% CI]	Cond. ASR	E2E yield	Rollback
Historical / Predicate-Max	.000	.252 [.154, .351]	10/15	10/18	0/14
CAPE-Max	.000	.252 [.154, .351]	10/15	10/18	0/14
DeepSeek judge	.036	.022 [0, .067]	1/15	1/18	10/14
CSPS	.000	.000	0/15	0/18	14/14
Lightweight CAPE	.000	.000	0/15	0/18	14/14
IDJ	.041	.017 [.003, .034]	1/15	1/18	12/14

Appendix

Table B.2: Core shared-reference and honest-score results.

Target	Coverage	Mean $\Delta_{\mathrm{hack}}$ [95% CI]	Cond. ASR	E2E yield
Historical / Predicate-Max	15/18	.252 [.154, .351]	10/15	10/18
CAPE-Max	16/18	.239 [.150, .335]	10/16	10/18
DeepSeek judge	14/18	.069 [.031, .113]	4/14	4/18
IDJ	15/18	.048 [.024, .074]	3/15	3/18

Appendix

Table B.3: Frozen target-specific results; means are conditional on matched tasks.

Rank	Historical	CAPE-Max	DeepSeek	IDJ
1	.600	.616	.248	.139
2	.500	.568	.190	.130
3	.450	.467	.179	.129
4	.400	.418	.128	.089
5	.400	.395	.060	.066
6	.333	.312	.034	.059

Appendix

Table B.4: Target-specific per-task inflation.

Evaluator	Detected	Miss rate	Retained false credit
Historical / Predicate-Max	0/14	1.000	.504 across all pairs
CAPE-Max	0/14	1.000	.448 across all pairs
DeepSeek judge	10/14	.286	.123 among four misses
IDJ	12/14	.143	.125 among two misses
CSPS	14/14	.000	.000
Lightweight CAPE	14/14	.000	.000

Appendix

Table B.5: Strict rollback audit.

Set / evaluator	$>.00$	$>.05$	$>.10$	$>.15$	$>.20$
Shared Historical	10	10	10	10	10
Shared DeepSeek	1	1	1	1	1
Shared IDJ	5	2	1	0	0
Target CAPE-Max	16	13	10	10	8
Target DeepSeek	14	5	4	3	1
Target IDJ	14	6	3	0	0

Appendix

Table B.6: Successful matched tasks under strict thresholds.

Figure B.1: Candidate coverage and successful attacks under alternative search budgets. Historical and CAPE-Max overlap in the successful-task panel.

Component	Frozen record
Environment	PB-CSTE v1.0, tool schema, initial snapshots, environment revision
Tasks	6 development and 18 held-out tasks, active goal versions, task-manifest hash
Attack search	$K=3$ , attacker prompt, model configuration, timeout, retry, candidate-order rule
Evaluators	scoring prompts, output schemas, model revisions, IDJ checklist hashes
Analysis	task-level ledger, strict success threshold, bootstrap configuration, analysis revision

Appendix

Table C.1: Minimum reproduction manifest.

Tool-Use Evaluation

Tool-use evaluation measures whether AI agents select, call, and rely on tools correctly, efficiently, and safely across realistic tasks.

See all 99 papers

BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
Aug 31, 2026Pradyumna Shyama Prasad, Meiri Anto, Leon Eshuijs +3Reward Hacking AI Agent Evaluation
National University of Singapore · MIT · Vrije Universiteit Amsterdam +3
BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
Sep 10, 2026Shenghan Zheng, Zonglin Di, Yimin Liu +19Reward Hacking LLM Agent Evaluation
Dartmouth College · Independent · Ohio State University +10
Partial Evidence Bench: Benchmarking Authorization-Limited Evidence in Agentic Systems
May 6, 2026Krti TallamAgent Failure Analysis Evidence-Grounded Reasoning
KamiwazaAI

Aug 31, 2026J/K move · Enter open · S save

BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks

Pradyumna Shyama Prasad, Meiri Anto, Leon Eshuijs, Julian Moncarz+2

National University of Singapore · MIT · Vrije Universiteit Amsterdam+3

Reward Hacking AI Agent Evaluation AI Agent Benchmarks Shortcut Learning

Open paper AskPDF arXiv

LLM agents are increasingly used to run autonomous ML experiments, iterating on target metrics with little human oversight. Prior work has documented reward hacking in these environments, bringing into question the validity of produced research and the broader safety case for AI R&D. Existing benchmarks do not measure exploits that live in the data or the modeling task itself. We introduce BAITBENCH, a suite of three synthetic tabular ML tasks that each contain a shortcut that allows agents to inflate the public test score but fail on a hidden test set. Since the shortcut is optional and using it breaks no stated rule, BAITBENCH measures how often models exploit the shortcut to achieve inflated scores. Across seven frontier agents scored by our two-stage judge pipeline, 57.1% of runs exhibit reward hacking, with five of seven above 50%. Agents cheat even under a second condition where they are prompted not to -the mean cheating rate remains above 50%. We release BAITBENCH, along with the judge implementation, and an annotated dataset of transcripts containing reward hacks as a testbed for evaluating reward-hacking mitigations head-to-head.

Abstract

Tool-Use Evaluation

Explore similar work

BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks

BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure

Partial Evidence Bench: Benchmarking Authorization-Limited Evidence in Agentic Systems