Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO

Shurun Li; Michael Hua Wang; Haibo Zeng; Long Wang

cs.LGOct 26, 2026

Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO

Authors: Shurun Li, Michael Hua Wang, Haibo Zeng, Long Wang

Organizations: REASONS Lab, Institute for Network Sciences and Cyberspace, Tsinghua University · Virginia Tech

Abstract

Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is too expensive to run on every rollout. We study how to spend a limited strong-verifier budget in GRPO, separating which weakly passing rollouts to audit from what reward a caught hack receives. At a fixed policy state with centered advantages, the expected update depends on a design only through each rollout's marginal audit probability and correction; reflecting a caught hack's reward about the failure reward negates the advantages of an all-hack group; and auditing whole groups with the fewest weak passes first (low-pass budgeted audits, LB) minimizes the advantage mass left on hacks among pass-count designs with a fixed correction when a label's value falls with the pass count. We tested this analysis in actual LLM training: from each run's starting state, the sign of this mass on a frozen panel predicted whether hacking would be suppressed in 33 of 39 runs, including 13 of 16 preregistered ones. In LeetCode (Wong et al., 2026) and Countdown-Code (Khalifa et al., 2026), LB drives hacking to near zero, while random audits with the same penalty and budget do not and use more labels. We also find that the randomness of the update affects training: in Countdown-Code, two unbiased corrections, per-item reward-level IPW and a group-level advantage correction, bring hacking down too but end 6.1-11.8 and 8.0-11.5 points lower in held-out accuracy than LB at the same budget. A full audit, which the group-level correction reproduces in expectation, stays within about two points of LB in three seeds, while LB uses 28% of its labels, and allocating the same correction by pass count recovers most of the loss (one seed). Training outcomes thus depend not only on the expected update but also on the randomness of its estimate.

Figures & tables

Figure 1: Low-pass budgeted audits ( LB ). (a) Groups of a batch sorted by their number of weak passes

k

: LB audits whole groups, fewest weak passes first, until a fraction

p

of the weak passes has been checked in expectation (the boundary group is audited with the remaining probability), and a caught hack receives the reflected penalty

2b-a

. (b) LeetCode hack rate with the same penalty and budget rule: LB , auditing whole groups in random order ( RB ), auditing single weak passes at random, and no audit; mean over three seeds with their range shaded, smoothed with a Gaussian kernel of one step. (c) Strong labels used: because the budget is a fraction of the weak passes, LB ’s spending falls as hacking is suppressed.

Figure 2: The two mechanisms behind LB , for a group of

K=16

responses with LeetCode rewards (

a=3.5

b=0.5

, normalized advantages). (a) The advantage of an uncaught hack among

k

weak passes,

\sqrt{(K-1)(K-k)/(Kk)}

, is largest when few siblings pass ( Proposition 2 ). (b) In a group with four hacks and twelve failures, auditing every weak pass with the strong-verdict reward

b

zeroes all advantages, while the reflected penalty

2b-a

negates them ( Proposition 1 ).

Figure 3: Paired differences, LB minus each comparator, with the same penalty and expected budget; the full audit checks every weak pass. Large markers are means over seeds; small dots and thin lines are each seed’s estimate and paired bootstrap 95% interval. Hollow markers count only parseable held-out answers. Held-out accuracy uses 119 LeetCode test problems with 32 samples each (against RB ), 353 held-out LeetCode training-split problems with 16 samples each (against the group-level correction), and 1000 Countdown-Code test problems with 8 samples each.

Figure 4: Countdown-Code training curves with the reflected target (

-0.8

) at

p=0.28

: LB , per-item reward-level IPW and the group-level advantage correction of Sakib et al. (2026) , each the mean over three seeds with the range shaded, smoothed with a Gaussian kernel of one step (hack rate) or three steps (correctness, length).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	LeetCode	Countdown-Code
Model	Qwen3-4B, LoRA rank 32, $\alpha=32$	Qwen2.5-3B-Instruct, all parameters
Framework	verl, code of Wong et al. (2026)	TRL 0.26.2 with vLLM
Prompts $\times$ responses per step	$16\times 16$	$32\times 8$
Optimizer updates per step	1	2
Learning rate (constant)	$1.75\times 10^{-5}$	$3\times 10^{-6}$
AdamW $(\beta_{1},\beta_{2})$ , weight decay	$(0.9,0.99)$ , $0.1$	$(0.9,0.999)$ , $0.01$

Appendix

Table 1: Training configurations.

Figure 5: LeetCode training curves with the reflected penalty (

-2.5

) and the same budget rule (

p=0.33

): mean over three seeds with their range shaded (correctness: means only), smoothed with a Gaussian kernel of one step (hack rate) or three steps (correctness).

Seed	$\Delta\mathrm{AUC}(h)$	$\Delta h$ , steps 31–40	$\Delta C$ , steps 31–40	$\Delta$ pass@16	Labels LB / RB
1	$-30.89^{+2.26}_{-2.30}$	$-36.25^{+4.57}_{-4.69}$	$-1.56^{+2.26}_{-2.35}$	$+$ 0.00	1385 / 2451
2	$-41.53^{+2.33}_{-2.44}$	$-51.68^{+4.96}_{-4.96}$	$-0.66^{+1.99}_{-2.00}$	$-$ 2.50	1314 / 2714
3	$-36.39^{+2.22}_{-2.18}$	$-44.84^{+4.72}_{-4.77}$	$+1.64^{+1.80}_{-1.68}$	$+$ 0.63	1359 / 2561
Mean ( $p$ )	$-$ 36.27 (0.007)	$-$ 44.26 (0.010)	$-$ 0.20 (0.857)	$-$ 0.62 (0.580)	53%

Appendix

Table 2: LeetCode: LB minus RB (reflected penalty

-2.5

p=0.33

), per seed. Paired bootstrap 95% intervals over (step, prompt) units. Labels are strong-verifier calls over 40 steps.

Seed	$\Delta\mathrm{AUC}(h)$	$\Delta h$ , steps 31–40	$\Delta C$ , steps 31–40	$\Delta$ pass@8	Labels LB / IPW
1	$+0.16^{+0.57}_{-0.57}$	$-0.74^{+0.35}_{-0.39}$	$+4.53^{+2.23}_{-2.15}$	$+$ 2.81	1403 / 1341
2	$-5.32^{+0.68}_{-0.72}$	$-0.82^{+0.35}_{-0.39}$	$+11.13^{+2.35}_{-2.34}$	$+$ 9.06	1393 / 1355
3	$-2.77^{+0.62}_{-0.64}$	$-0.31^{+0.23}_{-0.28}$	$+7.73^{+2.35}_{-2.34}$	$+$ 3.75	1395 / 1225
Mean ( $p$ )	$-$ 2.65 (0.236)	$-$ 0.62 (0.058)	$+$ 7.80 (0.055)	$+$ 5.21 (0.116)	$+$ 6.9%

Appendix

Table 3: Countdown-Code: LB minus per-item IPW (target

-0.8

p=0.28

), per seed. LB uses group std normalization, IPW does not.

Seed	LeetCode $\Delta H$	LeetCode $\Delta C$	Countdown-Code $\Delta C$	$\Delta C$ , parseable
1	$-50.4^{+5.5}_{-5.6}$	$-1.3^{+1.5}_{-1.6}$	$+6.1^{+1.3}_{-1.4}$	$+4.5^{+1.3}_{-1.4}$
2	$-62.2^{+5.4}_{-5.1}$	$+1.5^{+2.0}_{-1.9}$	$+11.8^{+1.4}_{-1.3}$	$+5.9^{+1.4}_{-1.5}$
3	$-49.1^{+5.3}_{-5.4}$	$-0.7^{+1.8}_{-1.6}$	$+7.3^{+1.4}_{-1.2}$	$+5.0^{+1.4}_{-1.4}$
Mean ( $p$ )	$-$ 53.9 (0.006)	$-$ 0.2 (0.871)	$+$ 8.4 (0.040)	$+$ 5.1

Appendix

Table 4: Held-out test problems, per seed: LB minus RB in LeetCode (119 problems

\times

32 samples) and LB minus IPW in Countdown-Code (1000 problems

\times

8 samples), in all answers and among parseable answers. Per-problem paired bootstrap 95% intervals; the last row gives the mean over seeds and the seed-level

p

	Arm	AUC $(h)$	$h$ , 31–40	$C$ , 31–40	Labels
LeetCode	None	50.28	53.40	36.95	0
	Full , $c=b$	12.37	3.12	42.11	5161
	Pos , $c=b$	41.35	42.46	40.43	4069
	Low , $c=b$	42.62	46.37	39.88	3348
	Low , $c=2b-a$	2.52	0.04	44.84	1779
	LB , $c=2b-a$ , $p=0.33$	3.04	0.08	41.84	1385

Appendix

Table 5: Single-seed arms (seed 1). AUC and rates in %. Top: LeetCode, fixed count of 8 groups per step or full audit. Bottom: Countdown-Code,

p=0.28

	Seed	$\Delta\mathrm{AUC}(h)$	$\Delta h$ , steps 31–40	$\Delta C$ , steps 31–40
D0	1	$+1.09^{+0.79}_{-0.86}$	$+1.99^{+1.74}_{-1.79}$	$-1.58^{+1.46}_{-1.51}$
	2	$+0.14^{+0.94}_{-0.94}$	$-3.16^{+2.03}_{-2.11}$	$-1.27^{+1.43}_{-1.41}$
	3	$+2.40^{+0.82}_{-0.89}$	$+3.30^{+1.93}_{-1.84}$	$-1.60^{+1.42}_{-1.51}$
D4	1	$-28.90^{+2.13}_{-2.23}$	$-39.26^{+4.38}_{-4.45}$	$+2.07^{+2.34}_{-2.38}$
D4	2	$-29.08^{+2.02}_{-2.07}$	$-31.72^{+3.87}_{-3.98}$	$+1.21^{+1.95}_{-1.91}$

Appendix

Table 6: LeetCode, independent per-item audits minus random whole-group audits at the same audit rate

p=0.4

(group std normalization). D0: penalty 0, 32 prompts per step, a fixed number of audited groups per step (

12

plus a Bernoulli

(0.8)

draw). D4: penalty

-11.5

, 16 prompts per step, each group audited with probability

p

. Paired bootstrap 95% intervals.

Arm	Strict H	Correct	Correct, attempt	Correct, no attempt	Attempt
LB , seed 1	0.08	41.84	0.12	41.72	0.39
LB , seed 2	0.00	38.20	0.08	38.12	0.12
LB , seed 3	0.16	34.22	0.00	34.22	0.35
RB , seed 1	36.33	43.40	42.19	1.21	84.38
RB , seed 2	51.68	38.87	38.40	0.47	94.53
RB , seed 3	45.00	32.58	32.58	0.00	83.67

Appendix

Table 7: LeetCode, steps 31–40, shares of all responses (%). “Attempt” counts every response that defines its own tests.

Model	Stratum (problems)	Strict H	Attempt	Correct
Start	hardest third (144)	7.68	30.30	51.95
Start	easiest third (112)	0.00	15.12	100.00
LB	hardest third	0.04	0.19	60.37
LB	easiest third	0.00	0.02	98.31
LB	labelled hard (44)	0.00	0.05	68.28
RB	hardest third	31.83	94.00	60.27

Appendix

Table 8: Held-out LeetCode problems by difficulty (%; LB and RB pooled over three seeds). Thirds are defined by the starting checkpoint’s correctness on each problem.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 182 papers

Sep 28, 2026

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls. We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. To address this limit, we construct a correction using additional feedback about correctness from audits. This correction achieves selective control: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness.

Christian Moya, Elliott Thornley, Guang Lin

Purdue University · National University of Singapore

Reward Hacking Reinforcement Learning Reinforcement Learning with Verifiable Rewards Reward Design for RL

Ask PDF

Jun 14, 2026

Auditing Reward Hackability in Code RL Training Environments

We measure the rate at which code RL environments accept incorrect solutions as correct. On a 49-task sample of SWE-bench Verified, 28.5% of tasks have test suites weak enough that a Docker-verified incorrect patch passes them. On 20 R2E-Gym tasks across 6 repositories, the same pipeline at single-shot exploit generation yields 25.0%. A random-effects meta-analysis over 134 frontier model submissions to SWE-bench Verified finds, within the same human-rated difficulty stratum, model Pass@1 is +14.14 percentage points higher on flagged-hackable tasks than on robust ones (95% CI [+11.80, +16.48]; one-sided p < 10^-6; I^2 = 0%; 123 of 134 models positive). We then describe a procedure for hardening the broken tasks. An inline LLM judge with a Docker gold-sanity gate runs each generated test against the gold solution before the judge is consulted. On the 11 broken tasks in the audit, the gate flags 65 of 105 decisive LLM-generated tests as failing on the gold patch itself, a 61.9% per-augmentation defect rate the LLM judge alone misses. With diversity-biased retry, the loop converges 9 of 11 tasks to a gated upgrade.

Shreshth Rajan

RL for Code Generation Reward Hacking Automated Software Testing Code Generation Evaluation

Ask PDF

May 12, 2026

Reward Hacking in Rubric-Based Reinforcement Learning

Reinforcement learning with verifiable rewards has enabled strong post-training gains in domains such as math and coding, though many open-ended settings rely on rubric-based rewards. We study reward hacking in rubric-based RL, where a policy is optimized against a training verifier but evaluated against a cross-family panel of three frontier judges, reducing dependence on any single evaluator. Our framework separates two sources of divergence: verifier failure, where the training verifier credits rubric criteria that reference verifiers reject, and rubric-design limitations, where even strong rubric-based verifiers favor responses that rubric-free judges rate worse overall. Across medical and science domains, weak verifiers produce large proxy-reward gains that do not transfer to the reference verifiers; exploitation grows over training and concentrates in recurring failures such as partial satisfaction of compound criteria, treating implicit content as explicit, and imprecise topical matching. Stronger verifiers substantially reduce, but do not eliminate, verifier exploitation. We also introduce a self-internalization gap, a verifier-free diagnostic based on policy log-probabilities, which tracks reference-verifier quality, detecting when the policy trained using the weak verifier stops improving. Finally, in our setting, stronger verification does not prevent reward hacking when the rubric leaves important failure modes unspecified: rubric-based verifiers prefer the RL checkpoint, while rubric-free judges prefer the base model. These disagreements coincide with gains concentrated in completeness and presence-based criteria, alongside declines in factual correctness, conciseness, relevance, and overall quality. Together, these results suggest that stronger verification reduces reward hacking, but does not by itself ensure that rubric gains correspond to broader quality gains.

Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang +3

Scale AI

Reward Hacking Reinforcement Learning Rubric-Based RL Reinforcement Learning with Verifiable Rewards Rubric-Based Evaluation Reward Design for RL

Ask PDF

Abstract

Reward Hacking

Explore similar work

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

Auditing Reward Hackability in Code RL Training Environments

Reward Hacking in Rubric-Based Reinforcement Learning