Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO

Shurun Li; Michael Hua Wang; Haibo Zeng; Long Wang

cs.LGOct 26, 2026

Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO

Authors: Shurun Li, Michael Hua Wang, Haibo Zeng, Long Wang

Organizations: REASONS Lab, Institute for Network Sciences and Cyberspace, Tsinghua University · Virginia Tech

Abstract

Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is too expensive to run on every rollout. We study how to spend a limited strong-verifier budget in GRPO, separating which weakly passing rollouts to audit from what reward a caught hack receives. At a fixed policy state with centered advantages, the expected update depends on a design only through each rollout's marginal audit probability and correction; reflecting a caught hack's reward about the failure reward negates the advantages of an all-hack group; and auditing whole groups with the fewest weak passes first (low-pass budgeted audits, LB) minimizes the advantage mass left on hacks among pass-count designs with a fixed correction when a label's value falls with the pass count. We tested this analysis in actual LLM training: from each run's starting state, the sign of this mass on a frozen panel predicted whether hacking would be suppressed in 33 of 39 runs, including 13 of 16 preregistered ones. In LeetCode (Wong et al., 2026) and Countdown-Code (Khalifa et al., 2026), LB drives hacking to near zero, while random audits with the same penalty and budget do not and use more labels. We also find that the randomness of the update affects training: in Countdown-Code, two unbiased corrections, per-item reward-level IPW and a group-level advantage correction, bring hacking down too but end 6.1-11.8 and 8.0-11.5 points lower in held-out accuracy than LB at the same budget. A full audit, which the group-level correction reproduces in expectation, stays within about two points of LB in three seeds, while LB uses 28% of its labels, and allocating the same correction by pass count recovers most of the loss (one seed). Training outcomes thus depend not only on the expected update but also on the randomness of its estimate.

Figures & tables

Figure 1: Low-pass budgeted audits ( LB ). (a) Groups of a batch sorted by their number of weak passes

k

: LB audits whole groups, fewest weak passes first, until a fraction

p

of the weak passes has been checked in expectation (the boundary group is audited with the remaining probability), and a caught hack receives the reflected penalty

2b-a

. (b) LeetCode hack rate with the same penalty and budget rule: LB , auditing whole groups in random order ( RB ), auditing single weak passes at random, and no audit; mean over three seeds with their range shaded, smoothed with a Gaussian kernel of one step. (c) Strong labels used: because the budget is a fraction of the weak passes, LB ’s spending falls as hacking is suppressed.

Figure 2: The two mechanisms behind LB , for a group of

K=16

responses with LeetCode rewards (

a=3.5

b=0.5

, normalized advantages). (a) The advantage of an uncaught hack among

k

weak passes,

\sqrt{(K-1)(K-k)/(Kk)}

, is largest when few siblings pass ( Proposition 2 ). (b) In a group with four hacks and twelve failures, auditing every weak pass with the strong-verdict reward

b

zeroes all advantages, while the reflected penalty

2b-a

negates them ( Proposition 1 ).

Figure 3: Paired differences, LB minus each comparator, with the same penalty and expected budget; the full audit checks every weak pass. Large markers are means over seeds; small dots and thin lines are each seed’s estimate and paired bootstrap 95% interval. Hollow markers count only parseable held-out answers. Held-out accuracy uses 119 LeetCode test problems with 32 samples each (against RB ), 353 held-out LeetCode training-split problems with 16 samples each (against the group-level correction), and 1000 Countdown-Code test problems with 8 samples each.

Figure 4: Countdown-Code training curves with the reflected target (

-0.8

) at

p=0.28

: LB , per-item reward-level IPW and the group-level advantage correction of Sakib et al. (2026) , each the mean over three seeds with the range shaded, smoothed with a Gaussian kernel of one step (hack rate) or three steps (correctness, length).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

	LeetCode	Countdown-Code
Model	Qwen3-4B, LoRA rank 32, $\alpha=32$	Qwen2.5-3B-Instruct, all parameters
Framework	verl, code of Wong et al. (2026)	TRL 0.26.2 with vLLM
Prompts $\times$ responses per step	$16\times 16$	$32\times 8$
Optimizer updates per step	1	2
Learning rate (constant)	$1.75\times 10^{-5}$	$3\times 10^{-6}$
AdamW $(\beta_{1},\beta_{2})$ , weight decay	$(0.9,0.99)$ , $0.1$	$(0.9,0.999)$ , $0.01$

Appendix

Table 1: Training configurations.

Figure 5: LeetCode training curves with the reflected penalty (

-2.5

) and the same budget rule (

p=0.33

): mean over three seeds with their range shaded (correctness: means only), smoothed with a Gaussian kernel of one step (hack rate) or three steps (correctness).

Seed	$\Delta\mathrm{AUC}(h)$	$\Delta h$ , steps 31–40	$\Delta C$ , steps 31–40	$\Delta$ pass@16	Labels LB / RB
1	$-30.89^{+2.26}_{-2.30}$	$-36.25^{+4.57}_{-4.69}$	$-1.56^{+2.26}_{-2.35}$	$+$ 0.00	1385 / 2451
2	$-41.53^{+2.33}_{-2.44}$	$-51.68^{+4.96}_{-4.96}$	$-0.66^{+1.99}_{-2.00}$	$-$ 2.50	1314 / 2714
3	$-36.39^{+2.22}_{-2.18}$	$-44.84^{+4.72}_{-4.77}$	$+1.64^{+1.80}_{-1.68}$	$+$ 0.63	1359 / 2561
Mean ( $p$ )	$-$ 36.27 (0.007)	$-$ 44.26 (0.010)	$-$ 0.20 (0.857)	$-$ 0.62 (0.580)	53%

Appendix

Table 2: LeetCode: LB minus RB (reflected penalty

-2.5

p=0.33

), per seed. Paired bootstrap 95% intervals over (step, prompt) units. Labels are strong-verifier calls over 40 steps.

Seed	$\Delta\mathrm{AUC}(h)$	$\Delta h$ , steps 31–40	$\Delta C$ , steps 31–40	$\Delta$ pass@8	Labels LB / IPW
1	$+0.16^{+0.57}_{-0.57}$	$-0.74^{+0.35}_{-0.39}$	$+4.53^{+2.23}_{-2.15}$	$+$ 2.81	1403 / 1341
2	$-5.32^{+0.68}_{-0.72}$	$-0.82^{+0.35}_{-0.39}$	$+11.13^{+2.35}_{-2.34}$	$+$ 9.06	1393 / 1355
3	$-2.77^{+0.62}_{-0.64}$	$-0.31^{+0.23}_{-0.28}$	$+7.73^{+2.35}_{-2.34}$	$+$ 3.75	1395 / 1225
Mean ( $p$ )	$-$ 2.65 (0.236)	$-$ 0.62 (0.058)	$+$ 7.80 (0.055)	$+$ 5.21 (0.116)	$+$ 6.9%

Appendix

Table 3: Countdown-Code: LB minus per-item IPW (target

-0.8

p=0.28

), per seed. LB uses group std normalization, IPW does not.

Seed	LeetCode $\Delta H$	LeetCode $\Delta C$	Countdown-Code $\Delta C$	$\Delta C$ , parseable
1	$-50.4^{+5.5}_{-5.6}$	$-1.3^{+1.5}_{-1.6}$	$+6.1^{+1.3}_{-1.4}$	$+4.5^{+1.3}_{-1.4}$
2	$-62.2^{+5.4}_{-5.1}$	$+1.5^{+2.0}_{-1.9}$	$+11.8^{+1.4}_{-1.3}$	$+5.9^{+1.4}_{-1.5}$
3	$-49.1^{+5.3}_{-5.4}$	$-0.7^{+1.8}_{-1.6}$	$+7.3^{+1.4}_{-1.2}$	$+5.0^{+1.4}_{-1.4}$
Mean ( $p$ )	$-$ 53.9 (0.006)	$-$ 0.2 (0.871)	$+$ 8.4 (0.040)	$+$ 5.1

Appendix

Table 4: Held-out test problems, per seed: LB minus RB in LeetCode (119 problems

\times

32 samples) and LB minus IPW in Countdown-Code (1000 problems

\times

8 samples), in all answers and among parseable answers. Per-problem paired bootstrap 95% intervals; the last row gives the mean over seeds and the seed-level

p

	Arm	AUC $(h)$	$h$ , 31–40	$C$ , 31–40	Labels
LeetCode	None	50.28	53.40	36.95	0
	Full , $c=b$	12.37	3.12	42.11	5161
	Pos , $c=b$	41.35	42.46	40.43	4069
	Low , $c=b$	42.62	46.37	39.88	3348
	Low , $c=2b-a$	2.52	0.04	44.84	1779
	LB , $c=2b-a$ , $p=0.33$	3.04	0.08	41.84	1385

Appendix

Table 5: Single-seed arms (seed 1). AUC and rates in %. Top: LeetCode, fixed count of 8 groups per step or full audit. Bottom: Countdown-Code,

p=0.28

	Seed	$\Delta\mathrm{AUC}(h)$	$\Delta h$ , steps 31–40	$\Delta C$ , steps 31–40
D0	1	$+1.09^{+0.79}_{-0.86}$	$+1.99^{+1.74}_{-1.79}$	$-1.58^{+1.46}_{-1.51}$
	2	$+0.14^{+0.94}_{-0.94}$	$-3.16^{+2.03}_{-2.11}$	$-1.27^{+1.43}_{-1.41}$
	3	$+2.40^{+0.82}_{-0.89}$	$+3.30^{+1.93}_{-1.84}$	$-1.60^{+1.42}_{-1.51}$
D4	1	$-28.90^{+2.13}_{-2.23}$	$-39.26^{+4.38}_{-4.45}$	$+2.07^{+2.34}_{-2.38}$
D4	2	$-29.08^{+2.02}_{-2.07}$	$-31.72^{+3.87}_{-3.98}$	$+1.21^{+1.95}_{-1.91}$

Appendix

Table 6: LeetCode, independent per-item audits minus random whole-group audits at the same audit rate

p=0.4

(group std normalization). D0: penalty 0, 32 prompts per step, a fixed number of audited groups per step (

12

plus a Bernoulli

(0.8)

draw). D4: penalty

-11.5

, 16 prompts per step, each group audited with probability

p

. Paired bootstrap 95% intervals.

Arm	Strict H	Correct	Correct, attempt	Correct, no attempt	Attempt
LB , seed 1	0.08	41.84	0.12	41.72	0.39
LB , seed 2	0.00	38.20	0.08	38.12	0.12
LB , seed 3	0.16	34.22	0.00	34.22	0.35
RB , seed 1	36.33	43.40	42.19	1.21	84.38
RB , seed 2	51.68	38.87	38.40	0.47	94.53
RB , seed 3	45.00	32.58	32.58	0.00	83.67

Appendix

Table 7: LeetCode, steps 31–40, shares of all responses (%). “Attempt” counts every response that defines its own tests.

Model	Stratum (problems)	Strict H	Attempt	Correct
Start	hardest third (144)	7.68	30.30	51.95
Start	easiest third (112)	0.00	15.12	100.00
LB	hardest third	0.04	0.19	60.37
LB	easiest third	0.00	0.02	98.31
LB	labelled hard (44)	0.00	0.05	68.28
RB	hardest third	31.83	94.00	60.27

Appendix

Table 8: Held-out LeetCode problems by difficulty (%; LB and RB pooled over three seeds). Thirds are defined by the starting checkpoint’s correctness on each problem.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 182 papers

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
Sep 28, 2026Christian Moya, Elliott Thornley, Guang LinReward Hacking Reinforcement Learning
Purdue University · National University of Singapore
Auditing Reward Hackability in Code RL Training Environments
Jun 14, 2026Shreshth RajanRL for Code Generation Reward Hacking
Reward Hacking in Rubric-Based Reinforcement Learning
May 12, 2026Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang +3Reward Hacking Reinforcement Learning
Scale AI

Sep 28, 2026J/K move · Enter open · S save

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

Christian Moya, Elliott Thornley, Guang Lin

Purdue University · National University of Singapore

Reward Hacking Reinforcement Learning Reinforcement Learning with Verifiable Rewards Reward Design for RL

Open paper AskPDF arXiv

In reinforcement learning with verifiable rewards (RLVR), imperfect verifiers can reward incorrect responses, creating opportunities for reward hacking. Using gradient flow with a fixed verifier, we characterize the conditions under which reward rises while correctness falls. We then show that the observations available during RLVR are, in general, insufficient to detect or identify accepted errors, or to guarantee their reduction without sacrificing correct responses. To address this limit, we construct a correction using additional feedback about correctness from audits. This correction achieves selective control: at the current policy, it lowers the probability of accepted errors and raises that of correct responses, provided it outweighs the pressure toward errors from verifier reward. Experiments with log linear and neural contextual bandits and with a language model support the analysis and show that selective control under partial auditing reduces accepted errors while increasing correctness.

Abstract

Reward Hacking

Explore similar work

Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

Auditing Reward Hackability in Code RL Training Environments

Reward Hacking in Rubric-Based Reinforcement Learning