Learned Reporting Preferences in RLVR Can Conflict with the Current Request

Yupeng Chang; Wenxuan Zhang; Yuan Wu

cs.LGSep 29, 2026

Learned Reporting Preferences in RLVR Can Conflict with the Current Request

Authors: Yupeng Chang, Wenxuan Zhang, Yuan Wu

Organizations: School of Artificial Intelligence, Jilin University · Singapore University of Technology and Design · Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach for improving language-model performance on reasoning tasks using automatically checked answers. Yet convention-matched evaluation cannot reveal whether reinforcing one reporting convention reduces adherence to a different request that the initial policy already follows. To test this, we train matched policies under two reporting conventions and evaluate each policy under both current requests, using the same initial policy as a shared reference. We complement this crossed design with controlled interventions and independent human calibration. On GSM8K, boxed-format RLVR reduces the fraction of Qwen2.5-7B responses containing the requested hash-format payload by 35.33--74.37 percentage points relative to a 95.45% initial baseline in four of five training seeds; the fifth improves by 2.50 points. In the four deteriorating runs, almost every response that omits the requested payload instead retains the trained boxed convention, and the same four seeds deteriorate under two fixed paraphrases. Changing only the final-answer marker in supervised targets reverses which reporting convention the model prefers across three seeds, providing controlled evidence that this preference is learnable. Across three settings with independent human calibration, gains under a convention-sensitive scorer exceed the corresponding gains in committed-answer correctness, i.e., the correctness of the answer the model actually commits to. Together, these results separate three distinct post-training outcomes: learned reporting preference, current-request adherence, and committed-answer correctness. They show that convention-matched accuracy alone does not fully characterize post-training behavior and motivate evaluating current-request adherence alongside convention-matched task accuracy.

Figures & tables

Model / seed	Prompt	$S_{0}$	$S_{\rm RL}$	$F$	$\Delta L$	$\Delta$ MV [-1pt]95% CI
A. Default prompt		Matched comparison across model families
Qwen2.5 1.5B / 83	Default	11.1	75.4	+59.7	+4.6	$+3.9$ $[+1.4,+6.4]$
SmolLM2 1.7B / 83	Default	10.2	41.5	+26.2	+5.2	$+4.1$ $[+1.6,+6.6]$
Qwen2.5 7B / 83	Default	77.7	91.6	+14.1	-0.2	$+0.0$ $[-1.2,+1.3]$
OLMo-2 7B / 83	Default	65.6	85.4	+15.8	+4.1	$+3.8$ $[+1.8,+5.7]$
OLMo-2 7B / 84	Default	65.6	85.1	+15.8	+3.7	$+3.5$ $[+1.5,+5.5]$

Table 1: Matched GSM8K scores and gains across models, prompt variants, and readers. Scores are percentages; changes are percentage points.

Figure 1: Current-request adherence across seeds and request wordings. (a) Requested hash-payload presence, irrespective of correctness, on all

1{,}319

GSM8K items; the dashed line marks the shared initial policy. (b) Change from each wording’s initial-policy baseline after boxed training on a fixed

300

-item subset using the original request and two paraphrases (A/B). Thin lines connect the same five seeds; thick lines show their equally weighted means. Four seeds deteriorate under all three wordings, while seed

85

improves under all three. Wordings are repeated evaluations, not additional training seeds. Details appear in Appendices B.2 – B.3 .

Figure 2: Complementary evidence separates reporting behavior, learned preference, and committed-answer correctness. (a) Across five Qwen2.5-7B GSM8K seed pairs, strict training–request interactions vary across seeds, while full-response MV interactions remain near zero. (b) In content-matched SFT, changing only the final-answer marker reverses convention preference across three seeds; the unchanged boxed-marker control remains near baseline. (c) Across three stratified human-calibration settings, strict-score gains exceed committed-answer gains; intervals and identification ranges reflect calibration uncertainty and unresolved-label sensitivity.

Intervention	Model	Strict gain		$\rho$ (%)	Seeds
		Control	Treatment
A. Remove marker requirement		reward control
Answer only	Qwen2.5 1.5B	$+62.68$	$+7.33$	$88.3$	$3$
Answer only	SmolLM2 1.7B	$+31.15$	$-0.78$	$102.5$	$2$
B. Swap requested/rewarded convention		convention control
#### $\rightarrow$ \boxed{}	Qwen2.5 1.5B †	$+69.07$	$+6.90$	$90.0$	$3$

Table 2: Reward and convention controls probe convention-sensitive gains. Gains are percentage points;

\rho

is attenuation relative to control.

	Strict gain $\Delta_{S}$	Human gain $\Delta_{H}$		Strict–human gap $\Delta_{S}-\Delta_{H}$
Setting	Strict gain $\Delta_{S}$	ID range	$95\%$ CI	ID range	$95\%$ CI
A. GSM8K		committed-answer calibration
Qwen2.5 1.5B	$+64.29$	$+2.13$ – $+4.84$	$[-27.22,+30.40]$	$+59.45$ – $+62.16$	$\mathbf{[+33.89,+91.51]}$
SmolLM2 1.7B	$+31.31$	$+2.42$ – $+9.80$	$[-20.05,+24.26]$	$+21.51$ – $+28.89$	$\mathbf{[+7.05,+51.36]}$
B. MATH500		symbolic-transfer calibration
Qwen2.5 7B	$+8.53$	$-9.74$ – $-6.58$	$[-33.41,+6.27]$	$+15.11$ – $+18.27$	$\mathbf{[+2.26,+41.94]}$

Table 3: Strict-score gains exceed human-calibrated committed-answer gains. Gains and gaps are percentage points. ID denotes identification ranges over unresolved labels; CIs are simultaneous

95\%

finite-population bounds.

A. Reader dependence MV readout
Test	Setting	$\Delta$ (pp)
		Strict	Alt. reader
Coverage ( $k=256$ )	Qwen	$+3.34$	$-0.53$
Fresh arithmetic	Qwen	$+87.03$	$+0.39$
Fresh arithmetic	SmolLM2	$+54.04$	$+5.89$
B. Aligned-interface boundary lenient readout

Table 4: Boundary tests constrain the interpretation.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset / model	State	Population	Sample
GSM8K / Qwen2.5 1.5B	Initial	$1319$	$105$
	RL, seed $83$	$1319$	$61$
GSM8K / SmolLM2 1.7B	Initial	$1319$	$106$
	RL, seed $83$	$1319$	$82$
MATH500 / Qwen2.5 7B	Shared initial	$500$	$69$
	RL, seed $83$	$500$	$62$

Appendix

Table 5: Human calibration sample and finite populations. Initial and trained states are sampled separately. The MATH500 initial state is shared across all three seed contrasts rather than replicated three times.

Setting	Missed-correct reduction $\mathrm{FN}_{0}-\mathrm{FN}_{\rm RL}$	False-credit increase $\mathrm{FP}_{\rm RL}-\mathrm{FP}_{0}$	Strict–human gap $\Delta_{S}-\Delta_{H}$
GSM8K / Qwen2.5 1.5B	$[+59.82,+62.16]$	$[-0.37,+0.00]$	$[+59.45,+62.16]$
GSM8K / SmolLM2 1.7B	$[+18.76,+26.14]$	$+2.75$	$[+21.51,+28.89]$
MATH500 / Qwen2.5 7B	$[+9.90,+12.77]$	$[+5.21,+5.50]$	$[+15.11,+18.27]$

Appendix

Table 6: Estimated error components of the strict–human gain gap. Population-weighted identification ranges over unresolved labels, in percentage points; these are estimates, not confidence intervals. The two signed components account for the gap in Table 3 .

Setting	State	Correctness agreement (%)	Final unresolved mass (%)
GSM8K / Qwen2.5 1.5B	Initial	$96.80$	$2.71$
	RL / $83$	$95.85$	$0.00$
GSM8K / SmolLM2 1.7B	Initial	$96.37$	$3.42$
	RL / $83$	$94.74$	$3.96$
MATH500 / Qwen2.5 7B	Shared initial	$96.54$	$0.00$
	RL / $83$	$95.78$	$5.52$

Appendix

Table 7: Independent annotation agreement and residual uncertainty. Population-weighted descriptive estimates under the frozen stratified design. Correctness agreement uses the original A/B labels before adjudication; final unresolved mass is measured after third review. Confidence intervals are not reported in this display.

	Hash request		Boxed request
Checkpoint	$S$	MV	$S$	MV	Trunc. (H/B)
GSM8K ( $n=1319$ per cell)
Initial	1025	1208	1209	1213	2/3
Seed 83	1208	1208	1217	1219	5/6
Seed 84	1213	1210	1171	1211	2/3
Seed 85	1218	1217	1215	1216	4/6

Appendix

Table 8: Complete secondary evaluation of the hash-trained checkpoints. Exact correct counts for requested-format strict (

S

) and full-response MV. Truncation counts are hash/boxed. The initial comparator is shared across training seeds. The boxed-trained intervention arm is outside this panel.

Dataset	Training seed	Strict gain	MV gain
GSM8K	83	$+13.87\;[11.68,16.07]$	$+0.00\;[-1.21,1.21]$
	84	$+14.25\;[12.05,16.45]$	$+0.15\;[-1.14,1.44]$
	85	$+14.63\;[12.51,16.83]$	$+0.68\;[-0.53,1.90]$
MATH500	83	$-1.20\;[-6.20,4.00]$	$-6.80\;[-10.60,-3.00]$
	84	$+11.00\;[6.40,15.60]$	$-11.00\;[-15.20,-7.00]$
	85	$+15.80\;[11.40,20.40]$	$-8.20\;[-11.80,-4.80]$

Appendix

Table 9: Hash-request gains remain strongly reader-dependent. Trained-minus-initial changes in percentage points, with

95\%

paired-item intervals (

20000

resamples, seed

20260910

). All are secondary endpoints conditional on fixed checkpoints and responses.

Seed	Cohort	$I_{P}$	$I_{q}$	$I_{S}$	$I_{\rm MV}$
83	Original	$+39.80$	$+39.88$	$+35.18$	$-0.99$
84	Original	$+74.30$	$+77.63$	$+71.80$	$+0.23$
85	Original	$+1.97$	$-2.50$	$-2.50$	$-0.45$
86	Extension	$+40.64$	$+40.86$	$+35.63$	$-0.91$
87	Extension	$+78.70$	$+79.00$	$+73.39$	$+1.14$
Mean		$+47.08$	$+46.97$	$+42.70$	$-0.20$

Appendix

Table 10: Complete five-seed GSM8K interactions. All

20

trained cells contain

1319

responses.

P

measures requested payload presence,

q

frozen-reader extractability,

S

strict accuracy, and MV full-response Math-Verify. All entries are percentage points. Mean and sample SD describe the five fixed seeds; no outcome is excluded.

Seed	Training	Request	Requested only	Other only	Both	Neither	$q_{c}$
—	Initial	Hash	1197	31	62	29	1109
—	Initial	Boxed	1316	0	0	3	1311
83	Hash	Hash	1009	1	306	3	1315
83	Hash	Boxed	1314	0	0	5	1312
83	Boxed	Hash	75	523	718	3	792
83	Boxed	Boxed	1317	0	0	2	1315

Appendix

Table 11: Full-sample reporting categories, independent of correctness. Each row contains

1319

responses. Four mutually exclusive payload categories sum to

1319

;

q_{c}

is the number extractable by the frozen requested-convention reader. Requested-payload presence is requested-only plus both.

Seed	Training	Request	Payload (%)	$q_{c}$ (%)	Lost	Gained	$\Delta P$ (pp)
—	Initial	Hash	95.45	84.08	—	—	—
—	Initial	Boxed	99.77	99.39	—	—	—
83	Hash	Hash	99.70	99.70	3	59	$+4.25$
83	Hash	Boxed	99.62	99.47	4	2	$-0.15$
83	Boxed	Hash	60.12	60.05	495	29	$-35.33$
83	Boxed	Boxed	99.85	99.70	1	2	$+0.08$

Appendix

Table 12: Requested reporting before and after training. Each cell has

1319

saved responses. Payload and

q_{c}

are percentages;

q_{c}

retains the frozen requested-convention reader. Lost/gained count same-item payload changes from the shared initial cache;

\Delta P

is the net percentage-point change. The initial rows are shared baselines, not separate replications.

Template	$\Delta P_{B,h}$	$I_{P}$	$I_{S}$	$I_{\rm MV}$
Original	$-41.53$ (30.73)	$+45.67$ (30.69)	$+39.87$ (31.22)	$-0.27$ (1.38)
A	$-34.27$ (26.58)	$+47.13$ (26.64)	$+43.60$ (25.88)	$+0.73$ (1.72)
B	$-41.07$ (29.70)	$+56.53$ (29.55)	$+50.27$ (28.61)	$-0.60$ (2.11)

Appendix

Table 13: Fixed wording summaries across all five seeds. Entries are mean (sample seed SD), in percentage points.

\Delta P_{B,h}

is boxed-trained minus initial hash-payload presence within each template.

I

denotes the training-by-request interaction. Each template uses the same

300

items; the original row is not the full

1319

-item result.

Template	Seed	$\Delta P_{B,h}$	$I_{P}$	$I_{q}$	$I_{S}$	$I_{\rm MV}$
Original	83	$-32.67$	$+36.67$	$+36.67$	$+31.67$	$-0.67$
Original	84	$-71.33$	$+75.33$	$+78.33$	$+69.67$	$-1.33$
Original	85	$+2.00$	$+2.33$	$-2.33$	$-5.00$	$-1.33$
Original	86	$-34.67$	$+38.67$	$+39.00$	$+33.33$	$0.00$
Original	87	$-71.00$	$+75.33$	$+76.00$	$+69.67$	$+2.00$
A	83	$-44.67$	$+57.67$	$+57.67$	$+52.33$	$-0.33$

Appendix

Table 14: All fifteen fixed seed/template contrasts. Columns use the definitions in Table 13 ;

I_{q}

additionally measures frozen-reader extractability. All values are pp. The sign of seed

85

’s strict interaction varies even though its boxed-to-hash payload change remains positive.

Figure 3: Reader dependence in an illustration and symbolic transfer. (a) An illustrative pair changes strict credit without changing the displayed answer value; each response is scored by both readers. (b) MATH500 hash-request gains for three Qwen2.5-7B training seeds: strict and Math-Verify (MV) score the same saved responses, with 95% paired-item intervals conditional on each checkpoint. Two seeds reverse gain direction between readers. Human calibration in Table 3 separately tests the strict gain against committed-answer correctness; neither analysis validates chains. The main controlled comparison appears in Figure 2 and Appendix Table 10 .

MATH500 transfer		seed $83$ ; $500$ items
	Request ####		Request \boxed{}
Checkpoint	$S$	MV	$S$	MV
Initial	53.0	73.4	75.0	75.4
Hash-trained	51.8	66.6	76.6	76.8
Boxed-trained	42.6	74.2	76.8	77.2

Appendix

Table 15: MATH500 transfer: request-sensitive readout.

Figure 4: Transfer gains depend on both request and reader. GSM8K hash-trained Qwen2.5-7B checkpoints evaluated on the same

500

MATH500 items. Changes are relative to the shared initial policy under each request. Points and bars show estimates and

95\%

paired-item intervals (

20{,}000

resamples), conditional on fixed checkpoints and responses. Under hash requests, strict and MV changes have opposite signs in two seeds; under boxed requests, all point estimates lie within

\pm 2

percentage points. These automatic scores do not establish committed-answer gains.

Checkpoint	Request	$q^{\rm pay}$	$q$	Trunc.	$\Delta S$ [CI]	$\Delta\mathrm{MV}$ [CI]
Initial	hash	63.4	60.0	0.6	—	—
Hash-trained	hash	99.4	98.2	1.0	$-1.2$ [ $-6.2$ , $3.8$ ]	$-6.8$ [ $-10.6$ , $-3.2$ ]
Boxed-trained	hash	48.6	47.8	0.6	$-10.4$ [ $-14.6$ , $-6.2$ ]	$+0.8$ [ $-2.4$ , $4.0$ ]
Initial	boxed	99.0	98.8	0.4	—	—
Hash-trained	boxed	99.4	99.4	0.4	$+1.6$ [ $-1.4$ , $4.6$ ]	$+1.4$ [ $-1.6$ , $4.4$ ]
Boxed-trained	boxed	99.4	99.4	0.2	$+1.8$ [ $-1.2$ , $4.8$ ]	$+1.8$ [ $-1.2$ , $4.8$ ]

Appendix

Table 16: Full MATH500 transfer diagnostics. Percentages over

500

items per cell;

q^{\rm pay}

is presence of a nonempty requested payload,

q

its symbolic parseability, and Trunc. the length-stop rate.

\Delta S

and

\Delta\mathrm{MV}

are changes in percentage points from the same-request initial policy, with paired

95\%

CIs.

Reference group	Checkpoint	Digit-only hash	Hash correct	MV correct
Nonnegative integer ( $n=320$ )	Initial	191	178	246
	Hash-trained	314	246	245
	Boxed-trained	149	138	243
Other ( $n=180$ )	Initial	3	87	121
	Hash-trained	159	13	88
	Boxed-trained	4	75	128

Appendix

Table 17: Exploratory answer-type diagnostics under hash requests. Exact counts; automatic grouping uses the reference’s symbolic parse. “Other” includes all references not parsed as a nonnegative integer. Digit-only means the hash payload matches an unsigned integer string. This is a post-hoc reporting diagnostic, not a matched difficulty comparison.

State	Prompt / $T$	$S_{f}$	$S_{l}$	$L_{f}$	$L_{n}$	MV
Qwen initial
	Original / 0.6	11.07	11.07	70.74	70.89	71.42
	Rephrased / 0.6	54.44	54.44	70.74	70.13	70.36
	One-shot / 0.6	55.50	55.50	68.69	67.93	67.78
	Original / 0	8.72	8.72	72.02	72.02	72.33
	Rephrased / 0	57.85	57.85	70.58	69.83	69.83

Appendix

Table 18: Complete matched GSM8K readouts (accuracy in percent), all

1319

items per cell.

S_{f}/S_{l}

: first-/last-hash strict;

L_{f}

: hash-first/fallback;

L_{n}

: unconditional last number; MV: Math-Verify. These are scores on identical saved responses within each row.

State	Readout	Strict	Fallback	MV	MV p@32	Output tokens
Qwen initial	Greedy	1.00	98.40	98.40	—	54,350
Qwen initial	Sampled	11.39	98.03	98.02	100.0	1,885,856
Qwen RL	Greedy	98.80	98.80	98.80	—	37,221
Qwen RL	Sampled	98.42	98.42	98.42	100.0	1,202,385
Smol initial	Greedy	18.40	62.20	77.80	—	55,821
Smol initial	Sampled	23.46	63.75	75.23	98.8	1,738,404

Appendix

Table 19: Fresh-arithmetic scores and measured generation cost.

500

items per state. Accuracy and pass@

32

are percentages; output token totals count all

16000

sampled responses. They describe generated work, not a latency or equal-compute claim.

State	Two operations ( $167$ )	Three ( $167$ )	Four ( $166$ )
Qwen initial	99.64	98.75	95.67
Qwen RL	99.48	98.56	97.21
Smol initial	82.17	77.75	65.72
Smol RL	91.69	83.29	68.32

Appendix

Table 20: Operation-stratified Math-Verify accuracy (percent),

32

samples per item. Strata are defined before generation.

State	$T$	$S_{f}$	$S_{l}$	$L_{f}$	$L_{n}$	MV
Initial	0.6	65.58	65.58	81.35	81.20	81.58
RL seed 83	0.6	85.44	85.44	85.44	85.44	85.37
RL seed 84	0.6	85.06	85.06	85.06	84.99	85.06
RL seed 85	0.6	86.05	86.05	86.05	86.05	85.97
Initial	0	66.72	66.72	82.94	82.87	82.94
RL seed 83	0	86.13	86.13	86.13	86.05	85.97

Appendix

Table 21: Matched OLMo-2-7B readouts for three completed seeds. All

1319

GSM8K items, the same prompt,

640

output tokens and

4096

context tokens; scores are percentages.

S_{f}/S_{l}

: first-/last-hash strict;

L_{f}

: hash-first/fallback;

L_{n}

: last number; MV: Math-Verify.

Reader	Initial	RL	Lost	Gained	$p$
First-hash strict	$95.30$	$98.64$	$5$	$49$	$3.89\times 10^{-10}$
Hash-first/fallback	$99.01$	$98.64$	$10$	$5$	$0.302$
Last number	$98.94$	$98.64$	$10$	$6$	$0.454$
Math-Verify	$99.17$	$98.64$	$10$	$3$	$0.092$

Appendix

Table 22: Coverage conclusions depend on the reader. Observed solved-set coverage at

k=256

is reported in percent. Lost and gained count item-level coverage changes from the initial to RL policy;

p

-values are exact paired McNemar tests.

Cell	Estimator	Convention	Step $0$	Step $100$	Gain	$\rho$
A	GRPO	####	$0.0713$	$0.7544$	$+0.6831$	$\mathbf{0.8846}$
B	GRPO	\boxed{}	$0.6876$	$0.7665$	$+0.0788$	$\mathbf{0.8846}$
C	Reinforce++	####	$0.0713$	$0.7582$	$+0.6869$	$\mathbf{0.9040}$
D	Reinforce++	\boxed{}	$0.6876$	$0.7536$	$+0.0660$	$\mathbf{0.9040}$

Appendix

Table 23: Estimator

\times

convention robustness panel. Qwen2.5-1.5B-Instruct, GSM8K, seed

83

100

training steps, with strict prompt sampling disabled in every cell. Values are mean training-validation reward on the frozen

1{,}319

-item split. No checkpoints were retained, so this panel does not support the single-response strict/lenient decomposition used elsewhere. The two #### cells share the same step-

0

value, as do the two \boxed{} cells, and all step-

0

values reproduce the corresponding frozen control runs exactly.

Related work	Finding already studied	Distinction tested here
The Invisible Leash ( Wu et al., 2025 )	Finite-budget support, including prompt-format misalignment and revised answer processing.	Acquisition of a convention versus following a new request; the crossed comparison pairs both trained arms within each seed.
From Accuracy to Robustness ( Huang et al., 2025 )	Verifier classification errors and behavior under RL training, including vulnerability to reward hacking.	Whether a request-dependent training advantage persists under another reader; selected gain contrasts receive a human semantic check.
Format correction / imitation ( Wang et al., 2026b ; Yang et al., 2025 )	Gains beyond format correction and effects of SFT format imitation.	Final-marker preference under exactly matched target content; acquisition remains distinct from the observed answer-score losses.
MathIF ( Fu et al., 2026 )	Instruction following and reasoning-oriented post-training.	Training convention crossed with request and reader; marker absence is distinguished from marker presence with conflicting extracted values.

Appendix

Table 24: Related questions and the evidence added by the present study. Distinctions describe designs, not priority claims.

Instruction Following

Instruction following studies how AI systems interpret and reliably carry out user goals, constraints, formats, and priorities.

See all 135 papers

Oct 13, 2026

Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR

Reinforcement learning with verifiable rewards (RLVR) has become a principal recipe for improving the reasoning of large language models (LLMs). It rests on the assumption that the checker scoring each response is itself correct. In practice, rule-based checkers reject correct answers in unexpected formats, unit tests accept incorrect programs, and LLM judges can be led to approve empty or adversarial responses. This work surveys 58 studies that measure, analyze, or mitigate such verifier errors. We ask what errors verifiers make, how they affect training, which mitigations work against which errors, and how the findings extend to self-generated, rubric- and judge-based, and verifier-free rewards. The evidence suggests that the structure of an error matters more than its rate. Random errors mostly slow learning, though under a fixed budget or KL regularization they also lower final accuracy. False positives that the policy can learn to trigger can cause plateaus or collapse, sometimes even with a low initial error rate. Whether a verifier wrongly rejects or accepts depends on what it checks and who built it, not on its family. Its errors shift as the policy improves, and most mitigations have been tested against only one kind of error. We compile the findings into comparison tables, reconcile apparent conflicts in the literature, and propose a reporting protocol to make future results comparable.

Savyasachi Gupta, Aniruddha Srinivas Joshi

Independent Researcher TX, USA · Independent Researcher CA, USA

Reinforcement Learning with Verifiable Rewards RL for Language Model Reasoning Reward Hacking

Ask PDF

Jul 20, 2025

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, the shrinkage of observed support outweighs its expansion: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in pass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.

Fang Wu, Weihao Xuan, Danny Xing +5

Stanford University, CA, USA · The University of Tokyo, Japan · RIKEN AIP, Tokyo, Japan +2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

Apr 6, 2026

Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR

Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs). While RLVR is designed for tasks with verifiable ground-truth answers, real-world verifiers (e.g., static code checkers) can introduce errors into the reward signal. Prior analyses have largely treated such errors as random and independent across samples, concluding that errors merely slow training with limited effect on final performance. However, practical verifiers tend to exhibit systematic errors. This introduces a risk of models learning unwanted consistent behavior from a structurally incorrect reward signal. In this work, we study the impact of such systematic verification errors on RLVR. Through controlled experiments on arithmetic tasks, we show that systematic false negatives lead to similar effects as random noise. On the other hand, systematic false positives can cause a wide range of behaviors from sub-optimal plateaus to performance collapse. Crucially, these outcomes are not determined by the overall error rate but by the specific pattern of introduced errors, making pre-hoc mitigation difficult. Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes and that verifier quality has to be understood beyond its sample-level error rate.

Kazuki Egashira, Mark Vero, Jasper Dekoninck +3

ETH Zurich · Max Planck Institute for Intelligent Systems, Tübingen

Reinforcement Learning with Verifiable Rewards RL for Language Model Reasoning

Ask PDF

Abstract

Instruction Following

Explore similar work

Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR