RLVR is a Kernel, Not a Function: Statistical Inference for pass@$k$ Crossovers

Chen Yang; Jun Chen

stat.MLSep 18, 2026

RLVR is a Kernel, Not a Function: Statistical Inference for pass@ $k$ Crossovers

Authors: Chen Yang, Jun Chen

Organizations: Department of Statistics, Texas A&M University · Division of Computational Biology, Department of Quantitative Health Sciences, Mayo Clinic

Abstract

Reinforcement learning with verifiable rewards (RLVR) often improves pass@1 while falling behind its base model at larger sampling budgets $k$ , a crossover read as evidence that RLVR only sharpens existing capability. We identify two limits to this reading. First, a visible crossing need not be statistically established: comparing models on the same prompts, we build confidence bands across sampling budgets $k$ that require evidence of both an early gain and a later loss. Across five public RLVR pairs no crossing is statistically established in the initial evaluations, while a 32k-token evaluation on fresh prompts locates a reversal with first loss between 11 and 61 samples; power analysis shows why failure to detect a crossing need not mean no crossing, and why more prompts can help more than more answers per prompt. Second, base success alone does not determine what RLVR does to a prompt: prompts with the same base success rate have different post-RL success rates, and these differences repeat across independent generation halves. The relationship is a conditional distribution---a Markov kernel---rather than a single curve, and fitting it predicts crossings in independent generations for the same prompts and corrects the simple model's power estimates. Theory further shows how losses on a minority of the hardest prompts can overturn an early lead even when training improves other prompts, separating evidence that a crossover exists from claims about what it means for capability.

Figures & tables

Figure 1: Four claims, not one: effects of the token limit. DeepScaleR vs. its base, 1000 mixture prompts, three per-response token limits:

\hat{D}(k)

with 95% pointwise and simultaneous (sup-

t

) bounds

L(k),U(k)

; the strip colours each

k

by the strongest supported claim. A visual sign change appears at 16k, dominance is rejected at 16k and 32k, and a statistically supported sign change (dark-red cell) exists only at 32k; the new-prompt confirmation is in Table 1 .

DeepScaleR: initial evaluation
pair / token limit	visual sign change	dominance rejected ( $p$ )	crossing supported	first-loss $\kappa\in$	follow-up status / design
8k	none	no (.99)	no	$[28,\infty)$	—
16k	$k{=}27$	yes, unadj. (.029) ∗	no	$[14,\infty)$	no (fresh 16k)
DeepScaleR: follow-up checks
16k fresh	$k{=}23$ (slice)	no (.91)	no	$[9,\infty)$	same token limit
32k mixture	$k{=}15$	yes (.009)	yes	$[9,33]$	adaptive

Table 1: Statistical evidence, grouped by the role of each measurement. The adaptive 32k crossing evidence is mixture-specific; the pre-specified 32k run confirms a crossing on fresh prompts. The fresh check at the same 16k limit does not replicate the original rejection. All bands and

\kappa

intervals are computed on the dense grid

k=1,\dots,K

(Appendix G ); the “visual sign change” column reports the smallest integer

k

at which the point estimate

\hat{D}(k)

is negative, “none” if it never is within

K

, and for the 16k fresh row the DeepScaleR-corpus slice (

m{=}250

; the pooled estimate stays positive throughout). ∗ Unadjusted; not family-wise significant under Holm over the five discovery tests (

\alpha{=}.05

, threshold

.01

). The 32k rows are a separate pre-specified family.

Figure 2: The operator is a kernel, not a function (ProRL-16k, $1000$ prompts, $n{=}n^{\prime}{=}256$ ). (a) $(\hat{p}_{i},\hat{q}_{i})$ against the best deterministic mean link; 40% of prompts fall outside its 95% binomial envelope (5% expected). (b) Residuals from two disjoint generation halves (cross-fitted mean removed); binomial noise is independent across halves, so the diagonal ( $\rho{=}.93$ ) is replicable heterogeneity. (c) The kernel fitted on generation half $A$ predicts half $B$ on the same prompts and vice versa: predicted crossing $67$ vs. $62$ held out; the inset magnifies the point crossings, not statistical evidence of a crossing.

Model pair	Fitted kernel $k^{\star}$	Same mean, no dispersion
DeepScaleR-16k	$27$	$267$
ProRL-16k	$58$	$845$
GRPO $G=4$	$140$	none
GRPO $G=16$	$168$	none

Table 2: Removing conditional dispersion removes the observed-window crossings. First positive-to-nonpositive crossings of fitted curves; “none” means none through

k=2\times 10^{5}

. These model comparisons are not statistical confirmation of population crossings. Full details are in Appendix E.2 .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

pair (token limit, $n$ )	base	$\hat{\beta}_{\mathrm{proj}}$ [95% CI]	$\hat{\gamma}$	$\hat{\pi}_{0}$	crossover	model crossing	M2
DeepScaleR (16k, 256)	R1-D	1.26 [1.20, 1.33]	1.23	.08	$k{=}27$ ( $p{=}\mathbf{.029}$ )	[20, 43]	rej.
ProRL (16k, 256)	R1-D	1.58 [1.40, 1.77]	2.56	.23	$k{=}56$ ( $p{=}.20$ )	[55, 70]	rej.
LUFFY (3k, 128) †	Q-Math	2.93 [2.54, 3.38]	4.37	.03	none $\leq 127$	12	rej. ∗∗
ORZ (8k, 128) ‡	Q2.5	4.51 [3.97, 5.41]	14.5	.03	none $\leq 127$	143	rej.
SimpleRL-Zoo (8k, 128) ‡	Q2.5	1.00 [0.96, 1.05]	1.20	.00	none	none	not rej.

Appendix

Table 3: Five public checkpoint pairs (grid EM; prompt-level bootstrap CIs, 2,000 refits).

\hat{\pi}_{0}

: operational excess-zero index. M2: zero-inflation-aware affine-link LRT. DeepScaleR, ProRL, LUFFY and SimpleRL-Zoo use group-relative objectives; ORZ is PPO-based with a critic, so its

G

is nominal (§ 4 ). † Moderate / ‡ severe base-side truncation (ORZ token-limit-confounded; its extreme estimates are upper bounds). ∗∗ Off-policy distillation. “Model crossing” mixes two fits and is not the crossing implied by the

\hat{\beta}_{\mathrm{proj}},\hat{\gamma},\hat{\pi}_{0}

columns, which come from the grid EM with

F

a nonparametric (NPMLE) mixing law. The bracketed entries are

95\%

profile-likelihood sets computed under the

\mathrm{Beta}(a,b)

-parametrized working model of § B.5 (DeepScaleR-16k

[20,43]

; ProRL-16k

[55,70]

, whose own point estimate under that fit is

\hat{k}^{\star}{=}60

); the single numbers are point estimates of the model crossing under the grid EM (LUFFY

12

, ORZ

143

). The two fits differ in how

F

is modelled, so their crossings are not directly comparable — see the

G

-sweep in § D for the grid-EM crossing of ProRL-16k.

	coverage of true $\kappa_{\mathrm{full}}$ / fraction unbounded
$\beta$	$\pi_{0}=0$	$\pi_{0}=.1$	$\pi_{0}=.2$
$1.00$	$1.00\,/\,1.00^{\dagger}$	$.93\,/\,.00$	$.97\,/\,.00$
$1.05$	$.92\,/\,.21$	$.96\,/\,.00$	$.94\,/\,.00$
$1.20$	$.95\,/\,.00$	$.94\,/\,.00$	$.94\,/\,.00$
$1.40$	$.93\,/\,.00$	$.94\,/\,.00$	$.93\,/\,.00$

Appendix

Table 4: Profile-LR coverage of the full-model crossing

\kappa_{\mathrm{full}}

, including the atom-driven

\pi_{0}>0

\beta\approx 1

cells (300 replicates per cell,

m{=}1000

n{=}n^{\prime}{=}64

G{=}8

\alpha=.05

). Membership of

+\infty

is decided by the likelihood ratio against the no-crossing boundary region (

D_{\mathrm{model}}\geq 0

throughout), not by any finite-

\kappa

proxy. † At

\beta=1

\pi_{0}=0

the true crossing is

+\infty

; coverage is the fraction of sets containing it. At

\beta=1

\pi_{0}>0

the true crossing is finite and the set correctly closes (unbounded fraction

0

); the

21\%

unbounded rate at

\beta=1.05

\pi_{0}=0

is honest weak-identification uncertainty near the boundary, and coverage of the finite truth there is

.92

Figure 3: The inference machinery, checked against its own claims. Left: size of the dominance test (2,000 replicates,

m{=}400

K{=}6

B{=}999

; bars are

\pm 1.96

Monte Carlo standard errors). The top four cells vary the null mean : the test is nominal at the all-binding boundary and conservative everywhere inside

H_{0}

, confirming that

D\equiv 0

is the worst case among mean drifts. The bottom three fix

D\equiv 0

and vary only the nuisance correlation ; all three sit at nominal, which no procedure calibrated to a single “

D\equiv 0

law” could achieve — the prompt-level multiplier bootstrap re-estimates the covariance from the data at hand. Middle: level and power against the number of prompts; the configuration fitted to ProRL-16k needs

m\approx 1000

for power near one, while a genuinely shallow crossing is never detected at these sample sizes. Right: coverage of three interval constructions for

\log k^{\star}

m{=}2000

. At the weak-identification boundary

\beta{=}1

the delta method covers

0\%

and the Wald-form Fieller set

53\%

, while profile-likelihood inversion holds

.95

; the

m{=}500

cells behave the same way.

Figure 4: Source-specific analysis of the 32k DeepScaleR result. Left: source-specific

D(k)

curves. Right: their contrast. Shading uses one 95% critical value jointly over both curves, the contrast, and all

k

. Both point estimates cross, but neither source has a statistically supported crossing separately; only the pre-specified 50/50 mixture does.

Figure 5: GRPO and SFT produce different response curves; group size has little effect. Monotone Bernstein-EM fits over the central 20–80% difficulty range: (a) success-rate change

\hat{T}(p)-p

, with average logit slopes in the legend; (b) local logit slope; (c) GRPO

-

SFT contrasts with 95% simultaneous bands. The supported ordering is integrated, not pointwise over the whole range.

pair	$m$	$S$	$\bar{S}_{\mathrm{null}}$	$z_{\mathrm{adj}}$	placebo $z$	$p_{\mathrm{det}}$	$p_{\mathrm{zi}}$	$\widehat{\mathrm{sd}}$
DeepScaleR-16k	1000	.0119	.0001	8.4	1.0	.001	.001	.108
ProRL-16k	1000	.0251	.0039	8.3	1.0	.001	.001	.146
DeepScaleR-8k	1000	.0143	.0005	9.4	0.8	.001	.001	.117
SimpleRL-Zoo	1000	.0020	.0003	7.2	0.8	.001	.001	.041
SimpleRL-Zoo-7B	1000	.0482	.0366	4.8	$-0.3$	.001	.001	.108
GRPO $G{=}4$	1000	.0615	.0390	6.8	$-0.6$	.001	.001	.150

Appendix

Table 5: Split-generation residual covariance test.

\bar{S}_{\mathrm{null}}

: mean of

S

under the fitted deterministic-spline null — the sign-definite bias term of the calibration paragraph; it ranges from

1\%

76\%

S

, largest where the fitted deterministic links are most contorted.

z_{\mathrm{adj}}=(S-\bar{S}_{\mathrm{null}})/\hat{\rm se}(S)

; inference is by the calibrated

p_{\mathrm{det}}

p_{\mathrm{zi}}

(

B{=}800

parametric-bootstrap replicates;

.001

is the resolution floor

1/(B{+}1)

), not by

z

\widehat{\mathrm{sd}}=\sqrt{S-\bar{S}_{\mathrm{null}}}

estimates the root-mean-square dispersion

\smash{\sqrt{\mathbb{E}\,\mathrm{Var}(Q\mid P)}}

— a global summary, not a pointwise function. The 170-prompt hard-tail slices (

n^{\prime}{=}2048

) are omitted: their unadjusted

z\approx 1.9

equals the placebo’s and we treat them as uninformative rather than confirmatory.

pair	$\hat{\beta}$	$\hat{\pi}_{0}$	$k^{\star}$	$\hat{\phi}$	$\hat{\beta}$	$\hat{\pi}_{0}$	$k^{\star}$	$\Delta\ell$
	$\phi=\infty$ (working model)			$\phi$ profiled
DeepScaleR-16k	1.26	.085	36	6.5	0.88	.028	54	$+2969$
ProRL-16k	1.58	.232	32	1.9	0.72	.120	80	$+6257$
GRPO $G{=}4$ ( $n{=}256$ )	2.08	.000	13	1.4	0.58	.000	85	$+13624$

Appendix

Table 6: Beta-binomial refit: the working model (

\phi{=}\infty

) against the same model with the conditional precision

\phi

profiled inside the grid EM.

k^{\star}

is the model-implied crossing under each fit;

\Delta\ell

is the log-likelihood gain from profiling

\phi

, computed in-sample on the full counts — not the held-out quantity of Table 8 , which carries the same symbol but scores untouched generations.

pair	kernel $k^{\star}$	mean-collapsed $k^{\star}$	degraded mass	lock-in bin	$\Pr(Q<10^{-3})$
DeepScaleR-16k	$27$	$267$	$.045$	$[10^{-4},10^{-3})$	$1.00$
ProRL-16k	$58$	$845$	$.049$	$[10^{-4},10^{-3})$	$.97$
GRPO $G{=}4$	$140$	none	$.044$	$[0,10^{-4})$	$.948$
GRPO $G{=}16$	$168$	none	$.056$	$[0,10^{-4})$	$.952$

Appendix

Table 7: Mean-collapse check.

k^{\star}

values are first positive-to-nonpositive crossings on the integer grid

k\leq 2\times 10^{5}

; “none” means no crossing there. “Degraded mass” is the total fitted

p

-marginal mass on nodes with

\mathbb{E}[Q\mid P{=}p]<p

— the

4

–

6\%

quoted in § 5 . The last two columns give the lowest base-difficulty bin whose conditional mass below

q=10^{-3}

exceeds

.9

, and that mass; the bin differs between the public pairs and the controlled arms because their

p

-marginals do.

pair	det-spline	zi-affine	zi-BB $\pm$ se	kernel $\pm$ se	$\hat{\phi}$	BB $\hat{\pi}_{0}$
DeepScaleR-16k	$-88$	$-80$	$998\pm 100$	$967\pm 100$	6.3	.025
ProRL-16k	$-6$	$-1612$	$562\pm 56$	$515\pm 56$	1.9	.127
DeepScaleR-8k	$29$	$-28$	$299\pm 36$	$275\pm 36$	5.4	.063
SimpleRL-Zoo	$6$	$0$	$29\pm 8$	$7\pm 8$	68.0	.003
SimpleRL-Zoo-7B	$0$	$-1238$	$361\pm 34$	$340\pm 35$	1.9	.000
GRPO $G{=}4$	$0$	$-3051$	$1054\pm 69$	$1058\pm 72$	1.4	.000

Appendix

Table 8: Held-out generation prediction: total

\Delta\log p(d_{B}\mid c,d_{A})

relative to the zero-inflated spline model (positive favours the row model); prompt-level standard errors for the two heterogeneity models.

\hat{\phi}

: profiled beta-binomial precision (large

=

near-deterministic); BB

\hat{\pi}_{0}

: the atom remaining once dispersion is allowed. h 170-prompt hard-tail slices (

n^{\prime}{=}2048

): the split test is uninformative there, but held-out prediction still favours dispersion.

pair	$A\to B$ pred./obs.	$B\to A$ pred./obs.	cross-fitted pred./obs.	mean fold curve MAE
DeepScaleR-16k	$26/32$	$33/25$	$29/28$	$.0062$
ProRL-16k	$74/56$	$61/68$	$67/62$	$.0013$
GRPO $G{=}4$	$--/--$	$--/115$	$--/--$	$.0015$
GRPO $G{=}16$	$--/--$	$--/--$	$--/--$	$.0025$
SFT	$--/--$	$--/--$	$--/--$	$.0015$
SimpleRL-Zoo	$--/--$	$--/--$	$--/--$	$.0063$

Appendix

Table 9: Two-fold generation-split validation of model-implied crossings. Entries are the first positive-to-nonpositive crossing; “–” means no crossing through the available held-out window (

k=127

, except SimpleRL-Zoo at

k=63

). The two public crossings survive without count reuse: the cross-fitted ensemble misses their held-out locations by

1

and

5

draws. Across the unaveraged directions,

11/12

predictions match held-out crossing presence; the exception is one shallow GRPO

G{=}4

fold. The lower block runs the spline-link beta-binomial through the same protocol, and is the criterion on which we prefer the kernel for

D(k)

and

k^{\star}

: its cross-fitted crossings sit

5

and

13

draws from the held-out values against the kernel’s

1

and

5

, and on the cross-fitted curves its MAE is

.0049

and

.0042

against the kernel’s

.0004

and

.0005

. On mean fold MAE the two tie on DeepScaleR-16k (

.0062

each, to four decimals) and the kernel leads on ProRL-16k. This ordering is the reverse of the per-prompt held-out likelihood of Table 8 , where the beta-binomial wins; the two criteria target different estimands and we report both. MAE compares each prediction with its untouched fold over every integer

k

in that pair’s available window, then averages the two direction-specific MAEs; it is not the smaller error obtained after fold errors cancel in the pooled curves.

pair	$m$	pass@1	trunc.	$z$	placebo $z$	cal. $p$	$\Delta\ell$	held-out $\Delta$
		base $\to$ RL	base/RL				(kernel)	kernel / beta-bin.
SimpleRL Llama-3.1-8B	$1000$	$.079\to.159$	$.17/.16$	$12.2$	$0.3$	$\leq.0025$	$+637$	$+66\pm 18$ / $+89\pm 18$
SimpleRL DeepSeek-Math-7B	$1000$	$.033\to.075$	$.47/.39$	$7.1$	$1.4$	$\leq.0025$	$+177$	$+32\pm 14$ / $+50\pm 13$

Appendix

Table 10: Non-Qwen replication.

z

: studentised split-generation residual covariance; placebo

z

is the same statistic on independent base quarters, which carries no operator signal. Calibrated

p

is the smaller of the deterministic-spline and zero-inflated-spline parametric-bootstrap

p

-values, and

.0025

is the floor at

400

null draws.

\Delta\ell

: kernel minus zero-inflated-spline log-likelihood. Held-out

\Delta

: gain over the zero-inflated spline on untouched generation halves,

\pm 1

SE.

checkpoint	public position	test $p$	95% set for $k^{\star}$	verdict
DeepScaleR	inversion-free at 8k c	$.029$ (16k)	$[20,43]$	dom. rejected at 16k b
ProRL	boundary expansion, no inversion ( Liu et al., 2025 )	$.20$ (16k)	$[55,70]$	neither claim established
SimpleRL-Zoo	zero-RL family; base superiority at large $k$ ( Yue et al., 2025 )	$1.0$	excl. $k^{\star}{\leq}2{\times}10^{4}$	no crossover
ORZ	zero-RL family ( Yue et al., 2025 )	$1.0$	n/a a	no crossing; confounded
LUFFY	off-policy; gains over on-policy RLVR	$1.0$	n/a a	no crossing $k{\leq}127$ ; out of scope

Appendix

Table 11: Re-analysis of published positions (dominance test of Theorem 1 , 1000 regenerated paired prompts; Fieller sets under the working model). a Set not reported where the model is rejected (LUFFY) or the pair is token-limit-confounded (ORZ). b Not family-wise significant (Holm threshold

.01

); the crossing criterion gives

\kappa\in[14,\infty)

; the fresh-prompt confirmation failed (dominance

p=.91

). c The public position is the absence of a reported inversion at an 8k token limit; the

p{=}.99

quoted elsewhere for this cell is our own 8k dominance test (Table 1 ), not an external claim.

Result	Pre-committed treatment
$\pi_{\varepsilon}$ (the realized excess-zero mass $\rho(p)=\pi_{0}(1-p)^{G}$ ) falls with $G$ as $(1-p)^{G}$ predicts	group-size hazard promoted to mechanism evidence
$G$ -sweep flat or irregular	“tied to $G$ ” language removed; zero-inflation kept as descriptive sensitivity component
weak-KL raises $\hat{\gamma}$ with $\hat{\beta}$ stable (primary contrast $\Delta\gamma>0$ ; realized KL reported)	supports reward-pressure/tempering knob separation
weak-KL moves both $\hat{\beta},\hat{\gamma}$	coordinates reported as empirical, no one-to-one mechanism map claimed
ProRL/DeepScaleR deep-resample candidates mostly revive	“support loss” removed from abstract; $\pi_{0}$ reported as finite-resolution near-zero mass
candidates still zero at $n{=}2048$	only exact binomial upper bounds reported ( $q<1.5\times 10^{-3}$ (95% one-sided exact binomial) at $0/2048$ ); no exact-zero claim

Appendix

Table 12: Pre-specified decision rules and the treatment committed to for each possible outcome, fixed before the corresponding results were examined.

population / repair	$m$	$\inf\hat{D}(1)$	$L(1)$	$\sup\hat{D}(128)$	$U(128)$	$\kappa\in$
as reported	$1060$	$.07596$	$.06547$	$-.01981$	$-.00370$	$[11,61]$
$\leq 5\%$ truncation, both sides	$647$	$.04043$	$.03198$	$-.00927$	$.00028$	$[3,25]$
zero truncation, both sides	$342$	$.01834$	$.01034$	$-.00292$	$.00463$	$[2,\infty)$
$f=0$ , joint repair band	$1060$	$.07596$	$.06421$	$-.01981$	$-.00177$	$[11,73]$
$f=.25$ , failed base truncations	$1060$	$.05709$	$.04562$	$-.10914$	$-.08239$	$[3,4]$
$f=1$ , failed base truncations	$1060$	$.00048$	$-.01361$	$-.11887$	$-.09039$	$[1,2]$

Appendix

Table 13: Truncation sensitivity under failed-output repair. The first three rows use separate bands. The repair rows use a common band jointly over

f

k

, and both admissible overlap endpoints. Repair applies only to failed base truncations. The displayed point values are the minimum

\hat{D}(1)

and maximum

\hat{D}(128)

over missing-overlap completions;

L

and

U

enclose both endpoints. The final row fails the early-positive requirement, although it bounds the first loss from above. “

\infty

” means no supported upper endpoint.

U(k)

need not be monotone: the low-truncation row has negative

U(25)

but positive

U(128)

$k$	$\hat{D}(k)$	$L(k)$	$U(k)$	sign
1	$.0760$	$.0655$	$.0864$	$+$
2	$.0524$	$.0422$	$.0625$	$+$
4	$.0308$	$.0214$	$.0402$	$+$
8	$.0145$	$.0052$	$.0237$	$+$
12	$.0077$	$-.0018$	$.0172$
16	$.0038$	$-.0060$	$.0136$

Appendix

Table 14: 32k confirmation on the pre-specified fresh population (

m{=}1060

n{=}n^{\prime}{=}128

\hat{D}(k)=\text{pass@}k^{\mathrm{RL}}-\text{pass@}k^{\mathrm{base}}

with a

95\%

sup-

t

simultaneous band (

B{=}4000

). “

+

” marks

L>0

and “

-

” marks

U<0

; the crossing criterion requires both, in that order. Band computed on the dense grid

k=1,\dots,128

; the pre-registered run used the sparse grid

\{1,2,3,4,6,8,12,16,24,32,48,64,96,128\}

and gave

\kappa\in[9,64]

(Appendix G ); densifying raises the critical value by

<0.001

and tightens the interval. Rows show a subset of the grid.

pair	power at realized design	observed $p$	cheapest design for power $.80$ (total generations)
DeepScaleR-16k	.63	.029	$1.0$ M ( $m{=}4000$ , $n{=}128$ )
ProRL-16k	.21	.20	$4.1$ M ( $m{=}8000$ , $n{=}256$ )
GRPO $G{=}8$	.07	$\geq.20$	$16.4$ M ( $m{=}32000$ , $n{=}256$ )
GRPO $G{=}4$	.01	$\geq.20$	none on the grid

Appendix

Table 15: Kernel-implied power of the dominance test at

\alpha{=}.05

m{=}1000

n{=}n^{\prime}{=}256

(1000 replicates per cell), and the cheapest design on a

(m,n)

grid that reaches power

.80

. The GRPO rows are the controlled training arms of § 6 .

	power at realized design		cheapest design for power $.80$
pair	$\alpha{=}.05$	$\alpha{=}.01$	$\alpha{=}.05$	$\alpha{=}.01$
DeepScaleR-16k	$.630$	$.358$	$1.0$ M ( $m{=}4000$ , $n{=}128$ )	$1.0$ M ( $m{=}4000$ , $n{=}128$ )
ProRL-16k	$.207$	$.074$	$4.1$ M ( $m{=}8000$ , $n{=}256$ )	$4.1$ M ( $m{=}8000$ , $n{=}256$ )
GRPO- $G{=}8$	$.074$	$.016$	$16.4$ M ( $m{=}32000$ , $n{=}256$ )	$16.4$ M ( $m{=}32000$ , $n{=}256$ )
GRPO- $G{=}4$	$.014$	$.003$	—	—

Appendix

Table 16: Kernel-simulated power of the dominance test and cheapest design reaching

.80

. These are an upper bound on the power to meet the two-sided crossing criterion of Proposition 1 , which additionally requires

L(j)>0

at some

j

and

U(\ell)<0

at some

\ell>j

. Cost is total generations

m(n+n^{\prime})

;

1000

replicates per grid cell. “—” means no grid cell reaches

.80

. Realized designs are

m{=}1000

n{=}n^{\prime}{=}256

. The cheapest design coincides on this grid; at DeepScaleR-16k’s cell (

m{=}4000

n{=}128

) power is

.967

\alpha{=}.05

and

.877

\alpha{=}.01

Reinforcement Learning with Verifiable Rewards

Reinforcement learning with verifiable rewards trains models using outcomes checked by code, proofs, simulators, or structured criteria.

See all 692 papers

Oct 29, 2026

There Is No Free Lunch in RLVR: Every pass@1 Gain Has a pass@k Price

Reinforcement learning with verifiable rewards (RLVR) makes a language model solve more problems on the first try and fewer when given many tries: it raises pass@1 while the base model overtakes it at large sampling budget k. Is this a failure of RL to add reasoning, or a transient that more training removes? We settle the question with a theorem. The reversal is a price, and we compute its exact rate. Modeling RLVR as an exponential tilt of a base policy over strategies shared across prompts, we prove that under exclusive specialization every update that raises pass@1 strictly lowers the success of some prompt; a strictly positive price on prompts (Stiemke duality) is the exact condition, and being merely the best strategy on a prompt is provably not enough. When the loss reaches the hardest prompts, it forces a pass@k crossover, and the number of crossings is at most the sign variation of the shift in per-prompt success, a totally positive bound that is tight and that recovers the known single-crossover result as its order-preserving case. The crossover is not a bug to patch: a constrained update traces the exact pass@1 versus pass@k frontier, and removing the crossover entirely costs the entire pass@1 gain. The same tilt identity yields entropy collapse and the entropy-performance law with an identifiable exponent, unifying two literatures under one mechanism. Reinforcement cannot widen the reasoning frontier; it can only move along it, at a price we make measurable by a diagnostic that reads the bound, the order-preserving test, and the price from any evaluation log. There is no free lunch, only a price.

Yinuo Liu

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

Jun 13, 2026

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models. However, RLVR often suffers from diversity collapse: Pass@

1

improves while high-

k

Pass@

k

degrades, which is viewed as a narrowing of the model's reasoning boundary. We formalize this diversity collapse through the lens of overtraining: once a problem's contribution to the reference metric has effectively saturated, further updates no longer expand what the model can solve but still concentrate probability mass on the trajectories favored by on-policy sampling. Under a standard setup with few rollouts per problem, even a single observed success places a problem in a nearly saturated regime for high-

k

Pass@

k

, so most updates in standard RLVR are overtraining from the boundary perspective. This perspective also suggests a reading of whether RLVR can expand the model's reasoning abilities beyond the base model: since RLVR is structurally biased against high-

k

Pass@

k

, its aggregate decline does not by itself mean that no new reasoning gains occurred. Interventionally, restricting updates to problems with zero observed success lifts Pass@

256

above the base model on difficult benchmarks; observationally, a non-trivial fraction of initially unsolvable problems become solvable during standard RLVR training. Building on these findings, we propose Bayesian Boundary Gating (BBG), which redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary. Across multiple reasoning benchmarks, BBG improves average Pass@

k

across a wide range of

k

Suqin Yuan, Jinkun Chen, Jiyang Zheng +6

Sydney AI Centre, The University of Sydney · Southeast University · Microsoft +3

Reinforcement Learning Reasoning Diversity RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

Jul 20, 2025

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, the shrinkage of observed support outweighs its expansion: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in pass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.

Fang Wu, Weihao Xuan, Danny Xing +5

Stanford University, CA, USA · The University of Tokyo, Japan · RIKEN AIP, Tokyo, Japan +2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Ask PDF

Abstract

Reinforcement Learning with Verifiable Rewards

Explore similar work

There Is No Free Lunch in RLVR: Every pass@1 Gain Has a pass@k Price

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

The Invisible Leash: Why RLVR May or May Not Escape Its Origin