What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training

Subham Rath; Raj Dandekar; Rajat Dandekar; Sreedath Panat

cs.LGOct 5, 2026

What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training

Authors: Subham Rath, Raj Dandekar, Rajat Dandekar, Sreedath Panat

Organizations: Accenture Strategy & Consulting · Vizuara AI Labs (vizuara.com) · Vizuara AI Labs (vizuara.ai)

Abstract

pass@ $k$ , the fraction of problems a model solves within $k$ sampled attempts, is the field's default protocol for deciding whether reinforcement-learning (RL) post-training on verifiable rewards improved a model. At the population level, pass@ $k$ depends only on a problem's probability of a correct sample, with no term for how it is distributed across outputs. We show this gap is not academic. Training Qwen2.5-1.5B-Instruct on grade-school math with Group Relative Policy Optimization (GRPO) and with rejection-sampling fine-tuning (RFT, training on the model's own shortest verifier-passed rollout) moves three complementary diversity measures (token-level entropy, answer-level entropy, unique answers per prompt) in opposite directions, with zero overlap across three seeds per arm. The gap survives restricting to verifier-correct completions only (lexical diversity among correct solutions is 15% lower for GRPO, after controlling for length) and a count-controlled check isolating diversity among incorrect answers alone, ruling out that GRPO's higher accuracy alone explains it. Yet pass@8 and pass@32 show no consistent winner on GSM8K, and a hard MATH-500 subset shows the same pattern: separation only at low $k$ . Compared against the starting checkpoint, no trained arm significantly improves hard-problem coverage: RFT is significantly worse, while GRPO is statistically indistinguishable from it - so GRPO's pass@1 edge over RFT reflects a smaller loss relative to Base, not a capability gain, a missing-control issue, not a failure of pass@ $k$ . On GSM8K, only pass@1, with no role in detecting diversity by construction, separates the arms cleanly, rewarding the arm whose correct solutions are least diverse. We argue this is a concrete instance of a standard evaluation protocol missing a property it is routinely used to certify.

Figures & tables

Figure 1: GSM8K, all three arms, 3 seeds each; shaded bands span the seed min–max, not a CI. (a–c) Token/answer entropy and unique answers over training: GRPO (green) and both RFT arms (blue, red) move in opposite directions, with no band overlap from step 75 onward. (d) Final-checkpoint pass@

k

: cleanly separated at

k{=}1

; no statistically detectable separation at

k{=}8

k{=}32

, same seeds and arms as (a–c).

Single seed each; context only, not a robustness claim:
Arm	Token entropy $\Delta$	Unique answers/problem $\Delta$	Answer entropy $\Delta$
RFT (shortest-correct)	$+0.028\pm 0.003$	$+0.56\pm 0.06$	$+0.091\pm 0.017$
RFT + KL brake	$+0.032\pm 0.001$	$+0.52\pm 0.11$	$+0.082\pm 0.009$
GRPO	$-0.055\pm 0.004$	$-1.30\pm 0.20$	$-0.164\pm 0.018$
GRPO + length penalty ( $\lambda{=}0.2$ )	$-0.055$	$-1.26$	$-0.163$
GRPO + length penalty ( $\lambda{=}0.5$ )	$-0.050$	$-1.20$	$-0.150$

Table 1: GSM8K diversity metrics, change from first to last checkpoint, mean

\pm

half-range across 3 seeds. All three move in the same direction within an arm, opposite across arms, with zero cross-arm overlap in any seed.

Arm	pass@1	pass@8	pass@32
RFT (shortest-correct)	$0.571$ [ $0.538$ , $0.605$ ]	$0.910$ [ $0.886$ , $0.935$ ]	$0.972$ [ $0.955$ , $0.990$ ]
RFT + KL brake	$0.574$ [ $0.541$ , $0.607$ ]	$0.908$ [ $0.882$ , $0.934$ ]	$0.967$ [ $0.945$ , $0.988$ ]
GRPO	$\mathbf{0.634}$ [ $\mathbf{0.598}$ , $\mathbf{0.669}$ ]	$0.914$ [ $0.887$ , $0.940$ ]	$0.969$ [ $0.951$ , $0.988$ ]
GRPO + length penalty ( $\lambda{=}0.2$ )	$0.630$ [ $0.565$ , $0.687$ ]	$0.913$ [ $0.865$ , $0.954$ ]	$0.960$ [ $0.920$ , $0.990$ ]
GRPO + length penalty ( $\lambda{=}0.5$ )	$0.625$ [ $0.560$ , $0.682$ ]	$0.908$ [ $0.858$ , $0.950$ ]	$0.950$ [ $0.900$ , $0.990$ ]
OPD	$0.599$ [ $0.532$ , $0.653$ ]	$0.911$ [ $0.866$ , $0.952$ ]	$0.970$ [ $0.930$ , $1.000$ ]

Table 2: GSM8K pass@

k

, final checkpoint. Top: 3-seed DerSimonian–Laird pooled mean, 95% CI. Bottom: single seed, own bootstrap CI. Only pass@1’s CI separates cleanly across arms.

Figure 2: MATH-500 levels 4–5 (100 problems,

n{=}16

/problem), chosen for headroom: pass@16 stays below

0.67

. Bands are min–max over 3 seeds (trained arms) or 3 sampling seeds of the shared pre-intervention checkpoint (Base). (a) Base sits above every trained arm at every

k

. (b) Arm

-

Base, 95% bootstrap CI clustered by problem: RFT is significantly below baseline through

k{=}4

; GRPO never significantly differs from it. (c) GRPO

-

RFT, using the same method; the interval excludes zero at

k{=}1,2

and includes it from

k{=}4

on.

Explore similar work

May 1, 2026cs.LG

Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity

Reinforcement Learning with Verifiable Rewards (RLVR) has achieved substantial gains in single-attempt accuracy (Pass@1) on reasoning tasks, yet often suffers from reduced multi-sample coverage (Pass@K), indicating diversity collapse. We identify a structural cause for this degradation: common RLVR objectives, such as GRPO, are indifferent to how probability mass is distributed among correct solutions. Combined with stochastic training dynamics, this indifference induces a self-reinforcing collapse, in which probability mass concentrates on a narrow subset of correct outputs while alternative valid solutions are suppressed. We formalize this collapse mechanism and further characterize the optimal policy structure under two complementary criteria: robustness and entropy-regularized optimality, which identify the Uniform-Correct Policy as uniquely optimal. Motivated by this analysis, we propose Uniform-Correct Policy Optimization (UCPO), a modification to GRPO that adds a conditional uniformity penalty on the policy's distribution over correct solutions. The penalty redistributes gradient signal toward underrepresented correct responses, encouraging uniform allocation of probability mass within the correct set. Across three models (1.5B-7B parameters) and five mathematical reasoning benchmarks, UCPO improves Pass@K and diversity while maintaining competitive Pass@1, achieving up to +10% absolute improvement on AIME24 at Pass@64 and up to 45% higher equation-level diversity within the correct set. The code is available at https://github.com/AnamikaLochab/UCPO.

Anamika Lochab, Bolian Li, Ruqi Zhang

Department of Computer Science Purdue University, West Lafayette

Reinforcement Learning With Verifiable Reward Verifiable Rewards Diversity Collapse

Ask PDF

Feb 6, 2026cs.LG

F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, computational limits often rule out very large groups, so training proceeds with finite rollout sets that can reinforce only the correct behavior they expose. At practical group sizes, updates can miss rare-correct trajectories while still containing mixed rewards, concentrating probability on more common sampled solutions. We derive the probability of such prompt-local tail-miss events as a function of group size, showing non-monotonic behavior, and in the categorical abstraction characterize how unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware scaling coefficient, inspired by Focal loss, that down-weights updates on high-success sampled groups. Empirically, categorical simulation illustrates the same effect in the categorical setting, Maze provides a single-solution test, and LLM experiments include a representative GRPO group-size sweep together with fixed-

N

transfer across GRPO, DAPO, and CISPO. On Qwen2.5-7B at

N{=}8

, our method improves average math pass@256 from 64.1

\rightarrow

70.3 (GRPO), 69.3

\rightarrow

72.5 (DAPO), and 73.2

\rightarrow

76.8 (CISPO); OOD pass@256 also improves in all three cases, without increasing group size or computational cost.

Daniil Plyusov, Alexey Gorbatovski, Boris Shaposhnikov +4

T-Tech · Saint Petersburg Electrotechnical University “LETI”

Reinforcement Learning With Verifiable Reward Flow-Grpo Verifiable Rewards Single Trajectory

Ask PDF

Jun 13, 2026cs.LG

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models. However, RLVR often suffers from \emph{diversity collapse}: Pass@

1

improves while high-

k

Pass@

k

degrades, which is viewed as a narrowing of the model's reasoning boundary. We formalize this diversity collapse through the lens of \emph{overtraining}: once a problem's contribution to the reference metric has effectively saturated, further updates no longer expand what the model can solve but still concentrate probability mass on the trajectories favored by on-policy sampling. Under a standard setup with few rollouts per problem, even a single observed success places a problem in a nearly saturated regime for high-

k

Pass@

k

, so most updates in standard RLVR are overtraining from the boundary perspective. This perspective also suggests a reading of whether RLVR can expand the model's reasoning abilities beyond the base model: since RLVR is structurally biased against high-

k

Pass@

k

, its aggregate decline does not by itself mean that no new reasoning gains occurred. Interventionally, restricting updates to problems with zero observed success lifts Pass@

256

above the base model on difficult benchmarks; observationally, a non-trivial fraction of initially unsolvable problems become solvable during standard RLVR training. Building on these findings, we propose \emph{Bayesian Boundary Gating} (BBG), which redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary. Across multiple reasoning benchmarks, BBG improves average Pass@

k

across a wide range of

k

Suqin Yuan, Jinkun Chen, Jiyang Zheng +6

Sydney AI Centre, The University of Sydney · Southeast University · Microsoft +3

Reinforcement Learning With Verifiable Reward Verifiable Rewards Reasoning Benchmark Diversity Collapse Diversity Lens

Ask PDF