Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR

Chandak Chakma; Syed Nazmus Sakib; Nafiul Haque; Shifat E. Arman

cs.AISep 30, 2026

Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR

Authors: Chandak Chakma, Syed Nazmus Sakib, Nafiul Haque, Shifat E. Arman

Organizations: Department of Robotics and Mechatronics Engineering, University of Dhaka

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become an important approach for improving reasoning during post-training. Recent work suggests that some difficult prompts remain resistant to learning even when they occasionally produce correct solutions. We revisit this unlearnability phenomenon and find that the affected prompts do improve, at roughly one third of the learnable rate, while the difficulty-defined set used to study them is much less reproducible than expected. These difficulty labels are estimated from a limited number of sampled responses. Combining them across seeds can further change which prompts are selected instead of simply reducing measurement noise. We develop a sampling-based framework for quantifying this instability and determining how much evaluation is required for difficulty assignments to reproduce reliably. We also revisit the gradient-similarity evidence proposed to explain unlearnability and show that part of the observed separation arises because difficult prompts provide fewer correct rollouts from which their gradients can be estimated. Matching this sample count weakens the gradient difference but does not remove it. Overall, the slow-learning phenomenon survives our reanalysis, while both the prompts used to define it and the evidence used to explain it require more careful measurement.

Figures & tables

Figure 1: The separation is present; the flatness is not. Training reward for the published easy, learnable, and unlearnable cohorts. Slopes per 100 optimizer steps are $+0.3800$ , $+0.1119$ , and $+0.0422\,[+0.0259,+0.0590]$ , respectively. The affected cohort remains markedly slower to improve, but its slope is not zero.

Panel of Figure 2 — Figure 2: The set is unstable in both seed count and aggregation operator. Top: set size changes as more seeds are combined; many prompts are flagged by only a subset of seeds, with disagreement concentrated near $\tau=0.1$ . Bottom: holding the same 292 sub-threshold candidates fixed, the published five-seed construction returns 74 prompts, individual seeds return 145–150 (mean 147.8), a zero-correct-in-128 rule returns 181, and the alternative five-seed aggregation returns 228.

Panel of Figure 3 — Figure 3: The sampling budget can be predicted before it is spent. (a) Predicted Jaccard agreement between two independent threshold-defined sets as a function of per-prompt budget $N$ ; the lower panel shows the discrete effective threshold that produces the small- $N$ sawtooth. (b) Predicted versus measured Jaccard. The filled diamonds are the two prospectively frozen $N=40$ and $N=64$ predictions. Retrospective and cross-model consistency checks are shown for context; the explicitly excluded 1.5B point is not treated as a validation result.

Figure 4: Gradient similarity after matching estimator sample count. Left: within-group mean cosine as the number $K$ of correct rollouts averaged per prompt is matched across cohorts. The fixed matched cohort contains 28 $D_{u}$ , 28 learnable, and 38 easy prompts. Right: the easy-to- $D_{u}$ similarity ratio. Matching reduces the unmatched 2.327 times ratio to approximately 1.5 times, while every tested matched level remains above one. Open markers show each group’s natural, unmatched sample count. At $K=11$ , matching retains only 28 of the 71 $D_{u}$ prompts in the gradient cohort; the matched values therefore demonstrate sensitivity to unequal estimator sample count in an eligible subset, rather than providing corrected full-cohort estimates.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Control	Comparison	Result	Interpretation
Same-model resampling	fixed checkpoint vs. cross-seed, $N=32$	0.798 vs. 0.751	Most apparent cross-seed disagreement can arise from evaluation sampling alone.
Token-cap sensitivity	1,024 vs. 5,120 tokens	0.898 Jaccard	Set movement is no larger than the measured repeatability scale.
Equal-cost allocation	one $N=128$ pass vs. $4\times 32$	0.877 vs. 0.791	A deeper estimate is more reproducible at the same rollout cost.

Appendix

Table 1: Key robustness controls. The first control separates evaluation noise from training variation; the second tests the generation-cap confound; the third asks how an equal sampling budget should be allocated.

Figure 5: Equal-budget comparison of one deep $N=128$ evaluation and four shallow blocks of 32. The deeper rule is more reproducible both before and after applying the never-rewarded exclusion. Error bars show 95% intervals.

Model / rule	Measured	Predicted	Error
Qwen2.5-0.5B, one pass $N=32$	0.7977	0.8000	+0.0023
Qwen2.5-0.5B, one pass $N=128$	0.8770	0.8759	-0.0011
Qwen2.5-0.5B, four blocks of 32	0.7906	0.8228	+0.0322
Qwen2.5-0.5B, two blocks of 16	0.7028	0.7248	+0.0220
Llama-3.2-3B, two blocks of 16	0.6844	0.6763	-0.0081
Llama-3.2-3B v2, two blocks of 16	0.7070	0.6926	-0.0144

Appendix

Table 2: Retrospective and cross-model consistency checks for the sampling-budget model. Six rows are treated as retrospective or consistency checks; the Qwen2.5-1.5B row is an excluded diagnostic and is not counted as validation evidence. The cross-model rows reuse the generation pass from which each pass-rate distribution was estimated and are therefore consistency checks rather than independent validation results.

Figure 6: Predicted versus measured Jaccard agreement across retrospective checks, cross-model consistency checks, and prospectively frozen predictions. Filled diamonds mark the two prospective measurements; the Qwen2.5-1.5B point is shown separately and is not counted as validation evidence.

Figure 7: Full 95% interval width of the easy-to- $D_{u}$ gradient-similarity ratio induced by redrawing the threshold-defined set from the validated sampling model, as a function of per-prompt budget $N$ . The $D_{u}$ -anchored variant is primary and the label-faithful variant secondary. The dotted line marks the pre-specified 0.25 materiality threshold; the dashed line marks the 0.50 sampling confidence-interval width already reported for the downstream statistic. At $N=32$ the two anchorings straddle the threshold.

Configuration constant	Ours	Original
Training
Dataset name	simplelr_qwen_level1to4_sub1k	simplelr_qwen_level1to4
Maximum response length	1,024	5,120
Train batch size	32	256
PPO mini-batch size	16	64
PPO micro-batch size	2	32

Appendix

Table 3: Configuration differences identified from the executed training and evaluation paths. The main text highlights differences that directly affect scientific interpretation; this table records the complete audit.

Explore similar work

The Unlearnability Phenomenon in RLVR for Language Models
May 16, 2026Yulin Chen, He He, Chen ZhaoReinforcement Learning With Verifiable Reward Unlearnable Examples
1New York University · 2NYU Shanghai
Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
May 27, 2026Yue Cheng, Jiajun Zhang, Xiaohui Gao +3Reinforcement Learning With Verifiable Reward LLM Reasoning Strategies
Beijing Jiaotong University · AntGroup · Northwestern Polytechnical University +2
On the Emergence of Implicit Curriculum in RLVR Learning Dynamics
Feb 16, 2026Yu Huang, Zixin Wen, Yuejie Chi +4Reinforcement Learning With Verifiable Reward Curriculum Reinforcement Learning
Department of Statistics and Data Science, Wharton School, University of Pennsylvania · Machine Learning Department, Carnegie Mellon University · Department of Statistics and Data Science, Yale University +1

May 16, 2026, cs.LGJ/K move · Enter open · S save

The Unlearnability Phenomenon in RLVR for Language Models

Yulin Chen, He He, Chen Zhao

1New York University · 2NYU Shanghai

Reinforcement Learning With Verifiable Reward Unlearnable Examples LLM Reasoning Strategies Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model's (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks. Code and data are available at \url{https://github.com/yulinchen99/unlearnability-rlvr}.

Abstract

Explore similar work

The Unlearnability Phenomenon in RLVR for Language Models

Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs

On the Emergence of Implicit Curriculum in RLVR Learning Dynamics