RLVR Sharpens the Pencil, Not the Mind
Organizations: Department of Physics, Indian Institute of Technology Roorkee
Abstract
Models like DeepSeek-R1 made Reinforcement Learning with Verifiable Rewards (RLVR) famous: reward a model for correct answers and it gets dramatically better at reasoning without being taught how. The leading explanation is sharpening---that the base model already knows how to solve these problems and RLVR just makes it more likely to reach for its best solution paths. This has only ever been checked by comparing before and after training, never by watching what changes inside across it. We tracked OLMo-2-1B across all 13 training checkpoints on GSM8K. First-try accuracy climbs from 58.0% to 61.3%, but ask the model 256 times and solution coverage actually rises from 94.5% to 96.9%, proving the knowledge was already there. We then planted reasoning mistakes into problems and trained simple probes to detect them. Those probes held between AUROC 0.798 and 0.840 across all 13 checkpoints and 3 error types, a spread smaller than the confidence interval; the model never builds a sharper internal picture of what a reasoning mistake looks like. The one place we find real internal change is at the final answer token, where a probe distinguishing early from late checkpoints hits an AUROC of 0.93, while the same probe on the actual reasoning steps lands near chance. RLVR makes the model lead with its best thinking. It does not change what that thinking is.
Figures & tables
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Layer | Mean AUROC | Spread | CI half-width | Invariant |
|---|---|---|---|---|
| 0 | 0.517 | 0.039 | 0.050 | ✓ |
| 1 | 0.868 | 0.020 | 0.031 | ✓ |
| 2 | 0.952 | 0.007 | 0.019 | ✓ |
| 3 | 0.949 | 0.014 | 0.018 | ✓ |
| 4 | 0.966 | 0.014 | 0.013 | ✓ |
| 5 | 0.971 | 0.009 | 0.013 | ✓ |
| Site | Mean AUROC | Spread | CI half-width | Invariant |
|---|---|---|---|---|
| L_inj_num | 0.998 | 0.002 | 0.002 | ✓ |
| L_inj_step | 1.000 | 0.000 | 0.000 | ✓ |
| L_dep_num | 0.933 | 0.034 | 0.020 | ✓ |
| L_dep_last | 0.998 | 0.003 | 0.003 | ✓ |
| L_dep_mean | 0.995 | 0.006 | 0.005 | ✓ |
| L_answer | 0.960 | 0.021 | 0.014 | ✓ |
| Checkpoint | Greedy/strict | Greedy/lenient | Sampled/strict | Sampled/lenient |
|---|---|---|---|---|
| step_200 | 0.625 | 0.625 | 0.595 | 0.590 |
| step_800 | 0.645 | 0.645 | 0.610 | 0.610 |
| step_1400 | 0.655 | 0.650 | 0.580 | 0.580 |
| step_2000 | 0.660 | 0.655 | 0.640 | 0.640 |
| step_2600 | 0.645 | 0.645 | 0.650 | 0.645 |
| mean | 0.646 | 0.644 | 0.615 | 0.613 |
RL for Language Model Reasoning
Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.
See all 1,171 papersThe Invisible Leash: Why RLVR May or May Not Escape Its Origin
pass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.