SpeCal: Spectral Calibration for Unsupervised RLVR
Organizations: Zhejiang University · Ant Group
Abstract
Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning abilities of large language models, but its dependence on ground-truth answers or external verifiers limits its ability to exploit abundant unlabeled reasoning data. Unsupervised RLVR alleviates this bottleneck by deriving scalar rewards from the model's own rollouts. However, the reliability of such self-generated supervision varies substantially across rollout groups and individual responses, while existing methods construct or refine their rewards mainly from decoded outcomes and reward statistics. These endpoint signals reveal what answers the model produces, but provide limited information about whether the underlying rollout representations form a reliable structure for learning. Our key observation is that hidden-state spectra offer a complementary view of supervision reliability by characterizing both the organization of a rollout group and the support received by each rollout within that group. Based on this observation, we propose SpeCal, a bi-level spectral calibration framework for unsupervised RLVR. At the group level, SpeCal combines spectral mass with a Gaussian band-pass gate over normalized entropy-based effective rank, favoring rollout groups with substantial representation variation whose effective dimensionality lies in a preferred regime. At the rollout level, the same group effective rank determines how many dominant directions are retained in a leave-one-out spectral subspace formed by the remaining rollouts, within which the conditional support of each held-out rollout is measured. The resulting signals calibrate the original reward-induced advantages at the group and rollout levels without redefining the underlying unlabeled reward.
Figures & tables
| Methods | In-Domain Performance | Out-of-Domain Performance | ||||||||
| AIME 24/25 | AMC | MATH-500 | Minerva | Olympiad | Avg. | ARC-c | GPQA | MMLU-Pro | Avg. | |
| Qwen3-8B-Base | ||||||||||
| Before RL | 9.4/7.3 | 38.0 | 64.4 | 25.6 | 32.5 | 29.5 | 29.7 | 11.5 | 46.6 | 29.3 |
| Fully Supervised | 23.1/19.8 | 55.1 | 85.0 | 46.5 | 50.9 | 46.7 | 93.9 | 50.5 | 65.5 | 70.0 |
| Tok-entropy | 12.3/11.1 | 44.8 | 78.5 | 34.6 | 41.8 | 37.2 | 43.1 | 10.2 | 56.7 | 36.7 |
| Seq-entropy | 14.2/15.8 | 51.9 | 78.3 | 38.0 | 43.1 | 40.2 | 81.3 | 20.8 | 59.5 | 53.9 |
| Method | Calibration Components | Benchmarks | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| AIME24 | AIME25 | AMC | MATH-500 | Minerva | Olympiad | Avg. | ||||
| Majority Vote RLVR | ✗ | ✗ | ✗ | 17.7 | 12.5 | 54.4 | 81.7 | 43.8 | 46.0 | 42.7 |
| SpeCal w/o Spread | ✗ | ✓ | ✓ | 19.2 | 15.7 | 56.0 | 82.0 | 44.3 | 49.7 | 44.5 |
| SpeCal w/o ER Gate | ✓ | ✗ | ✓ | 19.2 | 13.8 | 57.6 | 81.6 | 44.5 | 48.7 | 44.2 |
| SpeCal w/o Traj. Support | ✓ | ✓ | ✗ | 20.4 | 13.6 | 53.9 | 82.5 | 46.9 | 48.8 | 44.3 |
| SpeCal | ✓ | ✓ | ✓ | 20.9 | 16.3 | 58.1 | 83.9 | 44.7 | 49.5 | 45.6 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Time per Step (s) | ||
|---|---|---|
| Method | Qwen3-8B-Base | DeepSeek-R1-Distill-Llama-8B |
| TTRL | 192 | 254 |
| SCRL | 112 | 246 |
| Co-rewarding | 245 | 427 |
| SpeCal | 198 | 251 |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,862 papersThe Invisible Leash: Why RLVR May or May Not Escape Its Origin
pass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.