cs.LGNov 16, 2026

SpeCal: Spectral Calibration for Unsupervised RLVR

Authors: Kai Tang, Xing Zheng, Haobo Wang, Guangcheng Zhu, Shenzhi Yang, Zhengqing Zang, Yingfan Ma, Cheng Peng, 

Organizations: Zhejiang University · Ant Group

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning abilities of large language models, but its dependence on ground-truth answers or external verifiers limits its ability to exploit abundant unlabeled reasoning data. Unsupervised RLVR alleviates this bottleneck by deriving scalar rewards from the model's own rollouts. However, the reliability of such self-generated supervision varies substantially across rollout groups and individual responses, while existing methods construct or refine their rewards mainly from decoded outcomes and reward statistics. These endpoint signals reveal what answers the model produces, but provide limited information about whether the underlying rollout representations form a reliable structure for learning. Our key observation is that hidden-state spectra offer a complementary view of supervision reliability by characterizing both the organization of a rollout group and the support received by each rollout within that group. Based on this observation, we propose SpeCal, a bi-level spectral calibration framework for unsupervised RLVR. At the group level, SpeCal combines spectral mass with a Gaussian band-pass gate over normalized entropy-based effective rank, favoring rollout groups with substantial representation variation whose effective dimensionality lies in a preferred regime. At the rollout level, the same group effective rank determines how many dominant directions are retained in a leave-one-out spectral subspace formed by the remaining rollouts, within which the conditional support of each held-out rollout is measured. The resulting signals calibrate the original reward-induced advantages at the group and rollout levels without redefining the underlying unlabeled reward.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. How Far Can Unsupervised RLVR Scale LLM Training?

    Mar 9, 2026Bingxiang He, Yuxin Zuo, Zeyuan Liu +18RL for Language ModelsReinforcement Learning with Verifiable Rewards

  2. The Invisible Leash: Why RLVR May or May Not Escape Its Origin

    Jul 20, 2025Fang Wu, Weihao Xuan, Danny Xing +5RL for Language Model ReasoningReinforcement Learning with Verifiable Rewards

  3. JURY-RL: Votes Propose, Proofs Dispose for Label-Free RLVR

    Apr 28, 2026Xinjie Chen, Biao Fu, Jing Wu +4RL for Language Model ReasoningMathematical Reasoning