cs.AINov 2, 2026

On the Primacy Bias in RLVR Training

Authors: Phuc Minh Nguyen, Ngoc-Hieu Nguyen, Parshin Shojaee, Chinh Duc La, Duy Minh Ho Nguyen, Rui Zhang, Khoa D. Doan, Ting Hua, 

Organizations: University of Notre Dame · The Pennsylvania State University · Virginia Tech · VinUniversity · 

Abstract

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a promising approach for enhancing the reasoning capabilities of large language models. However, it still remains unclear whether RLVR can extend reasoning capabilities beyond what is already encoded in the base model. In this paper, we study this question through the lens of primacy bias: the tendency of models to over-rely on already-learned representations, disproportionately improving performance on problems they already handle well while underexploring harder ones that require departing from those representations. Through our controlled experiments and large-scale analysis, we show that RLVR training on mixed data tends to further sharpen performance on easier problems while yielding only marginal gains on harder ones. We find that although hard problems lead to modest gains overall, they are actually the ones that help most in expanding the model's reasoning capabilities and pushing the representations. However, prolonged training on these hard problems may also hurt and lead to forgetting on easy problems. These findings help to better understand some of the prior conflicting observations about RLVR, specifically, why on (OOD) domains where the base model performs poorly, improvements on hard problems can expand reasoning; while on well-trained domains, forgetting can reduce performance and mask genuine gains obtained from hard problems.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Understanding Diversity Collapse in RLVR via the Lens of Overtraining

    Jun 13, 2026Suqin Yuan, Jinkun Chen, Jiyang Zheng +6Reinforcement LearningReasoning Diversity

  2. The Unlearnability Phenomenon in RLVR for Language Models

    May 16, 2026Yulin Chen, He He, Chen ZhaoReinforcement LearningRL for Language Model Reasoning