cs.LGNov 19, 2026

RL Can Only Reinforce What It Sees Succeed

Authors: Ruben Sanchez-Corcuera, Unai Zulaika, Aritz Bilbao-Jayo, Miguel Fernandez-de-Retana

Organizations: University of Deusto

Abstract

When a familiar rule stops working, an agent has to weigh what pre-training taught it against what it now observes. We ask how outcome-reward reinforcement learning (RL) changes this balance, in a puzzle world where both sides are known: small language models trained from scratch on games played by a Bayesian player, so that the prior and the right answer are exact. Base models hedge as their corpus did, and RL turns that hedging into decisions, in whichever direction the reward favours. But RL can only reinforce what it sees succeed. When the right action is rare, each run either learns it everywhere or not at all, and which runs take off is predicted before training by one count: the expected number of groups of attempts in which the action wins. About 35 such groups give even odds. The count holds for new and repeated problems and for groups of 4 to 256 attempts, and across learning rates, optimizers, GRPO variants and model sizes from 1.3M to 85M parameters, which move the threshold but not what is counted.

Figures & tables

Explore similar work

CardsList
  1. Demystifying Reinforcement Learning Post-Training of Language Models

    Aug 24, 2026Donovan Clay, Saket Gollapudi, Sankar Harilal +4RL for Language ModelsRL Post-Training