On Language Drift during RLVR Post-Training

Michael Sullivan; Alexander Koller

cs.LGOct 1, 2026

On Language Drift during RLVR Post-Training

Authors: Michael Sullivan, Alexander Koller

Organizations: Department of Language Science and Technology, Saarland University, Saarbrücken, Germany.

Abstract

Recent advances in LLM reasoning models---driven primarily by the paradigm of post-training via reinforcement learning with verifiable reward (RLVR)---have enabled them to accomplish impressively complex tasks. However, in parallel with their rising capabilities, LLMs have increasingly displayed signs of language drift in their chains of thought (CoTs): unusual, non-standard, and seemingly nonsensical language use. Although it is well-documented---and can potentially impair CoT monitorability---the causes of language drift are thus far poorly understood. In this paper, we identify the conditions under which language drift occurs: we prove theoretically that RLVR optimization pressure permits unbounded language drift, while supervised fine-tuning does not. We then show empirically that language drift specifically arises during RLVR on novel reasoning tasks---i.e. when the target behavior cannot be drawn out of the base model. Finally, we prove that it is not possible to constrain language drift without constraining expected reward, suggesting that CoT monitorability cannot be improved without harming performance during RLVR post-training at the frontier.

Figures & tables

Figure 1 : SFT (left) on prompt/CoT/answer triples $(x,z,y)$ does not permit a model $\pi^{(\textit{SFT})}$ to drift arbitrarily far from a human language distribution ${P^{(\textit{HL})}}$ while minimizing cross-entropy with a target distribution ${{D}^{(\textit{SFT})}}$ . RLVR (right) places no optimization pressure on the CoT $z$ , allowing arbitrary language drift away from natural language in the reasoning trace. The example completions in this figure were drawn from Llama-3.2-1B models trained via SFT and RLVR (respectively) on GSM8K.

Figure 2 : “Claudish”—language in the idiosyncratic style of Anthropic’s Claude models—translated to its plain-English equivalent. This example was taken from the ProgramAsWeights Claudish-to-English translator .

Figure 3 : Example of illegible internal reasoning uncovered during the METR evaluation of OpenAI’s GPT-5 .

Figure 4 : For every radius

\epsilon_{n}

around

{P^{(\textit{HL})}}

, let

{\pi}_{n}

be the highest-scoring model on

{{D}^{(\textit{RL})}}

such that

\operatorname{LD}({\pi}_{n})\leq\epsilon_{n}

. By Theorem 3 , there is always some

{\pi}_{n+1}

with

\operatorname{LD}({\pi}_{n+1})>\epsilon_{n}

that scores higher than

{\pi}_{n}

{{D}^{(\textit{RL})}}

(illustrated here by an arrow

{\pi}_{n}\rightarrow{\pi}_{n+1}

Figure 5 : As a consequence of Theorem 3 , the maximum possible expected reward that a model can achieve on

{{D}^{(\textit{RL})}}

asymptotically approaches

\sup_{\pi}{\mathcal{R}}({{D}^{(\textit{RL})}},{\pi})

as language drift approaches

\infty

Figure 6 : Illustration of our trace legibility experiment. The target model

\pi

first generates a full reasoning trace and answer

(z,y)\sim{\pi}(\text{\textendash}\hskip 2.84526pt|\hskip 2.84526ptx)

(top). We then produce ten evenly spaced partial traces from

z

(bottom), and for each partial trace, prompt the instruct models to guess the submitted answer

y

based on the partial trace (with 0.0 as a control). In this highly legible example, the final answer

y=16

can be guessed with high probability from partial traces

\geq

0.6.

Model	Base		RLVR	SFT
Model	First-Ans.	Last-Ans.	RLVR	SFT
Llama	0.06	0.02	0.23 $\pm$ 0.04	0.32 $\pm$ 0.02
Gemma	0.02	0.01	0.12 $\pm$ 0.01	0.24 $\pm$ 0.02
Qwen	0.41	0.20	0.79 $\pm$ 0.01	0.63 $\pm$ 0.02

Table 1 : GSM8K validation accuracy for all three model types. For the RLVR- and SFT-trained models, we report the mean accuracy (

\pm

standard deviation) across all five training seeds. We additionally report first-answer accuracy for the base models, as a measure of behavior-sharpening potential. These two metrics are disjoint: cases where only one answer was submitted counted only towards last-answer score.

Figure 7 : Mean difference in trace legibility (first two rows) and self-legibility (last two rows) scores for the best (first and third rows) and iso-SFT/RLVR (second and fourth rows) checkpoints (95% confidence intervals). Each point

p\in\{0.0,0.1,\dots,0.9\}

reports the difference in the mean legibility of each trace

z

generated by the model in question, when cut off at

\textit{len}(z)\cdot p

tokens. Absolute values are given in Figure 9 in the Appendix.

Task	Checkpoint	Llama	Gemma	Qwen
Trace Legibility	Best-Perf	-0.24	-0.17	0.02
Trace Legibility	Iso-Perf	-0.28	-0.18	0.01
Self-Legibility	Best-Perf	-0.17	-0.09	0.56
Self-Legibility	Iso-Perf	-0.15	-0.14	0.39

Table 2 : AUC summary of each plot in Figure 7 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8 : Few-shot prompt used during SFT and RLVR training.

Training Type	Hyperparameter	Values
Both	Learning rate	1e-6, 5e-6, 1e-5, 3e-5, 5e-5
DAPO (RLVR)	Group size	64, 128
	Temperature	0.3, 0.5
	Batch size	128, 256
	Epochs	1, 2
SFT	Batch size	4, 8, 16, 32

Appendix

Table 3 : Hyperparameter search values by training type. We swept the same learning rates for both DAPO and SFT.

Figure 9 : Mean trace legibility (left) and self-legibility (right) scores for the best (top row) and iso-SFT/RLVR (bottom row) checkpoints (95% confidence intervals).

Figure 10 : Few-shot prompt used to elicit final-answer guessing in the self-legibility experiment.

Figure 11 : Prompt used with the instruct models in the trace legibility experiment.

Figure 12 : Mean difference between final-answer probabilities given partial-trace fractions (for

p\in\{0.0,0.1,\dots,1.0\}

) versus final-answer probabilities given full traces (95% confidence intervals). Differences at 1.0 (the full trace) are always 0.0 by design.

Model	Training Type	Seed 0	Seed 1	Seed 2	Seed 3	Seed 4
Llama	RLVR	$0.6866_{\pm 0.3530}$	$0.5873_{\pm 0.3509}$	$0.5205_{\pm 0.3794}$	$0.6903_{\pm 0.3477}$	$0.6149_{\pm 0.3642}$
	SFT	$0.6887_{\pm 0.3812}$	$0.7350_{\pm 0.3584}$	$0.7491_{\pm 0.3627}$	$0.7762_{\pm 0.3386}$	$0.7714_{\pm 0.3496}$
	Iso-SFT	$0.6973_{\pm 0.3979}$	$0.5807_{\pm 0.3771}$	$0.2999_{\pm 0.3272}$	$0.5924_{\pm 0.3751}$	$0.6650_{\pm 0.3406}$
Gemma	RLVR	$0.3365_{\pm 0.3317}$	$0.3965_{\pm 0.3296}$	$0.4571_{\pm 0.3699}$	$0.4726_{\pm 0.3438}$	$0.4549_{\pm 0.3956}$
	SFT	$0.5231_{\pm 0.3719}$	$0.7223_{\pm 0.3226}$	$0.3363_{\pm 0.3606}$	$0.6304_{\pm 0.3607}$	$0.3723_{\pm 0.4095}$
	Iso-SFT	$0.4721_{\pm 0.3435}$	$0.1419_{\pm 0.2569}$	$0.1605_{\pm 0.2534}$	$0.1185_{\pm 0.1835}$	$0.0505_{\pm 0.1464}$

Appendix

Table 4 : Mean difference (

\pm

standard deviation) between final-answer probability given the original CoT and final-answer probability given a swapped CoT. All values in this table are significant at

p<0.001

under a paired t-test.

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Explore similar work

The Unlearnability Phenomenon in RLVR for Language Models
May 16, 2026Yulin Chen, He He, Chen ZhaoReinforcement Learning With Verifiable Reward Unlearnable Examples
1New York University · 2NYU Shanghai
Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
Jun 22, 2026Stanley Wei, Juno KimLarge Reasoning Models Reinforcement Learning With Verifiable Reward
1Princeton University · University of California, Berkeley.
Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model
Jun 21, 2026Pengxiang Cai, Tianchen Fang, Xiaohan Li +3Curriculum Reinforcement Learning Large Language Model Reinforcement Learning
The Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Zhejiang University

May 16, 2026, cs.LGJ/K move · Enter open · S save

The Unlearnability Phenomenon in RLVR for Language Models

Yulin Chen, He He, Chen Zhao

1New York University · 2NYU Shanghai

Reinforcement Learning With Verifiable Reward Unlearnable Examples LLM Reasoning Strategies Verifiable Rewards

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Reward (RLVR) has proven effective in improving Large Language Model's (LLM) reasoning ability. However, the learning dynamics of RLVR remain underexplored. In this paper, we reveal a counterintuitive phenomenon: among hard examples that the model initially struggles with, a substantial subset remains unlearnable even when correct rollouts are present. To understand the phenomenon, we first demonstrate that existing optimization and sampling techniques fail to resolve unlearnability. With cross-example gradient analysis, we show that unlearnable examples have fundamental representation issue, characterized by low gradient similarity with the rest of the examples and ungeneralizable reasoning patterns. We further show that representation flaws are difficult to mitigate in RL, as data augmentation does not improve gradient similarity. Our study provides the first systematic characterization of unlearnable data in RLVR training and reveals fundamental limitations in current RL approaches for reasoning tasks. Code and data are available at \url{https://github.com/yulinchen99/unlearnability-rlvr}.

Abstract

Explore similar work

The Unlearnability Phenomenon in RLVR for Language Models

Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently

Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model