Erased, Rerouted, or Rescaled? Post-Training and the Causal Quotient of a Language Model's Belief State

Weihan Li; Tianshi Zheng; Junhao Wu; Xinlei Chen

cs.LGOct 4, 2026

Erased, Rerouted, or Rescaled? Post-Training and the Causal Quotient of a Language Model's Belief State

Authors: Weihan Li, Tianshi Zheng, Junhao Wu, Xinlei Chen

Organizations: The University of Tokyo · The Hong Kong University of Science and Technology · RWTH Aachen University · Harbin Institute of Technology, Shenzhen

Abstract

What happens to information a pretrained model already encodes when post-training no longer rewards using it? The common language of representation compression conflates three fates: information may be erased, rerouted away from the decision while still represented, or rescaled to occupy less variance while still represented and used. We make these fates identifiable in models whose pretraining recovers Bayesian belief states. A reward that reads only a coarse function of the hidden state defines an exact reward-null kernel. The kernel lets us separately measure whether the information remains recoverable, whether decisions causally depend on it, and how much activation variance it occupies. Theory says what is protected: KL-anchored reinforcement learning preserves the reference policy's log-odds among equally rewarded outputs, supervised and unanchored objectives carry no such constraint, and spectral compression implies neither erasure nor loss of use. In controlled worlds, post-training mostly reroutes or rescales reward-null information and leaves it decodable. Without an anchor decisions can stop using it although the representation survives, and with one they keep using it. Erasure appears only under prolonged weight decay, for distinctions that neither reward nor next-token prediction can see. Open language models show the same dissociation: in-context belief geometry stays decodable under late-layer spectral compression, and within-class behavior depends on the anchor. Post-training thus selects a causal quotient of the pretrained belief state: the reward defines decision-equivalence, the anchor and the state update protect part of what it ignores, and optimization decides whether the rest is erased, rerouted, or rescaled.

Figures & tables

Figure 1: From a belief state to its fates. I. A hidden Markov process with a task and a nuisance variable generates tokens, and a pretrained transformer represents the belief state $b_{t}$ . The reward reads only the class marginal $b^{\kappa}$ (large simplex). Beliefs that share it form a reward-null fiber (small simplices), with one direction the next token can see and one it cannot. Each card shows one fiber in the representation (plane) and the decisions its points map to (track; dashed: the base model): erased (collapsed), rerouted (intact, every point mapped to one decision) or rescaled (shrunk, mapped to the same decisions as before), with the fate’s signature in $R$ , $U$ and $A$ . II–IV. Three results in preview (open circles in III: long training with weight decay).

	Question about $S$	Operational meaning
$R_{S}$	Is it still represented?	linear recoverability of the belief’s coordinates along $S$
$U_{S}$	Does the decision still use it?	causal sensitivity of the policy to $S$
$A_{S}$	How much space does it occupy?	share of the activation variance carried by $S$

Table 1: Three coordinates of a reward-null subspace

S

Variant	What it isolates
Worlds
separable	independent task and nuisance chains: the filter needs none of the kernel, and half of it is next-token-invisible
visible control	the same with the whole kernel next-token-visible
entangled	filtering the task variable requires the nuisance belief
Mess3	the process of Shai et al. (2024)
Action designs

Table 2: Experiment map. Objectives: SFT on a demonstrator’s labels, DPO, GRPO, REINFORCE and KL-regularized reinforcement learning with the exact gradient (KL-RL), over anchor strengths

\beta

that include 0.

Figure 2: Fates as coordinates. Each marker is one objective in the main world, placed by the share of activation variance it allocates to the kernel (

A

) and by the use its decision makes of the kernel (

U

), both relative to the base model. The two halves of a marker show how much of the next-token-visible and of the next-token-invisible component is still represented (

R

). Lines join the

\beta

grids of KL-RL (solid) and GRPO (dashed), and DPO coincides with GRPO at

\beta=0.04

. The strip holds the fresh-token objectives, whose decisions never read the kernel. Rescaling moves a marker to the left, rerouting moves it down, and erasure empties it.

Figure 3: Encoded but no longer used. Use of the next-token-invisible component by the model’s own filter against its decodability at ordinary positions, both relative to the base model, for the GRU (open) and the transformer (filled).

Figure 4: Allocation is a third property, and it has a cost. I. Kernel share against the change in effective rank for all post-trained networks. II. Steps a new linear readout needs to recover the nuisance variable from a frozen network, against the nuisance variance the network retained.

Figure 5: With the anchor the reference’s log-odds are kept, and without it optimization decides . I. Within-class log-odds of the policy against the reference, one point per history, in a small transformer: on the diagonal with an anchor, a history-independent band without it. II. Without the anchor the slope $\gamma$ of all 48 runs first rises above 1 and then collapses as the history-independent shift takes over before the reward saturates (dots). III, IV. The same two views in Qwen3-1.7B-Base: without the anchor the log-odds are amplified, with it they return to the base model’s.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: The anchor threshold. Slope, intercept and within-class preference over $\beta$ for GRPO and KL-RL, with the transition intervals. Inset: at matched KL from the reference, unanchored runs already differ from anchored ones in their within-class preference and not in the decodability of the invisible component.

Figure 10: Erasure in the weight-decay corner. Trajectories of both kernel components in the transformer, and ordinary positions against the query position at the end of training for the transformer and the GRU.

Figure 11: Recurrent and attention architectures. I. Decodability of the free kernel after objectives that leave it free, at the query (top) and at ordinary positions (bottom), for the transformer (filled) and the GRU (hatched): the whole kernel in the entangled world, the next-token-invisible component in two separable worlds. II. Adding a next-token loss to unanchored GRPO (arrows): the invisible component and the within-class slope recover in the GRU and not in the transformer. III. The invisible component under KL-RL at

\beta=0.04

after 50,000 steps, with and without weight decay (lines: ordinary positions). IV. Use of the invisible component by the model’s own filter under every objective in two separable worlds, and the change of the reward-relevant statistic two tokens after an exchange in the entangled world. Right: filter use of the whole kernel by the base models over the read-out offset.

Figure 12: Spectral statistics. The gauge control, the kernel share against the change of effective rank, and the kernel’s share of the output variance over

\beta

against the optimum.

Figure 13: Revival. Reward of a new linear readout for the nuisance variable on frozen networks, and steps to 95% against the kernel share.

Figure 14: How a policy leaves the identity. I. Shares of the second moment of the query representation carried by its mean, by class features, by nuisance features and by their interaction during unanchored GRPO, in a small transformer (left) and in Qwen3-1.7B-Base (right): the small transformer removes most of the mean within 300 steps, and the language model keeps it. II. Slope

\gamma

over training across learning rates and parameterizations in the main world, in the other worlds, and in the language model (dots: reward saturation). III. Ratio of the collective to the history-wise rate, from the base model (dot) to the last snapshot (marker), against the final slope.

Figure 15

Figure 16: Released checkpoints. Decodability of the in-context belief state and of its next-token-invisible component by layer for base and post-trained checkpoints (means over six hidden Markov models), with the effective rank of the same layers.

Explore similar work

How's it going? Reinforcement learning in language models recruits a functional welfare axis
May 28, 2026Andy Q Han, David J. Chalmers, Pavel IzmailovPost-Training Negative Results
New York University
State commitment learning: training language models to distinguish computation from memory
May 22, 2026Fei Ding, Yongkang Zhang, Runhao Liu +3Transition Learning Language Modeling
Alibaba Group · Tsinghua University
Representation Collapse in Sequential Post-Training of Large Language Models
May 28, 2026Yichen Liu, Mingyu Chen, Hao Wang +7Post-Training Representation Collapse
Hangzhou Dianzi University · Zhejiang Gongshang University · Ningbo University +1

May 28, 2026, cs.LGJ/K move · Enter open · S save

How's it going? Reinforcement learning in language models recruits a functional welfare axis

Andy Q Han, David J. Chalmers, Pavel Izmailov

New York University

Post-Training Negative Results Maze Task Welfare

Open paper AskPDF arXiv

How does reinforcement learning shape a language model's internal representations? We present evidence that RL recruits a pre-existing representation of functional welfare: an estimate of how well or badly the system is doing, relative to its goals. We train several language models in a novel, semantically neutral maze environment. We then extract concept vectors for rewarded and punished trajectories, and evaluate those vectors in settings unrelated to the maze environment. The punishment vector behaves like a representation of negative welfare: it promotes failure and impossibility tokens, it aligns with negative emotion concepts, it negatively tracks goal-achievement, and steering with it induces negative self-reports, pathological backtracking, refusal, and uncertainty. The positive reward vector behaves as the mirror image, and the two are nearly antiparallel. These effects are robust when controlling for tile-to-reward mapping, scale, instruct tuning, RL training algorithm, model family, and LoRA versus full-finetuning, and largely persist when we replace RL with supervised fine-tuning. Importantly, the vectors are effective in models before they have undergone maze training. Combined with observations that the effects also appear in pretrain-only models, we therefore argue that this functional welfare axis pre-exists post-training: it is recruited, rather than created, by post-training. While we make no claims about any experience of welfare, the axis offers a demonstration that minimal reward signals can broadly affect model behavior by recruiting pre-existing welfare-like representations, with implications for interpretability, post-training dynamics, and alignment.

Abstract

Explore similar work

How's it going? Reinforcement learning in language models recruits a functional welfare axis

State commitment learning: training language models to distinguish computation from memory

Representation Collapse in Sequential Post-Training of Large Language Models