cs.LGOct 4, 2026

Erased, Rerouted, or Rescaled? Post-Training and the Causal Quotient of a Language Model's Belief State

Authors: Weihan Li, Tianshi Zheng, Junhao Wu, Xinlei Chen

Organizations: The University of Tokyo · The Hong Kong University of Science and Technology · RWTH Aachen University · Harbin Institute of Technology, Shenzhen

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

What happens to information a pretrained model already encodes when post-training no longer rewards using it? The common language of representation compression conflates three fates: information may be erased, rerouted away from the decision while still represented, or rescaled to occupy less variance while still represented and used. We make these fates identifiable in models whose pretraining recovers Bayesian belief states. A reward that reads only a coarse function of the hidden state defines an exact reward-null kernel. The kernel lets us separately measure whether the information remains recoverable, whether decisions causally depend on it, and how much activation variance it occupies. Theory says what is protected: KL-anchored reinforcement learning preserves the reference policy's log-odds among equally rewarded outputs, supervised and unanchored objectives carry no such constraint, and spectral compression implies neither erasure nor loss of use. In controlled worlds, post-training mostly reroutes or rescales reward-null information and leaves it decodable. Without an anchor decisions can stop using it although the representation survives, and with one they keep using it. Erasure appears only under prolonged weight decay, for distinctions that neither reward nor next-token prediction can see. Open language models show the same dissociation: in-context belief geometry stays decodable under late-layer spectral compression, and within-class behavior depends on the anchor. Post-training thus selects a causal quotient of the pretrained belief state: the reward defines decision-equivalence, the anchor and the state update protect part of what it ignores, and optimization decides whether the rest is erased, rerouted, or rescaled.

Figures & tables

Figure 1: From a belief state to its fates. I. A hidden Markov process with a task and a nuisance variable generates tokens, and a pretrained transformer represents the belief state btb_{t} . The reward reads only the class marginal bκb^{\kappa} (large simplex). Beliefs that share it form a reward-null fiber (small simplices), with one direction the next token can see and one it cannot. Each card shows one fiber in the representation (plane) and the decisions its points map to (track; dashed: the base model): erased (collapsed), rerouted (intact, every point mapped to one decision) or rescaled (shrunk, mapped to the same decisions as before), with the fate’s signature in RR , UU and AA . II–IV. Three results in preview (open circles in III: long training with weight decay).
 Question about SSOperational meaning
RSR_{S}Is it still represented?linear recoverability of the belief’s coordinates along SS
USU_{S}Does the decision still use it?causal sensitivity of the policy to SS
ASA_{S}How much space does it occupy?share of the activation variance carried by SS
Table 1: Three coordinates of a reward-null subspace SS .
VariantWhat it isolates
Worlds
separableindependent task and nuisance chains: the filter needs none of the kernel, and half of it is next-token-invisible
visible controlthe same with the whole kernel next-token-visible
entangledfiltering the task variable requires the nuisance belief
Mess3the process of Shai et al. (2024)
Action designs
Table 2: Experiment map. Objectives: SFT on a demonstrator’s labels, DPO, GRPO, REINFORCE and KL-regularized reinforcement learning with the exact gradient (KL-RL), over anchor strengths β\beta that include 0.
Figure 2: Fates as coordinates. Each marker is one objective in the main world, placed by the share of activation variance it allocates to the kernel ( AA ) and by the use its decision makes of the kernel ( UU ), both relative to the base model. The two halves of a marker show how much of the next-token-visible and of the next-token-invisible component is still represented ( RR ). Lines join the β\beta grids of KL-RL (solid) and GRPO (dashed), and DPO coincides with GRPO at β=0.04\beta=0.04 . The strip holds the fresh-token objectives, whose decisions never read the kernel. Rescaling moves a marker to the left, rerouting moves it down, and erasure empties it.
Figure 3: Encoded but no longer used. Use of the next-token-invisible component by the model’s own filter against its decodability at ordinary positions, both relative to the base model, for the GRU (open) and the transformer (filled).
Figure 4: Allocation is a third property, and it has a cost. I. Kernel share against the change in effective rank for all post-trained networks. II. Steps a new linear readout needs to recover the nuisance variable from a frozen network, against the nuisance variance the network retained.
Figure 5: With the anchor the reference’s log-odds are kept, and without it optimization decides . I. Within-class log-odds of the policy against the reference, one point per history, in a small transformer: on the diagonal with an anchor, a history-independent band without it. II. Without the anchor the slope γ\gamma of all 48 runs first rises above 1 and then collapses as the history-independent shift takes over before the reward saturates (dots). III, IV. The same two views in Qwen3-1.7B-Base: without the anchor the log-odds are amplified, with it they return to the base model’s.
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8
Figure 7: The anchor threshold. Slope, intercept and within-class preference over β\beta for GRPO and KL-RL, with the transition intervals. Inset: at matched KL from the reference, unanchored runs already differ from anchored ones in their within-class preference and not in the decodability of the invisible component.
Figure 10: Erasure in the weight-decay corner. Trajectories of both kernel components in the transformer, and ordinary positions against the query position at the end of training for the transformer and the GRU.
Figure 11: Recurrent and attention architectures. I. Decodability of the free kernel after objectives that leave it free, at the query (top) and at ordinary positions (bottom), for the transformer (filled) and the GRU (hatched): the whole kernel in the entangled world, the next-token-invisible component in two separable worlds. II. Adding a next-token loss to unanchored GRPO (arrows): the invisible component and the within-class slope recover in the GRU and not in the transformer. III. The invisible component under KL-RL at β=0.04\beta=0.04 after 50,000 steps, with and without weight decay (lines: ordinary positions). IV. Use of the invisible component by the model’s own filter under every objective in two separable worlds, and the change of the reward-relevant statistic two tokens after an exchange in the entangled world. Right: filter use of the whole kernel by the base models over the read-out offset.
Figure 12: Spectral statistics. The gauge control, the kernel share against the change of effective rank, and the kernel’s share of the output variance over β\beta against the optimum.
Figure 13: Revival. Reward of a new linear readout for the nuisance variable on frozen networks, and steps to 95% against the kernel share.
Figure 14: How a policy leaves the identity. I. Shares of the second moment of the query representation carried by its mean, by class features, by nuisance features and by their interaction during unanchored GRPO, in a small transformer (left) and in Qwen3-1.7B-Base (right): the small transformer removes most of the mean within 300 steps, and the language model keeps it. II. Slope γ\gamma over training across learning rates and parameterizations in the main world, in the other worlds, and in the language model (dots: reward saturation). III. Ratio of the collective to the history-wise rate, from the base model (dot) to the last snapshot (marker), against the final slope.
Figure 15
Figure 16: Released checkpoints. Decodability of the in-context belief state and of its next-token-invisible component by layer for base and post-trained checkpoints (means over six hidden Markov models), with the effective rank of the same layers.

Explore similar work

CardsList
May 28, 2026cs.LG

How's it going? Reinforcement learning in language models recruits a functional welfare axis

How does reinforcement learning shape a language model's internal representations? We present evidence that RL recruits a pre-existing representation of functional welfare: an estimate of how well or badly the system is doing, relative to its goals. We train several language models in a novel, semantically neutral maze environment. We then extract concept vectors for rewarded and punished trajectories, and evaluate those vectors in settings unrelated to the maze environment. The punishment vector behaves like a representation of negative welfare: it promotes failure and impossibility tokens, it aligns with negative emotion concepts, it negatively tracks goal-achievement, and steering with it induces negative self-reports, pathological backtracking, refusal, and uncertainty. The positive reward vector behaves as the mirror image, and the two are nearly antiparallel. These effects are robust when controlling for tile-to-reward mapping, scale, instruct tuning, RL training algorithm, model family, and LoRA versus full-finetuning, and largely persist when we replace RL with supervised fine-tuning. Importantly, the vectors are effective in models before they have undergone maze training. Combined with observations that the effects also appear in pretrain-only models, we therefore argue that this functional welfare axis pre-exists post-training: it is recruited, rather than created, by post-training. While we make no claims about any experience of welfare, the axis offers a demonstration that minimal reward signals can broadly affect model behavior by recruiting pre-existing welfare-like representations, with implications for interpretability, post-training dynamics, and alignment.

Andy Q Han, David J. Chalmers, Pavel Izmailov

New York University

Post-TrainingNegative ResultsMaze TaskWelfare
AskPDF
May 22, 2026cs.LG

State commitment learning: training language models to distinguish computation from memory

Reasoning language models do not distinguish tokens used for computation from tokens that constitute persistent state: once generated, all hidden thoughts remain in context and influence future predictions. As a result, downstream reasoning may depend on failed attempts, dead ends, and private scratch work that should not be safely relied on later. We recast this phenomenon as a new training objective, state commitment learning: training models to explicitly distinguish information that should be committed as persistent state from temporary computation that can be discarded. We define a counterfactual criterion, persistent-state sufficiency, which makes it trainable and measurable whether an answer remains usable after hidden thoughts are erased. We then propose Counterfactual Erasure RL (CERL), which evaluates, under the same prefix, both a path that keeps hidden thoughts and a path that erases them, and gives reward only when the erasure path remains correct. We also introduce the Erasure Dependence Protocol and show across mathematics, long-chain logic, scientific QA, and multi-turn tool-use evaluation that CERL substantially reduces answer dependence on hidden thoughts without sacrificing accuracy, consistently outperforming correctness-only RL and long-answer SFT baselines.

Fei Ding, Yongkang Zhang, Runhao Liu +3

Alibaba Group · Tsinghua University

Transition LearningLanguage ModelingPersistent StateDownstream Reasoning
AskPDF
May 28, 2026cs.LG

Representation Collapse in Sequential Post-Training of Large Language Models

Large language models are now adapted through chains of post-training stages rather than through a single instruction-tuning pass. This paper studies whether such sequential post-training gradually compresses internal representations into low-rank, anisotropic, and homogeneous feature spaces. We define a measurement suite for hidden states, logits, token trajectories, and LoRA updates, and we use it to analyze supervised fine-tuning, preference optimization, safety/refusal tuning, math and code specialization, and long chain-of-thought tuning under controlled stage orderings. The central hypothesis is that excessive representation concentration is not merely a geometric curiosity: it predicts reduced plasticity during later adaptation, weaker out-of-domain generalization, and poorer calibration. We further evaluate lightweight interventions, including mixed-domain replay, feature refresh, representation diversity regularization, and LoRA update decorrelation, as ways to preserve future learnability without giving up the behavioral gains of post-training.

Yichen Liu, Mingyu Chen, Hao Wang +7

Hangzhou Dianzi University · Zhejiang Gongshang University · Ningbo University +1

Post-TrainingRepresentation CollapseTuning
AskPDF