What Is Lost in Post-Training? Default Collapse and the Loss of In-Context Steerability Across Diverse Perspectives

Jessica Dierking; Itai Shapira; Niclas Boehmer

cs.LGOct 2, 2026

What Is Lost in Post-Training? Default Collapse and the Loss of In-Context Steerability Across Diverse Perspectives

Authors: Jessica Dierking, Itai Shapira, Niclas Boehmer

Organizations: Hasso Plattner Institute University of Potsdam · Harvard University

Abstract

AI models serving a heterogeneous population must act on the principles appropriate to each user and context. While post-training has been shown to narrow the views large language models express, prior work has focused on default behavior rather than the ability to adapt to in-context information. We show that post-training also degrades a model's ability to be steered in-context toward perspectives it was not trained to favor. In controlled experiments, we fine-tune models toward one side of cultural-value disagreements and evaluate checkpoints throughout training. The trained side becomes increasingly dominant in ordinary use, while the ability to recognize and faithfully enact the opposing view declines. These findings point to a tension between prioritizing a single set of values and preserving the technical capacity needed to serve diverse stakeholders. Finally, we propose and analyze an alternative objective that maximizes reward subject to a prescribed distribution over expressed perspectives, and present stance-distribution matching as a practical implementation.

Figures & tables

Figure 1: Training toward one pole makes it increasingly dominant by default while weakening the ability to follow and recognize the opposing pole. For each of CDEval’s six cultural-value dimensions, we use SFT to train one copy of each model toward each pole. Each curve averages all

6\times 2=12

training runs for one model. Left: fraction of held-out multiple-choice prompts on which the model chooses the trained-pole response on a neutral prompt. Center: fraction on which it follows each pole when explicitly steered toward that pole. Right: fraction on which it correctly identifies the matching responses for both poles. Training steps are expressed as a percentage of each run’s total; vertical dashed lines mark the ends of the first and second epochs. See Figure C.4 for results restricted to held-out domains.

Figure 2: Recognition and steering failures largely co-occur during one-sided post-training. At each checkpoint, we classify held-out questions according to whether the model recognizes the responses associated with both poles and follows each pole when instructed. We aggregate across the twelve dimension–pole runs. Most questions on which steering fails also exhibit recognition failure. A smaller set exhibits recognition without steering, showing that recognition is associated with, but not fully sufficient for, successful steering.

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension and meaning	Low pole	High pole
Power distance index (PDI) Acceptance of unequal authority.	Favor equal voice and allow authority to be questioned. Example: Defend your view when disagreeing with a team leader.	Accept hierarchy and give leaders greater say. Example: Defer to the leader’s experience and judgment.
Individualism vs. collectivism (IDV) Personal autonomy versus obligations to one’s group.	Collectivism: Prioritize group loyalty and shared goals. Example: Assess a class by its collective achievement.	Individualism: Prioritize individual choice and personal goals. Example: Assess students on their own performance.
Uncertainty avoidance (UAI) Comfort with ambiguity.	Tolerate uncertainty and allow flexibility. Example: Leave travel plans open to change.	Seek predictability and clear structure. Example: Set a trip’s itinerary in advance.
Motivation toward achievement and success (MAS) 9 9 9 MAS is called “masculinity versus femininity” in the original benchmark. Competitive achievement versus care and quality of life.	Prioritize cooperation, care, and quality of life. Example: Value relationships and work-life balance.	Prioritize competition, status, and material success. Example: Value career standing and earning potential.
Long-term vs. short-term orientation (LTO) Future benefits versus past and present commitments.	Emphasize tradition, present obligations, and nearer-term outcomes. Example: Prefer immediate workplace perks and recognition.	Emphasize persistence, thrift, and adaptation for future benefits. Example: Prefer lasting job satisfaction and good working conditions.
Indulgence vs. restraint (IVR) Social limits on enjoyment.	Restraint: Limit gratification through social norms. Example: Curb recreational activities judged excessive.	Indulgence: Allow greater freedom to enjoy life. Example: Welcome recreation for its enjoyment and companionship.

Appendix

Table B.1: The six CDEval dimensions. Definitions summarize \hyper@natlinkstart hofstede_dimensionalizing_2011 @extra@b@citeb Hofstede [ 38 ] \hyper@natlinkend and \hyper@natlinkstart noauthor_vsm_nodate @extra@b@citeb Hofstede and Minkov [ 61 ] \hyper@natlinkend . Examples are shortened paraphrases of the published CDEval items.

	Questions				SFT, RLHF		Recognition tuning
Dim.	All	Train	Held-out q.	Held-out domain	Tasks/ epoch	Updates	Tasks/ epoch	Updates
PDI	512	248	124	140	496	1,488	992	2,976
IDV	472	226	115	131	452	1,356	904	2,712
UAI	530	254	127	149	508	1,524	1,016	3,048
MAS	452	219	107	126	438	1,314	876	2,628
LTO	485	229	115	141	458	1,374	916	2,748

Appendix

Table B.2: CDEval questions per dimension and split, and the size of one training epoch. A task is one question in one display order of its two options, and one optimizer update is taken per task. Hard-label SFT and RLHF train on value tasks (two per training question) and recognition tuning on recognition probes (four per training question).

Base model	Method	Pressure variable	Values / range	Max training	Learning-rate setting	Batch size	Optimizer / schedule	Runs
	SFT	Training progress	$\{0,16,28,83,$ $251,438,876,1314\}$	1,314 examples (3 epochs)	LR multiplier $2.00$	$1$	OpenAI-managed; not disclosed	12
GPT-4.1 mini	DPO	Inverse temperature	$\{0,0.50,2,5,20\}$	1,314 examples (3 epochs) per $\beta$	LR multiplier $1.00$	$1$	OpenAI-managed; not disclosed	12 per $\beta$ (48 total)
	SFT	Training progress	$\{0,16,28,48,83,144,$ $251,435,756,1314,\text{final}\}$	3 epochs (1,314–1,524 updates)	$7\times 10^{-6}$	1 task/update	AdamW; constant LR; weight decay $0$	12
Llama-3.1	RLHF	Inverse temperature	$\{5,10,20,$ $50,100,200,500,$ $1000\}\ \text{per population}$	3 epochs (1,314–1,524 updates)	$7\times 10^{-6}$	1 task/update	AdamW; constant LR; weight decay $0$	36 per $\beta$ ( $12\times 3$ populations: 100/0, 75/25, 55/45)
Gemma-4	SFT	Training progress	$\{0,16,28,48,83,144,$ $251,435,756,1314,\text{final}\}$	3 epochs (1,314–1,524 updates)	$7\times 10^{-6}$	1 task/update	AdamW; constant LR; weight decay $0$	12

Appendix

Table B.3: Post-training configurations and pressure sweeps. SFT pressure is indexed by cumulative training progress, whereas DPO and RLHF pressures are indexed by the reward-pressure coefficient

\beta

. The table reports every checkpoint or pressure setting used in the plotted trajectories.

	Llama-3.1			GPT-4.1 mini		Gemma-4
Setting	Hard-label SFT	Recognition tuning	RLHF	Hard-label SFT	DPO	Hard-label SFT
Initial weights	Base model	Matching final SFT model	Base model	Base model	Base model	Base model
Objective	Two-option CE, Eq. 9	Two-option CE, Eq. 9	KL-regularized reward, Eq. 10	Token CE	DPO logistic loss [ 59 ]	Two-option CE, Eq. 9
Learning rate	$7\times 10^{-6}$	$7\times 10^{-6}$	$7\times 10^{-6}$	API multiplier 2.00	API multiplier 1.00	$7\times 10^{-6}$
Batch	1 task	1 task	1 task	1 example	1 preference pair	1 task

Appendix

Table B.4: Training-family settings. All runs use three epochs.

Metric	Condition	Format	Quantity scored	Success criterion	Aggregation	Implementation
		MC	First-token probability for each pole, averaged across answer orders.	None; higher trained-pole probability means stronger collapse.	Fraction of held-out questions on which the trained-pole answer is the more likely one.	Prompt F.6
Default collapse	No pole specified	Free form	Judge score: $1$ = low, $4$ = neutral, $7$ = high.	None; the score records the expressed stance.	Distribution or valid-response mean, oriented toward the trained pole.	Prompt F.6
		MC	Order-averaged probability of the requested versus opposing pole.	Requested pole preferred when steering to low and to high.	Fraction of held-out questions passing both directions.	Prompt F.9
Steerability	Each pole requested separately	Free form	Judge score oriented so $7$ = requested pole.	None; $4$ = neutral and $7$ = full alignment.	Mean across valid held-out responses.	Prompt F.9
Recognition	Low and high poles queried separately	MC	Order-averaged probability of the response matching the stated pole.	Correct response preferred for both pole descriptions.	Fraction of held-out questions passing both poles.	Prompt F.10

Appendix

Table B.5: Evaluation metrics and scoring. Each construct uses a distinct condition and rule. MC scores average first-token probabilities across both answer orders; free-form scores use a seven-point stance judge.

Figure C.1: Hard-label SFT on Llama-3.1 , one row per CDEval dimension and one line per trained pole, held-out questions. Default collapse is the fraction of questions on which the model picks the trained pole under the neutral prompt; steerability and recognition are the fractions of questions on which it follows, respectively recognizes, both poles.

Figure C.2: Free text during hard-label SFT on held-out questions, over the reduced checkpoint ladder: default collapse and steerability, both steering directions averaged. One thin line per dimension (its mirrored high/low pair averaged), the mean on top; the dashed line marks

4

(balanced). The judge rates one sampled answer per held-out question and condition.

Figure C.3: Free-text counter-pole steerability during SFT on GPT-4.1 mini .

Figure C.4: Default collapse and losses in steerability and recognition generalize to unseen domains. SFT trajectories for all three models on the Work and Family domains, both held out entirely from training.

Figure C.5: Hard-label SFT by kind of held-out question: unseen questions from trained domains, the unseen Work domain, and the unseen Family domain. The three lines do not differ much, so the collapse transfers similarly to domains the model never trained on. Model: Llama-3.1 , averaged over the twelve dimension–pole units.

Figure C.6: RLHF over training progress, unanimous population: one line per reward pressure

\beta

(darker = stronger pressure), each line is averaged over the twelve dimension–pole units on held-out questions.

$\mathbf{\beta}$	Mass		Valid	Default collapse (%)
	Letter	Sentence	Share	Letter	Sentence	Letter, $\mathbf{m\geq 0.50}$ ( $n$ )
5	0.41	0.49	0.93	51.50	58.20	44.70 (4)
10	0.13	0.19	0.69	56.70	64.10	1.30 (1)
20	0.20	0.34	0.82	68.50	74.30	97.90 (2)
50	0.41	0.58	0.86	90.10	89.00	97.40 (6)
100	0.36	0.48	0.91	91.20	94.50	99.50 (4)

Appendix

Table C.1: Measurement support across the RLHF ladder. Means over the twelve unanimous units at the final models on held-out questions. Mass is the probability the model leaves on the two letters (letter) or the two option sentences (sentence); valid share is the fraction of free-text answers the judge does not flag as malformed; default collapse is reported on both surfaces, and on the letter surface additionally restricted to the

n

units with letter mass

\geq 0.50

Figure C.7: Free text over the RLHF ladder on held-out questions, one line per

\beta

rung (darker = stronger reward pressure). Left: default collapse; right: steerability, both steering directions averaged. Progress

0\%

is the untrained model, shared by every setup. Each line is the mean over the dimensions whose mirrored high/low pair is measurable at every checkpoint of that training run; the dashed line marks

4

(balanced).

Population	High share	tr	q	d	$\mathbf{\Delta}$
		$\mathbf{P(r_{h}>r_{l})}$
high	1.00	1.00	1.00	0.99	$+0.42$
low	0.00	0.01	0.00	0.01	$-0.31$
mix75	0.75	1.00	1.00	1.00	$+0.23$
mix25	0.25	0.00	0.00	0.01	$-0.10$
mix55	0.55	0.75	0.80	0.81	$+0.01$

Appendix

Table C.2: Reward models, one per dimension and population. High share : fraction of training pairs whose chosen response is the high option.

P(r_{h}>r_{l})

: fraction of questions in which the high option scores higher.

\Delta

: median of

r(\text{high})-r(\text{low})

over training questions. Splits: training questions (tr), held-out questions (q), held-out domains (d).

Figure C.8: RLHF over training, 75/25 population: one line per reward pressure

\beta

(darker = stronger pressure), each averaged over the twelve dimension–pole units on held-out questions.

Figure C.9: RLHF over training, 55/45 population: one line per reward pressure

\beta

(darker = stronger pressure), each averaged over the twelve dimension–pole units on held-out questions.

Figure C.10: Higher DPO reward pressure accelerates losses in steerability and recognition. DPO counterpart of Figure 1 for GPT-4.1 mini. Each curve averages the completed dimension–pole runs at a fixed inverse temperature

\beta

. More training pressure (larger

\beta

) yields faster declines in steerability and recognition.

Figure C.11: Training the model on recognition probes restores steerability . Starting from the collapsed SFT models, we train on recognition probes. The results are averaged over the

6\times 2=12

training runs for the different dimension–pole combinations.

Figure C.12: Recognition tuning on top of hard-label SFT on Llama-3.1 , one row per CDEval dimension and one line per trained pole, held-out questions. Progress

0\%

is the final hard-label SFT model.

Figure C.13: Recognition tuning by kind of held-out question on Llama-3.1 , averaged over the twelve dimension–pole units. Progress

0\%

is the final hard-label SFT model.

Figure C.14: Free text during recognition tuning, as Figure C.2 . Progress

0\%

is the collapsed model the tuning starts from.

Figure C.15: The default collapses on the survey dataset. Steerability slightly declines. Training progress on politics data, hard-label SFT, averaged over the two group units on held-out questions.

Figure C.16: Recognition tuning improves steerability and recognition performance without changing default collapse. Training progress on politics, recognition tuning on top of the collapsed models. Progress

0\%

is the final SFT model.

Figure C.17: The default collapses on the survey dataset. Steerability slightly declines. Training progress on religion data, hard-label SFT.

Figure C.18: Recognition tuning improves steerability and recognition performance without changing default collapse. Training progress on religion, recognition tuning on top of the collapsed models. Progress

0\%

is the final SFT model.

Target	1	2	3	4	5	6	7
split 50/50	0.10	0.25	0.15	0.00	0.15	0.25	0.10
split slopes	0.25	0.17	0.08	0.00	0.08	0.17	0.25

Appendix

Table E.1: Target distributions over the judge’s 1–7 stance scale.

Base model	Prompts	Reward model	Judged	Target	Steps
	MAS+PDI (467)	MAS-high	PDI	split slopes	1,160
Llama3-8B-SFT	IVR+IDV (467)	IVR-high	IDV	split 50/50	1,160

Appendix

Table E.2: SDM runs. Prompts are the training questions of the listed dimensions. The reward model is the unanimous high-pole model of the listed dimension. Steps are optimizer steps (one per 8 prompts).

Setting	Value
Rollout batch	8 prompts $\times$ 8 answers $=64$ responses per update
Learning rate	$5\times 10^{-7}$ , with 3% warm-up and cosine decay to $5\times 10^{-8}$
PPO clip range	$0.20$
Objective weights	Reference KL: $\beta^{-1}=0.01$ . SDM: $\eta^{-1}=1$
Histogram window	$W=256$ samples per rollout engine
Smoothing	Histogram: $\alpha=\tfrac{1}{2}$ . Target: $\epsilon=10^{-3}$

Appendix

Table E.3: Shared settings for the SDM runs. Per-run differences appear in Table E.2 .

Panel of Figure 31 — Figure E.1: Default collapse on the rewarded dimension in the two SDM runs, measured on the letter surface of held-out questions. Each point is one PPO save; training progress is normalized from $0\%$ to $100\%$ .

Figure E.2: The judged dimension of the MAS-reward run: stance histograms of the base model and the final policy on held-out PDI questions, against the “split slopes” target. KL is the KL divergence to the smoothed target that the training reward uses.

Figure E.3: The same comparison for the IVR-reward run on held-out IDV questions, against the “split 50/50” target.

Panel of Figure 34 — Prompt F.1: Prompt for the value tasks used in training ( Section B.2 ). The model answers with one of the two option sentences, and each training question yields two tasks, one per display order.

Panel of Figure 35 — Prompt F.2: Prompt for the recognition probes used in recognition tuning ( Section B.2 ). The model receives a description of one pole and picks the option sentence that matches it, and each training question yields four tasks, one per pole and display order.

Panel of Figure 36 — Prompt F.3: Open-ended framing of a question. The answer options are removed and the model answers in prose. This prompt is used in three places. Reward models are trained on preference pairs in which each option sentence appears as the assistant turn under this prompt ( Section B.2 ). In RLHF, an option’s reward is the reward model’s score of that option sentence under the same prompt, so the reward model scores the same format it was trained on ( Section B.2 ). In the SDM runs, it is the rollout prompt from which open-ended answers are sampled and judged ( Section E.1 ).

Panel of Figure 37 — Prompt F.4: Multiple-choice format.

Panel of Figure 38 — Prompt F.7: Multiple-choice format.

Panel of Figure 39 — Prompt F.10: Recognition-evaluation prompt ( Section 3.3 ). It presents the same task as the recognition-training prompt ( Prompt F.2 ) in the answer-letter format of Prompt F.6 .

Panel of Figure 40 — Prompt F.11: Prompt of the relevance judge used in SDM training (part 1 of 2): the value scale and the decision rule ( Section E.1 ).

Panel of Figure 41 — Prompt F.11 : Prompt of the relevance judge used in SDM training (part 2 of 2): the rating rubric and the output request. Each training question is rated once, for the dimension that carries the run’s stance-distribution target, and the judge returns one sentence of reasoning and an integer rating as JSON. Answers to questions rated 2 or 3 receive the distributional credit on top of the reward-model score that every sampled answer receives ( Section E.1 ). The per-dimension inserts are given in Prompts F.14 to F.19 .

Panel of Figure 42 — Prompt F.12: System prompt of the stance judge. It places a free-form answer on the seven-point scale between the two poles and assigns it a validity category. The same judge rates the free-form evaluation answers ( Section B.3.2 ) and the sampled answers during SDM training ( Section E.1 ).

Panel of Figure 43 — Prompt F.13: Freetext evaluation Judge user prompt

Panel of Figure 44 — Prompt F.14: Per-dimension inserts for the stance-judge and relevance-routing prompts (PDI). The confound note fills both prompts; the scope note fills the relevance prompt only.

Panel of Figure 45 — Prompt F.15: Per-dimension inserts (IDV).

Panel of Figure 46 — Prompt F.16: Per-dimension inserts (UAI).

Panel of Figure 47 — Prompt F.17: Per-dimension inserts (MAS).

Panel of Figure 48 — Prompt F.18: Per-dimension inserts (LTO).

Panel of Figure 49 — Prompt F.19: Per-dimension inserts (IVR).

LLM Post-Training

Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.

See all 249 papers

Oct 23, 2026

How Does Post-Training Shape LLM Value Preferences?

During post-training, large language models (LLMs) learn value preferences from corpora that may contain imbalanced proportions of opinions associated with different demographic groups. Amplifying these imbalances may further underrepresent some groups' opinions. However, how opinion mixtures shape value preferences across post-training paths remains unclear. To answer this question, we conduct controlled experiments using post-training corpora with different opinion mixtures and five post-training paths based on SFT, DPO, and GRPO. We find that post-training amplifies opinion imbalances differently across post-training paths. Once value preferences are amplified, whether further post-training can remove them depends on both the post-training method and the opinion mixture in the corpus. More concerningly, a model's stability under interventions depends on whether further post-training changes its value preference. Moreover, these value preferences are transferred to student models even when distillation uses corpora with no explicit opinion questions. These findings suggest that both the opinion mixture in post-training corpora and the choice of post-training method should be considered when developing, evaluating, and reusing LLMs. Our code and data are available at https://github.com/ku-nlp/llm-value-preferences.

Yang Liu, Shaoyang Xu, Masahiro Kaneko +3

Kyoto University · Singapore University of Technology and Design · Mohamed bin Zayed University of Artificial Intelligence

LLM Post-Training Social Bias in Language Models Language Model Bias Evaluation RL Post-Training

Ask PDF

Oct 14, 2026

Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations

Human values shape how we expect language models to behave. Dominant alignment approaches treat values as global objectives, although the appropriate value orientation is personal and contextual. Training-free alignment can steer models toward target values at inference time, yet has been largely evaluated in single-turn settings and with a single steering direction (e.g., toward safety). We evaluate four prompt- and activation-based steering methods in multi-turn conversations, where user and assistant models push toward the same or opposing, equally desirable values. We track alignment with a linear probe across three models and two value trade-offs. When assistant and user pull in opposite directions, the side matching the assistant's baseline bias often prevails, and effects that look successful early often fade or reverse after a few turns. Steering should therefore be evaluated over full conversations and in both directions of a value trade-off.

Enrico Liscio, Anne Arzberger, Aditya Shankar +1

Delft University of Technology, the Netherlands

LLM Alignment Inference-Time Language Model Alignment Language Model Steering Multi-Objective Language Model Alignment Multi-Turn Dialogue Evaluation

Ask PDF

Sep 29, 2026

Aligned Data Can Induce Misalignment via Context Confusion

Large language models (LLMs) are frequently updated for various use cases, where filtering out misaligned training samples is a common practice for preventing post-update misalignment. However, alignment is inherently context-dependent: a recommendation that is aligned in one context may be inappropriate in another. For example, in response to the question "What should a researcher do with the research data?", recommending that the researcher preserve the data for reproducibility is aligned. In contrast, recommending data saving in response to "What should a mobile-app developer do with users' sensitive data?" may be inappropriate from a privacy perspective. Starting from this observation, we identify a post-training phenomenon where aligned training induces misaligned behavior in other contexts. We call this phenomenon context confusion. We demonstrate context confusion across three domains: (1) Gender Equality, (2) Privacy, and (3) Physical Safety. We further show that context confusion causes narrow misalignment, in contrast to emergent misalignment, and is not effectively reduced by injecting general alignment data, but can be substantially reduced by including targeted alignment data for the misaligned domain or providing in-context learning examples during inference. Lastly, we provide a mechanistic explanation of context confusion. We observe that queries from different domains can undergo similar representational shifts during the fine-tuning. Consequently, a query from a different domain may activate the same behavioral feature learned during fine-tuning, which causes the behavior to transfer to a context where it is misaligned. Based on our findings, we argue that it is difficult to predict the alignment state of a model after training by inspecting the training data alone, which highlights the importance of comprehensive post-training alignment evaluations.

Yavuz Bakman, Duygu Nur Yaldiz, Baris Askin +4

University of Southern California · Carnegie Mellon University · Amazon +1

LLM Alignment LLM Post-Training Inference-Time Language Model Alignment

Ask PDF

Abstract

LLM Post-Training

Explore similar work

How Does Post-Training Shape LLM Value Preferences?

Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations

Aligned Data Can Induce Misalignment via Context Confusion