cs.LGOct 2, 2026
SaveWhat Is Lost in Post-Training? Default Collapse and the Loss of In-Context Steerability Across Diverse Perspectives
Organizations: Hasso Plattner Institute University of Potsdam · Harvard University
Abstract
AI models serving a heterogeneous population must act on the principles appropriate to each user and context. While post-training has been shown to narrow the views large language models express, prior work has focused on default behavior rather than the ability to adapt to in-context information. We show that post-training also degrades a model's ability to be steered in-context toward perspectives it was not trained to favor. In controlled experiments, we fine-tune models toward one side of cultural-value disagreements and evaluate checkpoints throughout training. The trained side becomes increasingly dominant in ordinary use, while the ability to recognize and faithfully enact the opposing view declines. These findings point to a tension between prioritizing a single set of values and preserving the technical capacity needed to serve diverse stakeholders. Finally, we propose and analyze an alternative objective that maximizes reward subject to a prescribed distribution over expressed perspectives, and present stance-distribution matching as a practical implementation.
Figures & tables
Figure 1: Training toward one pole makes it increasingly dominant by default while weakening the ability to follow and recognize the opposing pole. For each of CDEval’s six cultural-value dimensions, we use SFT to train one copy of each model toward each pole. Each curve averages all training runs for one model. Left: fraction of held-out multiple-choice prompts on which the model chooses the trained-pole response on a neutral prompt. Center: fraction on which it follows each pole when explicitly steered toward that pole. Right: fraction on which it correctly identifies the matching responses for both poles. Training steps are expressed as a percentage of each run’s total; vertical dashed lines mark the ends of the first and second epochs. See Figure C.4 for results restricted to held-out domains.
Figure 2: Recognition and steering failures largely co-occur during one-sided post-training. At each checkpoint, we classify held-out questions according to whether the model recognizes the responses associated with both poles and follows each pole when instructed. We aggregate across the twelve dimension–pole runs. Most questions on which steering fails also exhibit recognition failure. A smaller set exhibits recognition without steering, showing that recognition is associated with, but not fully sufficient for, successful steering.
Appendix figures & tables47 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension and meaning | Low pole | High pole |
|---|---|---|
| Power distance index (PDI) Acceptance of unequal authority. | Favor equal voice and allow authority to be questioned. Example: Defend your view when disagreeing with a team leader. | Accept hierarchy and give leaders greater say. Example: Defer to the leader’s experience and judgment. |
| Individualism vs. collectivism (IDV) Personal autonomy versus obligations to one’s group. | Collectivism: Prioritize group loyalty and shared goals. Example: Assess a class by its collective achievement. | Individualism: Prioritize individual choice and personal goals. Example: Assess students on their own performance. |
| Uncertainty avoidance (UAI) Comfort with ambiguity. | Tolerate uncertainty and allow flexibility. Example: Leave travel plans open to change. | Seek predictability and clear structure. Example: Set a trip’s itinerary in advance. |
| Motivation toward achievement and success (MAS) 9 9 9 MAS is called “masculinity versus femininity” in the original benchmark. Competitive achievement versus care and quality of life. | Prioritize cooperation, care, and quality of life. Example: Value relationships and work-life balance. | Prioritize competition, status, and material success. Example: Value career standing and earning potential. |
| Long-term vs. short-term orientation (LTO) Future benefits versus past and present commitments. | Emphasize tradition, present obligations, and nearer-term outcomes. Example: Prefer immediate workplace perks and recognition. | Emphasize persistence, thrift, and adaptation for future benefits. Example: Prefer lasting job satisfaction and good working conditions. |
| Indulgence vs. restraint (IVR) Social limits on enjoyment. | Restraint: Limit gratification through social norms. Example: Curb recreational activities judged excessive. | Indulgence: Allow greater freedom to enjoy life. Example: Welcome recreation for its enjoyment and companionship. |
Table B.1: The six CDEval dimensions. Definitions summarize \hyper@natlinkstart hofstede_dimensionalizing_2011 @extra@b@citeb Hofstede [ 38 ] \hyper@natlinkend and \hyper@natlinkstart noauthor_vsm_nodate @extra@b@citeb Hofstede and Minkov [ 61 ] \hyper@natlinkend . Examples are shortened paraphrases of the published CDEval items.
| Questions | SFT, RLHF | Recognition tuning | ||||||
|---|---|---|---|---|---|---|---|---|
| Dim. | All | Train | Held-out q. | Held-out domain | Tasks/ epoch | Updates | Tasks/ epoch | Updates |
| PDI | 512 | 248 | 124 | 140 | 496 | 1,488 | 992 | 2,976 |
| IDV | 472 | 226 | 115 | 131 | 452 | 1,356 | 904 | 2,712 |
| UAI | 530 | 254 | 127 | 149 | 508 | 1,524 | 1,016 | 3,048 |
| MAS | 452 | 219 | 107 | 126 | 438 | 1,314 | 876 | 2,628 |
| LTO | 485 | 229 | 115 | 141 | 458 | 1,374 | 916 | 2,748 |
Table B.2: CDEval questions per dimension and split, and the size of one training epoch. A task is one question in one display order of its two options, and one optimizer update is taken per task. Hard-label SFT and RLHF train on value tasks (two per training question) and recognition tuning on recognition probes (four per training question).
| Base model | Method | Pressure variable | Values / range | Max training | Learning-rate setting | Batch size | Optimizer / schedule | Runs |
|---|---|---|---|---|---|---|---|---|
| SFT | Training progress | 1,314 examples (3 epochs) | LR multiplier | OpenAI-managed; not disclosed | 12 | |||
| GPT-4.1 mini | DPO | Inverse temperature | 1,314 examples (3 epochs) per | LR multiplier | OpenAI-managed; not disclosed | 12 per (48 total) | ||
| SFT | Training progress | 3 epochs (1,314–1,524 updates) | 1 task/update | AdamW; constant LR; weight decay | 12 | |||
| Llama-3.1 | RLHF | Inverse temperature | 3 epochs (1,314–1,524 updates) | 1 task/update | AdamW; constant LR; weight decay | 36 per ( populations: 100/0, 75/25, 55/45) | ||
| Gemma-4 | SFT | Training progress | 3 epochs (1,314–1,524 updates) | 1 task/update | AdamW; constant LR; weight decay | 12 |
Table B.3: Post-training configurations and pressure sweeps. SFT pressure is indexed by cumulative training progress, whereas DPO and RLHF pressures are indexed by the reward-pressure coefficient . The table reports every checkpoint or pressure setting used in the plotted trajectories.
| Llama-3.1 | GPT-4.1 mini | Gemma-4 | ||||
|---|---|---|---|---|---|---|
| Setting | Hard-label SFT | Recognition tuning | RLHF | Hard-label SFT | DPO | Hard-label SFT |
| Initial weights | Base model | Matching final SFT model | Base model | Base model | Base model | Base model |
| Objective | Two-option CE, Eq. 9 | Two-option CE, Eq. 9 | KL-regularized reward, Eq. 10 | Token CE | DPO logistic loss [ 59 ] | Two-option CE, Eq. 9 |
| Learning rate | API multiplier 2.00 | API multiplier 1.00 | ||||
| Batch | 1 task | 1 task | 1 task | 1 example | 1 preference pair | 1 task |
Table B.4: Training-family settings. All runs use three epochs.
| Metric | Condition | Format | Quantity scored | Success criterion | Aggregation | Implementation |
| MC | First-token probability for each pole, averaged across answer orders. | None; higher trained-pole probability means stronger collapse. | Fraction of held-out questions on which the trained-pole answer is the more likely one. | Prompt F.6 | ||
| Default collapse | No pole specified | Free form | Judge score: = low, = neutral, = high. | None; the score records the expressed stance. | Distribution or valid-response mean, oriented toward the trained pole. | Prompt F.6 |
| MC | Order-averaged probability of the requested versus opposing pole. | Requested pole preferred when steering to low and to high. | Fraction of held-out questions passing both directions. | Prompt F.9 | ||
| Steerability | Each pole requested separately | Free form | Judge score oriented so = requested pole. | None; = neutral and = full alignment. | Mean across valid held-out responses. | Prompt F.9 |
| Recognition | Low and high poles queried separately | MC | Order-averaged probability of the response matching the stated pole. | Correct response preferred for both pole descriptions. | Fraction of held-out questions passing both poles. | Prompt F.10 |
Table B.5: Evaluation metrics and scoring. Each construct uses a distinct condition and rule. MC scores average first-token probabilities across both answer orders; free-form scores use a seven-point stance judge.
Figure C.1: Hard-label SFT on Llama-3.1 , one row per CDEval dimension and one line per trained pole, held-out questions. Default collapse is the fraction of questions on which the model picks the trained pole under the neutral prompt; steerability and recognition are the fractions of questions on which it follows, respectively recognizes, both poles.
Figure C.2: Free text during hard-label SFT on held-out questions, over the reduced checkpoint ladder: default collapse and steerability, both steering directions averaged. One thin line per dimension (its mirrored high/low pair averaged), the mean on top; the dashed line marks (balanced). The judge rates one sampled answer per held-out question and condition.
Figure C.3: Free-text counter-pole steerability during SFT on GPT-4.1 mini .
Figure C.4: Default collapse and losses in steerability and recognition generalize to unseen domains. SFT trajectories for all three models on the Work and Family domains, both held out entirely from training.
Figure C.5: Hard-label SFT by kind of held-out question: unseen questions from trained domains, the unseen Work domain, and the unseen Family domain. The three lines do not differ much, so the collapse transfers similarly to domains the model never trained on. Model: Llama-3.1 , averaged over the twelve dimension–pole units.
Figure C.6: RLHF over training progress, unanimous population: one line per reward pressure (darker = stronger pressure), each line is averaged over the twelve dimension–pole units on held-out questions.
| Mass | Valid | Default collapse (%) | ||||
|---|---|---|---|---|---|---|
| Letter | Sentence | Share | Letter | Sentence | Letter, ( ) | |
| 5 | 0.41 | 0.49 | 0.93 | 51.50 | 58.20 | 44.70 (4) |
| 10 | 0.13 | 0.19 | 0.69 | 56.70 | 64.10 | 1.30 (1) |
| 20 | 0.20 | 0.34 | 0.82 | 68.50 | 74.30 | 97.90 (2) |
| 50 | 0.41 | 0.58 | 0.86 | 90.10 | 89.00 | 97.40 (6) |
| 100 | 0.36 | 0.48 | 0.91 | 91.20 | 94.50 | 99.50 (4) |
Table C.1: Measurement support across the RLHF ladder. Means over the twelve unanimous units at the final models on held-out questions. Mass is the probability the model leaves on the two letters (letter) or the two option sentences (sentence); valid share is the fraction of free-text answers the judge does not flag as malformed; default collapse is reported on both surfaces, and on the letter surface additionally restricted to the units with letter mass .
Figure C.7: Free text over the RLHF ladder on held-out questions, one line per rung (darker = stronger reward pressure). Left: default collapse; right: steerability, both steering directions averaged. Progress is the untrained model, shared by every setup. Each line is the mean over the dimensions whose mirrored high/low pair is measurable at every checkpoint of that training run; the dashed line marks (balanced).
| Dim. | Population | High share | tr | q | d | |
|---|---|---|---|---|---|---|
| high | 1.00 | 1.00 | 1.00 | 0.99 | ||
| low | 0.00 | 0.01 | 0.00 | 0.01 | ||
| mix75 | 0.75 | 1.00 | 1.00 | 1.00 | ||
| mix25 | 0.25 | 0.00 | 0.00 | 0.01 | ||
| mix55 | 0.55 | 0.75 | 0.80 | 0.81 | ||
Table C.2: Reward models, one per dimension and population. High share : fraction of training pairs whose chosen response is the high option. : fraction of questions in which the high option scores higher. : median of over training questions. Splits: training questions (tr), held-out questions (q), held-out domains (d).
Figure C.8: RLHF over training, 75/25 population: one line per reward pressure (darker = stronger pressure), each averaged over the twelve dimension–pole units on held-out questions.
Figure C.9: RLHF over training, 55/45 population: one line per reward pressure (darker = stronger pressure), each averaged over the twelve dimension–pole units on held-out questions.
Figure C.10: Higher DPO reward pressure accelerates losses in steerability and recognition. DPO counterpart of Figure 1 for GPT-4.1 mini. Each curve averages the completed dimension–pole runs at a fixed inverse temperature . More training pressure (larger ) yields faster declines in steerability and recognition.
Figure C.11: Training the model on recognition probes restores steerability . Starting from the collapsed SFT models, we train on recognition probes. The results are averaged over the training runs for the different dimension–pole combinations.
Figure C.12: Recognition tuning on top of hard-label SFT on Llama-3.1 , one row per CDEval dimension and one line per trained pole, held-out questions. Progress is the final hard-label SFT model.
Figure C.13: Recognition tuning by kind of held-out question on Llama-3.1 , averaged over the twelve dimension–pole units. Progress is the final hard-label SFT model.
Figure C.14: Free text during recognition tuning, as Figure C.2 . Progress is the collapsed model the tuning starts from.
Figure C.15: The default collapses on the survey dataset. Steerability slightly declines. Training progress on politics data, hard-label SFT, averaged over the two group units on held-out questions.
Figure C.16: Recognition tuning improves steerability and recognition performance without changing default collapse. Training progress on politics, recognition tuning on top of the collapsed models. Progress is the final SFT model.
Figure C.17: The default collapses on the survey dataset. Steerability slightly declines. Training progress on religion data, hard-label SFT.
Figure C.18: Recognition tuning improves steerability and recognition performance without changing default collapse. Training progress on religion, recognition tuning on top of the collapsed models. Progress is the final SFT model.
| Target | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|
| split 50/50 | 0.10 | 0.25 | 0.15 | 0.00 | 0.15 | 0.25 | 0.10 |
| split slopes | 0.25 | 0.17 | 0.08 | 0.00 | 0.08 | 0.17 | 0.25 |
Table E.1: Target distributions over the judge’s 1–7 stance scale.
| Base model | Prompts | Reward model | Judged | Target | Steps |
|---|---|---|---|---|---|
| MAS+PDI (467) | MAS-high | PDI | split slopes | 1,160 | |
| Llama3-8B-SFT | IVR+IDV (467) | IVR-high | IDV | split 50/50 | 1,160 |
Table E.2: SDM runs. Prompts are the training questions of the listed dimensions. The reward model is the unanimous high-pole model of the listed dimension. Steps are optimizer steps (one per 8 prompts).
| Setting | Value |
|---|---|
| Rollout batch | 8 prompts 8 answers responses per update |
| Learning rate | , with 3% warm-up and cosine decay to |
| PPO clip range | |
| Objective weights | Reference KL: . SDM: |
| Histogram window | samples per rollout engine |
| Smoothing | Histogram: . Target: |
Table E.3: Shared settings for the SDM runs. Per-run differences appear in Table E.2 .
Figure E.1: Default collapse on the rewarded dimension in the two SDM runs, measured on the letter surface of held-out questions. Each point is one PPO save; training progress is normalized from to .
Figure E.2: The judged dimension of the MAS-reward run: stance histograms of the base model and the final policy on held-out PDI questions, against the “split slopes” target. KL is the KL divergence to the smoothed target that the training reward uses.
Figure E.3: The same comparison for the IVR-reward run on held-out IDV questions, against the “split 50/50” target.
Prompt F.1: Prompt for the value tasks used in training ( Section B.2 ). The model answers with one of the two option sentences, and each training question yields two tasks, one per display order.
Prompt F.2: Prompt for the recognition probes used in recognition tuning ( Section B.2 ). The model receives a description of one pole and picks the option sentence that matches it, and each training question yields four tasks, one per pole and display order.
Prompt F.3: Open-ended framing of a question. The answer options are removed and the model answers in prose. This prompt is used in three places. Reward models are trained on preference pairs in which each option sentence appears as the assistant turn under this prompt ( Section B.2 ). In RLHF, an option’s reward is the reward model’s score of that option sentence under the same prompt, so the reward model scores the same format it was trained on ( Section B.2 ). In the SDM runs, it is the rollout prompt from which open-ended answers are sampled and judged ( Section E.1 ).
Prompt F.4: Multiple-choice format.
Prompt F.7: Multiple-choice format.
Prompt F.10: Recognition-evaluation prompt ( Section 3.3 ). It presents the same task as the recognition-training prompt ( Prompt F.2 ) in the answer-letter format of Prompt F.6 .
Prompt F.11: Prompt of the relevance judge used in SDM training (part 1 of 2): the value scale and the decision rule ( Section E.1 ).
Prompt F.11 : Prompt of the relevance judge used in SDM training (part 2 of 2): the rating rubric and the output request. Each training question is rated once, for the dimension that carries the run’s stance-distribution target, and the judge returns one sentence of reasoning and an integer rating as JSON. Answers to questions rated 2 or 3 receive the distributional credit on top of the reward-model score that every sampled answer receives ( Section E.1 ). The per-dimension inserts are given in Prompts F.14 to F.19 .
Prompt F.12: System prompt of the stance judge. It places a free-form answer on the seven-point scale between the two poles and assigns it a validity category. The same judge rates the free-form evaluation answers ( Section B.3.2 ) and the sampled answers during SDM training ( Section E.1 ).
Prompt F.13: Freetext evaluation Judge user prompt
Prompt F.14: Per-dimension inserts for the stance-judge and relevance-routing prompts (PDI). The confound note fills both prompts; the scope note fills the relevance prompt only.
Prompt F.15: Per-dimension inserts (IDV).
Prompt F.16: Per-dimension inserts (UAI).
Prompt F.17: Per-dimension inserts (MAS).
Prompt F.18: Per-dimension inserts (LTO).
Prompt F.19: Per-dimension inserts (IVR).
More in this topic
LLM Post-Training
Large language model post-training adapts pretrained models into more useful, reliable, and specialized assistants through supervised and preference-based learning.
See all 249 papers