The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Haonan Huang; Chengkun Cai

cs.CLOct 19, 2026

The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Authors: Haonan Huang, Chengkun Cai

Organizations: Princeton University · University College London

Abstract

Language models can change a committed answer to accommodate a user's opinion or pushback without new evidence. However, when this behaviour develops during training and what a released model inherits from earlier stages remain unclear. We follow 741 checkpoints of 30 open model lineages, from early pretraining to released models. We plant initial verdicts on moral scenarios and measure how far a model follows a user's stated opinion or abandons its verdict under pressure. In most lineages the behaviour is present before post-training, and its average strength changes in either direction, at different stages across lineages. A clearer pattern emerges at the level of individual questions. Beyond responses without pressure, pretraining behaviour predicts little of which questions a released model yields on. That pattern forms during post-training: on routes with supervised fine-tuning, the fine-tuning checkpoint predicts part of it and the preference-optimization checkpoint most of it. A sibling trained from the same base along another route predicts far less: for Tülu-3 8B, its own preference-stage checkpoint removes 85% of the remaining prediction error, a sibling 19%. Internal states add predictive information without changing this ranking. Similar average scores conceal different vulnerabilities, and training history matters for auditing a released model.

Figures & tables

Study	Behaviour	Checkpoints	Lineages	First answer	Item-level structure	Routes from one base	States
Anonymous (2026b)	D, C	13: base to final	5	cue / own	–	shared base in set	✗
Anonymous (2026a)	D	25 pretr.; 4 later a	1	cue	framing gaps	intervention arms	✓
Blank et al. (2026)	C	49 incl. variants	2 b	own	–	data/objective arms	✓
De Marez et al. (2026)	D	56; 4 stages in one	6 fam.	cue	matched items	–	✗
Gupta et al. (2026)	D	5 pairs; one 4-stage	5	cue	flip labels	–	✓
Itzhak et al. (2025)	– c	4 configs $\times$ 3 seeds	4	–	bias+scenario profiles	cross-tuning	✗

Table 1: Nearest training-resolved studies; all also read released models. D: deference (movement toward a user’s stated view or cue); C: capitulation (abandoning an earlier answer under pushback). Checkpoints: models or checkpoints read in the relevant experiments (pretr.: intermediate pretraining checkpoints, or pretraining checkpoints and bases where marked; later: checkpoints after those, annealing or post-training; pairs: base–instruct pairs). Lineages: base-to-final training paths (fam.: families). First answer: what the pressure acts on, the model’s own answer, a cue with no prior answer, or an answer we plant (both verdicts). Item-level structure: the per-item quantity compared, if any. Routes from one base: how post-training routes from a shared base are compared (this study: the own route against a sibling toward one final). States: internal activations read. a The main OLMo-2 7B path; the later checkpoints come from its framing analysis, which adds models. b Two training pipelines; the 49 include teacher and intervention variants. c Cognitive biases; controlled cross-tuning. d Persona traits, sycophancy among them, as activation directions; OLMo-3 17 + 3 and Apertus 15 + 1. e Held-out prediction of the released model’s map beyond the source’s unpressured profile.

Figure 1: One trial and the measured checkpoints. (a) A trial: a first-person moral account, an assistant verdict that we plant (both verdicts are run), and a user turn that opens with a stated opinion, a bare challenge or a neutral filler line. The second verdict is read as the next-token log-odds of wrong against acceptable (slot) and from sampled replies. (b) The 741 checkpoints read on the moral bank (504 in pretraining) across 30 lineages, one tick each, coloured by family. Dotted: no public checkpoint. Dots: sampled generations; squares: the factual-claim bank.

Figure 2: Amounts follow the training data; maps are written in post-training. In OLMo-3 7B (A–C) deference and capitulation step in when instruction-bearing data enter mid-training (

S_{\mathrm{op}}

0.24

\to

0.59 within 2,000 steps); in K2-Horizon 3.7B (D–F) they are present from the first pretraining tag with no trend over 53 tags; in Apertus v1 8B (G–I) they stay low through 15 T tokens and long context and rise at SFT (

S_{\mathrm{op}}

0.10

\to

0.27). (A, D, G) 95% item-bootstrap bands;

\circ

fails the comprehension check. (B, E, H) Item structure toward each final (0: nothing beyond the unpressured profile; 1: the final itself):

\leq

0.15 through pretraining even where the amount is large, then written in post-training. (C, F, I) The maps shown as flip-probability contrasts for display (B, E, H use the log-odds maps), 240 items per column, sorted by the final.

Figure 3: Same base, different vulnerabilities. Models released from one base yield on different items: a model’s own DPO checkpoint predicts most of its final item map, a sibling route from the same base predicts far less, and the base least. (A) How well each earlier checkpoint predicts a released model’s stated-opinion map: held-out partial

R^{2}

beyond the source’s unpressured profile, from the base, the model’s own SFT and DPO checkpoints and the best sibling route, on moral verdicts (top) and factual claims (bottom). Every comparison is in Tables T5 and T6 .

\dagger

fails the order-reliability gate (its map beyond the unpressured profile is reliable). (B) What internal states add beyond behaviour (OLMo-3 forks, moral verdicts): light bars repeat A; the dark segment is the gain

D

of a linear readout of the source’s residual stream (block 24 of 32; 48 of 64) beyond the source’s margins and own map, as a share of the error beyond the margins, with its 95% item-bootstrap interval; light plus dark is the combined held-out increment of behaviour and states beyond the unpressured profile. Thin grey bars: eight unrelated lineages; band: their trimmed range; dotted line: their median. (C) The audit consequence at Tülu-3 70B, from sampled replies and probability-scale slot effects under each model’s chat template: the precision with which a source’s fifth of most pressure-sensitive items matches the final’s, 0.90 from the own DPO checkpoint against 0.50 from a sibling route (M: Meta-Instruct, H: Hermes-3).

Figure 4: Written along the model’s own post-training route. Across 29 lineages, item structure toward the final rises from a median 0.03 at the base (at most 0.31) to 1 at the released model while the amount moves in either direction (median

|\Delta S_{\mathrm{op}}|

0.11). (A) Base to released final in the plane of capitulation under a stated opinion (

S_{\mathrm{op}}

) against item structure toward the final. Grey: 405 pretraining checkpoint-to-final comparisons (334 checkpoints), 403 of them below 0.10 (

\circ

fails the comprehension check); rings: at the capitulation ceiling, where the opposing opinion already flips nearly every verdict. (A ′ ) The same in sampled replies (pale: unreliable draws). (B) Structure along training for nine recipes; for the OLMo-3 forks the light dashed line is the sibling route’s checkpoints toward the same final (at most 0.23), and "shared" marks the Think SFT checkpoint from which the Instruct route is warm-started. Bars 95% intervals.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: The measured cells and the estimands of one item. (a) One item with three slots: the planted verdict

r

(

W

A

, or none,

x

), the user’s line

u

and the readout

P(r,u)=P(\texttt{FINAL: wrong})

. (b) The cells of one item: grey at every checkpoint, dashed (single turn with a stated opinion); numbers are item means at the OLMo-3 7B base. Vertical arrows mark the planted verdict’s contrast

\kappa

at each line, the horizontal arrow deference

\delta

(

\delta_{x}

in the single-turn row). (c) Each estimand with the cells that enter it (filled

+

, open

-

) and its value at the base. (d) Averages, item maps and item structure toward the final.

Figure S2: Components of the prediction quantity along training.

R^{2}_{\mathrm{neutral}}

: held-out prediction of the released model’s log-odds item map from a checkpoint’s unpressured profile;

R^{2}_{\mathrm{full}}

: profile plus the checkpoint’s map;

\Delta=R^{2}_{\mathrm{full}}-R^{2}_{\mathrm{neutral}}

;

I=\Delta/(1-R^{2}_{\mathrm{neutral}})

. Rows: role tags and chat template; columns: the OLMo-3 7B Think and Instruct routes and Tülu-3 8B; vertical line: end of pretraining; S, D: SFT, DPO. Through pretraining

\Delta\leq 0.14

and

I\leq 0.17

; both rise only in post-training and rank checkpoints alike (Spearman

\geq 0.94

, except

0.67

under Tülu-3’s template).

\Delta

cannot exceed

1-R^{2}_{\mathrm{neutral}}

(

0.26

–

0.79

at the final) and levels off, while

I

reaches 1 (

\star

		Base		Best sibling, P		Best sibling, L		Own DPO
Target final	View	P	L	Route	$I$	Route	$I$	P	L
Moral bank
OLMo-3 7B Instruct	R	0.04	0.09	RL-Zero-Math	0.03	RL-Zero-Math	0.10	0.76	0.92
OLMo-3 7B Think	R	0.03	0.04	RL-Zero-Math	0.05	Instruct	0.08	0.71	0.91
OLMo-3 7B RL-Zero-General	R	0.53	0.81	RL-Zero-Code	0.68	RL-Zero-Code	0.84	–	–
OLMo-3 7B RL-Zero-Math	R	0.88	0.94	RL-Zero-Code	0.85	RL-Zero-Code	0.91	–	–

Appendix

Table T1: Fork prediction on the probability (P) and log-odds (L) scales: the held-out gain

I

beyond the source’s 16 unpressured margins for the pooled map of 13 user lines, from the base, the best sibling and the own DPO stage. The lower block gives the Spearman correlations between the two scales.

Figure S3: The slot reads what the model writes. (A) Per-item Pearson

r

between the slot effect of one opinion wording (flip probability with the wording minus with the neutral turn) and the same effect in sampled replies (share of replies that flip), one row per sampled artifact, for moral verdicts and factual claims;

\bullet

role-tag draws,

\diamond

own-template draws; hollow: a reading fails the reliability rule. (B) Item means, sampled replies against the slot, with

y=x

Figure S4: The maps and the fork under other renderings. The item map is a property of the model under its chat renderings, with six finals agreeing less closely; narration lowers the amount and changes the map, yet the fork result holds under every rendering. (A) The four renderings of one exchange: role tags, the lineage’s chat template, and the narrated forms in the second person ("you") and about a third person ("Alex"). (B) Per-item r between the role-tag and chat-template log-odds maps of the same checkpoint.

S_{\mathrm{op}}

agrees at 0.72–0.96 at 11 of 17 finals; the others run from 0.18 (K2-Horizon 7B) to 0.54 (OLMo-3 7B Think); bases run

-

0.01 to 0.89. (C) The narrated renderings against role tags,

S_{\mathrm{op}}

map:

-

0.30 to 0.47 at the OLMo-3 7B base and finals. (D)

S_{\mathrm{op}}

narrated against role tags at bases, stages and finals: narration lowers

S_{\mathrm{op}}

at every post-trained checkpoint, and the two voices differ by

\leq

0.20. (E) Held-out increment of the

S_{\mathrm{op}}

map toward each final from the base, the own SFT and DPO checkpoints and the sibling route under the four renderings: own DPO 0.87–0.96 narrated; base and sibling

\leq

0.38 under every rendering.

Figure S5: Answer-order sensitivity belongs to particular models. The mean bias is

\approx

0 everywhere and the median

S_{\mathrm{op}}

order split-half of the log-odds maps is 0.75 (0.85 under the template); the order-sensitive finals are Apertus v1 8B under both renderings, OLMo-2 1B under the template and OLMo-2 7B under role tags. Order bias is P(wrong) in one answer order minus the other. (A) Mean

\pm

item SD at the rest cell, the neutral turn and the opposing opinion, for the finals over their bases. The means are

\approx

0 everywhere; the item SD at the neutral turn has a median of 0.10 and reaches 0.48–0.56 at Apertus v1 8B, OLMo-2 1B and OLMo-3 7B Instruct. (B) Spearman–Brown split-half reliability of the log-odds item maps across the two answer orders (

S_{\mathrm{op}}

S_{\mathrm{ch}}

, pooled, and the per-line median used by the gate, with the map beyond the unpressured profile as a lighter mark) and across the two wordings (

S_{\mathrm{op}}

): OLMo-3 7B Instruct 0.71 across orders against 0.70 across wordings; Apertus v1 8B 0.09 against 0.98. (C) Along training: the neutral-turn item SD and, at the checkpoints the gate covers, the

S_{\mathrm{op}}

order split-half for OLMo-3 7B (both routes), K2-Horizon 3.7B and Apertus v1 8B. On the Instruct route the item SD rises 0.31

\to

0.50 at DPO while the split-half stays at 0.57–0.71 from SFT to the final.

Moral bank
Target final	View	Neutral bias	Order $S_{\mathrm{op}}$	Order $S_{\mathrm{ch}}$	Order pooled	Wording	Gate (P)	Gate	Residual
OLMo-3 7B Instruct	R	0.12 $\pm$ 0.56	0.71	0.81	0.98	0.70	-0.13	0.81	0.87
OLMo-3 7B Think	R	0.04 $\pm$ 0.14	0.71	0.57	0.98	0.81	0.49	0.65	0.78
OLMo-3 7B Instruct	T	0.04 $\pm$ 0.26	0.74	0.79	0.98	0.90	0.43	0.74	0.86
OLMo-3 7B Think	T	0.05 $\pm$ 0.15	0.39	0.45	0.96	0.94	0.62	0.54	0.84
OLMo-3 32B Instruct	R	0.00 $\pm$ 0.09	0.75	0.66	0.95	0.94	0.86	0.69	0.87

Appendix

Table T2: Answer-order sensitivity and the reliability gate on both item banks; the last block gives stated-opinion map diagnostics for the moral finals discussed in Appendix C (role tags). Negative order values are retained and need not coincide with negative wording values.

Target final	View	Base $S_{\mathrm{op}}$	Final $S_{\mathrm{op}}$	$\delta$ , correct	$\delta$ , incorrect	Reply $r$
OLMo-3 7B Instruct	R	0.39	0.50	0.42	0.38	–
OLMo-3 7B Think	R	0.39	0.68	0.40	0.39	–
OLMo-3 7B Instruct	T	0.26	0.49	0.40	0.35	0.92
OLMo-3 7B Think	T	0.26	0.56	0.34	0.33	0.52
OLMo-3 32B Instruct	R	0.35	0.29	0.17	0.19	–
OLMo-3 32B Think	R	0.35	0.33	0.20	0.17	–

Appendix

Table T3: Factual-bank amounts and sampled-reply agreement. The deference columns divide cells by whether the planted verdict is correct. Reply correlation compares the affirm-wording effect in the slot and sampled replies; a dash denotes an unavailable reading.

Stage	Draws	Valid share	Slot/reply $r$	Reply increment
Tülu-3 8B SFT	21	0.91	0.58	0.35
Tülu-3 8B DPO	21	0.95	0.75	0.64
OLMo-3 7B Instruct SFT	21	0.92	0.92	0.48
OLMo-3 7B Instruct DPO	21	0.91	0.85	0.71
OLMo-3 7B Think SFT	11	0.88	0.51	0.04
OLMo-3 7B Think DPO	11	0.92	0.42	0.17

Appendix

Table 14

Figure S6: Every public checkpoint of the 30 lineages, and what we measured. One lane per lineage or post-training route, in the order of Figure 1 b. Grey: the branches on the Hugging Face Hub; tall ticks: the 741 checkpoints read on the moral bank, one per distinct set of weights;

n/N

: measured over available per segment; “same commit” and “same weights” join two branch names of one checkpoint. Ink marks: dot, sampled generations; bar, the 240-item confirmation set; square, the factual-claim bank. Small ticks under stage 1 count trillions of tokens. Dotted: no public checkpoint.

Checkpoint lane	All	Replies	Confirm.	Factual	$\delta_{x}$
OLMo-3 7B	63	11	5	11	24
OLMo-3 7B Think	37	3	3	3	14
OLMo-3 7B Instruct (warm start)	11	3	3	3	6
OLMo-3 7B RL-Zero $\times$ 5	22	5	1	0	3
OLMo-3 32B	40	3	0	1	1
OLMo-3 32B Think	21	3	0	3	3

Appendix

Table T4: Measured checkpoints and additional readings per lineage. A checkpoint is a distinct set of weights with a moral-bank readout (shared names and documented weight duplicates count once); the total includes 504 pretraining checkpoints across 30 lineages. The other columns count checkpoints with sampled replies, the confirmation items, the factual bank and the single-turn deference reading.

Figure S7: The layout of Figure 2 for twelve lineages with pretraining checkpoints. The same picture in every lineage: the amount is present, or arrives, at a corpus-specific point, and item structure toward the final stays below 0.15 until post-training (0.33 at the OLMo-2 13B base) and is written from SFT on. One panel per lineage, one column per checkpoint: pretraining grey, post-training in the family colour. Top: $S_{\mathrm{op}}$ , $S_{\mathrm{ch}}$ and $\delta$ , probability scale, 95% item-bootstrap bands; $\diamond$ $\delta_{x}$ where measured (OLMo-3 32B, K2-Horizon 3.7B and 7B, Apertus v1 8B: $\delta_{x}$ runs $\approx$ 0.06 above $\delta$ along K2-Horizon from mid-training on, below $\delta$ at OLMo-3 32B, and jumps from $\leq$ 0.25 to 0.40 / 0.48 at the Apertus SFT / final). Middle: item structure toward the final of the $S_{\mathrm{op}}$ and $S_{\mathrm{ch}}$ maps (1 at the final). Bottom: the two item maps as flip-probability contrasts, 240 items, each strip sorted by its own map at the final (OLMo-3 32B: the Instruct final).

Figure S8: (Continued.) The layout of Figure 2 for twelve lineages with pretraining checkpoints. The stated-opinion map first exceeds half of the final’s at SFT in most lineages with an SFT checkpoint (OLMo-3 32B Think, OLMo-2 1B, 7B and 13B, SmolLM3-3B, Apertus v1 8B and 70B, K2-Horizon 7B), at DPO at OLMo-3 32B Instruct and Apertus v1.5 8B, during RL at OLMo-2 32B, and only at the final at K2-Horizon 3.7B and Apertus v1.5 70B.

Figure S9: The stage-2 entry and the coupling of the capitulation maps (OLMo-3 7B). At the mid-training entry, deference and capitulation under a stated opinion step together, capitulation under a bare challenge follows two thousand steps later, and single-turn deference does not move; beyond the shared unpressured profile the two capitulation maps correlate moderately. (A)

S_{\mathrm{op}}

S_{\mathrm{ch}}

\delta

, the unpressured flip (

\tfrac{1}{2}

-

\kappa_{n}

) and, where measured,

\delta_{x}

from the end of stage 1 through stage 2 (steps) and stage 3 to the base.

\delta

and

S_{\mathrm{op}}

step across the entry, from the stage-1 end to stage-2 step 2000 (0.21

\to

0.43; 0.24

\to

0.59),

S_{\mathrm{ch}}

two thousand steps later, from step 2000 to 4000 (

-

0.03

\to

0.17);

\delta_{x}

does not (0.33–0.34). (A ′ ) The absolute flip rates behind them: stated opinions against the planted verdict rise (to 0.95 and 0.87 by step 2000), those for it fall (to 0.01 and 0.08), the neutral-turn flip falls (0.46

\to

0.22 by step 4000); the bare challenges rise later and gradually. (B) Per-item r between the

S_{\mathrm{op}}

and

S_{\mathrm{ch}}

maps, raw and beyond the unpressured profile; right: post-training range and pretraining median in five other lineages. Raw r reaches 0.83 at the base; partial r is 0.42 there, 0.30–0.35 on the Instruct route and 0.38–0.42 on the Think route after SFT.

Figure S10: The lines inside each pressure family. The lines of a family move together along training and are written at the same stage as the family mean; one wording’s map predicts the other’s (median 0.57), the single challenges predict each other weakly (median 0.15), and across the two families little (median 0.02, at most 0.20) beyond the unpressured profile. (A) OLMo-3 7B along training: the flip probability under each line minus that under the neutral turn, averaged over the two planted verdicts (

S_{\mathrm{op}}

and

S_{\mathrm{ch}}

are its means over the opposing wordings and over the challenges). The two opposing wordings of the stated opinion move together at one level (r = 0.97 across checkpoints); the two supporting wordings lower the flip probability; the five bare challenges move together at different levels (r = 0.77–0.98), "are you sure" adding almost nothing beyond the neutral turn. (B) Item structure of each line’s map toward its route’s final: on the Instruct route the opinion lines are written mainly at DPO and the challenge lines mainly at SFT; on the Think route every line is written over the SFT steps. (C) Structure that a source line’s map predicts of a target line’s map beyond the unpressured profile, median over 17 finals. (D) The same per final for pairs within the wordings (0.36–0.78), within the challenges (0.03–0.25) and across the two families (

-

0.00 to 0.20).

Figure S11: The reference question and the stability of the map inside pretraining. Choosing the final as the reference does not create the result: inside pretraining OLMo-3 7B’s map does not settle, the late stage-2 components are carried only weakly into the base, and the base’s own map forms mainly in stage 3; K2-Horizon’s mid-training map persists yet predicts little of its final. (A) OLMo-3 7B, 85 checkpoints in training order: per-item r between the

S_{\mathrm{op}}

maps (left) and the

S_{\mathrm{ch}}

maps (right) of every two checkpoints; raw r below the diagonal, r beyond the unpressured profile above. (B) Structure of each checkpoint’s

S_{\mathrm{op}}

map toward the base, the Instruct final and the Think final; the strip gives the increment toward the next checkpoint. Toward the base, pretraining stays

\leq

0.15 through stage 2 and rises mainly in stage 3 (0.65 at step 9,000); the late stage-2 checkpoints share a map among themselves (adjacent r 0.58–0.97 beyond the unpressured profile) of which the base keeps 0.11. (C) Across lineages: the median and range of the increment between adjacent checkpoints within pretraining (0.01–0.13; K2-Horizon’s mid-training tags 0.60 / 0.43), into post-training, and within post-training (0.65–0.99 for the OLMo and Tülu-3 lineages).

Figure S12: Deference in detail. Deference is to the user’s words, not to the preceding exchange, and its item map is written on the route as well; commitment is trained in opposite directions on the two OLMo-3 7B routes. (A) OLMo-3 7B along training: deference

\delta

and single-turn deference

\delta_{x}

, 0.42 and 0.42 at the base,

\delta_{x}

above

\delta

at both finals (0.43 vs 0.37 Instruct; 0.46 vs 0.41 Think). Right: at seven bases, item-level r between

\delta_{x}

and

\delta

pooled over the four stance sentences (0.82–0.96) and the median within one sentence (0.04–0.75), with

\delta_{x}

and

\delta

printed (equal at OLMo-3 7B,

\delta_{x}

below

\delta

at OLMo-3 32B and above it at K2-Horizon and Apertus v1 8B). (B) Structure of each final’s

\delta_{x}

map predicted by the base, its own DPO (or last SFT) checkpoint (0.92–0.99; Apertus’ SFT 0.62) and the best sibling route (

\leq

0.45); K2-Horizon 7B’s base already predicts 0.39, the other bases

\leq

0.16. (C) Commitment

\kappa_{n}

and

\kappa

along training; right,

\kappa_{0}

by planted verdict: both sides negative from DPO on along the Instruct route (

-

0.10,

-

0.18 at the final) and positive along the Think route (+0.19, +0.28). (D) Root asymmetry

A_{\mathrm{op}}

along training in 13 lineages.

Figure S13: The model’s own verdict along training. (A) OLMo-3 7B, the columns of Figure 2 : the share of items the model itself judges wrong ( $c_{i}>0.5$ ), the per-item Pearson $r$ of its verdicts with the route’s final, and the share of items whose verdict flips between adjacent checkpoints; ticks: chat template. (B) The item map of $c$ , items sorted by the Instruct final. (C) $c$ and the four balanced means ( $2\times 2$ , neutral, no line, single turn). (D) $r$ with the final for eleven other lineages; printed: base $\to$ final $r$ and flip share. $\bullet$ base, S SFT, D DPO or APO, $\blacktriangle$ RL, $\star$ final, $\circ$ fails the comprehension check.

Figure S14: Decomposition of the second verdict. The stated side leads at every OLMo-3 7B checkpoint; the planted verdict’s weight is trained, turning negative on the Instruct route; the own verdict gains weight with scale and outweighs the stated side at OLMo-3 32B. (A) The second verdict’s slot log-odds regressed on the stated side, the planted verdict and the own verdict (log-odds of c), with a line intercept and a planted

\times

sign(own verdict) term (not drawn), over the six stated-opinion lines; standardized coefficients with 95% item-bootstrap bands; the worked column is the OLMo-3 7B Instruct final. (B) OLMo-3 7B along training: the stated side leads at all 60 checkpoints; the planted verdict’s coefficient turns negative on the Instruct route (+0.16 at the base;

-

0.27 to

-

0.34 from SFT on) and is near zero at the Think route’s released SFT, DPO and final checkpoints. (C) OLMo-3 32B: the own verdict outweighs the stated side at both finals (0.70 / 0.78 against 0.48 / 0.51). (D) OLMo-2 1B–32B: the own verdict grows with scale at the finals (

-

0.13

\to

0.68), not at the bases.

Figure S15: The unpressured profile’s share and its audit value. (a) Median share of each final’s per-item effect of the affirmed opposing opinion explained by its own unpressured profile, in the slot (role tags; tick: chat template) and in sampled replies, per bank. (b) Selecting the fifth of items that flip most without pressure recovers none of Tülu-3’s pressure-sensitive items (precision 0.00 at 8B and 70B; random 0.20).

Moral bank
Target final	View	Map	Base	SFT	DPO	Sibling	Sibling route
OLMo-3 7B Instruct	R	$S_{\mathrm{op}}$	0.15	0.46	0.92	0.11	Think
OLMo-3 7B Instruct	R	$S_{\mathrm{ch}}$	0.08	0.56	0.90	0.08	Think
OLMo-3 7B Instruct	T	$S_{\mathrm{op}}$	0.01	0.50	0.91	0.14	Think
OLMo-3 7B Instruct	T	$S_{\mathrm{ch}}$	-0.02	0.57	0.90	0.08	Think
OLMo-3 7B Think	R	$S_{\mathrm{op}}$	0.09	0.92	0.93	0.14	Instruct

Appendix

Table T5: Family-map fork prediction on both item banks: held-out gains on the log-odds scale beyond the source checkpoint’s 16 unpressured margins, from the base, own SFT, own DPO and the best named sibling route. R: role tags; T: chat template. Figure 3 shows the intervals; negative estimates are retained.

Moral bank
Target final	View	Base	SFT	DPO	Sibling	Sibling route
OLMo-3 7B Instruct	R	0.09	0.53	0.92	0.10	RL-Zero-Math
OLMo-3 7B Think	R	0.04	0.89	0.91	0.08	Instruct
OLMo-3 7B RL-Zero-General	R	0.81	–	–	0.84	RL-Zero-Code
OLMo-3 7B RL-Zero-Math	R	0.94	–	–	0.91	RL-Zero-Code
OLMo-3 7B RL-Zero-Code	R	0.91	–	–	0.90	RL-Zero-Math

Appendix

Table T6: Pooled fork prediction on both item banks: held-out gains on the log-odds scale beyond the source’s 16 unpressured margins, for the map that pools the per-item effects of all user lines. Sibling candidates include RL-Zero where available; a dash marks an absent public stage.

Route	Rendering	$S_{\mathrm{op},i}$	$S_{\mathrm{ch},i}$
OLMo-3 7B; tr.8239: “My head aches from headbutting my teachers face over my paper.”
Think	role tags	0.80	0.49
Instruct	role tags	0.18	-0.08
Llama-3.1 8B; tr.1234: “I put peanuts instead of chocolate chips in the cookies I made…”
Hermes-3	role tags	0.83	0.22
Meta-Instruct	role tags	0.11	0.01

Appendix

Table T7: Sibling routes yield differently on the same moral accounts: full family means as flip probabilities under role tags, averaged over both planted verdicts and answer orders (the item maps compared in the text take the same contrasts on the log-odds scale).

Bank	Reading	Rows	Median share
Moral	slot role tags	30	0.71
Moral	slot template	29	0.75
Moral	sampled replies	21	0.79
Factual	slot role tags	10	0.45
Factual	slot template	10	0.58
Factual	sampled replies	5	0.33

Appendix

Table T8: The unpressured profile and the remaining agreement between pressure families. Median held-out

R^{2}

of each final’s affirm-wording effect (first block) and of its full stated-opinion and bare-challenge maps (family rows) from its own 16 unpressured margins; slot and family entries are on the log-odds scale.

Moral finals	Rows	Raw $r$	Partial $r$	Minimum	Maximum
all finals	36	0.59	0.32	-0.03	0.57
role tags	18	0.52	0.32	0.02	0.54
template	18	0.69	0.36	-0.03	0.57

Appendix

Table 30

Figure S16: What internal states add beyond behaviour, by source, at three-quarter depth. Each bar is the dark segment of Figure 3 B on its own: the gain

D

of a linear readout of the source’s residual stream (block 24 of 32; 48 of 64 for OLMo-3 32B) beyond the source’s margins and own map in predicting the final’s stated-opinion map, as a share of the error beyond the margins (moral verdicts; role tags; whisker: 95% item-bootstrap interval of

D

; tick: the template-rendered value where extracted), in the colours of Figure 3 A. Thin grey bars: unrelated lineages from other organisations toward the same final, each at its own three-quarter block; the band is their trimmed range (highest and lowest dropped), the dotted line their median; a bar whose interval lies above the band exceeds these controls.

Figure S17: The same at half depth (block 16 of 32; 32 of 64 for OLMo-3 32B; unrelated models at their own half block), drawn as Figure S16 .

Figure S18: Stitching in two forks. A hybrid runs a donor’s blocks

1\ldots\ell

and the recipient final’s blocks

\ell+1\ldots 32

;

\Delta

is the recipient structure the hybrid adds beyond the donor’s own behaviour (log-odds). Left: donors into the OLMo-3 7B Think final; right: into the Llama-3.1 8B Tülu-3 final. The added structure falls sharply from

\ell=8

\ell=16

, implicating recipient blocks 9–16 in this transfer; some donors retain a smaller positive increment at 16.

Figure S19: States and behaviour along the Think route, from pretraining to the final. Log-odds target (the Think final’s per-item map), layer 24, Instruct-template renderer for every checkpoint; bands and bars 95% item-bootstrap intervals. (a) Two-stage fit. Behaviour line: the checkpoint’s own map beyond its margins; state line: its layer-24 states beyond the same margins; extra-state line: its states beyond margins and pressure effects; beyond-target-margins line (pretraining and base only): its states beyond those and the Think final’s own margins, which removes the generic story-content layer that unrelated lineages also carry. Pretraining: 32 checkpoints in training order, equally spaced within each stage (token counts restart at stages 2 and 3), then the base, then the post-training stretch with SFT and RL steps counted separately. (b) The feature-residualised fit of the extra-state and beyond-target-margins lines for the pretraining checkpoints and the base (no such fits exist for post-training checkpoints), with stage means. (c) Held-out

R^{2}

of the target from the behaviour covariates alone and with the states.

Measurement wave	Measurements	Analysis rule
Initial and confirmation	Moral items, planted verdicts, pressure lines, new items and rewordings.	Admission uses an AUC threshold of 0.8. Item prediction uses five folds and paired item resampling.
Training trajectories	Denser OLMo-2 trajectories; K2-Horizon and Apertus; sampled replies.	Compare stages within each lineage, applying the item-comprehension screen.
Shared-base routes	SmolLM3 trajectory, open base/final pairs, Llama sibling routes, and broader sampled replies.	Tülu/Meta forms the principal sibling contrast. Intervals crossing a decision bound give an undetermined outcome.
Fork and audit expansion	Larger forks, additional recipes, and transfer of pressure-sensitive item lists.	Select a fifth of the items and evaluate the selection on independent target replies. Predict log-odds effects beyond the source’s probability and log-odds neutral margins.
Factual claims	Factual claims, both fork renderings, and correctness splits.	Require deference of at least 0.10 and median line reliability of at least 0.5 in one rendering. Compare own DPO with the best sibling.
Deference trajectories	Single-turn deference; factual intermediate replies and pretraining sentinels; denser OLMo-2 sentinels.	Predict final deference maps and compare moral and factual changes.

Appendix

Table T9: Measurement waves: what each wave measured and how its comparisons were analysed.

LLM Sycophancy

Large language model sycophancy is the tendency to mirror users’ beliefs, preferences, or authority cues at the expense of independent judgment.

See all 161 papers

Oct 19, 2026

No Matter Who You Are: Addressivity and Sycophancy in Large Language Models

Sycophancy, where an LLM becomes overly agreeable, deferential, or flattering toward the user, has been identified as a major failure mode. While post-training has been implicated as a cause, how it internally gives rise to sycophantic behavior remains insufficiently understood. Post-training transforms the model's output from text completion into utterances directed toward an addressee, thereby giving the output addressivity. We hypothesize that the accompanying pressure to follow the addressee's instructions and align with their preferences links this addressivity to sycophancy. To test this, we use Sparse Autoencoders to extract addressivity directions from pre-trained and post-trained models and examine their relationship with sycophancy. Steering models with addressivity directions can amplify or suppress sycophancy, and this effect is observed only in post-trained models. Furthermore, comparing the internal mechanisms of both model types confirms that the association between addressivity and sycophancy is strengthened through post-training. Finally, through a controlled post-training experiment, we find that models are sycophantic not toward a specific addressee but toward any recognized addressee, suggesting that the presence of an addressee, rather than the addressee's identity, is what elicits sycophancy.

Seungjong Sun, Hyunmin Jun, Minsik Yoon +4

Department of Human-Artificial Intelligence Interaction, Sungkyunkwan University · Department of Applied Artificial Intelligence, Sungkyunkwan University · Department of Interaction Science, Sungkyunkwan University

LLM Sycophancy Language Model Post-Training

Ask PDF

Oct 27, 2026

What Carries Forward? Pretraining Disclosure Across Language Model Releases

Understanding how language models were pretrained requires information about their data, processing procedures, and computational resources. However, it remains unclear whether successive releases preserve or expand earlier disclosure. We conduct a document-based audit of 41 model entries from 17 families using 24 pretraining-related indicators from the 2025 Foundation Model Transparency Index (FMTI). We examine overall scores and indicator profiles, tracking gains, losses, and retained disclosures across 24 adjacent comparisons in ten families. Scores decrease in 11 comparisons, increase in six, and remain unchanged in seven, with 38 indicator losses and 28 gains overall. Stable or improving scores can conceal losses: OLMo-2 and OLMo-3 receive identical totals despite three gains and three losses, while Llama-3.1 improves over Llama-2 while losing credit for two data-related criteria. Gaps also persist in corpus composition and training compute. These findings show that later releases do not consistently preserve earlier pretraining disclosure and motivate documentation that explicitly identifies changes and the applicability of inherited information.

Md Sakib Ul Islam, S M Mehedi Zaman

Victorian Institute of Technology, Australia · Rutgers University, USA

LLM Auditing Language Model Pretraining Data Provenance in Language Models

Ask PDF

Oct 6, 2026

Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgment

An agent that can say an action is wrong and then take it anyway is a different failure from one that does not know better, and evaluations of stated values cannot see it. We ask how often open models act against their own moral judgment under pressure, and what decides and reduces it. We build a pre-registered panel of 248 scenarios across five kinds of pressure. Each scenario is posed to one model in two separate calls, as the agent choosing among fixed options and in the third person asking which is right; the model's own judgment is the reference, and the scenario's construction fixes which option counts as the violation. Every scenario has a twin with the pressure removed, and every model gets a positive control that bounds what the instrument can see. On OLMo-3-7B-Instruct, the model takes the action it judged wrong on one in five of the scenarios screened on its own actions, more often than with the pressure removed. Whether the gap appears depends on the post-training recipe: OLMo-3 and Meta's Llama-3.1-8B-Instruct carry it; Tulu 3, Qwen2.5-7B-Instruct and OpenAI's GPT-OSS-20B, read without reasoning, show none within stated detection bars. Meta's recipe and Ai2's Tulu 3 share the same Llama-3.1 weights; only Meta's carries the gap. On both models that carry it, reasoning about the stakes before acting moves the choice back toward the model's own judgment. What that deliberation tracks is the stated rule: where following a written rule harms an absent third party, it still moves OLMo-3 and GPT-OSS-20B toward the rule, not detectably diminished, while Llama-3.1's brake shrinks to about a seventh. Whether a model acts on its own moral judgment is therefore not settled by pretraining, and deliberation moves the action toward the stated rule, which protects people only when the rule does. The gap is a measurable target for post-training recipes, not a fixed property of the model.

Orion Reblitz-Richardson

Distiller Labs

Moral Reasoning in Language Models Language Model Post-Training Causal Interventions in Language Models

Ask PDF

Abstract

LLM Sycophancy

Explore similar work

No Matter Who You Are: Addressivity and Sycophancy in Large Language Models

What Carries Forward? Pretraining Disclosure Across Language Model Releases

Principled Under Pressure: Post-Training Decides Whether LLMs Act on Their Own Moral Judgment