cs.CLSep 26, 2026

Is the Judge Its Twin? Post-training makes a model's writing more predictable but barely moves its taste, as a judge, toward predictable writing

Authors: Arman Nik Khah, Arvin Bahreini

Organizations: The University of Texas at Dallas · University of Oregon

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Language models now routinely grade other language models. If post-training makes a model's writing more predictable, it may also teach the same model, as a judge, to reward predictable writing. We test this directly. We follow two open 7-billion-parameter model families, OLMo-2 and Zephyr, from their base models through supervised fine-tuning (SFT) and direct preference optimization (DPO), and measure every stage as a writer of short stories and as a judge of story pairs. As writers, the models drift as feared. Each family's trained model finds its stages' stories more familiar in the order base, SFT, DPO, in all ten prompts, and a model from the other family finds the trained stories 7.0 to 9.2 percent (OLMo-2) and 24 to 25 percent (Zephyr) less surprising per token. As judges, they barely move toward predictable writing. Asked which story is better, a question every judge passes a competence check on, no trained judge's preference for the more predictable story grows by as much as one point of pick probability at its estimate. A one-sided 95% upper bound on that growth is about 5 points on the widest pairs. Training instead makes judges more swayed by story length and by display order, and it breaks "more creative" as a question. Trained judges no longer reliably prefer a story to its own words in scrambled order. For these two families, the practical hazard in a small trained judge of creative writing lies in these measurable habits.

Figures & tables

FamilyStage that wrote the storyStranger’s viewFamily’s view
OLMo-2base1.915+0.434
 SFT−- 9.2%−- 0.121
 DPO−- 7.0%−- 0.567
 final−- 7.7%−- 0.581
Zephyrbase2.105+0.243
 SFT−- 24.9%−- 0.139
Table 1
Judge“better” slope [95% CI]“creative” slope [95% CI]“creative”, holding the judge’s own “better” fixed“creative” minus “better”
OLMo-2 base−- .057 [ −- .12, −- .01]−- .096 [ −- .18, −- .02]−- .022−- .039 [ −- .07, −- .01]
OLMo-2 SFT−- .048 [ −- .11, +.01]−- .058 [ −- .16, +.03]+.001−- .010 [ −- .05, +.02]
OLMo-2 DPO−- .064 [ −- .16, +.03]−- .051 [ −- .19, +.07]+.019+.013 [ −- .04, +.06]
OLMo-2 final−- .069 [ −- .18, +.03]−- .054 [ −- .21, +.08]+.020+.016 [ −- .03, +.06]
Zephyr base+.024 [ −- .01, +.06]+.040 [ −- .01, +.10]+.007+.016 [ −- .01, +.04]
Zephyr SFT+.023 [ −- .02, +.08]+.052 [ −- .00, +.11]+.021+.029 [+.01, +.05]
Table 2
Judgeasked “more creative”asked “better”
OLMo-2 base.951.00
OLMo-2 SFT.47.97
OLMo-2 DPO.47.98
OLMo-2 final.48.98
Zephyr base.88.97
Zephyr SFT.13.88
Table 3
RouteHow the treated story was madeGap in surprise (nats per token)Blind read, treated sideBlind read, partner
Askingan added instruction to write a response with under 10 percent probability−- 0.031 [ −- 0.097, +0.037] ; 11 of 29 pairs in the intended directionnot runnot run
Heatingsampling temperature raised from 1.0 to 1.3+0.768 [+0.595, +0.908] ; 19 of 20 pairs2, 3, 4, 45, 5, 5, 5
Selectingordinary stories at temperature 1.0, paired by natural spread+0.176 [+0.053, +0.302] , measured on held-out tokens4.5, 1, 2, 42, 3.5, 4, 3
Table 4
More in this topic

Language Model Post-Training

Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.

See all 195 papers

Explore similar work

CardsList
May 27, 2026

Narrative Flattening: How Post-Training Compresses Thematic, Affective, and Stylistic Variation in LLM Fiction

Large language models produce fluent fiction, yet their creative output is widely seen as flat. We ask where this quality originates in the training and whether it affects different domains of human fiction equally. We construct a matched story-continuation paradigm across StoryStar (public-platform), TMAS (prompt-guided), and The New Yorker (professional literary)-and compare continuations from four OLMo 32B checkpoints (Base, SFT, DPO, RLVR) against matched human text. Because these checkpoints share architecture, scale, tokenizer, and pretraining, the design isolates the post-training effect. We measure each continuation along three sentence-level dimensions: thematic motion, affective prevalence, and linguistic diversity. Across all three, post-training compresses dynamic variation: thematic transitions become more uniform, high-intensity emotions give way to neutrality, and stylistic diversity across stories shrinks. We term this progressive loss narrative flattening. The effect is directionally stable across story domains but gap size depends on the human baseline: professional literary fiction is compressed most, while public-platform and prompt-guided stories show smaller gaps, consistent with their human baselines sitting closer to the model's default rhythm. Post-trained endpoints converge across domains, suggesting alignment produces a continuation regime largely insensitive to the source domain's narrative texture.

Zehan Li, Yutong Zhu, Siyang Wu +2

Knowledge Lab, University of Chicago

Language Model Post-TrainingText Generation
AskPDF
Oct 19, 2026

The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Language models can change a committed answer to accommodate a user's opinion or pushback without new evidence. However, when this behaviour develops during training and what a released model inherits from earlier stages remain unclear. We follow 741 checkpoints of 30 open model lineages, from early pretraining to released models. We plant initial verdicts on moral scenarios and measure how far a model follows a user's stated opinion or abandons its verdict under pressure. In most lineages the behaviour is present before post-training, and its average strength changes in either direction, at different stages across lineages. A clearer pattern emerges at the level of individual questions. Beyond responses without pressure, pretraining behaviour predicts little of which questions a released model yields on. That pattern forms during post-training: on routes with supervised fine-tuning, the fine-tuning checkpoint predicts part of it and the preference-optimization checkpoint most of it. A sibling trained from the same base along another route predicts far less: for Tülu-3 8B, its own preference-stage checkpoint removes 85% of the remaining prediction error, a sibling 19%. Internal states add predictive information without changing this ranking. Similar average scores conceal different vulnerabilities, and training history matters for auditing a released model.

Haonan Huang, Chengkun Cai

Princeton University · University College London

LLM SycophancyLanguage Model Post-TrainingLLM Auditing
AskPDF
Jun 6, 2026

Self- and Other-Labels Induce Bidirectional Bias in LLM Judges

As LLM-as-a-judge becomes increasingly widespread, self-preference -- the tendency of a judge to favor its own outputs -- raises growing concerns about evaluation reliability. However, this bias has been studied predominantly on generated text, where stylistic features and response quality are inevitably conflated. As a result, existing measurements cannot separate genuine self-preference from these confounds. We address this limitation by changing the object of evaluation: instead of judging generated text, ten LLMs assess sets of narrative constraints selected from a shared pool, which carry no stylistic fingerprint yet retain a recoverable model-specific signature. Two experiments on this task yield complementary findings. Under blind evaluation, self-preference disappears, with a small effect remaining in the opposite direction once selection quality and judge severity are controlled. Under matched quality, however, self- and other-labels alone -- without naming any model -- shift scores bidirectionally. LLM judges inflate scores for self-labeled selections and deflate those for other-labeled ones regardless of the selection's actual source. We make two contributions: 1) authorship attribution is a distinct driver of evaluation bias, and 2) ground-truth-free tasks can serve as controlled instruments for studying LLM judge behavior.

Songeun Chae, Min Kim, Donghoon Jung +2

School of Digital Humanities and Computational Social Sciences, KAIST, South Korea

Bias in LLM-as-a-JudgeLanguage Model Bias EvaluationLLM-as-a-Judge
AskPDF