Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs

Kirill Skobelev; Eric Fithian; X. Y. Han

cs.AISep 15, 2026

Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs

Authors: Kirill Skobelev, Eric Fithian, X. Y. Han

Organizations: Northwestern University · University of Chicago

Abstract

Recent work by Doshi and Hauser (2024), Bisbee et al. (2024), and Xie et al. (2026) raises concerns that outputs from large language models (LLMs) tend to be under-diverse: they repeat or resemble one another more often than responses from the population they are meant to represent, a phenomenon known as mode collapse. In this work, we show that whether mode-collapse, or its opposite, occurs depends on the specific model and dataset used. Further, with sufficient supervised fine-tuning (SFT) data, LLM output diversity converges toward that of the target distribution from which fine-tuning data are sampled. To quantify this comparison, we measure the probability that two responses sampled independently from the same fixed prompt coincide (collide), or their expected similarity under a kernel. We derive a bias-variance decomposition of the expected gap between the model's and target's collision probabilities, showing that SFT is not inherently biased toward mode collapse or its opposite: finite-sample SFT can leave a model either under- or over-dispersed, depending on the model and dataset. Finally, we show that the absolute gap is bounded by the square root of the Kullback-Leibler (KL) divergence from the target distribution to the model. Consequently, a model sufficiently close to optimal under population cross-entropy cannot exhibit arbitrarily miscalibrated diversity. We test the decomposition and the bound in three experiments: small transformers on synthetic languages, four LLMs fine-tuned on human surveys, and these LLMs fine-tuned on CodeNet, a dataset of human code solutions. More target data moves model diversity toward the human (or synthetic target) level in all experiments, consistent with our theoretical predictions. These results show that diversity miscalibration can arise from finite-sample error and shrink as SFT better approximates the target distribution.

Figures & tables

Figure 1: Large Langauge Models (LLMs) can produce diverse or colliding outputs given the same prompt. The extent to LLMs do so can be measured. Higher average output similarity than the target’s is referred to as mode-collapse. However, contrary to wide-spread belief, mode-collapse is not guaranteed, and depending on model and dataset, models can instead be over-diverse (the opposite of mode-collapse). It can be shown that fine-tuning calibrates the models, that is, they become not too under- and over-diverse. (a.i) Independent outputs from the same LLM and prompt can collide or differ. (a.ii) The notation used in this figure for collision. (a.iii) The definition of the collision ratio

R

. (b) Shows the decomposition of Eq. ( 6 ) using data from Experiment 1 (§ 4.1 ). The two plots correspond to two different model and data set ups. The plots demonstrate that Alignment term of Eq. ( 6 ) can make a model either under- or over-diverse. Here

b_{h}=\mathbb{E}[q_{h}]-p_{h}

is the bias at context

h

, with

\mathbb{E}

over independent fits. The plotted curves average contexts and start at

R=1

, with each contribution divided by

C(p_{h})

. (c) Plots median collision ratio

R

for early-stopped survey and final CodeNet adapters against fine-tuning examples from Experiments 2–3 (§§ 4.2 – 4.3 ). The experiments show that additional fine-tuning leads to diversity calibration. (d) Theorem 1 bounds the absolute model–target collision gap by the square root of their Kullback-Leibler divergence. Consequently, a model sufficiently close to optimal under population cross-entropy cannot exhibit arbitrarily miscalibrated diversity.

Figure 2: Decomposition of the normalized gap between model and target collision probabilities. The two synthetic languages have known target distributions (Appendix H.1 ), so repeated fits let us estimate the contributions of variance, squared bias, and target–bias alignment to

\mathbb{E}[R]-1

using Eq. ( 6 ). Contributions are averaged over contexts and plotted against sampled sequences

N

. (a) shows pre-training from the diffuse prior (

6.135

M-parameter GPT,

N

from

32

k to

1

2000

common contexts); (b) shows fine-tuning from the mode-aligned prior (

27.97

M-parameter GPT,

N

from

500

10^{6}

). The black line shows the estimated

\mathbb{E}[R]-1

, averaged over contexts: positive values indicate mode collapse and negative values over-dispersion. The three contributions sum to this value. The two panels carry independent vertical and horizontal scales.

Figure 3: Experiment 2. Collision ratio

R_{\mathrm{U}}=C(q)/\hat{C}_{\mathrm{U}}

(

1

= calibrated) against fine-tuning examples (

20

sample sizes for the GSS,

18

each for the WVS and ANES).

\hat{C}_{\mathrm{U}}

estimates how often two distinct human respondents give the same answer;

C(q)

is the probability that two independent model answers match, using probabilities normalized over the listed options. Each panel shows the median collision ratio across evaluated question–group pairs for early-stopped adapters (blue line), with shaded bands spanning the

25

th–

75

th and

5

th–

95

th percentiles across pairs. The marker at

n=0

is the zero-shot base model, with vertical whiskers repeating the two bands; the horizontal axis is symmetric-log so that

n=0

is shown on the same axis as the sweep, and each panel has its own y-axis limits.

Figure 4: Experiment 3. CodeNet diversity under normalized Zhang–Shasha similarity. Collision ratio

C_{\mathrm{ZS}}(q)/C_{\mathrm{ZS}}(p)

against SFT examples (

1

= calibrated). Lines show median finite collision ratios across problems; shaded bands span the

25

th–

75

th and

5

th–

95

th percentiles. At the full fine-tuning dataset, each model’s median uses all

2{,}647

candidate problems, while base markers cover

2{,}646

2{,}569

986

, and

2{,}646

problems in panel order. The text separately compares the same problems before fine-tuning and after fine-tuning on the full dataset. Panels have separate y-axes and broken horizontal axes.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Starting distribution $\bar{q}_{0}$	Diffuse (regime (a))	Mode-aligned (regime (b))
$\|V\|$	$1{,}024$	$50{,}000$
$\Tr\Cov(q_{\theta_{0},h})/C(p_{h})$	$0.068$	$1.2\times 10^{-3}$
	$[0.067,0.069]$	$[1.14,1.22]{\times}10^{-3}$
$\\|b_{0,h}\\|_{2}^{2}/C(p_{h})$	$0.826$	$1.829$
	$[0.823,0.829]$	$[1.720,1.941]$
$2p_{h}^{\top}b_{0,h}/C(p_{h})$	$-1.651$	$\mathbf{+2.413}$

Appendix

Table 1: Initialization realizes opposite alignment signs. The four rows from variance through normalized gap form the

C(p_{h})

-normalized decomposition for each

100

-seed starting ensemble; the final two rows are unnormalized. Each quantity is averaged over contexts after any indicated normalization. Brackets give

95\%

bootstrap intervals over common evaluation contexts. The mode-aligned start satisfies

p_{h}^{\top}\bar{q}_{0,h}>C(p_{h})

, whereas the diffuse start does not. This is an initial-condition comparison, not a claim that initialization fixes alignment after fitting.

Survey	Model	Base ( $n=0$ )	Max over grid (final)	Largest $n$ (final)
GSS	gemma-2-2b-it	$2.11$	$1.95$	$1.00$
GSS	Qwen3.5-2B	$0.99$	$2.22$	$0.99$
GSS	Qwen3.5-4B	$1.61$	$1.18$	$1.00$
GSS	gemma-3-4b-it	$2.28$	$2.19$	$1.01$
WVS	gemma-2-2b-it	$2.42$	$1.92$	$1.00$
WVS	Qwen3.5-2B	$1.04$	$1.39$	$1.02$

Appendix

Table 2: Final-adapter survey medians. Values are

R_{\mathrm{U}}=C(q_{\mathrm{opt}})/\hat{C}_{\mathrm{U}}

across evaluated question–group pairs, where

\hat{C}_{\mathrm{U}}

is the distinct-pair U-statistic. “Base” is the zero-shot checkpoint; the other columns summarize final adapters over the SFT data-size grid. At the largest pool, all medians round to

0.99

–

1.02

Figure 5: The

\ell_{2}

bound holds everywhere and is occasionally near-tight. Across

150{,}000

held-in contexts, the median measured-to-bound ratio is

0.140

and the maximum is

0.980

. (a) Median measured error against median bound for each

N

; all points lie below

y=x

. (b) Median, interquartile band, and full range of the ratio across

1{,}500

contexts per

N

Examples (fine-tuning / validation), pooled (respondent, question) pairs: GSS $5{,}224{,}403$ ( $4{,}706{,}940$ / $517{,}463$ ); WVS $1{,}620{,}333$ ( $1{,}462{,}645$ / $157{,}688$ ); ANES $1{,}846{,}292$ ( $1{,}662{,}788$ / $183{,}504$ ). Minimum-eval threshold $=60$ respondents per $(\text{question},X)$ pair. The $n$ -grid is anchored per survey to the full fine-tuning pool $N$ and halves down to the smallest $n\geq 8$ : GSS $20$ points $[4{,}706{,}940,\dots,8]$ ; WVS $18$ points $[1{,}462{,}645,\dots,11]$ ; ANES $18$ points $[1{,}662{,}788,\dots,12]$ (roughly six orders of magnitude on the GSS).
Survey	Respondents	Countries	Groups	Eval groups ( $\geq 60$ )	Eval pairs	Questions
GSS	$74{,}485$	— (US)	$2{,}512$	$345$	$18{,}270$	$376$
WVS	$32{,}227$	$22$	$956$	$293$	$8{,}015$	$53$
ANES	$69{,}784$	— (US)	$2{,}942$	$352$	$8{,}211$	$44$

Appendix

Table 3: Experiment 2 datasets (measurable-only). Each survey is restricted to its measurable opinion questions (a question with

\geq 1

group of

\geq 60

respondents). “Groups” is the number of demographic tuples

X

; “eval groups” counts groups holding

\geq 60

respondents; “eval pairs” counts evaluated

(\text{question},X)

pairs. “Examples” are pooled (respondent, question) pairs, split at the respondent level; the

n

-grid is anchored to each survey’s full pool

N

and halves down to the smallest

n\geq 8

, so each survey has its own grid.

Item ( var )	Graded answer options	Ex.
Death penalty for murder ( cappun )	A) favor; B) oppose	A
General happiness ( happy )	A) very happy; B) pretty happy; C) not too happy	B
Financial satisfaction ( satfin )	A) pretty well satisfied; B) more or less satisfied; C) not satisfied at all	B
Political views ( polviews )	A) extremely liberal; B) liberal; C) slightly liberal; D) moderate; E) slightly conservative; F) conservative; G) extremely conservative	D
Party identification ( partyid )	A) strong democrat; B) not very strong democrat; C) independent, close to democrat; D) independent; E) independent, close to republican; F) not very strong republican; G) strong republican; H) other party	A

Appendix

Table 4: GSS prompt format. The persona template (top) is instantiated from a group’s variables; the model then receives the question with its lettered (graded) options and must reply with one letter. Five curated items are shown with their graded options and an illustrative answer; the full measurable set of

376

items is evaluated.

Item ( var )	Graded answer options	Ex.
Interpersonal trust ( trust )	A) most people can be trusted; B) need to be very careful	B
Feeling of happiness ( happy )	A) very happy; B) rather happy; C) not very happy; D) not at all happy	B
Confidence in government ( conf_govt )	A) a great deal; B) quite a lot; C) not very much; D) none at all	C
Justifiability of divorce ( divorce_just )	A) 1 (never justifiable); B) 2; …; I) 9; J) 10 (always justifiable)	C
Left–right self-placement ( left_right )	A) 1 (left); B) 2; …; I) 9; J) 10 (right)	E

Appendix

Table 5: WVS prompt format. Same template as the GSS up to the persona variables (country / year / urban-rural / age / sex / education). Five curated items are shown with their graded options and an illustrative answer; a random third of the measurable items (

53

159

, seed

42

) is evaluated.

Item ( var )	Graded answer options	Ex.
Party identification ( VCF0301 )	A) strong Democrat; B) weak Democrat; C) independent–Democrat; D) independent; E) independent–Republican; F) weak Republican; G) strong Republican	D
Liberal–conservative ( VCF0803 )	A) extremely liberal; …; D) moderate; …; G) extremely conservative	D
Government health insurance ( VCF0806 )	A) 1 (government insurance plan); B) 2; …; F) 6; G) 7 (private insurance plan)	D
Interest in elections ( VCF0310 )	A) not much interested; B) somewhat interested; C) very much interested	B
Presidential approval ( VCF0450 )	A) approve; B) disapprove	A

Appendix

Table 6: ANES prompt format. Same template up to the persona variables (decade / age / sex / race / region / education). Five illustrative items are shown with their graded options and an answer; the full measurable set of

44

items is evaluated. Endpoint-anchored issue scales keep numbered interior points with labelled poles.

Hyperparameter	Value
LoRA rank $r$	$16$
LoRA $\alpha$	$32$
LoRA dropout	$0.05$
LoRA bias	none
Task type	causal language modeling
Base precision	bf16

Appendix

Table 7: Experiment 2 shared fine-tuning hyperparameters.

Model	LoRA targets	Trainable params	% of total
gemma-2-2b-it	7 proj., all decoder layers	$20{,}766{,}720$	$0.79\%$
Qwen3.5-2B	7 proj., all decoder layers	$10{,}911{,}744$	$0.49\%$
Qwen3.5-4B	7 proj., all decoder layers	$21{,}233{,}664$	$0.47\%$
gemma-3-4b-it	7 proj., language_model only	$29{,}802{,}496$	$0.69\%$

Appendix

Table 8: Experiment 2 per-model LoRA configuration. The shared hyperparameters of Table 7 apply to all; the LoRA target scope differs. Target projections are q_proj , k_proj , v_proj , o_proj , gate_proj , up_proj , down_proj ; for gemma-3-4b-it these are matched only within the language_model decoder (vision tower excluded). Trainable counts are from get_peft_model .

Code A: s613176273 ⬇ A , B = map ( int , input (). split ()) if B % A ==0: print ( A + B ) else : print ( B - A )

Complete canonical AST A ⬇ root : Module body []: Assign targets []: Tuple elts []: Name : VAR_0_0 elts []: Name : VAR_0_1 value : Call func : Name : map args []: Name : int args []: Call func : Attribute : split value : Call func : Name : input body []: If test : Compare left : BinOp left : Name : VAR_0_1 op : Mod right : Name : VAR_0_0 ops []: Eq comparators []: Constant : INT body []: Expr value : Call func : Name : print args []: BinOp left : Name : VAR_0_0 op : Add right : Name : VAR_0_1 orelse []: Expr value : Call func : Name : print args []: BinOp left : Name : VAR_0_1 op : Sub right : Name : VAR_0_0

Code B: s053364634 ⬇ a , b = map ( int , input (). split ()) if b % a == 0: print ( b + a ) else : print ( b - a )

Complete canonical AST B ⬇ root : Module body []: Assign targets []: Tuple elts []: Name : VAR_0_0 elts []: Name : VAR_0_1 value : Call func : Name : map args []: Name : int args []: Call func : Attribute : split value : Call func : Name : input body []: If test : Compare left : BinOp left : Name : VAR_0_1 op : Mod right : Name : VAR_0_0 ops []: Eq comparators []: Constant : INT body []: Expr value : Call func : Name : print args []: BinOp left : Name : VAR_0_1 op : Add right : Name : VAR_0_0 orelse []: Expr value : Call func : Name : print args []: BinOp left : Name : VAR_0_1 op : Sub right : Name : VAR_0_0

Appendix

Table 9: High-similarity accepted pair for problem p03125 . The four cells contain the complete source and complete automatically exported canonical AST for submissions s613176273 and s053364634 . Node classes in the AST are highlighted in purple; Python keywords, comments, and strings use distinct syntax colors.

Code A: s227737564 ⬇ L = input () L = L . split () for i , val in enumerate ( L ): L [ i ] = int ( val ) t = L [0]/ L [1] t = format ( t , ’0.4f’ ) print ( t )

Complete canonical AST A ⬇ root : Module body []: Assign targets []: Name : VAR_0_0 value : Call func : Name : input body []: Assign targets []: Name : VAR_0_0 value : Call func : Attribute : split value : Name : VAR_0_0 body []: For target : Tuple elts []: Name : VAR_0_1 elts []: Name : VAR_0_2 iter : Call func : Name : enumerate args []: Name : VAR_0_0 body []: Assign targets []: Subscript value : Name : VAR_0_0 slice : Name : VAR_0_1 value : Call func : Name : int args []: Name : VAR_0_2 body []: Assign targets []: Name : VAR_0_3 value : BinOp left : Subscript value : Name : VAR_0_0 slice : Constant : INT op : Div right : Subscript value : Name : VAR_0_0 slice : Constant : INT body []: Assign targets []: Name : VAR_0_3 value : Call func : Name : format args []: Name : VAR_0_3 args []: Constant : STRING body []: Expr value : Call func : Name : print args []: Name : VAR_0_3

Code B: s089174134 ⬇ import sys def solve ( T : int , X : int ): print ( T / X ) return def main (): def iterate_tokens (): for line in sys . stdin : for word in line . split (): yield word tokens = iterate_tokens () T = int ( next ( tokens )) X = int ( next ( tokens )) solve ( T , X ) if name == ’main’ : main ()

Complete canonical AST B ⬇ root : Module body []: Import names []: alias : sys body []: FunctionDef : FUNC_0 args : arguments args []: arg : VAR_1_0 annotation : Name : int args []: arg : VAR_1_1 annotation : Name : int body []: Expr value : Call func : Name : print args []: BinOp left : Name : VAR_1_0 op : Div right : Name : VAR_1_1 body []: Return body []: FunctionDef : FUNC_1 args : arguments body []: FunctionDef : FUNC_2 args : arguments body []: For target : Name : VAR_3_0 iter : Attribute : stdin value : Name : LIB_sys body []: For target : Name : VAR_3_1 iter : Call func : Attribute : split value : Name : VAR_3_0 body []: Expr value : Yield value : Name : VAR_3_1 body []: Assign targets []: Name : VAR_2_0 value : Call func : Name : FUNC_2 body []: Assign targets []: Name : VAR_2_1 value : Call func : Name : int args []: Call func : Name : next args []: Name : VAR_2_0 body []: Assign targets []: Name : VAR_2_2 value : Call func : Name : int args []: Call func : Name : next args []: Name : VAR_2_0 body []: Expr value : Call func : Name : FUNC_0 args []: Name : VAR_2_1 args []: Name : VAR_2_2 body []: If test : Compare left : Name : name ops []: Eq comparators []: Constant : STRING body []: Expr value : Call func : Name : FUNC_1

Appendix

Table 10: Low-similarity accepted pair for problem p03135 . The problem asks how much time passes in World A when a student studies for

T

hours in World B, where time passes

X

times as fast. Both accepted submissions compute

T/X

. Solution A converts a split input list in a loop and formats the result; solution B uses nested generator loops, helper functions, and type annotations. Their normalized Zhang–Shasha similarity is

0.407407

(edit distance

64

across trees with

44

and

64

nodes). The four cells contain the source (comments removed) and the complete automatically exported canonical AST for submissions s227737564 and s089174134 ; the AST difference is not a manually prepared summary.

Panel of Figure 6 — Figure 6: Tabular collision inflation decays with local coverage. The exact expected collision ratio $1+(N_{2}-1)/n_{h}$ for $N_{2}=10,50,100$ .

Panel of Figure 7 — Figure 7: Homogeneous-tree illustration. With independent tabular estimates, $N_{2}=100$ , and identical continuation collision at every branch, the per-step factor compounds multiplicatively. The generic recursion need not factor this way.

Panel of Figure 8 — Figure 8: Temperature preserves outcome order. For $q=(0.50,0.25,0.12,0.07,0.04,0.02)$ , increasing $T$ flattens the distribution but cannot reproduce the off-order target $p=(0.30,0.05,0.05,0.05,0.25,0.30)$ , even though $N_{2}(p)=4<S=6$ .

Panel of Figure 9 — Figure 9: Global temperature traces a fixed diversity interval. For $q=(0.7,0.2,0.07,0.03)$ , $N_{2}^{T}$ increases from the mode count $1$ toward the support size $4$ ; finite temperatures attain neither endpoint.

Panel of Figure 10 — Figure 10: Local temperature can increase sequence collision. In the two-stage EOS/continue example, raising $T$ moves mass toward the short deterministic branch and reduces sequence diversity.

Output Diversity in Language Models

See all 32 papers

Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs
May 11, 2026Amin Banayeeanzade, Qingchuan Yang, Dhruv Tarsadiya +6LLM Evaluation Language Model Decoding
♠University of Southern California · ♣Capital One
Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
Oct 1, 2025Jiayi Zhang, Simon Yu, Derek Chong +4LLM Prompting Model Collapse
Northeastern University · Stanford University · West Virginia University
Diversity in Large Language Models under Supervised Fine-Tuning
Apr 30, 2026Roman Klypa, Oleksandr CherednichenkoSupervised Fine-Tuning Focal Loss
Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK, 38000 Grenoble, France · Department of Mathematics and Mathematical Statistics, Integrated Science Lab, Umeå University, Sweden.

May 11, 2026J/K move · Enter open · S save

Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs

Amin Banayeeanzade, Qingchuan Yang, Dhruv Tarsadiya, Fatemeh Bahrani+5

♠University of Southern California · ♣Capital One

LLM Evaluation Language Model Decoding Language Model Calibration Diverse Text Generation Output Diversity in Language Models

Open paper AskPDF arXiv

Diversity is essential for language-model applications ranging from creative generation to scientific discovery, yet modern LLMs often collapse into a narrow subset of plausible outputs. While prior work has developed benchmarks for measuring this lack of diversity, less is known about how the step-by-step probability distributions at inference time cause the problem. We introduce a validity--diversity framework that attributes diversity collapse to how an LLM allocates probability mass across valid and invalid continuations during decoding. This framework decomposes the bottleneck into two complementary forms of miscalibration. First, order calibration: valid tokens are not reliably ranked above invalid tokens, so rank-based cutoff rules must trade off between recovering valid continuations and admitting invalid ones. Second, shape calibration: probability mass is overly concentrated only on few valid continuations while having a heavy-tail of mixed valid and invalid tokens, so maintaining high validity limits diversity. We formalize both mechanisms and show that local failures compound across decoding steps, producing strong sequence-level losses in diversity. Empirically, we develop controlled diagnostics for probing these bottlenecks, including tasks with exactly known valid sets and oracle cutoff baselines. Across 14 language models spanning multiple families and scales, we find that diversity collapse is not merely a limitation of particular sampling heuristics, but a consequence of order and shape miscalibration in the LLM distribution.

Abstract

Output Diversity in Language Models

Explore similar work

Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

Diversity in Large Language Models under Supervised Fine-Tuning