Language models suffer from a curse of ambiguity

Nicolas Zucchet; Hyun Dong Lee; Scott Linderman

cs.CLAug 15, 2026

Language models suffer from a curse of ambiguity

Authors: Nicolas Zucchet, Hyun Dong Lee, Scott Linderman

Organizations: Stanford University

Abstract

Large language models increasingly rely on sampling as a driver of their own improvement, making the fidelity of their learned distributions more critical than ever. Yet, not all distributions are equally easy to learn. In this work, we identify a curse of ambiguity: in large language models, and more broadly in all neural networks that produce discrete probability distributions, the more ambiguous a next-token distribution is, the harder it is to learn accurately. Through an extensive theoretical analysis, we trace this curse to architectural and learning roots. More ambiguous distributions require more capacity to be stored, larger embeddings to be represented, more steps to be fitted, and amplify token-sampling noise. We validate these findings on synthetic tasks with controlled ground truth and observe the same signatures in language models trained on real data. Our results provide a new perspective on the statistical capabilities of large language models and a practical framework for when to trust their output distribution.

Figures & tables

Figure 1: Architectural roots of the curse of ambiguity: ambiguous distributions require more parameters to be stored and larger embeddings to be represented. Schematic description of the theoretical analysis of Section 3 , depicting how the network capacity (left) and the required embedding dimension (right) depend on our measure of ambiguity, which is the number of possible next tokens

k

(the support size) under the ground-truth distribution. The capacity is proportional to the number of parameters and inversely proportional to

k

, while the required embedding dimension grows proportionally with

k

, with a larger constant when the backbone cannot produce arbitrary embeddings (realistic vs. ideal backbone). Capacity predictions are empirically validated in Appendix A.6 .

Figure 2: Learning roots of the curse of ambiguity: ambiguous distributions require more optimization steps to be learned and amplify the noise coming from next-token sampling. Simulation of the dynamics theoretically analyzed in Section 4 . Increasing the support size

k

slows down initial gradient descent on the expected loss and yields larger losses in the long run (left), and widens the gap between sampled-batch trajectories and the expected-loss dynamics (right). The insights from this simplified one-context toy model extend to a linear head trained on many contexts of mixed ambiguity, as shown in Appendices C.2 and D.6 .

Figure 3: Transformers trained autoregressively on a $3$ -gram task suffer from the curse of ambiguity. The data follows the empirical $3$ -gram distributions from the TinyStories dataset, at the word level. Each dot is a different context (a pair of words), characterized by how many times it appears in the dataset (context frequency) and the effective support size of the true next-token distribution. For each context, we report the average KL divergence between the model’s prediction and the true distribution (left), the average probability mass assigned to the support (middle), and the average entropy gap between the true and the predicted distribution (right). The first two panels confirm that the model suffers from the curse of ambiguity: performance degrades as effective support size increases at fixed context frequency. The model’s failure mode systematically consists of leaking probability mass outside the support.

Figure 4: Transformers suffer from the curse of ambiguity in the same way as linear models. Each line is an iso-loss contour at

D_{\mathrm{KL}}=0.5

, obtained from a log-log linear fit of the per-context loss of Figure 3 (left) in effective support size and context frequency, which describes the measurements well (Appendix F ). Increasing capacity or the number of training steps improves performance uniformly across all effective support sizes. Increasing the embedding dimension or removing sampling noise by directly optimizing the expected loss over outputs (KL divergence) significantly improves modeling abilities at large effective support sizes, but barely affects deterministic next-token distributions, consistent with the theory developed for the linear case. Setup is the same as in Figure 3 ; see Appendix F for experimental details.

Figure 5: Large language models suffer from the curse of ambiguity. Same per-context evaluation as Figure 3 (left), except that we now evaluate OLMo-2 1B [ 66 ] on natural data sampled from Dolma v1.7 [ 91 ] . The effective support size and semantic context frequency are estimated from the larger OLMo-2 13B model, a proxy for the ground-truth distribution. Overall, we recover the same behavior as in Figure 3 : at fixed frequency, more ambiguous contexts are harder to predict. Appendix G.3 quantitatively shows that the log KL divergence and the log effective support size at fixed semantic context frequency are correlated. See Section 5.2 and Appendix G.2 for a discussion of what the outliers in the bottom left of the plot correspond to, and Appendix G.1 for experimental details.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The Hebbian model captures the same dependency on ambiguity as learned linear heads. The left and middle panels show the number of contexts whose top-

k

predictions exactly match the ground-truth support, as a function of the dataset size

n

, for a linear head trained with Adam (left) and the closed-form Hebbian construction of Section 3.1 (middle). The right panel shows the corresponding capacity

n^{\ast}

as a function of the support size

k

, where

n^{\ast}

is the largest

n

for which the median top-

k

accuracy stays above

0.99

. Both models exhibit similar dependencies on ambiguity and only approximately differ by a constant factor.

Figure 7: The capacity of a trained linear head grows proportionally to the number of parameters

hd

, with the

1/k

offset predicted by Theorem 1 . Same capacity definition and learning-rate selection as Figure 6 , up to the reductions described in the experimental setup below, with the capacity

n^{\ast}

reported as a function of the number of parameters

hd

for

k\in\{1,4,16\}

, sweeping

h=d\in\{32,64,128,256\}

. The gray dotted line indicates a slope of

1

, that is, a capacity directly proportional to

hd

Architecture	linear head, $h=d=128$ ( $hd=16{,}384$ parameters)
Loss	exact KL divergence to the target distributions
Optimizer	Adam, batch size $2{,}048$ , constant learning rate, $150{,}000$ steps
Support size $k$	$\{1,\,2,\,4,\,8,\,16,\,32\}$
Number of contexts $n$	$21$ values, log-spaced between $32$ and $32{,}768$
Learning rate	$\{10^{-3},\,3{\cdot}10^{-3},\,10^{-2},\,3{\cdot}10^{-2},\,10^{-1},\,3{\cdot}10^{-1},\,1\}$
Seeds	$5$

Appendix

Table 1: Sweep configuration for the capacity validation experiment of Figure 6 .

\{\cdots\}

indicate the values that we sweep over.

Figure 8: The tetrahedron is

3

-neighborly: every subset of its four vertices is a face. The cube is only

1

-neighborly: antipodal vertices are not connected by an edge, so not every pair of vertices forms a face.

Figure 9: Gradient-flow theory qualitatively captures learning below the capacity threshold, even with mixed ambiguity within the data. Each panel shows the mean KL divergence on the size-

k

subset of contexts, against normalized step

\mathrm{step}/n

, for one value of the number of contexts

n

. Below the capacity threshold (top row,

n\in\{32,64,128\}

), the curves stack in increasing order of

k

and decay at similar rate, mirroring the theoretical prediction that more ambiguous distributions are harder to learn but share the same asymptotic slope; see Figure 2 . As we approach and cross the capacity threshold (bottom row,

n\in\{256,512,1024\}

), the dynamics flatten and the simplified theory no longer applies: ambiguous distributions plateau at high loss while less ambiguous ones are still learned, consistent with the capacity-based picture of Appendix A .

Figure 10: Local log-log slope

\Delta\log\mathrm{loss}/\Delta\log t

for each panel of Figure 9 . Below the capacity threshold, all

k

converge to slopes close to the dashed reference at

-1

, matching the gradient-flow prediction

D_{\mathrm{KL}}\sim k(d-k)/(td)\approx k/t

for small

k

values.

Architecture	linear head, $h=d=256$
	( $hd=65{,}536$ parameters)
Optimizer	stochastic gradient descent, batch size $1{,}024$ ,
	constant learning rate $1$ , $6.4{\cdot}10^{6}$ steps
Support size $k$	$\{1,\,2,\,4,\,8,\,16,\,32\}$ , mixed in equal proportion
Contexts per ambiguity level $n$	$\{32,\,64,\,128,\,256,\,512,\,1024\}$
Seeds	$5$

Appendix

Table 2: Sweep configuration for the gradient-flow validation experiment of Figures 9 and 10 . Each ambiguity level forms a group of

n

contexts, for a total of

6n

context-distribution pairs per run.

Figure 11: The empirical sampling gap grows with

k

asymptotically, in line with the theoretical prediction of Section 4.2 . Top row: evolution of the mean per-

k

KL divergence under stochastic gradient descent on the cross-entropy loss (with sampled targets), against the training step normalized by the number of contexts per ambiguity level

n

, for

n\in\{32,256\}

. Bottom row: gap to the matched distributional-target reference (full-batch gradient descent on the same task);

k=1

is omitted as its gap is below the seed-noise floor. The late-time gap increases with

k

, matching the asymptote of Equation 4.2 .

Figure 12: Same as Figure 3 , but using the raw support size of the ground-truth distribution as the measure of ambiguity instead of the effective support size.

Figure 13: Visual representation of how we summarize the per-context loss landscape of Figure 3 (left) into the iso-loss line we use in Figure 4 .

	Default	Capacity	Embedding	Optimization	Sampling
Architecture	$4$ -layer transformer, $4$ attention heads
Vocabulary	$2{,}000$ most frequent words of TinyStories
Data	$256$ -token sequences, batch size $128$ , one seed
Optimizer	AdamW, cosine schedule, weight decay $0.01$
Learning rate	$\{10^{-4},\,3\times 10^{-4},$ $10^{-3},\,3\times 10^{-3}\}$	$\{3\times 10^{-4},\,10^{-3},\,3\times 10^{-3}\}$
$d_{\mathrm{model}}$	$256$	$\mathbf{\{32,\,64,}$ $\mathbf{128,\,256,}$ $\mathbf{512,\,1024\}}$	$256$	$256$	$256$

Appendix

Table 3: Configuration of ablation experiments on the synthetic

3

-gram task. Each column corresponds to a sweep that varies a single parameter (highlighted in bold) while holding all others at the default value. Learning rate is tuned for every configuration in every sweep: the number of runs is the number of swept values times the size of the learning-rate grid, and the reported curves use the best learning rate (

3\times 10^{-3}

for the default configuration).

Figure 14: Per-context losses across the capacity ablation. Each row corresponds to one value of $d_{\mathrm{model}}\in\{32,\,64,\,128,\,256,\,512,\,1024\}$ (annotated as parameter count in millions). All other parameters are held at the defaults of Table 3 : embedding dimension $256$ , $3000$ training steps, sampled cross-entropy loss, with learning rate selected per configuration as the best of $\{3\times 10^{-4},\,10^{-3},\,3\times 10^{-3}\}$ .

Figure 15: Per-context losses across the embedding-dimension ablation. Each row corresponds to one value of $d_{\mathrm{embed}}\in\{32,\,64,\,128,\,256,\,512,\,1024\}$ . All other parameters are held at the defaults of Table 3 : $d_{\mathrm{model}}=256$ , $3000$ training steps, sampled cross-entropy loss, with learning rate selected per configuration as the best of $\{3\times 10^{-4},\,10^{-3},\,3\times 10^{-3}\}$ .

Figure 16: Per-context losses across the training-duration ablation. Each row corresponds to one number of training steps in $\{356,\,725,\,1500,\,3000,\,6000,\,12000\}$ . All other parameters are held at the defaults of Table 3 : $d_{\mathrm{model}}=d_{\mathrm{embed}}=256$ , sampled cross-entropy loss, with learning rate selected per configuration as the best of $\{3\times 10^{-4},\,10^{-3},\,3\times 10^{-3}\}$ .

Figure 17: Per-context losses across the loss ablation. The two rows compare optimizing the full KL divergence $D_{\mathrm{KL}}(p_{x}\,\|\,\hat{p}_{x})$ (top) against the sampled cross-entropy with one label drawn per step (bottom). All other parameters are held at the defaults of Table 3 : $d_{\mathrm{model}}=d_{\mathrm{embed}}=256$ , $3000$ training steps, with learning rate selected per configuration as the best of $\{3\times 10^{-4},\,10^{-3},\,3\times 10^{-3}\}$ .

Figure 18: Effect of changing the kernel bandwidth $\lambda$ on the results of Figure 5 . Same setup as Figure 5 , varying the bandwidth $\lambda$ of the exponential kernel $\exp(-d_{i}/\lambda)$ used to compute $n_{\mathrm{est}}$ from kNN distances. Larger $\lambda$ shifts the x-axis to the right (more neighbors are weighted in), but the cloud shape and the per-context KL coloring are essentially invariant under $\lambda$ .

Figure 19: Joint distribution of semantic context frequency and estimated effective support size, alongside the average position in the sequence at which each context occurs. Both panels share axes with Figure 5 . Left: each hex is colored by the geometric-mean position (in tokens) of the contexts that fall in it, with opacity modulated by the log of the per-hex count so sparse cells fade out. Low-frequency, low-effective-support cells are dominated by early-sequence positions, where local context is short and the kNN density estimate is least reliable. Right: log-density of contexts in the same plane. Most of the mass concentrates in a narrow band where $n_{\mathrm{est}}$ and effective support size are positively correlated.

Figure 20: The kNN density proxy correlates with the true context frequency on the $3$ -gram task. Log-density (hexbin, $10^{6}$ positions subsampled uniformly from the held-out evaluation set) of the true frequency of each context in the training corpus (x-axis) against its kernel density $\mathrm{e}^{-d_{i}/\lambda}$ averaged over its retrieved nearest neighbors in the last-layer embedding space of the trained $4$ -layer transformer (y-axis), at $\lambda=2$ . On this task we plot the average rather than the sum of Appendix G.1 , which is why the y-axis lies below one; the two differ by the constant number of retrieved neighbors, so both correlations below are identical for either choice. The two quantities are strongly rank-correlated across positions (Spearman correlation of $0.92$ ), supporting the use of $n_{\mathrm{est}}$ as a context-frequency proxy on natural data, where the true count is not observable.

Figure 21: The curse of ambiguity persists when ambiguity is measured by raw support size instead of effective support size. Same setup as Figure 5 , except that the y-axis now shows the raw support size of the OLMo-2 13B reference distribution. The qualitative picture is unchanged.

Figure 22: Effect of varying the size of the evaluated model on the results of Figure 5 . Same setup as Figure 5 , with OLMo-2 1B (left) and OLMo-2 7B (middle) as the model evaluated against the OLMo-2 13B reference. The 7B model tracks the reference more closely (KL is lower across the cloud), mirroring the improvement observed when increasing model size in the synthetic $3$ -gram task; cf. Figures 4 and 14 .

population (1994 est.) Net migration rate: -12.05 migrant(s)/ 1, 000 population (1994 est.) Infant mortality rate: 18.5 deaths/1 ,

RESSIDA. I will not, uncle. I have forgot my father; I know no touch of consanguinity, No kin, no love, no blood, no soul so near me As the sweet Troil us. O you gods divine, Make Cressid’s name the very crown of falsehood, If ever she leave Tro il

from the men of Shechem, and from the house of Millo, and devour Abimele ch. 9:21 And Jotham ran away, and fled, and went to Beer, and dwelt there, for fear of Abimelech his brother. 9:22 When Abime le

:014:017 And I, behold, I will harden the hearts of the Egyptians, and they shall follow them: and I will get me honour upon Ph araoh, and upon all his host, upon his chariots, and upon his horsemen. 02:014:018 And the Egyptians shall know that I am the LORD, when I have gotten me honour upon Ph

/1, 000 population (1994 est.) Net migration rate: -1.78 migrant(s)/1,000 population (1994 est.) Infant mortality rate: 8.9 deaths/1 ,

God, that I may be at once avenged of the Philistines for my two eyes. 16:29 And Sam son took hold of the two middle pillars upon which the house stood, and on which it was borne up, of the one with his right hand, and of the other with his left. 16:30 And Sam

the tone of fairy tales, and it is certainly not lawlessness or even liberty, though men under a mean modern tyranny may think it liberty by comparison. People out of Portland Gaol might think Fleet Street free; but closer study will prove that both fairies and journalists are the slaves of duty. Fairy godmo thers seem at least as strict as other god mo

Appendix

Table 4: Examples of low-loss tokens from the band highlighted in Figure 5 (

n_{\mathrm{est}}<10

). The token to predict is shown in italics and the matching

n

-gram earlier in the same sequence is shown in dark red (a trigram when a trigram match exists, a bigram otherwise).

Figure 23: Per-context loss and effective support size are correlated at fixed semantic context frequency. Each panel slices the data of Figure 5 by a log-spaced bin of semantic context frequency, indicated in brackets in the title, and shows the 2D log-density of the KL divergence between the predictions of the 13B reference model and the 1B probe against the estimated effective support size; the value next to each title is the squared Pearson correlation $R^{2}$ between $\log D_{\mathrm{KL}}$ and $\log$ effective support size within the bin. The correlation is strongest for both small (rare contexts, $n_{\mathrm{est}}\lesssim 10$ ) and large (common contexts, $n_{\mathrm{est}}\gtrsim 100$ ) semantic context frequencies. It drops for intermediate values of $n_{\mathrm{est}}$ , which is partially explained by the high overall loss in that regime: when most contexts already sit near the maximum KL, effective support size has little remaining variance to predict.

Figure 24: Per-context loss vs. raw support size at fixed semantic context frequency. Same setup as Figure 23 , with raw support size in place of effective support size on the x-axis. $R^{2}$ is higher than in the effective-support-size version, partly because the relationship is more linear in this plane.

Explore similar work

May 12, 2026

Probabilistic Calibration Is a Trainable Capability in Language Models

Language models are increasingly used in settings where outputs should follow an intended distribution, yet their generation probabilities are often poorly calibrated to it. In many such requests the distribution is implicit and the valid outputs cannot be listed in advance, so the task cannot be passed to an external random number generator. We study whether this capability can be improved directly through fine-tuning. Concretely, we fine-tune language models on synthetic prompts that require sampling from mathematical distributions, and compare two Calibration Fine-Tuning variants: a soft-target method that converts the desired output distribution into trie-derived next-token targets, and a hard-target method that trains on sampled completions from the same target distribution. Across 12 models spanning four families, both methods substantially improve structured-sampling fidelity on held-out distribution families and unseen parameter settings, showing that training on cheap synthetic mathematical distributions improves sampling on unseen distributions, even in natural language. The gains also transfer to broader stochastic generation benchmarks, including open-ended random generation, where fine-tuned models broaden their first-token support by one to two orders of magnitude, and NoveltyBench. The gains sometimes reduce downstream capability, especially arithmetic reasoning, with costs varying by model. Overall, our results show that probabilistic calibration can be improved through fine-tuning with either objective, and that calibration learned on mathematical distributions transfers to natural-language stochastic generation. Code is available at https://github.com/chandar-lab/calibration-finetuning.

Davide Baldelli, Sruthi Kuriakose, Maryam Hashemzadeh +2

Chandar Research Lab · Mila – Quebec AI Institute · LAMA-WeST Lab +3

LLM Fine-Tuning Controllable Text Generation Language Model Calibration

Ask PDF

Sep 20, 2026

The Probabilistic Structure of Large Language Models

This paper presents a probabilistic perspective on large language models (LLMs), developed with the aim of bringing together, in a single self-contained account, tools that are usually treated separately across the literature. LLMs are described through probability measures on the set of sequences of tokens, specified via their autoregressive conditional distributions. Training is formulated as a maximum-likelihood estimation problem, addressed by stochastic gradient methods, while text generation is viewed as the sequential simulation of the resulting stochastic process. The role of the asymmetry of the Kullback--Leibler divergence in text generation is examined in relation with characteristic phenomena such as hallucination and the distinction between statistical plausibility and truth. As a complementary illustration of the same viewpoint, we also discuss diffusion models, built around the score function, which cast generation not as sequential token prediction but as the simulation of a reverse-time stochastic process transforming noise into data both in discrete and continuous time.

Adnan Aboulalaâ

Language Modeling Autoregressive Language Modeling LLM Training

Ask PDF

Dec 14, 2024

Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

Tokenization is a necessary component within the current architecture of many language mod-els, including the transformer-based large language models (LLMs) of Generative AI, yet its impact on the model's cognition is often overlooked. We argue that LLMs demonstrate that the Distributional Hypothesis (DH) is sufficient for reasonably human-like language performance (particularly with respect to inferential lexical competence), and that the emergence of human-meaningful linguistic units among tokens and current structural constraints motivate changes to existing, linguistically-agnostic tokenization techniques, particularly with respect to their roles as (1) vehicles for conveying salient distributional patterns from human language to the model and as (2) semantic primitives. We explore tokenizations from a BPE tokenizer; extant model vocabularies obtained from Hugging Face and tiktoken; and the information in exemplar token vectors as they move through the layers of a RoBERTa (large) model. Besides creating suboptimal semantic building blocks and obscuring the model's access to the necessary distributional patterns, we describe how tokens and pretraining can act as a backdoor for bias and other unwanted content, which current alignment practices may not remediate. Additionally, we relay evidence that the tokenization algorithm's objective function impacts the LLM's cognition, despite being arguably meaningfully insulated from the main system intelligence. Finally, we discuss implications for architectural choices, meaning construction, the primacy of language for thought, and LLM cognition. [First uploaded to arXiv in December, 2024.]

Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer +9

Computational Story Lab, Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Vermont Complex Systems Institute, University of Vermont, Burlington, VT 05405, USA · Department of Computer Science, St. Michael’s College, Colchester, VT 05439, USA +3

LLM Interpretability Distributional Semantics Subword Tokenization

Ask PDF