Compact Language, Complex Model Shifts: How and Where Ambiguity and Underspecification Affect LLMs

Michaela Regneri; Nina Scheller; Sören Laue

cs.CLSep 30, 2026

Compact Language, Complex Model Shifts: How and Where Ambiguity and Underspecification Affect LLMs

Authors: Michaela Regneri, Nina Scheller, Sören Laue

Organizations: HAW Hamburg · Universität Hamburg

Abstract

We analyze how lexical ambiguity and underspecification affect language model training. We create artificial homonyms and artificial hypernyms as pseudowords and analyze the generative performance of language models as they are trained with increasing amounts of these ambiguous or underspecified pseudoword types. We further analyze whether the models disambiguate ambiguous or underspecified statements and provide a first mechanistic account of how ambiguity and disambiguation are represented internally. Our main results show that both ambiguity and underspecification increase model performance in ways that scale with their influence on the language's type-token ratio. However, the accuracy of generating sequences containing ambiguous words or their synonyms decreases compared to other texts. We also show that internal representations of pseudowords reflect disambiguation of pseudo-homonyms, but underspecification of pseudo-hypernyms is maintained during the generative process.

Figures & tables

Ambiguity Type	recipe	tiny
homonyms	0.28	0.32
polysemes	0.24	0.30
monosemes	0.21	0.28
pseudo-homonyms	0.36	0.40
pseudo-hypernyms	0.29	0.29

Table 1: Mean neighborhood distance (20 nearest neighbors, Word2Vec) for standard homonyms, polysemes, and monosemes, and our pseudowords. A larger distance means that a word’s neighbors are more spread out in semantic space.

Figure 1: Changes in accuracy (left) and perplexity (right) compared to the unmodified base corpora after introducing N pseudo-homonyms (HOM) and pseudo-hypernyms (HYP), with 1–50 pseudoword types. Results are averaged over both corpora; regions indicate min/max over 3 runs. Changes for models with full component replacement are significant (

p<0.01

) from N=20, HALF models do not show significant changes.

Figure 2: Linear regression predicting changes in LM accuracy from TTR for all models; regression coefficient for TTR change is 0.86.

Figure 3: Comparing accuracy for batches containing pseudowords or components with batches containing neither of them, averaged over both corpora and either all HALF-models (left) or models with full replacement (right), measured as delta with other batches ("neither"). All results, except HYP in HALF models, are significant (

p<0.01

Figure 4: Sense selectivity in the 50-HALF models. Pseudo-homonyms are disambiguated; pseudo-hypernyms remain underspecified.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Pseudoword	Word 1	Freq	Word 2	Freq	Cos. Sim.	Total Freq
simmoil	boil	465,659	simmer	365,022	0.729	830,681
liemon	lemon	268,966	lime	80,588	0.781	349,554
poreef	beef	191,114	pork	146,549	0.752	337,663
dryet	dry	244,075	wet	34,358	0.737	278,433
poettle	pot	255,925	kettle	11,061	0.823	266,986
thinick	thick	144,560	thin	77,768	0.728	222,328

Appendix

Table 2: Pseudoword mappings for hypernyms on RecipeNLG, including word frequencies on the raw corpus and cosine similarity of the component words.

Pseudoword	Word 1	Freq	Word 2	Freq	Cos. Sim.	Total Freq
wilookie	cookie	153,461	wild	9,416	-0.170	162,877
livide	divide	83,732	lid	60,620	-0.147	144,352
wintress	press	117,501	winter	3,509	-0.163	121,010
strorrange	arrange	106,223	strong	3,642	-0.163	109,865
microwight	microwave	103,087	slight	3,413	-0.158	106,500
glaender	glass	78,152	render	1,902	-0.145	80,054

Appendix

Table 3: Pseudoword mappings for homonyms on RecipeNLG, including word frequencies on the raw corpus and cosine similarity of the component words.

Pseudoword	Word 1	Freq	Word 2	Freq	Cos. Sim.	Total Freq
fimp	fish	331,746	shrimp	16,052	0.878	347,798
wirty	wet	111,774	dirty	92,013	0.826	203,787
harf	hat	159,558	scarf	28,519	0.837	188,077
purplink	pink	87,515	purple	57,877	0.894	145,392
towack	tower	76,352	stack	12,493	0.831	88,845
fanch	farm	69,515	ranch	9,316	0.927	78,831

Appendix

Table 4: Pseudoword mappings for hypernyms on TinyStories, including word frequencies on the raw corpus and cosine similarity of the component words.

Pseudoword	Word 1	Freq	Word 2	Freq	Cos. Sim.	Total Freq
slark	dark	118,492	slice	3,691	-0.162	122,183
earthquink	pink	87,515	earthquake	6,802	-0.188	94,317
wrise	wise	89,254	wrist	2,511	-0.172	91,765
colorision	colorful	85,866	decision	3,184	-0.193	89,050
harair	chair	78,373	harm	1,796	-0.169	80,169
bitinue	bite	60,784	continue	3,010	-0.192	63,794

Appendix

Table 5: Pseudoword mappings for homonyms on TinyStories, including word frequencies on the raw corpus and cosine similarity of the component words

Panel of Figure 5 — Figure 5: Accuracy (upper) and Perplexity (lower) changes in the models for increasing ambiguous (HOM) or underspecified (HYP) pseudowords, split by corpus.

Figure 7: Absolute number of tokens in the models depending on the number of pseudoword types.

Figure 8: Von-Neumann-Entropy change in the models depending on increasing ambiguity or abstraction (left), and the layer-wise localization of that change (right).

ID	Neutral	Disambiguating
1	This is another $x$ , like the other one.	This is another $x$ , like the other $y$ .
2	Here we have that $x$ , just an ordinary one.	Here we have that $x$ , just an ordinary $y$ .
3	She likes that $x$ , like all other things.	She likes that $x$ , like all other $y$ s.
4	Look for my $x$ , it looks like one of your things.	Look for my $x$ , it looks like one of your $y$ s.
5	He talked about the $x$ after other topics.	He talked about the $x$ after other $y$ s.

Appendix

Table 6: Matched patterns to test disambiguation.

x

= pseudoword;

y

= component word (pluralized in frames 3–5). Each frame also generates a reference sentence with

y

as subject (

y

-neutral ) and a sanity-check sentence with

y

as subject and

x

as comparison item (

y

-swapped ).

Figure 9: Sense bias for ambiguous and underspecified words. The dots show the differences in cosine similarity between the target word and the two senses, all in neutral contexts. A positive value indicates a bias towards the more frequent sense.

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 353 papers

Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing
Oct 6, 2026Jing Chen, Giulia Loca, Simona Amenta +1Language Modeling Computational Psycholinguistics
Department of Psychology, University of Milano-Bicocca, Piazza dell’Ateneo Nuovo 1, 20126 Milano, Italy · Department of Informatics, Systems and Communication – DISCo, University of Milano-Bicocca, Milano, Italy
Evaluating Chinese Ambiguity Understanding in Large Language Models
May 15, 2026Junwen Mo, Yuanzhi Lu, Yifang Xue +2LLM Evaluation
Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · School of Software Engineering, South China University of Technology, Guangzhou Higher Education Mega Centre, Panyu District, Guangzhou, 510006, Guangdong, China
Language models suffer from a curse of ambiguity
Aug 15, 2026Nicolas Zucchet, Hyun Dong Lee, Scott LindermanLLM Training LLM Reliability
Stanford University

Oct 6, 2026J/K move · Enter open · S save

Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing

Jing Chen, Giulia Loca, Simona Amenta, Marco Marelli

Department of Psychology, University of Milano-Bicocca, Piazza dell’Ateneo Nuovo 1, 20126 Milano, Italy · Department of Informatics, Systems and Communication – DISCo, University of Milano-Bicocca, Milano, Italy

Language Modeling Computational Psycholinguistics Language Model Probing

Open paper AskPDF arXiv

Systematicity, the probabilistic mapping of form to meaning, permeates language at all levels, and sublexical cues have been shown to govern human pseudoword processing. Yet whether LLMs exhibit comparable sensitivity to these cues remains unclear. We tested five LLMs on two Italian two-alternative forced-choice pseudoword experiments and compared their responses with a human behavioural baseline. LLMs aligned more reliably with humans when real-word options provided a lexical familiarity cue than in the pseudoword-only condition, where they fell substantially below fastText, a character-n-gram model. In addition, the sublexical cosine-similarity cue that reliably drove human--fastText agreement did not consistently transfer to human--LLM alignment, and reasoning-token expenditure bore no consistent relation to human processing difficulty. These findings suggest that LLMs do not necessarily share the sublexical cues that govern human pseudoword processing; we discuss tokenization and training-data coverage as candidate explanations.

Abstract

LLM Training

Explore similar work

Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing

Evaluating Chinese Ambiguity Understanding in Large Language Models

Language models suffer from a curse of ambiguity