How Do Language Models Represent and Use Phonological Information for Allomorph Selection?

Sangwoo Kim; Sangah Lee

cs.CLSep 4, 2026

How Do Language Models Represent and Use Phonological Information for Allomorph Selection?

Authors: Sangwoo Kim, Sangah Lee

Organizations: Department of Linguistics Seoul National University

Abstract

Language models are trained on tokenized text that obscures the sound structure of words, yet they reliably produce morphemes whose form is phonologically conditioned. It remains unclear whether they rely on item-specific memorization or rule-like generalization and, if the latter, how that generalization is implemented. We therefore ask whether this phonological condition is represented within language models and how it is causally used for allomorph selection. For the English indefinite article a/an, we show that the phonological condition is encoded along a single linear direction in trigger-token embeddings, that this direction causally drives article selection in token-level wug tests, and that, at the article-prediction position, the model forecasts the upcoming trigger token and uses the forecasted trigger's phonological feature to choose the article. We then ask whether this rule-like generalization extends beyond English article selection, both to allomorph selection in other languages and to explicit phonological judgment. Together, these results provide a mechanistic account of phonologically conditioned allomorph selection in language models, and dissociate this generation-time ability from explicit metalinguistic judgments.

Figures & tables

Allomorph	Initial sound	Examples
a	consonant	a banana ; a university /j-/
an	vowel	an apple ; an hour /aU-/

Table 1: Allomorphs of the English indefinite article a / an , conditioned by the initial sound of the trigger token rather than its orthography.

Orthography	Phonology
Orthography	consonant-start	vowel-start
consonant-start	Train banana , city	Test hour , honor , heir
vowel-start	Test one , unique , university	Train apple , ordinary

Table 2: Train/test split for fitting the phonological direction. Training tokens are cases where orthography and corpus-derived phonological label agree; test tokens are exceptions where orthography predicts the wrong allomorph.

Model	Train $n$	CV acc.	Test correct	Misses
Llama	8,934	99.89	41/41	–
Qwen	8,915	99.85	39/41	heir , honors
Gemma	9,873	99.93	49/50	hei

Table 3: Phonological-direction fitting results. Train

n

counts retained trigger-token vocabulary items and CV accuracy is a percentage; the test column reports correctly classified tokens out of the full exception set. The test set consists only of orthography–phonology exception tokens.

Model	$\mathbf{d}_{1}$		Residual after removing $\mathbf{d}_{1}$
Model	CV acc.	Test acc.	CV acc.	Test acc.
Llama	99.89	100.0	40.82	73.2
Qwen	99.85	95.1	38.31	61.0
Gemma	99.93	98.0	51.74	42.0

Table 4: Dimensionality test for the phonological embedding direction. All entries are accuracies in percent. The first two columns evaluate the original classifier direction

\mathbf{d}_{1}

; the last two evaluate a new residual classifier trained after projecting

\mathbf{d}_{1}

out of all embeddings. Test accuracy is measured on the same held-out exception set as Table 3 .

Table 5: Translation template for the token-level wug test. The <nonce> placeholder is replaced by the reserved nonce token for each model.

Figure 1: Effect of varying the

\mathbf{d}

component of nonce trigger tokens in the token-level wug test. Each line shows one model; the plotted value is

P(an)-P(a)

, averaged over 500 nonce bases in the Chinese-to-English translation template. The

x

-axis normalizes

\alpha

so that 0 is the consonant-start class mean and 1 is the vowel-start class mean.

Panel of Figure 2 — Figure 2: Schematic of the FutureLens map. From the article prediction position $t$ , $F_{\ell}$ maps an intermediate state $h_{t}^{\ell}$ to a forecast of the article-position final state $h_{t+1}^{L}$ , whose decoding predicts the trigger token.

Figure 3: FutureLens top-5 agreement across layers. The probe maps the hidden state before a target token to the final state at the target-token position, whose decoding predicts the following token. The a / an condition is compared with random-token controls.

Figure 4: Counterfactual steering along the FutureLens-transposed phonological direction. For each layer, baseline- a examples are steered to the an class mean and baseline- an examples are steered to the a class mean. The plot reports top-1 flip rates on held-out examples.

Figure 5: Cross-linguistic token-level wug interventions. Each panel plots top-1 rates for the two allomorph families and for other outputs. The

x

-axis normalizes

\alpha

so that 0 and 1 are the two class means along the learned direction.

Language	Model	Conditioning feature
Korean	EXAONE-4.0-1.2B ( LG AI Research, 2025 )	presence of a final consonant
Turkish	Trendyol-LLM-8B-T1 ( Trendyol LLM & Core NLP Team, 2025 )	backness of the preceding vowel
Italian	Minerva-7B-instruct-v1.0 ( Orlando et al., 2024 )	phonological shape of the onset
French	Lucie-7B-Instruct-v1.1 ( Gouvert et al., 2025 )	vowel-initiality of the trigger

Table 6: Cross-linguistic allomorphy cases used in the token-level wug intervention. The last column gives the phonological feature that conditions the choice in each language.

Question	Prompt
Sound judgment	Answer yes or no only. Consider the English phrase “ the <nonce> ”. Does the noun begin with a { consonant , vowel } sound in English pronunciation?

Table 7: Explicit sound-judgment prompts. The <nonce> placeholder is replaced by the reserved nonce token for each model (with no space after the ), and { consonant , vowel } indicates two prompt variants used separately.

Figure 6: Effect of varying the

\mathbf{d}

component of nonce trigger tokens in explicit sound-judgment prompts. The upper panel asks whether the nonce noun begins with a consonant sound, and the lower panel asks whether it begins with a vowel sound. The plotted value is

P(yes)-P(no)

, averaged over 500 nonce embeddings.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Stability of $\mathbf{d}_{1}$		MLP bal. acc.
Model	Bootstrap cos.	CV-shuffle acc.	Full	Resid.
Llama	$0.989{\pm}0.001$	$0.9993{\pm}0.0001$	1.000	0.542
Qwen	$0.990{\pm}0.0004$	$0.9990{\pm}0.0001$	0.983	0.542
Gemma	$0.992{\pm}0.0004$	$0.9993{\pm}0.0001$	1.000	0.486

Appendix

Table 8: Robustness checks for the phonological direction. Bootstrap cosine is the cosine similarity between the reported

\mathbf{d}_{1}

and the directions refit on 200 resampled training sets; CV-shuffle accuracy is five-fold cross-validation accuracy over 50 redrawn splits. The last two columns give the balanced accuracy of the MLP probe on the exception set, before and after

\mathbf{d}_{1}

is projected out. All entries are mean

{\pm}

standard deviation where a distribution is available.

Model	Set	Centroid dist.	Nearest cos.
Llama3.2	Words	$1.111{\pm}0.076$	$0.684{\pm}0.126$
	Nonce	$1.114{\pm}0.026$	$0.226{\pm}0.076$
Qwen2.5	Words	$1.068{\pm}0.072$	$0.686{\pm}0.140$
	Nonce	$1.069{\pm}0.023$	$0.169{\pm}0.043$
Gemma3	Words	$0.919{\pm}0.038$	$0.717{\pm}0.129$
	Nonce	$0.918{\pm}0.028$	$0.242{\pm}0.038$

Appendix

Table 9: Validation of the sampled nonce bases. Centroid distance is measured from the mean of the retained trigger-token embeddings. Nearest-vocabulary cosine is the maximum cosine similarity to any vocabulary embedding row; for real trigger tokens, the token’s own row is excluded.

Figure 7: Top-1 next-token choices in the token-level wug test. Rates are computed over 500 nonce bases in the Chinese-to-English translation template. The

x

-axis normalizes

\alpha

so that 0 is the consonant-start class mean and 1 is the vowel-start class mean.

Condition	Context excerpt	Target	Next
a / an	…Natasha Irons / Steel appears in Multiversity as a member of the Justice League …	a	member
Random	…explained the properties of the nucleus in terms of these sub- atomic particles and the forces …	atomic	particles

Appendix

Table 10: Representative context examples. Bold marks the target token, and underlining marks the following token predicted from the target-token position.

Figure 8: Top-5 trigger agreement for ordinary and orthogonal FutureLens maps on matched strict a / an runs. The orthogonal maps preserve the late-layer forecasting accuracy relevant to the inverse-mapping analysis.

Figure 9: Article-probability shifts under counterfactual steering. The top row steers baseline- a examples toward the an class mean; the bottom row steers baseline- an examples toward the a class mean. Values are mean probability after intervention minus mean probability before intervention, over up to 1,000 held-out examples per direction.

Figure 10: The forecasting mechanism in Llama-3.3-70B-Instruct. Left: FutureLens top-5 agreement with the ground-truth trigger token at

t+2

, for the a / an condition and the random-token control. Right: top-1 flip rate under counterfactual steering along

\mathbf{s}_{\ell}

, for baseline- a and baseline- an examples. Probes are fit at every second layer from 40 to 80.

A. Allomorphy systems
Language	Allomorphy	Examples
French	Definite article contraction: le / la before consonant-initial nouns, l’ before vowel-initial nouns.	le livre ‘the book’; la maison ‘the house’; l’école ‘the school’
Korean	Object particle alternation: after consonant-final nouns, after vowel-final nouns.	‘book-OBJ’; ‘apple-OBJ’
Turkish	Locative vowel harmony: front-vowel -de / -te vs. back-vowel -da / -ta .	evde ‘at home’; okulda ‘at school’; parkta ‘at the park’
Italian	Masculine indefinite article alternation between un and uno .	un libro ‘a book’; uno studente ‘a student’
B. Evaluation templates

Appendix

Table 11: Cross-linguistic allomorphy examples and translation templates. The <nonce> placeholder denotes the nonce token whose embedding is replaced during the sweep. Korean forms are shown in Hangul.

Figure 11: The forecasting mechanism in French and Italian. Top: FutureLens top-5 agreement with the ground-truth trigger token, for the allomorph condition and the random-token control. Bottom: top-1 flip rate under counterfactual steering along

\mathbf{s}_{\ell}

, in both directions. Probes are fit at every layer from 16 to 32.

Artifact	Creator/source	Size	License or terms	URL
Llama-3.2-3B-Instruct ( Grattafiori et al., 2024 )	Meta	3B	Llama 3.2 Community License	https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct
Llama-3.3-70B-Instruct ( Grattafiori et al., 2024 )	Meta	70B	Llama 3.3 Community License	https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct
Qwen2.5-3B-Instruct ( Qwen Team, 2024 )	Alibaba Cloud/Qwen Team	3B	Qwen Research License Agreement	https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
Gemma-3-1b-it ( Gemma Team, 2025 )	Google	1B	Gemma Terms of Use	https://huggingface.co/google/gemma-3-1b-it
EXAONE-4.0-1.2B ( LG AI Research, 2025 )	LG AI Research	1.2B	EXAONE AI Model License Agreement 1.2 – NC	https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-1.2B
Trendyol-LLM-8B-T1 ( Trendyol LLM & Core NLP Team, 2025 )	Trendyol	8B	Apache-2.0	https://huggingface.co/Trendyol/Trendyol-LLM-8B-T1

Appendix

Table 12: External artifacts used in the experiments. License and terms are reported as listed on the linked model or dataset cards.

Representation Geometry in Language Models

Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.

See all 417 papers

Do Audio Language Models Hear and Read Distinctive Features Alike?
Sep 24, 2026Yuanhao Chen, Peter ChinAudio-Text Alignment Audio-Language Models
Thayer School of Engineering, Dartmouth College, Hanover, NH, USA
How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them
Apr 18, 2026Disen Liao, Freda ShiLLM Fine-Tuning Language Modeling
University of Waterloo · Vector Institute
Language Models Generalize to Human-like Word Order Preferences
Aug 5, 2026Amanda Popadich, Shane Steinert-ThrelkeldSyntactic Generalization in Language Models Language Modeling
Department of Linguistics, University of Washington

Sep 24, 2026J/K move · Enter open · S save

Do Audio Language Models Hear and Read Distinctive Features Alike?

Yuanhao Chen, Peter Chin

Thayer School of Engineering, Dartmouth College, Hanover, NH, USA

Audio-Text Alignment Audio-Language Models Self-Supervised Speech Representation Learning Cross-Lingual Consistency in Language Models

Open paper AskPDF arXiv

Audio language models pass speech and text through a single decoder. We ask whether that decoder represents a distinctive feature in the same direction when a phoneme is heard and when it is read. For minimal pairs of phonemes differing in one feature, we take the offset between the two members' mean representations. Averaging those offsets gives a direction for each stream, and we measure the cosine between the two. Because the two streams already agree about arbitrary phoneme pairs, we compare every measure against a reference built from random pairings rather than against zero. We apply this to 6 models, 7 features and 15 languages from 11 families. Only voicing in the two Qwen2.5-Omni models exceeds that reference after correction for multiple testing, and the reference varies by a factor of seven between models. In three of the six models, voicing has one direction in audio across the 14 languages with enough minimal pairs to measure it, and every language pair agrees in two of them. The model family, not the model size, predicts which stream represents a feature.

Abstract

Representation Geometry in Language Models

Explore similar work

Do Audio Language Models Hear and Read Distinctive Features Alike?

How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them

Language Models Generalize to Human-like Word Order Preferences