Language models are trained on tokenized text that obscures the sound structure of words, yet they reliably produce morphemes whose form is phonologically conditioned. It remains unclear whether they rely on item-specific memorization or rule-like generalization and, if the latter, how that generalization is implemented. We therefore ask whether this phonological condition is represented within language models and how it is causally used for allomorph selection. For the English indefinite article a/an, we show that the phonological condition is encoded along a single linear direction in trigger-token embeddings, that this direction causally drives article selection in token-level wug tests, and that, at the article-prediction position, the model forecasts the upcoming trigger token and uses the forecasted trigger's phonological feature to choose the article. We then ask whether this rule-like generalization extends beyond English article selection, both to allomorph selection in other languages and to explicit phonological judgment. Together, these results provide a mechanistic account of phonologically conditioned allomorph selection in language models, and dissociate this generation-time ability from explicit metalinguistic judgments.
Figures & tables
Allomorph
Initial sound
Examples
a
consonant
a banana ; a university /j-/
an
vowel
an apple ; an hour /aU-/
Table 1: Allomorphs of the English indefinite article a / an , conditioned by the initial sound of the trigger token rather than its orthography.
Orthography
Phonology
consonant-start
vowel-start
consonant-start
Train banana , city
Test hour , honor , heir
vowel-start
Test one , unique , university
Train apple , ordinary
Table 2: Train/test split for fitting the phonological direction. Training tokens are cases where orthography and corpus-derived phonological label agree; test tokens are exceptions where orthography predicts the wrong allomorph.
Model
Train n
CV acc.
Test correct
Misses
Llama
8,934
99.89
41/41
–
Qwen
8,915
99.85
39/41
heir , honors
Gemma
9,873
99.93
49/50
hei
Table 3: Phonological-direction fitting results. Train n counts retained trigger-token vocabulary items and CV accuracy is a percentage; the test column reports correctly classified tokens out of the full exception set. The test set consists only of orthography–phonology exception tokens.
Model
d1
Residual after removing d1
CV acc.
Test acc.
CV acc.
Test acc.
Llama
99.89
100.0
40.82
73.2
Qwen
99.85
95.1
38.31
61.0
Gemma
99.93
98.0
51.74
42.0
Table 4: Dimensionality test for the phonological embedding direction. All entries are accuracies in percent. The first two columns evaluate the original classifier direction d1 ; the last two evaluate a new residual classifier trained after projecting d1 out of all embeddings. Test accuracy is measured on the same held-out exception set as Table 3 .
Table 5: Translation template for the token-level wug test. The <nonce> placeholder is replaced by the reserved nonce token for each model.
Figure 1: Effect of varying the d component of nonce trigger tokens in the token-level wug test. Each line shows one model; the plotted value is P(an)−P(a) , averaged over 500 nonce bases in the Chinese-to-English translation template. The x -axis normalizes α so that 0 is the consonant-start class mean and 1 is the vowel-start class mean.
Figure 2: Schematic of the FutureLens map. From the article prediction position t , Fℓ maps an intermediate state htℓ to a forecast of the article-position final state ht+1L , whose decoding predicts the trigger token.
Figure 3: FutureLens top-5 agreement across layers. The probe maps the hidden state before a target token to the final state at the target-token position, whose decoding predicts the following token. The a / an condition is compared with random-token controls.
Figure 4: Counterfactual steering along the FutureLens-transposed phonological direction. For each layer, baseline- a examples are steered to the an class mean and baseline- an examples are steered to the a class mean. The plot reports top-1 flip rates on held-out examples.
Figure 5: Cross-linguistic token-level wug interventions. Each panel plots top-1 rates for the two allomorph families and for other outputs. The x -axis normalizes α so that 0 and 1 are the two class means along the learned direction.
Table 6: Cross-linguistic allomorphy cases used in the token-level wug intervention. The last column gives the phonological feature that conditions the choice in each language.
Question
Prompt
Sound judgment
Answer yes or no only. Consider the English phrase “ the <nonce> ”. Does the noun begin with a { consonant , vowel } sound in English pronunciation?
Table 7: Explicit sound-judgment prompts. The <nonce> placeholder is replaced by the reserved nonce token for each model (with no space after the ), and { consonant , vowel } indicates two prompt variants used separately.
Figure 6: Effect of varying the d component of nonce trigger tokens in explicit sound-judgment prompts. The upper panel asks whether the nonce noun begins with a consonant sound, and the lower panel asks whether it begins with a vowel sound. The plotted value is P(yes)−P(no) , averaged over 500 nonce embeddings.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Stability of d1
MLP bal. acc.
Bootstrap cos.
CV-shuffle acc.
Full
Resid.
Llama
0.989±0.001
0.9993±0.0001
1.000
0.542
Qwen
0.990±0.0004
0.9990±0.0001
0.983
0.542
Gemma
0.992±0.0004
0.9993±0.0001
1.000
0.486
Appendix
Table 8: Robustness checks for the phonological direction. Bootstrap cosine is the cosine similarity between the reported d1 and the directions refit on 200 resampled training sets; CV-shuffle accuracy is five-fold cross-validation accuracy over 50 redrawn splits. The last two columns give the balanced accuracy of the MLP probe on the exception set, before and after d1 is projected out. All entries are mean ± standard deviation where a distribution is available.
Model
Set
Centroid dist.
Nearest cos.
Llama3.2
Words
1.111±0.076
0.684±0.126
Nonce
1.114±0.026
0.226±0.076
Qwen2.5
Words
1.068±0.072
0.686±0.140
Nonce
1.069±0.023
0.169±0.043
Gemma3
Words
0.919±0.038
0.717±0.129
Nonce
0.918±0.028
0.242±0.038
Appendix
Table 9: Validation of the sampled nonce bases. Centroid distance is measured from the mean of the retained trigger-token embeddings. Nearest-vocabulary cosine is the maximum cosine similarity to any vocabulary embedding row; for real trigger tokens, the token’s own row is excluded.
Figure 7: Top-1 next-token choices in the token-level wug test. Rates are computed over 500 nonce bases in the Chinese-to-English translation template. The x -axis normalizes α so that 0 is the consonant-start class mean and 1 is the vowel-start class mean.
Condition
Context excerpt
Target
Next
a / an
…Natasha Irons / Steel appears in Multiversity as a member of the Justice League …
a
member
Random
…explained the properties of the nucleus in terms of these sub- atomic particles and the forces …
atomic
particles
Appendix
Table 10: Representative context examples. Bold marks the target token, and underlining marks the following token predicted from the target-token position.
Figure 8: Top-5 trigger agreement for ordinary and orthogonal FutureLens maps on matched strict a / an runs. The orthogonal maps preserve the late-layer forecasting accuracy relevant to the inverse-mapping analysis.
Figure 9: Article-probability shifts under counterfactual steering. The top row steers baseline- a examples toward the an class mean; the bottom row steers baseline- an examples toward the a class mean. Values are mean probability after intervention minus mean probability before intervention, over up to 1,000 held-out examples per direction.
Figure 10: The forecasting mechanism in Llama-3.3-70B-Instruct. Left: FutureLens top-5 agreement with the ground-truth trigger token at t+2 , for the a / an condition and the random-token control. Right: top-1 flip rate under counterfactual steering along sℓ , for baseline- a and baseline- an examples. Probes are fit at every second layer from 40 to 80.
A. Allomorphy systems
Language
Allomorphy
Examples
French
Definite article contraction: le / la before consonant-initial nouns, l’ before vowel-initial nouns.
le livre ‘the book’; la maison ‘the house’; l’école ‘the school’
Korean
Object particle alternation: after consonant-final nouns, after vowel-final nouns.
evde ‘at home’; okulda ‘at school’; parkta ‘at the park’
Italian
Masculine indefinite article alternation between un and uno .
un libro ‘a book’; uno studente ‘a student’
B. Evaluation templates
Appendix
Table 11: Cross-linguistic allomorphy examples and translation templates. The <nonce> placeholder denotes the nonce token whose embedding is replaced during the sweep. Korean forms are shown in Hangul.
Figure 11: The forecasting mechanism in French and Italian. Top: FutureLens top-5 agreement with the ground-truth trigger token, for the allomorph condition and the random-token control. Bottom: top-1 flip rate under counterfactual steering along sℓ , in both directions. Probes are fit at every layer from 16 to 32.
Artifact
Creator/source
Size
License or terms
URL
Llama-3.2-3B-Instruct ( Grattafiori et al., 2024 )
Audio language models pass speech and text through a single decoder. We ask whether that decoder represents a distinctive feature in the same direction when a phoneme is heard and when it is read. For minimal pairs of phonemes differing in one feature, we take the offset between the two members' mean representations. Averaging those offsets gives a direction for each stream, and we measure the cosine between the two. Because the two streams already agree about arbitrary phoneme pairs, we compare every measure against a reference built from random pairings rather than against zero. We apply this to 6 models, 7 features and 15 languages from 11 families. Only voicing in the two Qwen2.5-Omni models exceeds that reference after correction for multiple testing, and the reference varies by a factor of seven between models. In three of the six models, voicing has one direction in audio across the 14 languages with enough minimal pairs to measure it, and every language pair agrees in two of them. The model family, not the model size, predicts which stream represents a feature.
Yuanhao Chen, Peter Chin
Thayer School of Engineering, Dartmouth College, Hanover, NH, USA
Tokenization is the first step in every language model (LM), yet it never takes the sounds of words into account. We investigate how tokenization influences text-only LMs' ability to represent phonological knowledge. Through a series of probing experiments, we show that subword-based tokenization systematically weakens the encoding of both local (e.g., rhyme) and global (e.g., syllabification) phonological features. To quantify this effect, we introduce the syllabification-tokenization alignment distance (STAD), a metric that measures the misalignment between a model's tokenization and the natural syllable boundaries of words, and find that higher misalignment correlates with poorer phonological representations, providing a simple diagnostic for phonology-aware tokenization. To address these limitations, we propose a lightweight IPA-based fine-tuning method that infuses phonological awareness into LMs, leading to consistent improvements across three phonology-related tasks while largely preserving math and general reasoning ability, with 1.1% and 0.9% drops on GSM8K and MMLU, respectively.
A central question in language acquisition is whether linguistic biases can emerge from general learning mechanisms operating over underdetermined input. Artificial Language Learning (ALL) studies have shown that human learners reliably generalize beyond the evidence provided, including by preferring scope-homomorphic noun phrase modifier orders. In this work, we investigate whether language models exhibit the same bias under similar conditions. We create a controlled learning environment in which models are trained on a corpus where all noun phrases containing multiple modifiers have been removed, eliminating direct evidence about modifier ordering, and are then evaluated on multiple modifier sentences. Across three model sizes, we find that they consistently prefer scope-homomorphic orders despite never observing them during training. These preferences vary in strength by modifier type. To investigate the source of these preferences, we examine noun-modifier association strength using pointwise mutual information (PMI). While PMI reflects known modifier-ordering patterns, it does not explain the models' ordering preferences. These findings demonstrate that LMs can recover human-like linguistic generalizations from impoverished input and provide a controlled framework for investigating the mechanisms underlying such biases.
Amanda Popadich, Shane Steinert-Threlkeld
Department of Linguistics, University of Washington