Multilingual language models (LMs) organize representations for typologically and orthographically diverse languages into a shared parameter space, yet the nature of this internal organization remains elusive. In this work, we investigate which linguistic properties - abstract language identity or surface-form cues - shape multilingual representations. To do so, we analyze language-associated units across different model families and scales using the Language Activation Probability Entropy (LAPE) metric, and further decompose activations with Sparse Autoencoders. We find that these units are strongly conditioned on orthography: romanization induces near-disjoint representations that align with neither native-script inputs nor English, while word-order shuffling has limited effect on unit identity. Probing shows that typological structure becomes increasingly accessible in deeper layers, while causal interventions indicate that generation is most sensitive to units that are invariant to surface-form perturbations rather than to units identified by typological alignment alone. Overall, our results suggest that multilingual LMs organize representations around surface form, with linguistic abstraction emerging gradually without collapsing into a unified interlingua.
Figures & tables
Figure 1: Overlap of language-associated units for Hindi under script variation in Llama-3.2-1B. Euler diagrams show units shared among up to three languages for (a) raw neurons and (b) SAE features. Native, Romanized (with diacritics), and Romanized (without diacritics) inputs activate largely disjoint sets in both representations. Corresponding results for all languages, for both raw neurons and SAE features, and for Gemma-2-2B are shown in Figures 10 and 9 in Appendix D.2 .
Figure 2: Jaccard similarity between Romanized and native-script or English language-associated units ( raw neurons and SAE features ) in Llama-3.2-1B (see Figure 8 for Gemma-2-2B). Romanized inputs exhibit low overlap with their native-script counterparts and near-zero overlap with English in both representations, indicating limited cross-script alignment without convergence to English.
Figure 3: Layer-wise alignment between language-associated units for Native and Romanized inputs in Llama-3.2-1B (see Figure 14 for Gemma-2-2B). The red line denotes average Jaccard similarity for raw neurons , and the blue line for SAE features ; shaded regions indicate standard deviation across languages. Raw neurons show a modest mid-layer increase in overlap, while SAE features remain uniformly low across depth. In all cases, alignment remains far from convergence, indicating that representational separation persists beyond input tokenization.
Figure 4: Jaccard similarity between language-associated units identified from original and word-shuffled text in Llama-3.2-1B (see Figure 23 for Gemma-2-2B). Raw neurons exhibit consistently moderate-to-high overlap across languages, indicating robustness to word-order perturbation. In contrast, SAE features show only selective instability : languages with distinctive scripts (e.g., Chinese, Japanese, Thai) remain highly stable, whereas several Latin-script languages exhibit somewhat reduced overlap, revealing sensitivity of sparse features to local distributional patterns disrupted by shuffling for these languages.
Figure 5: Average family-wise probing R2 scores across neuron subsets induced by romanization in Llama-3.2-1B (raw neurons). Neurons overlapping between native and romanized inputs exhibit the strongest typological alignment, while script-specific subsets encode weaker signal. Baseline denotes probing over the pooled set of all neurons that were selected for either native or romanized inputs (across all layers), serving as a non-selective reference. Corresponding results for Llama-3.2-1B (SAE features) and for Gemma-2-2B using both raw neurons and SAE features are shown in Figures 15 , 16 , and 17 in Appendix D.3 .
Figure 6: Average family-wise probing R2 scores across neuron subsets induced by word-order shuffling in Llama-3.2-1B (raw neurons). Neurons specific to original text, shuffled text, and their overlap exhibit comparable typological alignment, indicating that sensitivity to word order is largely decoupled from typological information. Baseline denotes probing over the pooled set of all neurons selected for either condition, serving as a non-selective reference. Corresponding results for Llama-3.2-1B (SAE features) and for Gemma-2-2B using both raw neurons and SAE features are shown in Figures 24 , 25 , and 26 in Appendix E.2 .
Figure 7: Average probing R2 scores across layers for SAE features in Llama-3.2-1B, grouped by typological family. Genealogical properties are accessible from early layers, while more abstract features such as phonology emerge mainly in deeper layers. Corresponding results for raw neurons and Gemma-2-2B show the same hierarchy (Figures 27 , 29 ).
Language
Neuron set
PPLratiotarget
PPLratiorandom
English
overlap
0.95
0.99
English
only-native
1.50
0.96
Hindi
overlap
1.05
0.98
Hindi
only-native
0.31
0.97
Table 1: Cross-language mean ablations for romanization-derived neuron sets in Llama-3.2-1B (see Table 8 for Gemma-2-2B). PPLratiotarget denotes perplexity relative to clean runs, and PPLratiorandom reports the same for matched random controls. All target effects are statistically significant ( p<0.05 ; Table 6 ). Ratios below 1 reflect language switching rather than improved modeling.
Language
Neuron set
PPLratiotarget
PPLratiorandom
English
overlap
1.12
0.95
English
only-unshuffled
0.96
1.04
Hindi
overlap
2.79
1.06
Hindi
only-unshuffled
1.08
0.95
Table 2: Zero-ablation results for shuffling-derived neuron sets in Llama-3.2-1B (see Table 7 for Gemma-2-2B). PPLratiotarget reports perplexity relative to clean runs after ablating the specified neuron set, and PPLratiorandom reports the same for matched random controls. All overlap-neuron effects differ significantly from random controls ( p<0.05 ; Table 5 ).
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Activation indicator
Latent z>0
Aggregation level
Token + example
Minimum example rate
0.98
Minimum HFL rate
0.10
Top- k threshold ratio
0.80
Entropy for invalid features
∞
Appendix
Table 3: Hyperparameters used for SAE-LAPE identification of language-associated sparse latent features. Rates are computed per layer over the multilingual corpus.
Parameter
Value
Activation indicator
a>0
Aggregation level
Token-level
Activation percentile (filter rate)
95th percentile
Entropy selection fraction
Lowest 1% neurons
Language assignment threshold
95th percentile (global)
Inactive neuron handling
Discarded
Appendix
Table 4: Hyperparameters used for LAPE-based identification of language-associated raw neurons. Activation percentiles are computed globally across all neurons and languages.
Figure 8: Jaccard similarity between language-associated units identified from Romanized inputs and those from Native-script or English inputs in Gemma-2-2B. Results are shown for both raw neurons and SAE features . Romanized inputs exhibit low overlap with their native-script counterparts and near-zero overlap with English in both representations, indicating limited cross-script alignment without convergence to English.
Figure 9: Aggregate degree-3 Venn diagrams of language-specific neurons under orthographic variation for Gemma-2-2B . Degree-3 denotes the union of neurons shared by up to three languages. Panels correspond to raw MLP and SAE representations under diacritics-preserving and diacritics-removed romanization.
Figure 10: Aggregate degree-3 Venn diagrams of language-specific neurons under orthographic variation for Llama-3.2-1B . Degree-3 denotes the union of neurons shared by up to three languages. Panels correspond to raw MLP and SAE representations under diacritics-preserving and diacritics-removed romanization.
Figure 11: Activation probability and entropy distributions for language-specific neurons under native vs. romanized inputs (Gemma-2-2B). Top: raw MLP; Bottom: SAE.
Figure 12: Activation probability and entropy distributions for language-specific neurons under native vs. romanized inputs (Llama-3.2-1B). Top: raw MLP; Bottom: SAE.
Figure 13: Mean activation statistics across languages for native and romanized inputs, for the raw MLP LAPE-identified features. Top: Gemma-2-2B; Bottom: Llama-3.2-1B.
Figure 14: Layer-wise alignment between language-associated units for Native and Romanized inputs in Gemma-2-2B. The red line denotes average Jaccard similarity for raw neurons , and the blue line for SAE features ; shaded regions indicate standard deviation across languages. Both raw neurons and SAE features show a mid-layer increase in overlap. However, in all cases, alignment remains far from convergence, indicating that representational separation persists beyond input tokenization.
Figure 15: Average family-wise maximum probing R2 scores across neuron subsets induced by romanization (Llama-3.2-1B, SAE). Overall probing scores are lower, but overlap neurons remain dominant.
Figure 16: Average family-wise maximum probing R2 scores across neuron subsets induced by romanization (Gemma-2-2B, raw MLP). Scores are closer across subsets, with native-only neurons occasionally falling below baseline.
Figure 17: Average family-wise maximum probing R2 scores across neuron subsets induced by romanization (Gemma-2-2B, SAE). Overlap neurons continue to show stronger typological alignment despite increased sparsity.
Figure 18: Aggregate degree-based Venn diagrams comparing features from original and shuffled inputs in Llama-3.2-1B. Top: raw MLP; Bottom: SAE. High overlap indicates stability of neuron identity under word-order perturbation.
Figure 19: Aggregate degree-based Venn diagrams comparing features from original and shuffled inputs in Gemma-2-2B. Top: raw MLP (degree 3); Bottom: SAE (degree 5). When sufficient neurons are identified, high overlap is preserved under shuffling.
Figure 20: Activation entropy and selection probability distributions for original and shuffled inputs in Llama-3.2-1B. Top: raw MLP; Bottom: SAE. The near-identical distributions indicate minimal distributional shift under shuffling.
Figure 21: Activation entropy and selection probability distributions for original and shuffled inputs in Gemma-2-2B. Top: raw MLP; Bottom: SAE. Distributional shifts remain small across representations.
Figure 22: Mean activation entropy and selection probability across languages before and after shuffling. Top: Llama-3.2-1B; Bottom: Gemma-2-2B (raw MLP). Mean-level changes are small, consistent with distribution-level stability.
Figure 23: Jaccard similarity between language-associated units identified from original and word-shuffled text in Gemma-2-2B. Raw neurons exhibit consistently moderate-to-high overlap across languages, indicating robustness to word-order perturbation. SAE features also show high overlap, revealing robustness of sparse features to local distributional patterns disrupted by shuffling.
Figure 24: Average family-wise maximum probing R2 scores across neuron subsets under shuffling (Llama-3.2-1B, SAE). Condition-specific subsets dominate overlap neurons; baseline scores remain lowest.
Figure 25: Average family-wise maximum probing R2 scores across neuron subsets under shuffling (Gemma-2-2B, raw MLP). Typological alignment is similar across normal-only, shuffled-only, and overlap subsets.
Figure 26: Average family-wise maximum probing R2 scores across neuron subsets under shuffling (Gemma-2-2B, SAE). Results use degree-5 aggregation due to low neuron counts. As in Llama, condition-specific subsets show stronger typological alignment than overlap neurons.
Figure 27: Layerwise probing performance in Llama-3.2-1B. Top: Raw MLP activations. Bottom: SAE features. SAE representations are comparatively stronger in early layers, while raw activations dominate in later layers.
Figure 28: Raw minus SAE probing score differences for Llama-3.2-1B. Negative values in shallow layers indicate higher SAE informativeness, while the gradual shift toward positive values reflects increasing raw dominance with depth.
Figure 29: Layerwise probing performance in Gemma-2-2B. Top: Raw MLP activations. Bottom: SAE features. Raw representations dominate for fam and syntax , while SAE features retain stronger phonological signals across layers.
Figure 30: Raw minus SAE probing score differences for Gemma-2-2B. Differences are stable across depth: positive for fam and syntax , and negative for phonology .
Figure 31: Cross-model comparison of probing performance. Top: Raw MLP activations. Bottom: SAE features. Raw representations show large early-layer differences, especially for phonology, followed by sharp convergence with depth, while SAE representations compress these disparities.
Lang
Category
PPLratiotarget
PPLratioctrl
p (ratio)
ΔPPLtarget
ΔPPLctrl
p ( Δ )
Llama-3.2-1B
en
overlap
1.116
0.954
1.5×10−50
+272.3
− 108.6
2.2×10−35
en
only-unshuffled
0.963
1.044
3.0×10−48
− 87.1
+99.6
1.9×10−34
hi
overlap
2.786
1.055
2.2×10−19
+1914.0
+204.9
1.6×10−12
hi
only-unshuffled
1.083
0.947
4.1×10−40
+228.6
− 133.3
4.6×10−10
fr
overlap
1.118
1.030
5.4×10−6
+114.6
+74.6
0.145
Appendix
Table 5: Causal zero-ablation results for shuffling-derived neuron sets across models (Llama models, raw neurons). Values report means over the first 100 FLORES+ examples per language. Control sets consist of matched random neurons with identical cardinality. Llama-3-8B shows qualitatively similar directional effects to Llama-3.2-1B, with stronger amplification in Hindi and Chinese, while maintaining robustness patterns under only-unshuffled conditions.
Figure 32: Representative Hindi generations under shuffling-based overlap-neuron ablation (Llama-3.2-1B, raw). Each row shows the input prefix, clean continuation, and ablated continuation for the same example. While clean generations preserve Devanagari script integrity, overlap-neuron ablations induce within-word mixed-script corruption, abrupt language switching, and topic drift. Such token-internal script mixing is not observed under matched random or Only-unshuffled neuron ablations.
Lang
Category
PPLratiotarget
PPLratioctrl
p (ratio)
ΔPPLtarget
ΔPPLctrl
p ( Δ )
Llama-3.2-1B
en
overlap
0.947
0.991
6.9×10−45
− 127.1
− 21.1
9.7×10−28
en
only-native
1.498
0.955
5.0×10−89
+1176.9
− 104.9
3.1×10−40
hi
overlap
1.047
0.982
9.5×10−34
+79.1
− 53.8
1.7×10−7
hi
only-native
0.312
0.970
1.2×10−38
− 1800.5
− 92.5
7.7×10−11
Llama-3-8B
Appendix
Table 6: Cross-language mean ablation results for romanization-derived neuron sets (raw neurons). Rows indicate forward-pass language; mean activations are taken from the opposite language. Results are averaged over the first 100 FLORES+ examples. Control sets consist of matched random neurons with identical cardinality. Llama-3-8B exhibits qualitatively similar patterns to Llama-3.2-1B: only-native Hindi neurons cause dramatic perplexity changes when ablated during Hindi inference ( PPLratiotarget=7.74 ), while overlap neurons produce modest effects across both models.
Lang
Category
PPLratiotarget
PPLratioctrl
p (ratio)
ΔPPLtarget
ΔPPLctrl
p ( Δ )
en
overlap
3.045
0.312
3.5×10−81
+588.9
− 191.2
2.5×10−39
en
only-unshuffled
0.799
0.312
2.8×10−130
− 56.5
− 191.2
1.1×10−47
hi
overlap
1.109
0.953
0.173
− 34.2
− 7.6
1.0×10−3
hi
only-unshuffled
0.397
2.557
8.1×10−50
− 102.7
+289.9
3.6×10−22
fr
overlap
1.547
0.952
1.2×10−89
+116.1
− 10.0
4.4×10−36
fr
only-unshuffled
1.260
1.403
3.1×10−40
+56.3
+90.4
4.4×10−24
Appendix
Table 7: Causal zero-ablation results for shuffling-derived neuron sets (Gemma-2-2B, raw). Values report means over the first 100 FLORES+ examples per language. Control sets consist of matched random neurons with identical cardinality.
Figure 33: Qualitative examples of model behavior under shuffling-based overlap ablation (Gemma-2-2B, raw). Ablation of overlap neurons induces systematic script mixing, including partial Latin insertions and mixed-script morphemes occurring within words. Such orthographic violations are not observed for random or only-normal neuron ablations.
Lang
Category
PPLratiotarget
PPLratioctrl
p (ratio)
ΔPPLtarget
ΔPPLctrl
p ( Δ )
en
only-native
5.208
1.405
2.2×10−65
+1222.6
+114.6
8.2×10−35
en
overlap
0.899
0.822
6.2×10−96
− 28.3
− 50.0
3.6×10−43
hi
only-native
0.684
1.136
1.2×10−54
− 55.9
+24.1
2.4×10−24
hi
overlap
2.228
1.036
5.8×10−45
+228.0
+6.5
7.1×10−21
Appendix
Table 8: Cross-language mean ablation results for romanization-derived neuron sets (Gemma-2-2B, raw). Rows indicate forward-pass language; mean activations are taken from the opposite language. Results are averaged over the first 100 FLORES+ examples.
Llama-3.2-1B
Llama-3-8B
Gemma-2-2B
Gemma-2-9B
Language
Nat
Rom
Nat
Rom
Nat
Rom
Nat
Rom
Bengali
0.40
0.08
0.67
0.23
0.60
0.11
0.72
0.33
Bulgarian
0.66
0.36
0.77
0.62
0.74
0.46
0.79
0.69
Chinese
0.60
0.12
0.69
0.29
0.68
0.12
0.72
0.33
Hindi
0.58
0.14
0.72
0.42
0.69
0.22
0.76
0.53
Japanese
0.54
0.08
0.67
0.14
0.65
0.11
0.71
0.19
Appendix
Table 9: Translation performance (BERTScore, roberta-large ) from Native (Nat) and Romanized (Rom) inputs to English (8-shot). Large BERTScores on romanized inputs confirm genuine semantic competence, proving that low neuron overlap is not a result of data sparsity.
Figure 34: Jaccard similarity between romanized and native-script or English units in Llama-3-8B. Representational isolation persists at the 8B scale.
Figure 35: Jaccard similarity between romanized and native-script or English units in Gemma-2-9B. Fragmentation remains a dominant feature despite increased model capacity.
Figure 36: Layer-wise alignment in Llama-3-8B. Mid-layer increases do not lead to cross-script convergence.
Figure 37: Layer-wise alignment in Gemma-2-9B, showing consistent representational separation in raw neurons across depth.
Figure 38: Jaccard similarity between units from original and shuffled text at scale. Robustness to word-order perturbation remains consistently high across larger architectures.
Multilingual language models transfer knowledge across languages through shared subword vocabulary, a mechanism that breaks down when related languages use different writing systems. Prior work addresses this via script equalization (romanization or IPA transcription), but direct comparisons are rare; the focus has been on encoder-only models, with most work adapting existing pretrained models. We systematically compare different input representations in autoregressive multilingual pretraining, comparing orthographic text, IPA, and romanization in a controlled setup across three scales (467M, 709M, and 1.03B) on eight languages in four typologically motivated pairs. Across a wide range of downstream tasks on seen and unseen languages, romanized pretraining yields the strongest cross-lingual transfer, and the advantage over text widens with scale. IPA improves over text in most settings but trails romanization. Surprisingly, finetuning a text-pretrained model on romanized data hurts performance on languages already covered by the base model, only marginally helping when the model lacks script coverage. Our results indicate that for multilingual models spanning typologically diverse scripts, to obtain maximum benefits, romanization should be treated as a core design choice applied at pretraining rather than a post hoc fix.
Muge Zhang, Aaron Jencks, Krishna Badikela +2
Department of Computer Science and Engineering, Ohio State University · Paul G. Allen School of Computer Science & Engineering, University of Washington
Previous work has argued that multilingual LLMs encode language-specific features most strongly in their initial and final layers, implying that intermediate layers support language-agnostic representations. This paper reconsiders this finding by performing four different experiments: three that undertake a fine-grained analysis of representations of Portuguese and Spanish, two closely related languages, and one large-scale language identification experiment that covers over 200 languages. The results of these experiments challenge the strict separation between language-specific and language-agnostic layers in multilingual LLMs, using Llama 3.2 1B as a test case. By analyzing and manipulating contextual sentence embeddings, we demonstrate that language identity remains strongly recoverable throughout the model rather than being restricted to initial and final layers. Moreover, we find that the model tends to classify ambiguous strings as closer to Spanish than to Portuguese, revealing a skew in model behavior that differs from human judgments.
Mikaela Martins, Jonathan Dunn
Department of Linguistics, University of Illinois Urbana-Champaign
Latent language identification is often used to argue that multilingual language models route computation through language-specific states, such as English pivots. However, existing probes infer latent language from different signals, such as the geometry of hidden states or what can be decoded from intermediate representations. Since such claims shape conclusions about how models share and route information across languages, we ask whether these probes measure the same phenomenon or expose distinct aspects of multilingual computation. We study this question across model families, training regimes, domains, tasks, checkpoints, and up to 27 languages. We find that identification probes systematically disagree: the GMM-based representation probe, which draws evidence from hidden state geometry, shows earlier cross-lingual mixing, whereas decoding-based probes, which rely on output-space decodability, retain sharper language-specific and more English-biased signals. These differences track model multilinguality and training progression, but are comparatively stable across domains. Our results suggest a more cautious interpretation of latent language identification, where current probes expose different aspects of multilingual processing, rather than directly revealing a single internal lingua franca.