Multilingual Language Models Encode Script Over Linguistic Structure

Aastha A K Verma; Anwoy Chatterjee; Mehak Gupta; Tanmoy Chakraborty

cs.CLApr 6, 2026

Multilingual Language Models Encode Script Over Linguistic Structure

Authors: Aastha A K Verma, Anwoy Chatterjee, Mehak Gupta, Tanmoy Chakraborty

Organizations: Indian Institute of Technology Delhi, New Delhi, India · Indian Institute of Technology Delhi, Abu Dhabi, UAE

Abstract

Multilingual language models (LMs) organize representations for typologically and orthographically diverse languages into a shared parameter space, yet the nature of this internal organization remains elusive. In this work, we investigate which linguistic properties - abstract language identity or surface-form cues - shape multilingual representations. To do so, we analyze language-associated units across different model families and scales using the Language Activation Probability Entropy (LAPE) metric, and further decompose activations with Sparse Autoencoders. We find that these units are strongly conditioned on orthography: romanization induces near-disjoint representations that align with neither native-script inputs nor English, while word-order shuffling has limited effect on unit identity. Probing shows that typological structure becomes increasingly accessible in deeper layers, while causal interventions indicate that generation is most sensitive to units that are invariant to surface-form perturbations rather than to units identified by typological alignment alone. Overall, our results suggest that multilingual LMs organize representations around surface form, with linguistic abstraction emerging gradually without collapsing into a unified interlingua.

Figures & tables

Panel of Figure 1 — Figure 1: Overlap of language-associated units for Hindi under script variation in Llama-3.2-1B. Euler diagrams show units shared among up to three languages for (a) raw neurons and (b) SAE features. Native, Romanized (with diacritics), and Romanized (without diacritics) inputs activate largely disjoint sets in both representations. Corresponding results for all languages, for both raw neurons and SAE features, and for Gemma-2-2B are shown in Figures 10 and 9 in Appendix D.2 .

Figure 2: Jaccard similarity between Romanized and native-script or English language-associated units ( raw neurons and SAE features ) in Llama-3.2-1B (see Figure 8 for Gemma-2-2B). Romanized inputs exhibit low overlap with their native-script counterparts and near-zero overlap with English in both representations, indicating limited cross-script alignment without convergence to English.

Figure 3: Layer-wise alignment between language-associated units for Native and Romanized inputs in Llama-3.2-1B (see Figure 14 for Gemma-2-2B). The red line denotes average Jaccard similarity for raw neurons , and the blue line for SAE features ; shaded regions indicate standard deviation across languages. Raw neurons show a modest mid-layer increase in overlap, while SAE features remain uniformly low across depth. In all cases, alignment remains far from convergence, indicating that representational separation persists beyond input tokenization.

Figure 4: Jaccard similarity between language-associated units identified from original and word-shuffled text in Llama-3.2-1B (see Figure 23 for Gemma-2-2B). Raw neurons exhibit consistently moderate-to-high overlap across languages, indicating robustness to word-order perturbation. In contrast, SAE features show only selective instability : languages with distinctive scripts (e.g., Chinese, Japanese, Thai) remain highly stable, whereas several Latin-script languages exhibit somewhat reduced overlap, revealing sensitivity of sparse features to local distributional patterns disrupted by shuffling for these languages.

Figure 5: Average family-wise probing

R^{2}

scores across neuron subsets induced by romanization in Llama-3.2-1B (raw neurons). Neurons overlapping between native and romanized inputs exhibit the strongest typological alignment, while script-specific subsets encode weaker signal. Baseline denotes probing over the pooled set of all neurons that were selected for either native or romanized inputs (across all layers), serving as a non-selective reference. Corresponding results for Llama-3.2-1B (SAE features) and for Gemma-2-2B using both raw neurons and SAE features are shown in Figures 15 , 16 , and 17 in Appendix D.3 .

Figure 6: Average family-wise probing

R^{2}

scores across neuron subsets induced by word-order shuffling in Llama-3.2-1B (raw neurons). Neurons specific to original text, shuffled text, and their overlap exhibit comparable typological alignment, indicating that sensitivity to word order is largely decoupled from typological information. Baseline denotes probing over the pooled set of all neurons selected for either condition, serving as a non-selective reference. Corresponding results for Llama-3.2-1B (SAE features) and for Gemma-2-2B using both raw neurons and SAE features are shown in Figures 24 , 25 , and 26 in Appendix E.2 .

Figure 7: Average probing

R^{2}

scores across layers for SAE features in Llama-3.2-1B, grouped by typological family. Genealogical properties are accessible from early layers, while more abstract features such as phonology emerge mainly in deeper layers. Corresponding results for raw neurons and Gemma-2-2B show the same hierarchy (Figures 27 , 29 ).

Language	Neuron set	$PPL_{\text{ratio}}^{\text{target}}$	$PPL_{\text{ratio}}^{\text{random}}$
English	overlap	0.95	0.99
English	only-native	1.50	0.96
Hindi	overlap	1.05	0.98
Hindi	only-native	0.31	0.97

Table 1: Cross-language mean ablations for romanization-derived neuron sets in Llama-3.2-1B (see Table 8 for Gemma-2-2B).

PPL_{\text{ratio}}^{\text{target}}

denotes perplexity relative to clean runs, and

PPL_{\text{ratio}}^{\text{random}}

reports the same for matched random controls. All target effects are statistically significant (

p<0.05

; Table 6 ). Ratios below

1

reflect language switching rather than improved modeling.

Language	Neuron set	$PPL_{\text{ratio}}^{\text{target}}$	$PPL_{\text{ratio}}^{\text{random}}$
English	overlap	1.12	0.95
English	only-unshuffled	0.96	1.04
Hindi	overlap	2.79	1.06
Hindi	only-unshuffled	1.08	0.95

Table 2: Zero-ablation results for shuffling-derived neuron sets in Llama-3.2-1B (see Table 7 for Gemma-2-2B).

PPL_{\text{ratio}}^{\text{target}}

reports perplexity relative to clean runs after ablating the specified neuron set, and

PPL_{\text{ratio}}^{\text{random}}

reports the same for matched random controls. All overlap-neuron effects differ significantly from random controls (

p<0.05

; Table 5 ).

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Activation indicator	Latent $z>0$
Aggregation level	Token + example
Minimum example rate	$0.98$
Minimum HFL rate	$0.10$
Top- $k$ threshold ratio	$0.80$
Entropy for invalid features	$\infty$

Appendix

Table 3: Hyperparameters used for SAE-LAPE identification of language-associated sparse latent features. Rates are computed per layer over the multilingual corpus.

Parameter	Value
Activation indicator	$a>0$
Aggregation level	Token-level
Activation percentile (filter rate)	$95^{\text{th}}$ percentile
Entropy selection fraction	Lowest 1% neurons
Language assignment threshold	$95^{\text{th}}$ percentile (global)
Inactive neuron handling	Discarded

Appendix

Table 4: Hyperparameters used for LAPE-based identification of language-associated raw neurons. Activation percentiles are computed globally across all neurons and languages.

Figure 8: Jaccard similarity between language-associated units identified from Romanized inputs and those from Native-script or English inputs in Gemma-2-2B. Results are shown for both raw neurons and SAE features . Romanized inputs exhibit low overlap with their native-script counterparts and near-zero overlap with English in both representations, indicating limited cross-script alignment without convergence to English.

Figure 9: Aggregate degree-3 Venn diagrams of language-specific neurons under orthographic variation for Gemma-2-2B . Degree-3 denotes the union of neurons shared by up to three languages. Panels correspond to raw MLP and SAE representations under diacritics-preserving and diacritics-removed romanization.

Figure 10: Aggregate degree-3 Venn diagrams of language-specific neurons under orthographic variation for Llama-3.2-1B . Degree-3 denotes the union of neurons shared by up to three languages. Panels correspond to raw MLP and SAE representations under diacritics-preserving and diacritics-removed romanization.

Figure 11: Activation probability and entropy distributions for language-specific neurons under native vs. romanized inputs (Gemma-2-2B). Top: raw MLP; Bottom: SAE.

Figure 12: Activation probability and entropy distributions for language-specific neurons under native vs. romanized inputs (Llama-3.2-1B). Top: raw MLP; Bottom: SAE.

Figure 13: Mean activation statistics across languages for native and romanized inputs, for the raw MLP LAPE-identified features. Top: Gemma-2-2B; Bottom: Llama-3.2-1B.

Figure 14: Layer-wise alignment between language-associated units for Native and Romanized inputs in Gemma-2-2B. The red line denotes average Jaccard similarity for raw neurons , and the blue line for SAE features ; shaded regions indicate standard deviation across languages. Both raw neurons and SAE features show a mid-layer increase in overlap. However, in all cases, alignment remains far from convergence, indicating that representational separation persists beyond input tokenization.

Figure 15: Average family-wise maximum probing

R^{2}

scores across neuron subsets induced by romanization (Llama-3.2-1B, SAE). Overall probing scores are lower, but overlap neurons remain dominant.

Figure 16: Average family-wise maximum probing

R^{2}

scores across neuron subsets induced by romanization (Gemma-2-2B, raw MLP). Scores are closer across subsets, with native-only neurons occasionally falling below baseline.

Figure 17: Average family-wise maximum probing

R^{2}

scores across neuron subsets induced by romanization (Gemma-2-2B, SAE). Overlap neurons continue to show stronger typological alignment despite increased sparsity.

Figure 18: Aggregate degree-based Venn diagrams comparing features from original and shuffled inputs in Llama-3.2-1B. Top: raw MLP; Bottom: SAE. High overlap indicates stability of neuron identity under word-order perturbation.

Figure 19: Aggregate degree-based Venn diagrams comparing features from original and shuffled inputs in Gemma-2-2B. Top: raw MLP (degree 3); Bottom: SAE (degree 5). When sufficient neurons are identified, high overlap is preserved under shuffling.

Figure 20: Activation entropy and selection probability distributions for original and shuffled inputs in Llama-3.2-1B. Top: raw MLP; Bottom: SAE. The near-identical distributions indicate minimal distributional shift under shuffling.

Figure 21: Activation entropy and selection probability distributions for original and shuffled inputs in Gemma-2-2B. Top: raw MLP; Bottom: SAE. Distributional shifts remain small across representations.

Figure 22: Mean activation entropy and selection probability across languages before and after shuffling. Top: Llama-3.2-1B; Bottom: Gemma-2-2B (raw MLP). Mean-level changes are small, consistent with distribution-level stability.

Figure 23: Jaccard similarity between language-associated units identified from original and word-shuffled text in Gemma-2-2B. Raw neurons exhibit consistently moderate-to-high overlap across languages, indicating robustness to word-order perturbation. SAE features also show high overlap, revealing robustness of sparse features to local distributional patterns disrupted by shuffling.

Figure 24: Average family-wise maximum probing

R^{2}

scores across neuron subsets under shuffling (Llama-3.2-1B, SAE). Condition-specific subsets dominate overlap neurons; baseline scores remain lowest.

Figure 25: Average family-wise maximum probing

R^{2}

scores across neuron subsets under shuffling (Gemma-2-2B, raw MLP). Typological alignment is similar across normal-only, shuffled-only, and overlap subsets.

Figure 26: Average family-wise maximum probing

R^{2}

scores across neuron subsets under shuffling (Gemma-2-2B, SAE). Results use degree-5 aggregation due to low neuron counts. As in Llama, condition-specific subsets show stronger typological alignment than overlap neurons.

Figure 27: Layerwise probing performance in Llama-3.2-1B. Top: Raw MLP activations. Bottom: SAE features. SAE representations are comparatively stronger in early layers, while raw activations dominate in later layers.

Figure 28: Raw minus SAE probing score differences for Llama-3.2-1B. Negative values in shallow layers indicate higher SAE informativeness, while the gradual shift toward positive values reflects increasing raw dominance with depth.

Figure 29: Layerwise probing performance in Gemma-2-2B. Top: Raw MLP activations. Bottom: SAE features. Raw representations dominate for fam and syntax , while SAE features retain stronger phonological signals across layers.

Figure 30: Raw minus SAE probing score differences for Gemma-2-2B. Differences are stable across depth: positive for fam and syntax , and negative for phonology .

Figure 31: Cross-model comparison of probing performance. Top: Raw MLP activations. Bottom: SAE features. Raw representations show large early-layer differences, especially for phonology, followed by sharp convergence with depth, while SAE representations compress these disparities.

Lang	Category	$PPL_{ratio}^{\text{target}}$	$PPL_{ratio}^{\text{ctrl}}$	$p$ (ratio)	$\Delta PPL^{\text{target}}$	$\Delta PPL^{\text{ctrl}}$	$p$ ( $\Delta$ )
Llama-3.2-1B
en	overlap	1.116	0.954	$1.5{\times}10^{-50}$	+272.3	$-$ 108.6	$2.2{\times}10^{-35}$
en	only-unshuffled	0.963	1.044	$3.0{\times}10^{-48}$	$-$ 87.1	+99.6	$1.9{\times}10^{-34}$
hi	overlap	2.786	1.055	$2.2{\times}10^{-19}$	+1914.0	+204.9	$1.6{\times}10^{-12}$
hi	only-unshuffled	1.083	0.947	$4.1{\times}10^{-40}$	+228.6	$-$ 133.3	$4.6{\times}10^{-10}$
fr	overlap	1.118	1.030	$5.4{\times}10^{-6}$	+114.6	+74.6	0.145

Appendix

Table 5: Causal zero-ablation results for shuffling-derived neuron sets across models (Llama models, raw neurons). Values report means over the first 100 FLORES+ examples per language. Control sets consist of matched random neurons with identical cardinality. Llama-3-8B shows qualitatively similar directional effects to Llama-3.2-1B, with stronger amplification in Hindi and Chinese, while maintaining robustness patterns under only-unshuffled conditions.

Figure 32: Representative Hindi generations under shuffling-based overlap-neuron ablation (Llama-3.2-1B, raw). Each row shows the input prefix, clean continuation, and ablated continuation for the same example. While clean generations preserve Devanagari script integrity, overlap-neuron ablations induce within-word mixed-script corruption, abrupt language switching, and topic drift. Such token-internal script mixing is not observed under matched random or Only-unshuffled neuron ablations.

Lang	Category	$PPL_{ratio}^{\text{target}}$	$PPL_{ratio}^{\text{ctrl}}$	$p$ (ratio)	$\Delta PPL^{\text{target}}$	$\Delta PPL^{\text{ctrl}}$	$p$ ( $\Delta$ )
Llama-3.2-1B
en	overlap	0.947	0.991	$6.9{\times}10^{-45}$	$-$ 127.1	$-$ 21.1	$9.7{\times}10^{-28}$
en	only-native	1.498	0.955	$5.0{\times}10^{-89}$	+1176.9	$-$ 104.9	$3.1{\times}10^{-40}$
hi	overlap	1.047	0.982	$9.5{\times}10^{-34}$	+79.1	$-$ 53.8	$1.7{\times}10^{-7}$
hi	only-native	0.312	0.970	$1.2{\times}10^{-38}$	$-$ 1800.5	$-$ 92.5	$7.7{\times}10^{-11}$
Llama-3-8B

Appendix

Table 6: Cross-language mean ablation results for romanization-derived neuron sets (raw neurons). Rows indicate forward-pass language; mean activations are taken from the opposite language. Results are averaged over the first 100 FLORES+ examples. Control sets consist of matched random neurons with identical cardinality. Llama-3-8B exhibits qualitatively similar patterns to Llama-3.2-1B: only-native Hindi neurons cause dramatic perplexity changes when ablated during Hindi inference (

PPL_{ratio}^{\text{target}}=7.74

), while overlap neurons produce modest effects across both models.

Lang	Category	$PPL_{ratio}^{\text{target}}$	$PPL_{ratio}^{\text{ctrl}}$	$p$ (ratio)	$\Delta PPL^{\text{target}}$	$\Delta PPL^{\text{ctrl}}$	$p$ ( $\Delta$ )
en	overlap	3.045	0.312	$3.5{\times}10^{-81}$	+588.9	$-$ 191.2	$2.5{\times}10^{-39}$
en	only-unshuffled	0.799	0.312	$2.8{\times}10^{-130}$	$-$ 56.5	$-$ 191.2	$1.1{\times}10^{-47}$
hi	overlap	1.109	0.953	0.173	$-$ 34.2	$-$ 7.6	$1.0{\times}10^{-3}$
hi	only-unshuffled	0.397	2.557	$8.1{\times}10^{-50}$	$-$ 102.7	+289.9	$3.6{\times}10^{-22}$
fr	overlap	1.547	0.952	$1.2{\times}10^{-89}$	+116.1	$-$ 10.0	$4.4{\times}10^{-36}$
fr	only-unshuffled	1.260	1.403	$3.1{\times}10^{-40}$	+56.3	+90.4	$4.4{\times}10^{-24}$

Appendix

Table 7: Causal zero-ablation results for shuffling-derived neuron sets (Gemma-2-2B, raw). Values report means over the first 100 FLORES+ examples per language. Control sets consist of matched random neurons with identical cardinality.

Figure 33: Qualitative examples of model behavior under shuffling-based overlap ablation (Gemma-2-2B, raw). Ablation of overlap neurons induces systematic script mixing, including partial Latin insertions and mixed-script morphemes occurring within words. Such orthographic violations are not observed for random or only-normal neuron ablations.

Lang	Category	$PPL_{ratio}^{\text{target}}$	$PPL_{ratio}^{\text{ctrl}}$	$p$ (ratio)	$\Delta PPL^{\text{target}}$	$\Delta PPL^{\text{ctrl}}$	$p$ ( $\Delta$ )
en	only-native	5.208	1.405	$2.2{\times}10^{-65}$	+1222.6	+114.6	$8.2{\times}10^{-35}$
en	overlap	0.899	0.822	$6.2{\times}10^{-96}$	$-$ 28.3	$-$ 50.0	$3.6{\times}10^{-43}$
hi	only-native	0.684	1.136	$1.2{\times}10^{-54}$	$-$ 55.9	+24.1	$2.4{\times}10^{-24}$
hi	overlap	2.228	1.036	$5.8{\times}10^{-45}$	+228.0	+6.5	$7.1{\times}10^{-21}$

Appendix

Table 8: Cross-language mean ablation results for romanization-derived neuron sets (Gemma-2-2B, raw). Rows indicate forward-pass language; mean activations are taken from the opposite language. Results are averaged over the first 100 FLORES+ examples.

	Llama-3.2-1B		Llama-3-8B		Gemma-2-2B		Gemma-2-9B
Language	Nat	Rom	Nat	Rom	Nat	Rom	Nat	Rom
Bengali	0.40	0.08	0.67	0.23	0.60	0.11	0.72	0.33
Bulgarian	0.66	0.36	0.77	0.62	0.74	0.46	0.79	0.69
Chinese	0.60	0.12	0.69	0.29	0.68	0.12	0.72	0.33
Hindi	0.58	0.14	0.72	0.42	0.69	0.22	0.76	0.53
Japanese	0.54	0.08	0.67	0.14	0.65	0.11	0.71	0.19

Appendix

Table 9: Translation performance (BERTScore, roberta-large ) from Native (Nat) and Romanized (Rom) inputs to English (8-shot). Large BERTScores on romanized inputs confirm genuine semantic competence, proving that low neuron overlap is not a result of data sparsity.

Figure 34: Jaccard similarity between romanized and native-script or English units in Llama-3-8B. Representational isolation persists at the 8B scale.

Figure 35: Jaccard similarity between romanized and native-script or English units in Gemma-2-9B. Fragmentation remains a dominant feature despite increased model capacity.

Figure 36: Layer-wise alignment in Llama-3-8B. Mid-layer increases do not lead to cross-script convergence.

Figure 37: Layer-wise alignment in Gemma-2-9B, showing consistent representational separation in raw neurons across depth.

Panel of Figure 38 — Figure 38: Jaccard similarity between units from original and shuffled text at scale. Robustness to word-order perturbation remains consistently high across larger architectures.

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography
Aug 26, 2026Muge Zhang, Aaron Jencks, Krishna Badikela +2Language Model Pretraining Multilingual Language Models
Department of Computer Science and Engineering, Ohio State University · Paul G. Allen School of Computer Science & Engineering, University of Washington
Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere
Oct 18, 2026Mikaela Martins, Jonathan DunnMultilingual Language Models Concept Localization in Language Models
Department of Linguistics, University of Illinois Urbana-Champaign
Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs
Aug 31, 2026Deniz Bayazit, Badr AlKhamissi, Antoine BosselutMultilingual Language Models Multilingual Language Model Evaluation
EPFL

Aug 26, 2026J/K move · Enter open · S save

One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography

Muge Zhang, Aaron Jencks, Krishna Badikela, Yulia Tsvetkov+1

Department of Computer Science and Engineering, Ohio State University · Paul G. Allen School of Computer Science & Engineering, University of Washington

Language Model Pretraining Multilingual Language Models Autoregressive Language Modeling Cross-Lingual Transfer

Open paper AskPDF arXiv

Multilingual language models transfer knowledge across languages through shared subword vocabulary, a mechanism that breaks down when related languages use different writing systems. Prior work addresses this via script equalization (romanization or IPA transcription), but direct comparisons are rare; the focus has been on encoder-only models, with most work adapting existing pretrained models. We systematically compare different input representations in autoregressive multilingual pretraining, comparing orthographic text, IPA, and romanization in a controlled setup across three scales (467M, 709M, and 1.03B) on eight languages in four typologically motivated pairs. Across a wide range of downstream tasks on seen and unseen languages, romanized pretraining yields the strongest cross-lingual transfer, and the advantage over text widens with scale. IPA improves over text in most settings but trails romanization. Surprisingly, finetuning a text-pretrained model on romanized data hurts performance on languages already covered by the base model, only marginally helping when the model lacks script coverage. Our results indicate that for multilingual models spanning typologically diverse scripts, to obtain maximum benefits, romanization should be treated as a core design choice applied at pretraining rather than a post hoc fix.

Abstract

Multilingual Language Models

Explore similar work

One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography

Where Are Language-Specific Features Represented in Multilingual LLMs? Everywhere

Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs