cs.CLAug 28, 2026

Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Models

Authors: Sasha Boguraev, Toshiki Nakai, Kyle Mahowald, Julius Steuer

Organizations: The University of Texas at Austin · Leipzig University, ScaDS.AI Dresden/Leipzig · Heidelberg Institute for Theoretical Studies

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Linguistic theory has long recognized cross-linguistic syntactic regularities, leading to claims that these similar structures are processed by similar mechanisms. However, this hypothesis has been difficult to test empirically due to our lack of fine-grained, manipulable access of human processing mechanisms. In this work, we take advantage of techniques from mechanistic interpretability to study such a question in multilingual LMs. We first isolate language-internal mechanisms before attempting to transfer them cross-lingually. Across four models and three well-studied constructions (subject--verb number agreement, anaphoric pronoun gender agreement, and filler--gap object extraction) we find consistent cross-lingual mechanism transfer. We further find transfer to be graded, with more transfer between more typologically similar languages. We believe our work provides novel hypotheses about cross-linguistic syntactic structures and multilingual processing, and more broadly shows how the study of language models can help inform linguistic theory.

Figures & tables

Figure 1: We localize LM’s syntactic mechanisms to language-internal subspaces. We then test such mechanism’s effects on other languages. If the intervention is still efficacious, this indicates the model’s syntactic mechanisms are language-agnostic. This provides evidence of cross-lingual shared mechanisms .
Subject–Verb Number AgreementNP 1PPNP 2VP
English The man / The mennearthe cabinetis / are
Italian L’uomo / Gli uominivicino allatavolaè / sono
Bulgarian Мъжът / Мъжетедомасатае / са
Anaphoric Pronoun Gender AgreementNameVPbecausePron.
English James / Maryliedbecausehe / she
Italian Mario / Mariariseperchélui / lei
Table 1: Exemplar minimal pairs for a representative subset of languages with a full set of exemplars are in § A . Our gender agreement stimuli are violable: “Mary was powerful. James lied because she told him to” , although we do not find this to affect our LMs behaviorally. Instead, we find our studied LMs to score at or above 90% accuracy on 90% (115/128) of templates and below 80% accuracy only once.
Figure 2: We calculate the Odds when evaluating our trained interventions on a held-out test set of the same language they are trained on (red line), all other languages (green line), all other languages within the same family (yellow line), and across two controls (grey and black lines). We find that across all positions, phenomena, and models, interventions consistently generalize across languages with pairwise t-tests on the Max Odds showing significant differences between both the cross-language and control conditions ( p<0.05p<0.05 ).
Figure 3: We visualize our languages on the top principal components of the transfer matrix. We find language family-based clusters not to be pure.
 Number AgreementGender AgreementFiller–Gap (Object)
TermNP 1 (.68)PP (.71)NP 2 (.85)Name (.69)VP (.75)because (.76)Filler (.73)NP (.61)Verb (.78)
Typological dist.−0.577∗∗∗\mathbf{-0.577}^{***}−0.558∗∗∗\mathbf{-0.558}^{***}−0.432∗∗∗\mathbf{-0.432}^{***}−0.462∗∗∗\mathbf{-0.462}^{***}−0.320∗∗∗\mathbf{-0.320}^{***}−0.498∗∗∗\mathbf{-0.498}^{***}−0.544∗∗∗\mathbf{-0.544}^{***}−0.781∗∗∗\mathbf{-0.781}^{***}−1.04∗∗∗\mathbf{-1.04}^{***}
Model size0.3630.3630.2680.2680.4810.4810.4000.4000.3790.3790.5500.550−0.013-0.013−0.151-0.1510.2240.224
Typological dist. ×\times model size−0.043-0.043−0.100∗∗∗\mathbf{-0.100}^{***}−0.078∗∗\mathbf{-0.078}^{**}−0.055-0.055−0.158∗∗∗\mathbf{-0.158}^{***}−0.063-0.0630.0150.0150.0100.010−0.103-0.103
Table 2: LMEM coefficients predicting the Max Odds from linguistic and architectural factors. Estimates significant under the LRT are bolded . ∗p<.05{}^{*}p<.05 ; ∗∗p<.01{}^{**}p<.01 ; ∗∗∗p<.001{}^{***}p<.001 . Rc2R^{2}_{c} values in parentheses next to each position. We find significant negative effects of typological distance, suggesting that transfer is stronger between typologically similar languages. Interaction effects between typological distance and model size further suggest such effects grow stronger in larger models for the tasks of Number Agreement and Gender Agreement.
Subject Verb Number Agreement
 NP 1PPNP 2
mGPT-0.0800.201-0.158
Llama-3.2-1b0.0650.029-0.093
Llama-3.2-3b-0.1790.0970.120
Tiny-Aya0.2360.039-0.229
Anaphoric Pronoun Gender Agreement
Table 3: Silhouette scores measuring language-family cluster purity in PCA space. Each cell is the mean silhouette width for the given model and position. Scores range from −1-1 (languages closer to other families) to +1+1 (tight within-family clusters).
Figure 4: We measure the correlation between different operationalizations of typological similarity, pooled across positions and phenomenon. All pairwise correlations are significantly positive (all cells p<.01p<.01 ) and broadly strong, suggesting these factors are entangled in the model’s training data and learned representations. Correlations with template overlap tend to be weaker, reflecting the hand-designed nature of our templates.
 Number AgreementGender AgreementFiller–Gap (Object)
Training Data
rr (mC4, Out-Degree)0.180.630.36
rr (mC4, In-Degree)0.250.77-0.06
Typological ‘Hubs’
rr (In, Out)0.700.890.44
rr (Centr., Hub Score)0.590.850.75
Table 4: Pearson correlation, rr , between different data-metrics. ‘Hub’-ness is most correlated with typological centrality, not by training data magnitude.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5: Exemplar minimal pairs per language for each construction. The minimal differences upstream (denoted by the two options separated by a ‘/’) in the NP 1 , Name , and Filler positions each correspond to the given labels in the VP , Pronoun , and Gap positions respectively. Such minimal pairs are used to evaluate LM linguistic competence, and to ensure that our causal interventions are successful. ‘-’ indicates positions where the given construction has no item. This appears only in German and Dutch Filler–Gap stimuli as they are V2 languages, mandating any embedded verbs appear clause-final. The contrast at the VP position for Russian gender agreement ( ходил / ходила ) reflects the past-tense verb’s obligatory agreement with the subject, and is not an additional label.
Table 6: Sampled exemplar minimal pairs per language for each construction for random labels : the stimulus is identical to the true construction, but the counterfactual target are arbitrary tokens. We expect a well-calibrated intervention to be specific and not generalize to these.
Table 7: Sampled exemplar minimal pairs per language for each construction in the OOD labels control. As inspired by Kumon and Yanaka (2026) : the stimulus is identical to the true construction, but the counterfactual target is a plausible yet out-of-distribution continuation. We expect a well-trained intervention to target the abstract feature of interest, and thus generalize to these.
Figure 5: Behavioral performance of each model on each of our templatic stimuli. We broadly see strong performance from all of our models on all templates. Generally multilingual LMs perform the best, but both Llama models show strong performance as well. All bars but thirteen are above 90% performance, with twelve of those above 80%. The only bar below 80% is Llama‑3.2‑1b on Bulgarian number agreement.
 Filler–Gap (Object)Subject Verb Number AgreementAnaphoric Pronoun Gender Agreement
ModelFillerNPVerbNP 1PPNP 2NAMEVPbecause
Other phenomena( Control: specificity)
Llama-3.2-1b1.2×10−9∗∗∗\mathbf{1.2\times 10^{-9}}^{***}5.2×10−7∗∗∗\mathbf{5.2\times 10^{-7}}^{***}1.5×10−7∗∗∗\mathbf{1.5\times 10^{-7}}^{***}3.0×10−40∗∗∗\mathbf{3.0\times 10^{-40}}^{***}2.2×10−28∗∗∗\mathbf{2.2\times 10^{-28}}^{***}1.9×10−34∗∗∗\mathbf{1.9\times 10^{-34}}^{***}1.3×10−32∗∗∗\mathbf{1.3\times 10^{-32}}^{***}5.6×10−30∗∗∗\mathbf{5.6\times 10^{-30}}^{***}2.7×10−22∗∗∗\mathbf{2.7\times 10^{-22}}^{***}
Llama-3.2-3b1.9×10−7∗∗∗\mathbf{1.9\times 10^{-7}}^{***}4.7×10−6∗∗∗\mathbf{4.7\times 10^{-6}}^{***}5.9×10−9∗∗∗\mathbf{5.9\times 10^{-9}}^{***}1.2×10−41∗∗∗\mathbf{1.2\times 10^{-41}}^{***}1.7×10−35∗∗∗\mathbf{1.7\times 10^{-35}}^{***}1.4×10−40∗∗∗\mathbf{1.4\times 10^{-40}}^{***}1.3×10−31∗∗∗\mathbf{1.3\times 10^{-31}}^{***}2.4×10−35∗∗∗\mathbf{2.4\times 10^{-35}}^{***}1.2×10−33∗∗∗\mathbf{1.2\times 10^{-33}}^{***}
mGPT1.0×10−5∗∗∗\mathbf{1.0\times 10^{-5}}^{***}6.1×10−6∗∗∗\mathbf{6.1\times 10^{-6}}^{***}3.9×10−7∗∗∗\mathbf{3.9\times 10^{-7}}^{***}2.8×10−35∗∗∗\mathbf{2.8\times 10^{-35}}^{***}8.3×10−36∗∗∗\mathbf{8.3\times 10^{-36}}^{***}2.4×10−33∗∗∗\mathbf{2.4\times 10^{-33}}^{***}3.1×10−22∗∗∗\mathbf{3.1\times 10^{-22}}^{***}8.3×10−25∗∗∗\mathbf{8.3\times 10^{-25}}^{***}1.9×10−32∗∗∗\mathbf{1.9\times 10^{-32}}^{***}
tiny-aya-base5.9×10−9∗∗∗\mathbf{5.9\times 10^{-9}}^{***}1.2×10−9∗∗∗\mathbf{1.2\times 10^{-9}}^{***}6.9×10−9∗∗∗\mathbf{6.9\times 10^{-9}}^{***}8.4×10−58∗∗∗\mathbf{8.4\times 10^{-58}}^{***}1.3×10−39∗∗∗\mathbf{1.3\times 10^{-39}}^{***}3.1×10−52∗∗∗\mathbf{3.1\times 10^{-52}}^{***}1.0×10−44∗∗∗\mathbf{1.0\times 10^{-44}}^{***}1.6×10−40∗∗∗\mathbf{1.6\times 10^{-40}}^{***}6.6×10−49∗∗∗\mathbf{6.6\times 10^{-49}}^{***}
Appendix
Table 8: Holm-bonferroni corrected pp -values (correction applied within each comparison type, pooled across phenomena/models/positions) for all pairwise tt -tests comparing cross-lingual transfer ( Cross Language (all) ) against our two selectivity controls and against family-restricted transfer, for Experiment One ( Section 4 ; Figure 6 ). Random Labels and Other phenomena are our controls: cross-lingual transfer should, and does, significantly exceed both. Cross Language (Within-Family) instead tests whether transfer is stronger among typologically related languages than to the full evaluated set; results here are mixed. †p<.1{}^{\dagger}p<.1 ; ∗p<.05{}^{*}p<.05 ; ∗∗p<.01{}^{**}p<.01 ; ∗∗∗p<.001{}^{***}p<.001 ; significant cells bolded.
 Filler–Gap (Object)Subject Verb Number Agreement
ModelFillerNPVerbNP 1PPNP 2
Other phenomena( Control: specificity)
Llama-3.2-1b3.3×10−7∗∗∗\mathbf{3.3\times 10^{-7}}^{***}2.5×10−6∗∗∗\mathbf{2.5\times 10^{-6}}^{***}1.2×10−6∗∗∗\mathbf{1.2\times 10^{-6}}^{***}5.2×10−39∗∗∗\mathbf{5.2\times 10^{-39}}^{***}7.2×10−32∗∗∗\mathbf{7.2\times 10^{-32}}^{***}3.3×10−40∗∗∗\mathbf{3.3\times 10^{-40}}^{***}
Llama-3.2-3b4.4×10−6∗∗∗\mathbf{4.4\times 10^{-6}}^{***}3.2×10−6∗∗∗\mathbf{3.2\times 10^{-6}}^{***}6.5×10−7∗∗∗\mathbf{6.5\times 10^{-7}}^{***}3.0×10−42∗∗∗\mathbf{3.0\times 10^{-42}}^{***}2.4×10−40∗∗∗\mathbf{2.4\times 10^{-40}}^{***}1.1×10−43∗∗∗\mathbf{1.1\times 10^{-43}}^{***}
mGPT0.042∗\mathbf{0.042^{*}}8.9×10−6∗∗∗\mathbf{8.9\times 10^{-6}}^{***}4.3×10−6∗∗∗\mathbf{4.3\times 10^{-6}}^{***}5.2×10−30∗∗∗\mathbf{5.2\times 10^{-30}}^{***}5.3×10−44∗∗∗\mathbf{5.3\times 10^{-44}}^{***}4.3×10−55∗∗∗\mathbf{4.3\times 10^{-55}}^{***}
tiny-aya-base8.9×10−6∗∗∗\mathbf{8.9\times 10^{-6}}^{***}1.2×10−8∗∗∗\mathbf{1.2\times 10^{-8}}^{***}1.6×10−9∗∗∗\mathbf{1.6\times 10^{-9}}^{***}1.1×10−51∗∗∗\mathbf{1.1\times 10^{-51}}^{***}5.7×10−45∗∗∗\mathbf{5.7\times 10^{-45}}^{***}2.4×10−59∗∗∗\mathbf{2.4\times 10^{-59}}^{***}
Appendix
Table 9: Holm-bonferroni corrected pp -values (correction applied within each comparison type, pooled across phenomena/models/positions) for the OOD labels ( § E ; Figure 7 ). As in Table 8 , Random Labels and Other phenomena are selectivity controls compared against Cross Language (all) transfer, and Cross Language (Within-Family) tests family-restricted vs. all-language transfer. †p<.1{}^{\dagger}p<.1 ; ∗p<.05{}^{*}p<.05 ; ∗∗p<.01{}^{**}p<.01 ; ∗∗∗p<.001{}^{***}p<.001 ; significant cells bolded.
Figure 6: We take the Max Odds across layers for each model at each position, before normalizing by the within-language value (red bar). We see strong performance in both our cross-language (green bar) and within-family (yellow bar) transfer conditions, with both being significantly above both controls in all cases when tested with t-tests with a Holm-Bonferroni correction. We further find the within-family transfer to be significantly higher than the cross-language bar in a select few comparisons, but for this to not hold more broadly.
Figure 7: We evaluate our interventions on syntactically appropriate but lexically disjoint labels from our training set. We find our results to broadly hold in this setting.
MetricEnglish (en)German (de)Dutch (nl)French (fr)Italian (it)Bulgarian (bg)Russian (ru)Spanish (es)Portuguese (pt)Romanian (ro)Galician (gl)Farsi (fa)Chinese (zh)Japanese (ja)Korean (ko)
Template Overlap✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Corpus Overlap✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Feature Overlap (Grambank)✓✗✓✓✓✗✓✗✓✗✓✓✓✓✓
Cognate Proportion✓✓✓✓✓✓✓✓✓✓✗✓✗✗✗
Typological Dist.✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Appendix
Table 10: Language coverage for each distance/overlap metric used in the typological-distance correlation analysis. Feature Overlap (Grambank) is missing for German, Bulgarian, Spanish, and Romanian because Grambank lacks entries for them. The cognate data we utilize, Cognate Proportion (IECOR) is limited to Indo-European languages, so it is unavailable for Galician, Chinese, Japanese, and Korean.
 TrainingEvaluation
Model / PhenomenonRuns# InterventionsSeconds per InterventionTime (h)Pairs# InterventionsSeconds per InterventionTime (h)
mGPT (24 layers)
number agreement12728.22.0579721.9422.4
gender agreement13728.22.1534722.8330.3
filler–gap7728.11.1294725.1430.2
total32––5.31407––82.9
Appendix
Table 11: Per-model compute cost on a single NVIDIA A40 GPU. A run is one language, with interventions at every position and layer. As there are three positions evaluated for each phenomenon, this just totals three times the number of layers in each model. Times reported reflect the median value of all runs performed during our experiments.
More in this topic

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Explore similar work

CardsList
Apr 24, 2026

Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Models

While language models demonstrate sophisticated syntactic capabilities, the extent to which their internal mechanisms align with cross-constructional principles studied in linguistics remains poorly understood. This study investigates whether models employ shared neural mechanisms across different syntactic constructions by applying causal interpretability methods at a granular level. Focusing on filler-gap dependencies and negative polarity item (NPI) licensing, we utilize activation patching to identify the functional roles of specific attention heads and MLP blocks. Our results reveal a highly localized and shared mechanism for filler-gap dependencies located in the early to middle layers, whereas NPI processing exhibits no such unified mechanism. Furthermore, we find that these mechanisms identified by activation patching generalize to out-of-distribution, while distributed alignment search, a supervised interpretability method, is susceptible to overfitting on narrow linguistic distributions. Finally, we validate our findings by demonstrating that the manipulation of the identified components improves model performance on acceptability judgment benchmarks.

Ryoma Kumon, Hitomi Yanaka

1The University of Tokyo · 2RIKEN · 3Tohoku University

Mechanistic InterpretabilityLanguage ModelingComputational Psycholinguistics
AskPDF
Aug 19, 2026

Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages

Multilingual large language models often generalize across languages, and prior work suggests that their internal mechanisms can overlap cross-lingually. It remains unclear, however, when such sharing emerges and whether it varies with the overt realization of the same grammatical operation. We investigate this question for present-tense subject-verb agreement, a morphosyntactic process that varies substantially across languages and is only weakly expressed in English. Using activation patching and attention analysis across 29 languages and five open-source model families, we identify the attention heads causally implicated in agreement and compare these head-level signatures across languages. We find that languages with overt person/number inflection exhibit more similar agreement circuitry than non-conjugating languages, with the strongest sharing appearing when the analysis isolates recovery of the inflectional contrast itself. English provides an informative bridge case, becoming more similar to conjugating languages precisely in contexts where overt agreement is required. Finally, many implicated heads display similar attention patterns across languages, suggesting that cross-lingual overlap reflects shared functional roles as well as shared localization. Together, these results indicate that multilingual LLMs reuse partially shared computational structure for morphosyntactic agreement rather than relying on fully separate language-specific solutions.

Isabella Gidi, Antonio Almudévar, Core Francisco Park +2

Harvard University · University of Zaragoza · Boston University +1

Multilingual Language ModelsCircuits in Language Models
AskPDF
May 26, 2026

An In-Vitro Study on Cross-Lingual Generalization in Language Models

Cross-lingual transfer in language models is difficult to study in natural corpora because lexical overlap, morphology, data imbalance, and tokenization are entangled. We introduce an in-vitro framework with two procedurally generated languages that share the same ontology, typed grammar, and compositional structure, but differ in surface realization. This lets us independently vary lexical distance, minority-language proportion, tokenizer training regime, and vocabulary size, while evaluating transfer on a masked minority-language condition whose lexical forms are never observed during training. Across 700 controlled runs, we find that transfer is governed less by tokenizer balance or raw lexical similarity than by whether tokenization preserves reusable cross-lingual substructure. Smaller vocabularies often improve masked transfer by keeping words decomposable into shared fragments, whereas larger vocabularies can turn forms into language-specific atoms. We further show that transfer emerges as a staged process: grammatical and type-level competence precede masked lexical generalization. Finally, we attempt to explain this mechanism through tokenizer bridges and show that bridge strength correlates strongly with masked reachability.

Adrian Cosma

Dalle Molle Institute for Artificial Intelligence (IDSIA)

Multilingual Language Model EvaluationCross-Lingual Transfer
AskPDF