cs.CLAug 28, 2026

Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Models

Authors: Sasha Boguraev, Toshiki Nakai, Kyle Mahowald, Julius Steuer

Organizations: The University of Texas at Austin · Leipzig University, ScaDS.AI Dresden/Leipzig · Heidelberg Institute for Theoretical Studies

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Linguistic theory has long recognized cross-linguistic syntactic regularities, leading to claims that these similar structures are processed by similar mechanisms. However, this hypothesis has been difficult to test empirically due to our lack of fine-grained, manipulable access of human processing mechanisms. In this work, we take advantage of techniques from mechanistic interpretability to study such a question in multilingual LMs. We first isolate language-internal mechanisms before attempting to transfer them cross-lingually. Across four models and three well-studied constructions (subject--verb number agreement, anaphoric pronoun gender agreement, and filler--gap object extraction) we find consistent cross-lingual mechanism transfer. We further find transfer to be graded, with more transfer between more typologically similar languages. We believe our work provides novel hypotheses about cross-linguistic syntactic structures and multilingual processing, and more broadly shows how the study of language models can help inform linguistic theory.

Figures & tables

Figure 1: We localize LM’s syntactic mechanisms to language-internal subspaces. We then test such mechanism’s effects on other languages. If the intervention is still efficacious, this indicates the model’s syntactic mechanisms are language-agnostic. This provides evidence of cross-lingual shared mechanisms .
Subject–Verb Number AgreementNP 1PPNP 2VP
English The man / The mennearthe cabinetis / are
Italian L’uomo / Gli uominivicino allatavolaè / sono
Bulgarian Мъжът / Мъжетедомасатае / са
Anaphoric Pronoun Gender AgreementNameVPbecausePron.
English James / Maryliedbecausehe / she
Italian Mario / Mariariseperchélui / lei
Table 1: Exemplar minimal pairs for a representative subset of languages with a full set of exemplars are in § A . Our gender agreement stimuli are violable: “Mary was powerful. James lied because she told him to” , although we do not find this to affect our LMs behaviorally. Instead, we find our studied LMs to score at or above 90% accuracy on 90% (115/128) of templates and below 80% accuracy only once.
Figure 2: We calculate the Odds when evaluating our trained interventions on a held-out test set of the same language they are trained on (red line), all other languages (green line), all other languages within the same family (yellow line), and across two controls (grey and black lines). We find that across all positions, phenomena, and models, interventions consistently generalize across languages with pairwise t-tests on the Max Odds showing significant differences between both the cross-language and control conditions ( p<0.05p<0.05 ).
Figure 3: We visualize our languages on the top principal components of the transfer matrix. We find language family-based clusters not to be pure.
 Number AgreementGender AgreementFiller–Gap (Object)
TermNP 1 (.68)PP (.71)NP 2 (.85)Name (.69)VP (.75)because (.76)Filler (.73)NP (.61)Verb (.78)
Typological dist.−0.577∗∗∗\mathbf{-0.577}^{***}−0.558∗∗∗\mathbf{-0.558}^{***}−0.432∗∗∗\mathbf{-0.432}^{***}−0.462∗∗∗\mathbf{-0.462}^{***}−0.320∗∗∗\mathbf{-0.320}^{***}−0.498∗∗∗\mathbf{-0.498}^{***}−0.544∗∗∗\mathbf{-0.544}^{***}−0.781∗∗∗\mathbf{-0.781}^{***}−1.04∗∗∗\mathbf{-1.04}^{***}
Model size0.3630.3630.2680.2680.4810.4810.4000.4000.3790.3790.5500.550−0.013-0.013−0.151-0.1510.2240.224
Typological dist. ×\times model size−0.043-0.043−0.100∗∗∗\mathbf{-0.100}^{***}−0.078∗∗\mathbf{-0.078}^{**}−0.055-0.055−0.158∗∗∗\mathbf{-0.158}^{***}−0.063-0.0630.0150.0150.0100.010−0.103-0.103
Table 2: LMEM coefficients predicting the Max Odds from linguistic and architectural factors. Estimates significant under the LRT are bolded . ∗p<.05{}^{*}p<.05 ; ∗∗p<.01{}^{**}p<.01 ; ∗∗∗p<.001{}^{***}p<.001 . Rc2R^{2}_{c} values in parentheses next to each position. We find significant negative effects of typological distance, suggesting that transfer is stronger between typologically similar languages. Interaction effects between typological distance and model size further suggest such effects grow stronger in larger models for the tasks of Number Agreement and Gender Agreement.
Subject Verb Number Agreement
 NP 1PPNP 2
mGPT-0.0800.201-0.158
Llama-3.2-1b0.0650.029-0.093
Llama-3.2-3b-0.1790.0970.120
Tiny-Aya0.2360.039-0.229
Anaphoric Pronoun Gender Agreement
Table 3: Silhouette scores measuring language-family cluster purity in PCA space. Each cell is the mean silhouette width for the given model and position. Scores range from −1-1 (languages closer to other families) to +1+1 (tight within-family clusters).
Figure 4: We measure the correlation between different operationalizations of typological similarity, pooled across positions and phenomenon. All pairwise correlations are significantly positive (all cells p<.01p<.01 ) and broadly strong, suggesting these factors are entangled in the model’s training data and learned representations. Correlations with template overlap tend to be weaker, reflecting the hand-designed nature of our templates.
 Number AgreementGender AgreementFiller–Gap (Object)
Training Data
rr (mC4, Out-Degree)0.180.630.36
rr (mC4, In-Degree)0.250.77-0.06
Typological ‘Hubs’
rr (In, Out)0.700.890.44
rr (Centr., Hub Score)0.590.850.75
Table 4: Pearson correlation, rr , between different data-metrics. ‘Hub’-ness is most correlated with typological centrality, not by training data magnitude.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Table 5: Exemplar minimal pairs per language for each construction. The minimal differences upstream (denoted by the two options separated by a ‘/’) in the NP 1 , Name , and Filler positions each correspond to the given labels in the VP , Pronoun , and Gap positions respectively. Such minimal pairs are used to evaluate LM linguistic competence, and to ensure that our causal interventions are successful. ‘-’ indicates positions where the given construction has no item. This appears only in German and Dutch Filler–Gap stimuli as they are V2 languages, mandating any embedded verbs appear clause-final. The contrast at the VP position for Russian gender agreement ( ходил / ходила ) reflects the past-tense verb’s obligatory agreement with the subject, and is not an additional label.
Table 6: Sampled exemplar minimal pairs per language for each construction for random labels : the stimulus is identical to the true construction, but the counterfactual target are arbitrary tokens. We expect a well-calibrated intervention to be specific and not generalize to these.
Table 7: Sampled exemplar minimal pairs per language for each construction in the OOD labels control. As inspired by Kumon and Yanaka (2026) : the stimulus is identical to the true construction, but the counterfactual target is a plausible yet out-of-distribution continuation. We expect a well-trained intervention to target the abstract feature of interest, and thus generalize to these.
Figure 5: Behavioral performance of each model on each of our templatic stimuli. We broadly see strong performance from all of our models on all templates. Generally multilingual LMs perform the best, but both Llama models show strong performance as well. All bars but thirteen are above 90% performance, with twelve of those above 80%. The only bar below 80% is Llama‑3.2‑1b on Bulgarian number agreement.
 Filler–Gap (Object)Subject Verb Number AgreementAnaphoric Pronoun Gender Agreement
ModelFillerNPVerbNP 1PPNP 2NAMEVPbecause
Other phenomena( Control: specificity)
Llama-3.2-1b1.2×10−9∗∗∗\mathbf{1.2\times 10^{-9}}^{***}5.2×10−7∗∗∗\mathbf{5.2\times 10^{-7}}^{***}1.5×10−7∗∗∗\mathbf{1.5\times 10^{-7}}^{***}3.0×10−40∗∗∗\mathbf{3.0\times 10^{-40}}^{***}2.2×10−28∗∗∗\mathbf{2.2\times 10^{-28}}^{***}1.9×10−34∗∗∗\mathbf{1.9\times 10^{-34}}^{***}1.3×10−32∗∗∗\mathbf{1.3\times 10^{-32}}^{***}5.6×10−30∗∗∗\mathbf{5.6\times 10^{-30}}^{***}2.7×10−22∗∗∗\mathbf{2.7\times 10^{-22}}^{***}
Llama-3.2-3b1.9×10−7∗∗∗\mathbf{1.9\times 10^{-7}}^{***}4.7×10−6∗∗∗\mathbf{4.7\times 10^{-6}}^{***}5.9×10−9∗∗∗\mathbf{5.9\times 10^{-9}}^{***}1.2×10−41∗∗∗\mathbf{1.2\times 10^{-41}}^{***}1.7×10−35∗∗∗\mathbf{1.7\times 10^{-35}}^{***}1.4×10−40∗∗∗\mathbf{1.4\times 10^{-40}}^{***}1.3×10−31∗∗∗\mathbf{1.3\times 10^{-31}}^{***}2.4×10−35∗∗∗\mathbf{2.4\times 10^{-35}}^{***}1.2×10−33∗∗∗\mathbf{1.2\times 10^{-33}}^{***}
mGPT1.0×10−5∗∗∗\mathbf{1.0\times 10^{-5}}^{***}6.1×10−6∗∗∗\mathbf{6.1\times 10^{-6}}^{***}3.9×10−7∗∗∗\mathbf{3.9\times 10^{-7}}^{***}2.8×10−35∗∗∗\mathbf{2.8\times 10^{-35}}^{***}8.3×10−36∗∗∗\mathbf{8.3\times 10^{-36}}^{***}2.4×10−33∗∗∗\mathbf{2.4\times 10^{-33}}^{***}3.1×10−22∗∗∗\mathbf{3.1\times 10^{-22}}^{***}8.3×10−25∗∗∗\mathbf{8.3\times 10^{-25}}^{***}1.9×10−32∗∗∗\mathbf{1.9\times 10^{-32}}^{***}
tiny-aya-base5.9×10−9∗∗∗\mathbf{5.9\times 10^{-9}}^{***}1.2×10−9∗∗∗\mathbf{1.2\times 10^{-9}}^{***}6.9×10−9∗∗∗\mathbf{6.9\times 10^{-9}}^{***}8.4×10−58∗∗∗\mathbf{8.4\times 10^{-58}}^{***}1.3×10−39∗∗∗\mathbf{1.3\times 10^{-39}}^{***}3.1×10−52∗∗∗\mathbf{3.1\times 10^{-52}}^{***}1.0×10−44∗∗∗\mathbf{1.0\times 10^{-44}}^{***}1.6×10−40∗∗∗\mathbf{1.6\times 10^{-40}}^{***}6.6×10−49∗∗∗\mathbf{6.6\times 10^{-49}}^{***}
Appendix
Table 8: Holm-bonferroni corrected pp -values (correction applied within each comparison type, pooled across phenomena/models/positions) for all pairwise tt -tests comparing cross-lingual transfer ( Cross Language (all) ) against our two selectivity controls and against family-restricted transfer, for Experiment One ( Section 4 ; Figure 6 ). Random Labels and Other phenomena are our controls: cross-lingual transfer should, and does, significantly exceed both. Cross Language (Within-Family) instead tests whether transfer is stronger among typologically related languages than to the full evaluated set; results here are mixed. †p<.1{}^{\dagger}p<.1 ; ∗p<.05{}^{*}p<.05 ; ∗∗p<.01{}^{**}p<.01 ; ∗∗∗p<.001{}^{***}p<.001 ; significant cells bolded.
 Filler–Gap (Object)Subject Verb Number Agreement
ModelFillerNPVerbNP 1PPNP 2
Other phenomena( Control: specificity)
Llama-3.2-1b3.3×10−7∗∗∗\mathbf{3.3\times 10^{-7}}^{***}2.5×10−6∗∗∗\mathbf{2.5\times 10^{-6}}^{***}1.2×10−6∗∗∗\mathbf{1.2\times 10^{-6}}^{***}5.2×10−39∗∗∗\mathbf{5.2\times 10^{-39}}^{***}7.2×10−32∗∗∗\mathbf{7.2\times 10^{-32}}^{***}3.3×10−40∗∗∗\mathbf{3.3\times 10^{-40}}^{***}
Llama-3.2-3b4.4×10−6∗∗∗\mathbf{4.4\times 10^{-6}}^{***}3.2×10−6∗∗∗\mathbf{3.2\times 10^{-6}}^{***}6.5×10−7∗∗∗\mathbf{6.5\times 10^{-7}}^{***}3.0×10−42∗∗∗\mathbf{3.0\times 10^{-42}}^{***}2.4×10−40∗∗∗\mathbf{2.4\times 10^{-40}}^{***}1.1×10−43∗∗∗\mathbf{1.1\times 10^{-43}}^{***}
mGPT0.042∗\mathbf{0.042^{*}}8.9×10−6∗∗∗\mathbf{8.9\times 10^{-6}}^{***}4.3×10−6∗∗∗\mathbf{4.3\times 10^{-6}}^{***}5.2×10−30∗∗∗\mathbf{5.2\times 10^{-30}}^{***}5.3×10−44∗∗∗\mathbf{5.3\times 10^{-44}}^{***}4.3×10−55∗∗∗\mathbf{4.3\times 10^{-55}}^{***}
tiny-aya-base8.9×10−6∗∗∗\mathbf{8.9\times 10^{-6}}^{***}1.2×10−8∗∗∗\mathbf{1.2\times 10^{-8}}^{***}1.6×10−9∗∗∗\mathbf{1.6\times 10^{-9}}^{***}1.1×10−51∗∗∗\mathbf{1.1\times 10^{-51}}^{***}5.7×10−45∗∗∗\mathbf{5.7\times 10^{-45}}^{***}2.4×10−59∗∗∗\mathbf{2.4\times 10^{-59}}^{***}
Appendix
Table 9: Holm-bonferroni corrected pp -values (correction applied within each comparison type, pooled across phenomena/models/positions) for the OOD labels ( § E ; Figure 7 ). As in Table 8 , Random Labels and Other phenomena are selectivity controls compared against Cross Language (all) transfer, and Cross Language (Within-Family) tests family-restricted vs. all-language transfer. †p<.1{}^{\dagger}p<.1 ; ∗p<.05{}^{*}p<.05 ; ∗∗p<.01{}^{**}p<.01 ; ∗∗∗p<.001{}^{***}p<.001 ; significant cells bolded.
Figure 6: We take the Max Odds across layers for each model at each position, before normalizing by the within-language value (red bar). We see strong performance in both our cross-language (green bar) and within-family (yellow bar) transfer conditions, with both being significantly above both controls in all cases when tested with t-tests with a Holm-Bonferroni correction. We further find the within-family transfer to be significantly higher than the cross-language bar in a select few comparisons, but for this to not hold more broadly.
Figure 7: We evaluate our interventions on syntactically appropriate but lexically disjoint labels from our training set. We find our results to broadly hold in this setting.
MetricEnglish (en)German (de)Dutch (nl)French (fr)Italian (it)Bulgarian (bg)Russian (ru)Spanish (es)Portuguese (pt)Romanian (ro)Galician (gl)Farsi (fa)Chinese (zh)Japanese (ja)Korean (ko)
Template Overlap✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Corpus Overlap✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Feature Overlap (Grambank)✓✗✓✓✓✗✓✗✓✗✓✓✓✓✓
Cognate Proportion✓✓✓✓✓✓✓✓✓✓✗✓✗✗✗
Typological Dist.✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
Appendix
Table 10: Language coverage for each distance/overlap metric used in the typological-distance correlation analysis. Feature Overlap (Grambank) is missing for German, Bulgarian, Spanish, and Romanian because Grambank lacks entries for them. The cognate data we utilize, Cognate Proportion (IECOR) is limited to Indo-European languages, so it is unavailable for Galician, Chinese, Japanese, and Korean.
 TrainingEvaluation
Model / PhenomenonRuns# InterventionsSeconds per InterventionTime (h)Pairs# InterventionsSeconds per InterventionTime (h)
mGPT (24 layers)
number agreement12728.22.0579721.9422.4
gender agreement13728.22.1534722.8330.3
filler–gap7728.11.1294725.1430.2
total32––5.31407––82.9
Appendix
Table 11: Per-model compute cost on a single NVIDIA A40 GPU. A run is one language, with interventions at every position and layer. As there are three positions evaluated for each phenomenon, this just totals three times the number of layers in each model. Times reported reflect the median value of all runs performed during our experiments.
More in this topic

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Explore similar work

CardsList

  1. Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Models

    Apr 24, 2026Ryoma Kumon, Hitomi YanakaMechanistic InterpretabilityLanguage Modeling

    1The University of Tokyo · 2RIKEN · 3Tohoku University

  2. Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages

    Aug 19, 2026Isabella Gidi, Antonio Almudévar, Core Francisco Park +2Multilingual Language ModelsCircuits in Language Models

    Harvard University · University of Zaragoza · Boston University +1

  3. An In-Vitro Study on Cross-Lingual Generalization in Language Models

    May 26, 2026Adrian CosmaMultilingual Language Model EvaluationCross-Lingual Transfer

    Dalle Molle Institute for Artificial Intelligence (IDSIA)