cs.CLAug 19, 2026

Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages

Authors: Isabella Gidi, Antonio Almudévar, Core Francisco Park, Naomi Saphra, Ricard Marxer

Organizations: Harvard University · University of Zaragoza · Boston University · Univ Toulon, Aix Marseille Univ, CNRS, LIS & ILLS

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multilingual large language models often generalize across languages, and prior work suggests that their internal mechanisms can overlap cross-lingually. It remains unclear, however, when such sharing emerges and whether it varies with the overt realization of the same grammatical operation. We investigate this question for present-tense subject-verb agreement, a morphosyntactic process that varies substantially across languages and is only weakly expressed in English. Using activation patching and attention analysis across 29 languages and five open-source model families, we identify the attention heads causally implicated in agreement and compare these head-level signatures across languages. We find that languages with overt person/number inflection exhibit more similar agreement circuitry than non-conjugating languages, with the strongest sharing appearing when the analysis isolates recovery of the inflectional contrast itself. English provides an informative bridge case, becoming more similar to conjugating languages precisely in contexts where overt agreement is required. Finally, many implicated heads display similar attention patterns across languages, suggesting that cross-lingual overlap reflects shared functional roles as well as shared localization. Together, these results indicate that multilingual LLMs reuse partially shared computational structure for morphosyntactic agreement rather than relying on fully separate language-specific solutions.

Figures & tables

Figure 1: Example of activation patching. The clean and corrupted prompts differ only in the subject pronoun and target inflection, with the contrast isolated to the final token. For each layer-head pair, we patch that head’s output from the clean run into the corrupted run and measure the resulting increase in the clean-target logit. The resulting heatmap summarizes the causal contribution of each head.
Figure 2: Inflection-specific sharing is stronger under the minimal-pair metric. Left: Pairwise similarity among conjugating languages under the minimal-pair metric, shown separately by model checkpoint. Similarity is broadly positive across families, indicating shared inflection-related circuitry across languages with overt agreement. Right: Pairwise similarity among conjugating languages under the minimal-pair and target-logit metrics. Similarity is higher and less variable under the minimal-pair metric, indicating that cross-lingual sharing is strongest when the analysis isolates recovery of the inflectional contrast itself.
Figure 3: Broader target-form recovery still tracks overt agreement structure. Left: Pairwise similarity between activation-patching heatmaps grouped by language comparison type under the target-logit metric. Each point is a comparison between two heatmaps matched for model, conjugation pair, and patching direction. Conjugating–conjugating comparisons are substantially more similar than conjugating–non-conjugating comparisons, showing that broader target-form recovery remains structured by whether the language overtly marks agreement. Right: English similarity to conjugating languages in non-inflecting versus 3sg-inflecting contexts. Each colored line corresponds to one model and connects its mean English–to–conjugating similarity across the two condition types. Black points and error bars show the across-model mean and bootstrap confidence interval for each condition. English becomes more similar to conjugating languages precisely in the contexts where overt agreement is required.
Figure 4: Shared heads are agreement-focused and functionally similar across languages. Left: Coarse attention-role composition of top heads across three groups: minimal-pair heads in conjugating languages, target-logit heads in conjugating languages, and target-logit heads in non-conjugating languages. The two conjugating groups show similar coarse routing profiles, whereas target-logit heads in non-conjugating languages allocate less attention to pronouns and more to other prompt material. Right: Cross-lingual similarity of attention-role vectors for overlapping top-20 heads, shown separately by metric. High similarity under both metrics indicates that implicated heads tend to preserve agreement-related routing roles across languages.
Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Contrastive examples of verb conjugation across three typologically distinct languages. English only inflects verbs in the third-person singular. Spanish exhibits rich person/number inflection. Swedish does not inflect for person in the tested present-tense contexts.
FamilyHugging Face NameCitation
BLOOMbigscience/bloom-7b1( BigScience Workshop et al., 2022 )
Gemmagoogle/gemma-2-9b( Gemma Team, 2024 )
LlamaLlama-2-7b-hf( Touvron and others, 2023 )
 meta-llama/Llama-3.2-3B( Grattafiori and others, 2024 )
Mistralmistralai/Mistral-7B-v0.1( Jiang and others, 2023 )
QwenQwen/Qwen2-7B( Yang and others, 2024 )
Appendix
Table 1: Models used in the main experiments.
Figure 6: Language-specific prompt templates for all tested languages.
Figure 7: Subject pronouns used in each tested language.
Figure 8: Examples retained and discarded by the tokenization filter.
Languages
Albanian, Bulgarian, Catalan, Chinese, Czech, Danish, English, Estonian, Finnish, French, German, Hindi, Hungarian, Icelandic, Irish, Italian, Latin, Latvian, Lithuanian, Modern Greek, Norwegian Bokmål, Persian, Polish, Portuguese, Romanian, Russian, Serbo-Croatian, Slovenian, Spanish, Swedish, Turkish, Ukrainian, Urdu, Welsh
Appendix
Table 2: Languages tested across all models (pulled from dataset). Not all of these languages were used in analysis if they did not pass the filtering.
GroupLanguages
ConjugatingAlbanian, Bulgarian, Catalan, Czech, Estonian, Finnish, French, German, Hindi, Hungarian, Icelandic, Italian, Latin, Latvian, Lithuanian, Modern Greek, Polish, Portuguese, Romanian, Russian, Serbo-Croatian, Spanish, Turkish, Urdu
NonconjugatingChinese, Danish, Norwegian Bokmål, Swedish
MiddleEnglish
Appendix
Table 3: Languages that survive the filtering and appear in at least one patch result (in either the target-logit or minimal-pair recovery) in at least one model, grouped by agreement type. These are the ones used in analysis.
Figure 9: Languages surviving for tested models.
Figure 10: Family-specific language-similarity matrices for conjugating languages under the minimal-pair recovery metric. Each panel shows mean pairwise Pearson similarity between flattened absolute heatmaps, computed at the model-pair level and then averaged within family. White cells denote missing comparisons.
ConjugationDirectionMeanSDnn
1sg–2sg1sg →\rightarrow 2sg0.68890.29501459
1sg–2sg2sg →\rightarrow 1sg0.63310.32691483
1sg–3sg1sg →\rightarrow 3sg0.60450.30171476
1sg–3sg3sg →\rightarrow 1sg0.63880.28871508
1pl–3pl1pl →\rightarrow 3pl0.51770.34671290
1pl–3pl3pl →\rightarrow 1pl0.50400.33121320
Appendix
Table 4: Cross-lingual activation-patching similarity among conjugating languages, disaggregated by conjugation contrast and patching direction, where nn denotes the number of matched language-pair observations contributing to each condition. Values report mean pairwise Pearson similarity between absolute minimal-pair recovery heatmaps, averaged across model checkpoints. The positive similarity observed in the aggregate analysis persists across all individual directions, although its magnitude varies across contrasts.
Figure 11: Model-specific cross-group similarity distributions. Within each panel, conjugating–conjugating comparisons are consistently more similar than conjugating–non-conjugating comparisons, although the size of the gap varies by model family.
Figure 12: English shift toward conjugating-language circuitry, by model.
Figure 13: Cross-lingual similarity of conjugating languages under alternative similarity metrics and head-selection schemes. Pearson and cosine recover the main pattern most consistently, whereas Spearman is visibly noisier.
Languages
Spanish, Portuguese, Italian, French, Catalan, Czech, Finnish, Russian, Hungarian, Polish, Serbo-Croatian, German, English, Swedish, Chinese, Mongolian
Appendix
Table 5: Languages included in the auxiliary experiments.
FamilyHugging Face NameCitation
BLOOMbigscience/bloom-560m( BigScience Workshop et al., 2022 )
BLOOMbigscience/bloom-1b1( BigScience Workshop et al., 2022 )
BLOOMbigscience/bloom-1b7( BigScience Workshop et al., 2022 )
BLOOMbigscience/bloom-3b( BigScience Workshop et al., 2022 )
BLOOMbigscience/bloom-7b1( BigScience Workshop et al., 2022 )
Gemmagoogle/gemma-2-9b( Gemma Team, 2024 )
Appendix
Table 6: Models used in the model-scale and accuracy robustness analyses.
Figure 14: Robustness checks for model scale and task accuracy.
BaseInstruction-tunedCitation
bloom-1b7bloomz-1b7( BigScience Workshop et al., 2022 ; Muennighoff et al., 2023 )
gemma-2-2bgemma-2-2b-it( Gemma Team, 2024 )
Llama-2-7b-hfLlama-2-7b-chat-hf( Touvron and others, 2023 )
Mistral-7B-v0.1Mistral-7B-Instruct-v0.1( Jiang and others, 2023 )
Qwen2-1.5BQwen2-1.5B-Instruct( Yang and others, 2024 )
Appendix
Table 7: Base and instruction-tuned models used in the instruction-tuning comparison.
Figure 15: Instruction tuning leaves the overall similarity structure largely unchanged.
Base modelbigscience/bloom-1b7
Checkpoint repobigscience/bloom-1b7-intermediate
Steps1k, 10k, 50k, 100k, 150k, 200k, 250k, 300k
Appendix
Table 8: Base model and intermediate checkpoints used in the BLOOM progression analysis ( BigScience Workshop et al., 2022 ) .
Figure 16: Supplementary views of the BLOOM checkpoint analysis. Across both metrics, cross-lingual similarity tends to increase over training, although the minimal-pair metric is noisier and more contrast-specific.
More in this topic

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Explore similar work

CardsList
May 20, 2026

Quantifying the cross-linguistic effects of syncretism on agreement attraction

Agreement attraction errors, in which a verb erroneously agrees with an intervening noun rather than its grammatical head, are amplified by morphological syncretism in some languages (English, German, Russian) but not others (Turkish, Armenian), a cross-linguistic pattern without a principled account. We use surprisal and attention entropy from large language models as processing proxies to investigate this variation across four languages. LLM-derived measures replicate behavioral findings in English and German (syncretism modulates attraction), align with Turkish null results (no modulation), and partially capture Russian patterns. We discuss further directions for better understanding why syncretism affects agreement attraction differently across languages.

Utku Turk, Eva Neu

University of Maryland, College Park · University of Massachusetts, Amherst

Topic pending
AskPDF
Aug 28, 2026

Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Models

Linguistic theory has long recognized cross-linguistic syntactic regularities, leading to claims that these similar structures are processed by similar mechanisms. However, this hypothesis has been difficult to test empirically due to our lack of fine-grained, manipulable access of human processing mechanisms. In this work, we take advantage of techniques from mechanistic interpretability to study such a question in multilingual LMs. We first isolate language-internal mechanisms before attempting to transfer them cross-lingually. Across four models and three well-studied constructions (subject--verb number agreement, anaphoric pronoun gender agreement, and filler--gap object extraction) we find consistent cross-lingual mechanism transfer. We further find transfer to be graded, with more transfer between more typologically similar languages. We believe our work provides novel hypotheses about cross-linguistic syntactic structures and multilingual processing, and more broadly shows how the study of language models can help inform linguistic theory.

Sasha Boguraev, Toshiki Nakai, Kyle Mahowald +1

The University of Texas at Austin · Leipzig University, ScaDS.AI Dresden/Leipzig · Heidelberg Institute for Theoretical Studies

Multilingual Language ModelsLinguistic Typology
AskPDF
Oct 23, 2026

The Emergence and Sparsification of a Syntactic Circuit

To understand the emergence of grammatical behavior in language models, we track circuits for subject-verb number agreement in pythia-1b and ten seeds of pythia-410m across training. We find that subject-verb agreement is learned early, and is initially handled by a diffuse collection of neurons. However, after agreement behavior has stabilized, the mechanisms responsible for it continue to reorganize. The attention heads which mediate agreement often shift during training: in the case of pythia-1b, behavior is controlled by two heads early in training but by a third (different) head later. Furthermore, the circuit shrinks over training, with pythia-1b's final circuit being strikingly sparse. We take these results to be informative about how linguistic knowledge emerges and is represented in a language model.

Qing Yao, Sasha Boguraev, Tiago Pimentel +1

The University of Texas at Austin · ETH Zurich

Circuits in Language ModelsSyntactic Generalization in Language Models
AskPDF