cs.CLAug 19, 2026
SaveShared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages
Organizations: Harvard University · University of Zaragoza · Boston University · Univ Toulon, Aix Marseille Univ, CNRS, LIS & ILLS
Abstract
Multilingual large language models often generalize across languages, and prior work suggests that their internal mechanisms can overlap cross-lingually. It remains unclear, however, when such sharing emerges and whether it varies with the overt realization of the same grammatical operation. We investigate this question for present-tense subject-verb agreement, a morphosyntactic process that varies substantially across languages and is only weakly expressed in English. Using activation patching and attention analysis across 29 languages and five open-source model families, we identify the attention heads causally implicated in agreement and compare these head-level signatures across languages. We find that languages with overt person/number inflection exhibit more similar agreement circuitry than non-conjugating languages, with the strongest sharing appearing when the analysis isolates recovery of the inflectional contrast itself. English provides an informative bridge case, becoming more similar to conjugating languages precisely in contexts where overt agreement is required. Finally, many implicated heads display similar attention patterns across languages, suggesting that cross-lingual overlap reflects shared functional roles as well as shared localization. Together, these results indicate that multilingual LLMs reuse partially shared computational structure for morphosyntactic agreement rather than relying on fully separate language-specific solutions.
Figures & tables
Figure 1: Example of activation patching. The clean and corrupted prompts differ only in the subject pronoun and target inflection, with the contrast isolated to the final token. For each layer-head pair, we patch that head’s output from the clean run into the corrupted run and measure the resulting increase in the clean-target logit. The resulting heatmap summarizes the causal contribution of each head.
Figure 2: Inflection-specific sharing is stronger under the minimal-pair metric. Left: Pairwise similarity among conjugating languages under the minimal-pair metric, shown separately by model checkpoint. Similarity is broadly positive across families, indicating shared inflection-related circuitry across languages with overt agreement. Right: Pairwise similarity among conjugating languages under the minimal-pair and target-logit metrics. Similarity is higher and less variable under the minimal-pair metric, indicating that cross-lingual sharing is strongest when the analysis isolates recovery of the inflectional contrast itself.
Figure 3: Broader target-form recovery still tracks overt agreement structure. Left: Pairwise similarity between activation-patching heatmaps grouped by language comparison type under the target-logit metric. Each point is a comparison between two heatmaps matched for model, conjugation pair, and patching direction. Conjugating–conjugating comparisons are substantially more similar than conjugating–non-conjugating comparisons, showing that broader target-form recovery remains structured by whether the language overtly marks agreement. Right: English similarity to conjugating languages in non-inflecting versus 3sg-inflecting contexts. Each colored line corresponds to one model and connects its mean English–to–conjugating similarity across the two condition types. Black points and error bars show the across-model mean and bootstrap confidence interval for each condition. English becomes more similar to conjugating languages precisely in the contexts where overt agreement is required.
Figure 4: Shared heads are agreement-focused and functionally similar across languages. Left: Coarse attention-role composition of top heads across three groups: minimal-pair heads in conjugating languages, target-logit heads in conjugating languages, and target-logit heads in non-conjugating languages. The two conjugating groups show similar coarse routing profiles, whereas target-logit heads in non-conjugating languages allocate less attention to pronouns and more to other prompt material. Right: Cross-lingual similarity of attention-role vectors for overlapping top-20 heads, shown separately by metric. High similarity under both metrics indicates that implicated heads tend to preserve agreement-related routing roles across languages.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Contrastive examples of verb conjugation across three typologically distinct languages. English only inflects verbs in the third-person singular. Spanish exhibits rich person/number inflection. Swedish does not inflect for person in the tested present-tense contexts.
| Family | Hugging Face Name | Citation |
|---|---|---|
| BLOOM | bigscience/bloom-7b1 | ( BigScience Workshop et al., 2022 ) |
| Gemma | google/gemma-2-9b | ( Gemma Team, 2024 ) |
| Llama | Llama-2-7b-hf | ( Touvron and others, 2023 ) |
| meta-llama/Llama-3.2-3B | ( Grattafiori and others, 2024 ) | |
| Mistral | mistralai/Mistral-7B-v0.1 | ( Jiang and others, 2023 ) |
| Qwen | Qwen/Qwen2-7B | ( Yang and others, 2024 ) |
Table 1: Models used in the main experiments.
Figure 6: Language-specific prompt templates for all tested languages.
Figure 7: Subject pronouns used in each tested language.
Figure 8: Examples retained and discarded by the tokenization filter.
| Languages |
| Albanian, Bulgarian, Catalan, Chinese, Czech, Danish, English, Estonian, Finnish, French, German, Hindi, Hungarian, Icelandic, Irish, Italian, Latin, Latvian, Lithuanian, Modern Greek, Norwegian Bokmål, Persian, Polish, Portuguese, Romanian, Russian, Serbo-Croatian, Slovenian, Spanish, Swedish, Turkish, Ukrainian, Urdu, Welsh |
Table 2: Languages tested across all models (pulled from dataset). Not all of these languages were used in analysis if they did not pass the filtering.
| Group | Languages |
|---|---|
| Conjugating | Albanian, Bulgarian, Catalan, Czech, Estonian, Finnish, French, German, Hindi, Hungarian, Icelandic, Italian, Latin, Latvian, Lithuanian, Modern Greek, Polish, Portuguese, Romanian, Russian, Serbo-Croatian, Spanish, Turkish, Urdu |
| Nonconjugating | Chinese, Danish, Norwegian Bokmål, Swedish |
| Middle | English |
Table 3: Languages that survive the filtering and appear in at least one patch result (in either the target-logit or minimal-pair recovery) in at least one model, grouped by agreement type. These are the ones used in analysis.
Figure 9: Languages surviving for tested models.
Figure 10: Family-specific language-similarity matrices for conjugating languages under the minimal-pair recovery metric. Each panel shows mean pairwise Pearson similarity between flattened absolute heatmaps, computed at the model-pair level and then averaged within family. White cells denote missing comparisons.
| Conjugation | Direction | Mean | SD | |
|---|---|---|---|---|
| 1sg–2sg | 1sg 2sg | 0.6889 | 0.2950 | 1459 |
| 1sg–2sg | 2sg 1sg | 0.6331 | 0.3269 | 1483 |
| 1sg–3sg | 1sg 3sg | 0.6045 | 0.3017 | 1476 |
| 1sg–3sg | 3sg 1sg | 0.6388 | 0.2887 | 1508 |
| 1pl–3pl | 1pl 3pl | 0.5177 | 0.3467 | 1290 |
| 1pl–3pl | 3pl 1pl | 0.5040 | 0.3312 | 1320 |
Table 4: Cross-lingual activation-patching similarity among conjugating languages, disaggregated by conjugation contrast and patching direction, where denotes the number of matched language-pair observations contributing to each condition. Values report mean pairwise Pearson similarity between absolute minimal-pair recovery heatmaps, averaged across model checkpoints. The positive similarity observed in the aggregate analysis persists across all individual directions, although its magnitude varies across contrasts.
Figure 11: Model-specific cross-group similarity distributions. Within each panel, conjugating–conjugating comparisons are consistently more similar than conjugating–non-conjugating comparisons, although the size of the gap varies by model family.
Figure 12: English shift toward conjugating-language circuitry, by model.
Figure 13: Cross-lingual similarity of conjugating languages under alternative similarity metrics and head-selection schemes. Pearson and cosine recover the main pattern most consistently, whereas Spearman is visibly noisier.
| Languages |
| Spanish, Portuguese, Italian, French, Catalan, Czech, Finnish, Russian, Hungarian, Polish, Serbo-Croatian, German, English, Swedish, Chinese, Mongolian |
Table 5: Languages included in the auxiliary experiments.
| Family | Hugging Face Name | Citation |
|---|---|---|
| BLOOM | bigscience/bloom-560m | ( BigScience Workshop et al., 2022 ) |
| BLOOM | bigscience/bloom-1b1 | ( BigScience Workshop et al., 2022 ) |
| BLOOM | bigscience/bloom-1b7 | ( BigScience Workshop et al., 2022 ) |
| BLOOM | bigscience/bloom-3b | ( BigScience Workshop et al., 2022 ) |
| BLOOM | bigscience/bloom-7b1 | ( BigScience Workshop et al., 2022 ) |
| Gemma | google/gemma-2-9b | ( Gemma Team, 2024 ) |
Table 6: Models used in the model-scale and accuracy robustness analyses.
Figure 14: Robustness checks for model scale and task accuracy.
| Base | Instruction-tuned | Citation |
|---|---|---|
| bloom-1b7 | bloomz-1b7 | ( BigScience Workshop et al., 2022 ; Muennighoff et al., 2023 ) |
| gemma-2-2b | gemma-2-2b-it | ( Gemma Team, 2024 ) |
| Llama-2-7b-hf | Llama-2-7b-chat-hf | ( Touvron and others, 2023 ) |
| Mistral-7B-v0.1 | Mistral-7B-Instruct-v0.1 | ( Jiang and others, 2023 ) |
| Qwen2-1.5B | Qwen2-1.5B-Instruct | ( Yang and others, 2024 ) |
Table 7: Base and instruction-tuned models used in the instruction-tuning comparison.
Figure 15: Instruction tuning leaves the overall similarity structure largely unchanged.
| Base model | bigscience/bloom-1b7 |
| Checkpoint repo | bigscience/bloom-1b7-intermediate |
| Steps | 1k, 10k, 50k, 100k, 150k, 200k, 250k, 300k |
Table 8: Base model and intermediate checkpoints used in the BLOOM progression analysis ( BigScience Workshop et al., 2022 ) .
Figure 16: Supplementary views of the BLOOM checkpoint analysis. Across both metrics, cross-lingual similarity tends to increase over training, although the minimal-pair metric is noisier and more contrast-specific.
More in this topic
Multilingual Language Models
Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.
See all 265 papers