Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models

Shin Yamaguchi; Masaya Taniguchi; Tatsuki Kuribayashi; Taro Watanabe; Yoshinobu Kano

cs.CLOct 22, 2026

Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models

Authors: Shin Yamaguchi, Masaya Taniguchi, Tatsuki Kuribayashi, Taro Watanabe, Yoshinobu Kano

Organizations: NAIST · RIKEN · MBZUAI · Tohoku University ·

Abstract

Language models (LMs) often exhibit successful cross-lingual transfer through continued (pre)training, still their underlying process remains unclear. One possibility is that LMs can leverage the evidence toward language-universal principles, namely, hierarchical structure, from a source language (L1) to a new language (L2), while many other non-linguistic explanations, e.g., improved initialization of position embeddings, can be potentially possible. In this study, to quantify the cross-linguistic transferability of linguistic evidence in cross-lingual LM training, we introduce a new experimental design, namely, a cross-lingual poverty-of-stimulus (CPOS) setting that ablates how LMs' hierarchical generalization in L2 can be led by structural evidence provided in L1. We contrast counterfactual L1 and L2 learning scenarios where the L2 corpus is ambiguous between linear and hierarchical grammar rules, while the amount of positive evidence for the hierarchical rule is controlled on the L1 side. Our experimental results show that LMs can partially, yet not perfectly, transfer L1 structural evidence to L2, yielding only a small but detectable hierarchical bias, while the strength of transfer depends on language pairs. We further ablate the conditions for successful transfer, motivated by recent reports on the substantial impact of vocabulary overlap. We indeed find that vocabulary overlap is crucial even for the cross-lingual transfer of hierarchical syntactic generalization. This suggests that this also explains the language-pair-dependent tendencies in the main experiments and suggests that the transfer of linguistic evidence in continued new-language learning is only conditionally achievable.

Figures & tables

Figure 1: Overview of the experimental design. A language model is first pretrained on an L1 corpus that contains positive evidence for the hierarchical rule ( Disambiguating data ), and then continued-pretrained on an L2 corpus from which such evidence has been removed ( Ambiguous data ). Any above-chance hierarchical-syntax performance at L2 must therefore reflect cross-lingual transfer of the rule acquired during L1 training.

Figure 2: Example of Hierarchical vs. Linear generalization for subject-verb number agreement. With an attractor ( donkey ) intervening between the plural subject ( Dogs ) and the verb, Linear predicts agreement with the linearly nearest noun ( plays ), whereas Hierarchical predicts agreement with the syntactic subject ( play ).

Figure 3: Learning trajectories on CLAMS (Left) and MultiBLiMP (Right) over L2-training epochs under the Full Overlap condition. Orange lines:

\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}}

; blue lines:

\textbf{{{\color[rgb]{0,0,0}Lin}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}}

baseline. Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds. The chance rate is

0.50

CLAMS
L1	Lin $\to$ Lin	Hier $\to$ Lin
En $\rightarrow$ En	56.04	61.49 ( $\Delta$ +5.45 )
De $\rightarrow$ En	52.83	54.52 ( $\Delta$ +1.68 )
Ru $\rightarrow$ En	51.07	52.11 ( $\Delta$ +1.04 )
Ja $\rightarrow$ En	51.24	51.77 ( $\Delta$ +0.53 )
MultiBLiMP

Table 1: Accuracy (seed-mean of the final checkpoint, scored over tokens seen in L1 and/or L2 training data (OOV excluded)) under full vocabulary overlap, for benchmarks CLAMS, MultiBLiMP. The

\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}}

column appends

\Delta=(\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}})-(\textbf{{{\color[rgb]{0,0,0}Lin}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}})

at the same overlap ( green =positive, red =negative).

Data	L1 $\rightarrow$ L2	full	1K	no	full	1K	no
		Lin $\to$ Lin			Hier $\to$ Lin
CLAMS	En $\rightarrow$ En	56.04	55.61	51.61	61.49 ( $\Delta$ +5.45 )	61.37 ( $\Delta$ +5.76 )	52.39 ( $\Delta$ +0.79 )
	De $\rightarrow$ En	52.83	52.78	51.42	54.52 ( $\Delta$ +1.68 )	55.46 ( $\Delta$ +2.69 )	52.19 ( $\Delta$ +0.77 )
	Ru $\rightarrow$ En	51.07	51.39	50.92	52.11 ( $\Delta$ +1.04 )	53.03 ( $\Delta$ +1.64 )	51.07 ( $\Delta$ +0.14 )
	Ja $\rightarrow$ En	51.24	51.62	51.85	51.77 ( $\Delta$ +0.53 )	51.75 ( $\Delta$ +0.13 )	51.59 ( $\Delta$ -0.26 )
MultiBLiMP	En $\rightarrow$ En	61.39	61.11	51.94	69.92 ( $\Delta$ +8.53 )	72.36 ( $\Delta$ +11.25 )	54.47 ( $\Delta$ +2.53 )

Table 2: Accuracy (seed-mean of the final checkpoint, scored over tokens seen in L1 and/or L2 training data (OOV excluded)) under each vocabulary overlap conditions, for benchmarks CLAMS, MultiBLiMP. The

\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}}

column appends

\Delta=(\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}})-(\textbf{{{\color[rgb]{0,0,0}Lin}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}})

at the same overlap ( green =positive, red =negative).

Figure 4:

\Delta\mathrm{acc}

on CLAMS (Left) and MultiBLiMP (Right) broken down by L1, vocabulary-overlap condition, and L2-training epoch. Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds.

Figure 5:

\Delta\mathrm{acc}

on CLAMS(Left) and MultiBLiMP(Right) for En

\to

En under extended training (8epoch in L1

\to

8epoch in L2). Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds.

Figure 6: Principal transfer contrasts on CLAMS from the mixed-effects model. Points are posterior medians of

\Delta_{\mathrm{hier}}

(log-odds) and error bars are 95% CrIs; the dashed line marks zero. In the axis labels, hier

=

0.5 and hier

=

0.0 correspond to the Hier

\to

Lin and Lin

\to

Lin conditions, respectively.

Figure 7:

\Delta\mathrm{acc}

on CLAMS(Left) and MultiBLiMP(Right) for De

\to

En under extended training (8epoch in L1

\to

8epoch in L2). Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds.

Figure 8:

\Delta\mathrm{acc}

on CLAMS(Left) and MultiBLiMP(Right) for Ru

\to

En under extended training (8epoch in L1

\to

8epoch in L2). Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Ambiguous data		Disambiguating data
	#Sents	#Words/Sent	#Sents	#Words/Sent
English (L1/L2)	2.42/2.42M	20.6/20.6	2.0M	24.6
German	2.7M	18.3	2.3M	21.9
Russian	2.8M	17.8	2.3M	21.3
Japanese	3.8M	13.1	2.8M	17.9

Appendix

Table 3: Statistics of the training corpora. Each language group reports the number of sentences and the average sentence length for the Ambiguous data and Disambiguating data partitions. The English row additionally reports the L2 column since English serves as the L2 in all experiments.

Language	Ambiguous data					Disambiguating data
	nsubj	Clausal	All	Depth	DLT Stor.	nsubj	Clausal	All	Depth	DLT Stor.
English	1.64	0.00	3.29	0.00	4.49	4.33	5.70	3.13	0.62	4.93
German	1.38	0.00	3.21	0.00	4.24	5.44	10.64	3.47	0.48	5.38
Russian	1.46	0.00	3.31	0.00	3.90	4.14	5.37	2.97	0.53	4.36
Japanese	2.30	0.00	2.87	0.00	2.55	7.06	5.27	2.88	0.66	2.96

Appendix

Table 4: Dependency-based complexity statistics of the 50M-token L1 training corpora: nsubj = mean subject–verb linear distance over nsubj relations; Clausal = mean head–dependent distance over clausal relations ( acl , advcl , ccomp , xcomp , csubj ); All = the same over all relations except punct ; Depth = mean per-sentence maximum clausal embedding depth; DLT Stor. = mean per-sentence maximum storage cost under the Dependency Locality Theory Gibson (2000) , i.e., the number of pending syntactic heads. Clausal and Depth are exactly

0

in the Lin

\to

Lin corpora by construction. The English L2 corpus matches the English Lin

\to

Lin columns to two decimals.

CLAMS (4 phenomena)
Phenomenon (subcategory)	# items
obj_rel_across_anim	11,200
obj_rel_within_anim	11,200
prep_anim	16,800
subj_rel	11,200
Total	50,400

Appendix

Table 5: Phenomena included in the hierarchical partition of CLAMS and MultiBLiMP, together with the number of minimal-pair items per phenomenon. These are the items aggregated into the accuracy reported in Section 5 .

		$\|V_{1}\|$		$\|V_{2}\|$		$\|O^{\prime}\|$		Vocabulary Size		Total Parameters (M)
Language Pair	Setting	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier
En $\to$ En	Full Overlap	63,839	63,789	63,825	63,820	63,768	63,733	64,000	64,000	134.60	134.60
	1k Overlap					1,000	1,000	126,768	126,733	182.81	182.78
	No Overlap					0	0	127,768	127,733	183.58	183.55
De $\to$ En	Full Overlap	62,676	62,238	56,707	55,048	55,473	53,374	64,000	64,000	134.60	134.60
	1k Overlap					1,000	1,000	118,473	116,374	176.44	174.82

Appendix

Table 6: Token statistics, vocabulary sizes, and parameter counts for each (language pair, overlap setting) combination, shown separately for the L1 Lin ( hier_ratio =0.0) and Hier ( hier_ratio =0.5) training corpora.

|V_{1}|

and

|V_{2}|

denote the L1- and L2-corpus vocabularies;

|O^{\prime}|

is the number of shared tokens retained after the disjoint strategy. Total parameters (in millions) are computed analytically from

\mathrm{vocab\_size}\times n_{\text{embd}}+n_{\text{pos}}\times n_{\text{embd}}+n_{\text{layer}}\times(12\,n_{\text{embd}}^{2}+13\,n_{\text{embd}})+2\,n_{\text{embd}}

Figure 9: A visualization of the three overlap conditions. Full Overlap , No Overlap , 1k Overlap .

		L1 corpus tokens (%)						L2 corpus tokens (%)
		Shared		Disjointed		L1-unique		Shared		Disjointed		L2-unique
Language Pair	Setting	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier
En $\to$ En	Full Overlap	100.00	100.00	0.00	0.00	0.00	0.00	100.00	99.99	0.00	0.00	0.00	0.01
	1k Overlap	60.06	59.26	39.94	40.74			60.05	59.98	39.94	40.01
	No Overlap	0.00	0.00	100.00	100.00			0.00	0.00	100.00	99.99
De $\to$ En	Full Overlap	94.32	92.56	0.00	0.00	5.68	7.44	99.04	98.70	0.00	0.00	0.96	1.30

Appendix

Table 7: Per-corpus token coverage (%) of the four dissociation categories. Cells are the share of the 50M-word L1 (resp. L2) training corpus whose tokens fall into each category, computed by tokenizing each corpus with its (language pair, hier_ratio)-specific joint Byte-Level BPE. Categories: Shared are intersection tokens whose vocabulary IDs are kept identical across L1 and L2; Disjointed are intersection tokens whose L2-side IDs are shifted into the

[V,V{+}D)

range, giving them separate embeddings; L1- / L2-unique are tokens that appear only in the respective corpus. L1-unique on the L1 side (and L2-unique on the L2 side) are constant across overlap settings within a (pair, hier) cell and shown via multirow ; the complementary cells (L2-unique on the L1 side and vice versa) are zero by definition and omitted. Within each (pair, hier) the three pct values (Shared, Disjointed, language-specific unique) sum to

\approx 100\%

on each side.

Hyperparameter	Setting
Training-corpus size per stage	50M tokens
Number of layers ( $n_{\text{layer}}$ )	12
Number of attention heads ( $n_{\text{head}}$ )	12
Hidden size ( $n_{\text{embd}}$ )	768
learning rate	5e-4
Per-device batch size	32

Appendix

Table 8: Hyperparameter settings used for the experiments.

Table 9: Model-estimated transfer contrasts and planned overlap-presence difference-in-differences by L1. Rows report the model estimate of

\Delta_{\mathrm{hier}}

within each vocabulary-overlap

\times

L1 combination; “Present

-

No” denotes

(\Delta_{\textsc{Full}}+\Delta_{\textsc{1K}})/2-\Delta_{\textsc{No}}

. Entries are estimates (posterior medians) with equal-tailed 95% CrIs on the log-odds scale, and positive estimates indicate a higher probability of a correct response under Hier

\to

Lin than under Lin

\to

Lin . Bold entries have CrIs that lie entirely above zero and are therefore described as credibly positive.

Figure 10: Results of the unfrozen-Transformer ablation under Full Overlap (top) and No Overlap (bottom); all Transformer layers are updated during L2 training, while the remaining settings match the main experiment.

Cross-Lingual Transfer

Cross-lingual transfer enables models to apply knowledge or capabilities learned in one language to tasks in another.

See all 258 papers

An In-Vitro Study on Cross-Lingual Generalization in Language Models
May 26, 2026Adrian CosmaMultilingual Language Model Evaluation Cross-Lingual Transfer
Dalle Molle Institute for Artificial Intelligence (IDSIA)
Leaving the Cave by Breaking the Symmetry: Identifiable Structure Can Enable Cross-Lingual Generalization
Oct 28, 2026Frederick Riemenschneider, Anette FrankCross-Lingual Representation Alignment Multilingual Language Models
Department of Computational Linguistics, Heidelberg University, Germany
Why Pretraining Fails to Share Cross-Lingual Knowledge
Sep 16, 2026Adam Gaber, Uriel Dolev, Elisabeth Fittschen +3Language Model Pretraining Multilingual Language Models
Weizmann Institute of Science · Bar-Ilan University · Johns Hopkins University +4

May 26, 2026J/K move · Enter open · S save

An In-Vitro Study on Cross-Lingual Generalization in Language Models

Adrian Cosma

Dalle Molle Institute for Artificial Intelligence (IDSIA)

Multilingual Language Model Evaluation Cross-Lingual Transfer

Open paper AskPDF arXiv

Cross-lingual transfer in language models is difficult to study in natural corpora because lexical overlap, morphology, data imbalance, and tokenization are entangled. We introduce an in-vitro framework with two procedurally generated languages that share the same ontology, typed grammar, and compositional structure, but differ in surface realization. This lets us independently vary lexical distance, minority-language proportion, tokenizer training regime, and vocabulary size, while evaluating transfer on a masked minority-language condition whose lexical forms are never observed during training. Across 700 controlled runs, we find that transfer is governed less by tokenizer balance or raw lexical similarity than by whether tokenization preserves reusable cross-lingual substructure. Smaller vocabularies often improve masked transfer by keeping words decomposable into shared fragments, whereas larger vocabularies can turn forms into language-specific atoms. We further show that transfer emerges as a staged process: grammatical and type-level competence precede masked lexical generalization. Finally, we attempt to explain this mechanism through tokenizer bridges and show that bridge strength correlates strongly with masked reachability.

Abstract

Cross-Lingual Transfer

Explore similar work

An In-Vitro Study on Cross-Lingual Generalization in Language Models

Leaving the Cave by Breaking the Symmetry: Identifiable Structure Can Enable Cross-Lingual Generalization

Why Pretraining Fails to Share Cross-Lingual Knowledge