Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models

Shin Yamaguchi; Masaya Taniguchi; Tatsuki Kuribayashi; Taro Watanabe; Yoshinobu Kano

cs.CLOct 22, 2026

Once a Tree, Always a Tree? Cross-lingual Transfer of Hierarchical Generalization in Language Models

Authors: Shin Yamaguchi, Masaya Taniguchi, Tatsuki Kuribayashi, Taro Watanabe, Yoshinobu Kano

Organizations: NAIST · RIKEN · MBZUAI · Tohoku University ·

Abstract

Language models (LMs) often exhibit successful cross-lingual transfer through continued (pre)training, still their underlying process remains unclear. One possibility is that LMs can leverage the evidence toward language-universal principles, namely, hierarchical structure, from a source language (L1) to a new language (L2), while many other non-linguistic explanations, e.g., improved initialization of position embeddings, can be potentially possible. In this study, to quantify the cross-linguistic transferability of linguistic evidence in cross-lingual LM training, we introduce a new experimental design, namely, a cross-lingual poverty-of-stimulus (CPOS) setting that ablates how LMs' hierarchical generalization in L2 can be led by structural evidence provided in L1. We contrast counterfactual L1 and L2 learning scenarios where the L2 corpus is ambiguous between linear and hierarchical grammar rules, while the amount of positive evidence for the hierarchical rule is controlled on the L1 side. Our experimental results show that LMs can partially, yet not perfectly, transfer L1 structural evidence to L2, yielding only a small but detectable hierarchical bias, while the strength of transfer depends on language pairs. We further ablate the conditions for successful transfer, motivated by recent reports on the substantial impact of vocabulary overlap. We indeed find that vocabulary overlap is crucial even for the cross-lingual transfer of hierarchical syntactic generalization. This suggests that this also explains the language-pair-dependent tendencies in the main experiments and suggests that the transfer of linguistic evidence in continued new-language learning is only conditionally achievable.

Figures & tables

Figure 1: Overview of the experimental design. A language model is first pretrained on an L1 corpus that contains positive evidence for the hierarchical rule ( Disambiguating data ), and then continued-pretrained on an L2 corpus from which such evidence has been removed ( Ambiguous data ). Any above-chance hierarchical-syntax performance at L2 must therefore reflect cross-lingual transfer of the rule acquired during L1 training.

Figure 2: Example of Hierarchical vs. Linear generalization for subject-verb number agreement. With an attractor ( donkey ) intervening between the plural subject ( Dogs ) and the verb, Linear predicts agreement with the linearly nearest noun ( plays ), whereas Hierarchical predicts agreement with the syntactic subject ( play ).

Figure 3: Learning trajectories on CLAMS (Left) and MultiBLiMP (Right) over L2-training epochs under the Full Overlap condition. Orange lines:

\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}}

; blue lines:

\textbf{{{\color[rgb]{0,0,0}Lin}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}}

baseline. Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds. The chance rate is

0.50

CLAMS
L1	Lin $\to$ Lin	Hier $\to$ Lin
En $\rightarrow$ En	56.04	61.49 ( $\Delta$ +5.45 )
De $\rightarrow$ En	52.83	54.52 ( $\Delta$ +1.68 )
Ru $\rightarrow$ En	51.07	52.11 ( $\Delta$ +1.04 )
Ja $\rightarrow$ En	51.24	51.77 ( $\Delta$ +0.53 )
MultiBLiMP

Table 1: Accuracy (seed-mean of the final checkpoint, scored over tokens seen in L1 and/or L2 training data (OOV excluded)) under full vocabulary overlap, for benchmarks CLAMS, MultiBLiMP. The

\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}}

column appends

\Delta=(\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}})-(\textbf{{{\color[rgb]{0,0,0}Lin}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}})

at the same overlap ( green =positive, red =negative).

Data	L1 $\rightarrow$ L2	full	1K	no	full	1K	no
		Lin $\to$ Lin			Hier $\to$ Lin
CLAMS	En $\rightarrow$ En	56.04	55.61	51.61	61.49 ( $\Delta$ +5.45 )	61.37 ( $\Delta$ +5.76 )	52.39 ( $\Delta$ +0.79 )
	De $\rightarrow$ En	52.83	52.78	51.42	54.52 ( $\Delta$ +1.68 )	55.46 ( $\Delta$ +2.69 )	52.19 ( $\Delta$ +0.77 )
	Ru $\rightarrow$ En	51.07	51.39	50.92	52.11 ( $\Delta$ +1.04 )	53.03 ( $\Delta$ +1.64 )	51.07 ( $\Delta$ +0.14 )
	Ja $\rightarrow$ En	51.24	51.62	51.85	51.77 ( $\Delta$ +0.53 )	51.75 ( $\Delta$ +0.13 )	51.59 ( $\Delta$ -0.26 )
MultiBLiMP	En $\rightarrow$ En	61.39	61.11	51.94	69.92 ( $\Delta$ +8.53 )	72.36 ( $\Delta$ +11.25 )	54.47 ( $\Delta$ +2.53 )

Table 2: Accuracy (seed-mean of the final checkpoint, scored over tokens seen in L1 and/or L2 training data (OOV excluded)) under each vocabulary overlap conditions, for benchmarks CLAMS, MultiBLiMP. The

\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}}

column appends

\Delta=(\textbf{{{\color[rgb]{0,0,0}Hier}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}})-(\textbf{{{\color[rgb]{0,0,0}Lin}}}\to\textbf{{{\color[rgb]{0,0,0}Lin}}})

at the same overlap ( green =positive, red =negative).

Figure 4:

\Delta\mathrm{acc}

on CLAMS (Left) and MultiBLiMP (Right) broken down by L1, vocabulary-overlap condition, and L2-training epoch. Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds.

Figure 5:

\Delta\mathrm{acc}

on CLAMS(Left) and MultiBLiMP(Right) for En

\to

En under extended training (8epoch in L1

\to

8epoch in L2). Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds.

Figure 6: Principal transfer contrasts on CLAMS from the mixed-effects model. Points are posterior medians of

\Delta_{\mathrm{hier}}

(log-odds) and error bars are 95% CrIs; the dashed line marks zero. In the axis labels, hier

=

0.5 and hier

=

0.0 correspond to the Hier

\to

Lin and Lin

\to

Lin conditions, respectively.

Figure 7:

\Delta\mathrm{acc}

on CLAMS(Left) and MultiBLiMP(Right) for De

\to

En under extended training (8epoch in L1

\to

8epoch in L2). Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds.

Figure 8:

\Delta\mathrm{acc}

on CLAMS(Left) and MultiBLiMP(Right) for Ru

\to

En under extended training (8epoch in L1

\to

8epoch in L2). Lines show the seed mean; shaded bands indicate

\pm 1

SD across three random seeds.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Ambiguous data		Disambiguating data
	#Sents	#Words/Sent	#Sents	#Words/Sent
English (L1/L2)	2.42/2.42M	20.6/20.6	2.0M	24.6
German	2.7M	18.3	2.3M	21.9
Russian	2.8M	17.8	2.3M	21.3
Japanese	3.8M	13.1	2.8M	17.9

Appendix

Table 3: Statistics of the training corpora. Each language group reports the number of sentences and the average sentence length for the Ambiguous data and Disambiguating data partitions. The English row additionally reports the L2 column since English serves as the L2 in all experiments.

Language	Ambiguous data					Disambiguating data
	nsubj	Clausal	All	Depth	DLT Stor.	nsubj	Clausal	All	Depth	DLT Stor.
English	1.64	0.00	3.29	0.00	4.49	4.33	5.70	3.13	0.62	4.93
German	1.38	0.00	3.21	0.00	4.24	5.44	10.64	3.47	0.48	5.38
Russian	1.46	0.00	3.31	0.00	3.90	4.14	5.37	2.97	0.53	4.36
Japanese	2.30	0.00	2.87	0.00	2.55	7.06	5.27	2.88	0.66	2.96

Appendix

Table 4: Dependency-based complexity statistics of the 50M-token L1 training corpora: nsubj = mean subject–verb linear distance over nsubj relations; Clausal = mean head–dependent distance over clausal relations ( acl , advcl , ccomp , xcomp , csubj ); All = the same over all relations except punct ; Depth = mean per-sentence maximum clausal embedding depth; DLT Stor. = mean per-sentence maximum storage cost under the Dependency Locality Theory Gibson (2000) , i.e., the number of pending syntactic heads. Clausal and Depth are exactly

0

in the Lin

\to

Lin corpora by construction. The English L2 corpus matches the English Lin

\to

Lin columns to two decimals.

CLAMS (4 phenomena)
Phenomenon (subcategory)	# items
obj_rel_across_anim	11,200
obj_rel_within_anim	11,200
prep_anim	16,800
subj_rel	11,200
Total	50,400

Appendix

Table 5: Phenomena included in the hierarchical partition of CLAMS and MultiBLiMP, together with the number of minimal-pair items per phenomenon. These are the items aggregated into the accuracy reported in Section 5 .

		$\|V_{1}\|$		$\|V_{2}\|$		$\|O^{\prime}\|$		Vocabulary Size		Total Parameters (M)
Language Pair	Setting	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier
En $\to$ En	Full Overlap	63,839	63,789	63,825	63,820	63,768	63,733	64,000	64,000	134.60	134.60
	1k Overlap					1,000	1,000	126,768	126,733	182.81	182.78
	No Overlap					0	0	127,768	127,733	183.58	183.55
De $\to$ En	Full Overlap	62,676	62,238	56,707	55,048	55,473	53,374	64,000	64,000	134.60	134.60
	1k Overlap					1,000	1,000	118,473	116,374	176.44	174.82

Appendix

Table 6: Token statistics, vocabulary sizes, and parameter counts for each (language pair, overlap setting) combination, shown separately for the L1 Lin ( hier_ratio =0.0) and Hier ( hier_ratio =0.5) training corpora.

|V_{1}|

and

|V_{2}|

denote the L1- and L2-corpus vocabularies;

|O^{\prime}|

is the number of shared tokens retained after the disjoint strategy. Total parameters (in millions) are computed analytically from

\mathrm{vocab\_size}\times n_{\text{embd}}+n_{\text{pos}}\times n_{\text{embd}}+n_{\text{layer}}\times(12\,n_{\text{embd}}^{2}+13\,n_{\text{embd}})+2\,n_{\text{embd}}

Figure 9: A visualization of the three overlap conditions. Full Overlap , No Overlap , 1k Overlap .

		L1 corpus tokens (%)						L2 corpus tokens (%)
		Shared		Disjointed		L1-unique		Shared		Disjointed		L2-unique
Language Pair	Setting	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier	Lin	Hier
En $\to$ En	Full Overlap	100.00	100.00	0.00	0.00	0.00	0.00	100.00	99.99	0.00	0.00	0.00	0.01
	1k Overlap	60.06	59.26	39.94	40.74			60.05	59.98	39.94	40.01
	No Overlap	0.00	0.00	100.00	100.00			0.00	0.00	100.00	99.99
De $\to$ En	Full Overlap	94.32	92.56	0.00	0.00	5.68	7.44	99.04	98.70	0.00	0.00	0.96	1.30

Appendix

Table 7: Per-corpus token coverage (%) of the four dissociation categories. Cells are the share of the 50M-word L1 (resp. L2) training corpus whose tokens fall into each category, computed by tokenizing each corpus with its (language pair, hier_ratio)-specific joint Byte-Level BPE. Categories: Shared are intersection tokens whose vocabulary IDs are kept identical across L1 and L2; Disjointed are intersection tokens whose L2-side IDs are shifted into the

[V,V{+}D)

range, giving them separate embeddings; L1- / L2-unique are tokens that appear only in the respective corpus. L1-unique on the L1 side (and L2-unique on the L2 side) are constant across overlap settings within a (pair, hier) cell and shown via multirow ; the complementary cells (L2-unique on the L1 side and vice versa) are zero by definition and omitted. Within each (pair, hier) the three pct values (Shared, Disjointed, language-specific unique) sum to

\approx 100\%

on each side.

Hyperparameter	Setting
Training-corpus size per stage	50M tokens
Number of layers ( $n_{\text{layer}}$ )	12
Number of attention heads ( $n_{\text{head}}$ )	12
Hidden size ( $n_{\text{embd}}$ )	768
learning rate	5e-4
Per-device batch size	32

Appendix

Table 8: Hyperparameter settings used for the experiments.

Table 9: Model-estimated transfer contrasts and planned overlap-presence difference-in-differences by L1. Rows report the model estimate of

\Delta_{\mathrm{hier}}

within each vocabulary-overlap

\times

L1 combination; “Present

-

No” denotes

(\Delta_{\textsc{Full}}+\Delta_{\textsc{1K}})/2-\Delta_{\textsc{No}}

. Entries are estimates (posterior medians) with equal-tailed 95% CrIs on the log-odds scale, and positive estimates indicate a higher probability of a correct response under Hier

\to

Lin than under Lin

\to

Lin . Bold entries have CrIs that lie entirely above zero and are therefore described as credibly positive.

Figure 10: Results of the unfrozen-Transformer ablation under Full Overlap (top) and No Overlap (bottom); all Transformer layers are updated during L2 training, while the remaining settings match the main experiment.

Cross-Lingual Transfer

Cross-lingual transfer enables models to apply knowledge or capabilities learned in one language to tasks in another.

See all 258 papers

May 26, 2026

An In-Vitro Study on Cross-Lingual Generalization in Language Models

Cross-lingual transfer in language models is difficult to study in natural corpora because lexical overlap, morphology, data imbalance, and tokenization are entangled. We introduce an in-vitro framework with two procedurally generated languages that share the same ontology, typed grammar, and compositional structure, but differ in surface realization. This lets us independently vary lexical distance, minority-language proportion, tokenizer training regime, and vocabulary size, while evaluating transfer on a masked minority-language condition whose lexical forms are never observed during training. Across 700 controlled runs, we find that transfer is governed less by tokenizer balance or raw lexical similarity than by whether tokenization preserves reusable cross-lingual substructure. Smaller vocabularies often improve masked transfer by keeping words decomposable into shared fragments, whereas larger vocabularies can turn forms into language-specific atoms. We further show that transfer emerges as a staged process: grammatical and type-level competence precede masked lexical generalization. Finally, we attempt to explain this mechanism through tokenizer bridges and show that bridge strength correlates strongly with masked reachability.

Adrian Cosma

Dalle Molle Institute for Artificial Intelligence (IDSIA)

Multilingual Language Model Evaluation Cross-Lingual Transfer

Ask PDF

Oct 28, 2026

Leaving the Cave by Breaking the Symmetry: Identifiable Structure Can Enable Cross-Lingual Generalization

Why do multilingual language models generalize across languages even without shared vocabulary? The Platonic Representation Hypothesis posits that models trained on different data, objectives, and modalities converge toward a shared statistical model of reality. Extending this intuition to representations inside a single multilingual model, we propose the Relational Identifiability Hypothesis: relations in a shared world can promote entity alignment by ruling out alternative cross-lingual correspondences, making corresponding entities more likely to become directly readable without an additional transformation. We test this hypothesis in a controlled bilingual Transformer whose two token-disjoint languages describe the same modular-arithmetic world. We find that the languages can develop a shared answer subspace without aligning corresponding numbers. Cross-lingual alignment emerges selectively when the available relations constrain number correspondences: subtraction makes zero task-level identifiable, inducing selective alignment, while an order-sensitive operation produces broad alignment across numbers. These patterns predict success in zero-shot cross-lingual transfer. Together, our results show that relational structure in a shared world can promote entity alignment and enable transfer.

Frederick Riemenschneider, Anette Frank

Department of Computational Linguistics, Heidelberg University, Germany

Cross-Lingual Representation Alignment Multilingual Language Models Representation Learning Cross-Lingual Transfer Representation Geometry in Language Models

Ask PDF

Sep 16, 2026

Why Pretraining Fails to Share Cross-Lingual Knowledge

Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain 360M- and 7B-parameter LLMs and show that poor cross-lingual knowledge generalization emerges during pretraining and persists under standard interventions. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross-lingual knowledge generalization. Guided by this understanding, we suggest mapping languages into a shared token space by simple word-wise translation and find it substantially improves cross-lingual knowledge generalization, recovering up to 12.6% of native-language learning efficiency --- 14

\times

the baseline.

Adam Gaber, Uriel Dolev, Elisabeth Fittschen +3

Weizmann Institute of Science · Bar-Ilan University · Johns Hopkins University +4

Language Model Pretraining Multilingual Language Models Cross-Lingual Transfer

Ask PDF

Abstract

Cross-Lingual Transfer

Explore similar work

An In-Vitro Study on Cross-Lingual Generalization in Language Models

Leaving the Cave by Breaking the Symmetry: Identifiable Structure Can Enable Cross-Lingual Generalization

Why Pretraining Fails to Share Cross-Lingual Knowledge