Leaving the Cave by Breaking the Symmetry: Identifiable Structure Can Enable Cross-Lingual Generalization

Frederick Riemenschneider; Anette Frank

cs.CLOct 28, 2026

Leaving the Cave by Breaking the Symmetry: Identifiable Structure Can Enable Cross-Lingual Generalization

Authors: Frederick Riemenschneider, Anette Frank

Organizations: Department of Computational Linguistics, Heidelberg University, Germany

Abstract

Why do multilingual language models generalize across languages even without shared vocabulary? The Platonic Representation Hypothesis posits that models trained on different data, objectives, and modalities converge toward a shared statistical model of reality. Extending this intuition to representations inside a single multilingual model, we propose the Relational Identifiability Hypothesis: relations in a shared world can promote entity alignment by ruling out alternative cross-lingual correspondences, making corresponding entities more likely to become directly readable without an additional transformation. We test this hypothesis in a controlled bilingual Transformer whose two token-disjoint languages describe the same modular-arithmetic world. We find that the languages can develop a shared answer subspace without aligning corresponding numbers. Cross-lingual alignment emerges selectively when the available relations constrain number correspondences: subtraction makes zero task-level identifiable, inducing selective alignment, while an order-sensitive operation produces broad alignment across numbers. These patterns predict success in zero-shot cross-lingual transfer. Together, our results show that relational structure in a shared world can promote entity alignment and enable transfer.

Figures & tables

Panel of Figure 1 — Figure 1: Modular addition produces a shared answer subspace without number-level alignment. Adding an order-sensitive operation broadly aligns those numbers despite disjoint vocabularies.

Training Condition	Relabelings	Fixed by All
add-only	$p(p-1)$	none
add+add-copy	$p(p-1)$	none
sub-only	$p(p-1)$	$0$
add+sub	$p-1$	$0$
add+max	$1$	all

Table 1: Task-level predictions for all conditions. An answer is fixed if every task-preserving output relabeling

\tau

maps it to itself. For

p=113

p(p-1)=12{,}656

and

p-1=112

Broad Identity Alignment
Metric	add-only	add+add-copy	sub-only	add+sub	add+max
Identity Cross-Readout	$.011$ $\pm.008$	$.012$ $\pm.007$	$\hphantom{-}.018$ $\pm.014$	$.012$ $\pm.007$	$.125$ $\pm.075$
Matched-Answer Cosine	$.015$ $\pm.074$	$.003$ $\pm.014$	$-.008$ $\pm.026$	$.039$ $\pm.125$	$.843$ $\pm.221$
Shared Causal Answer Subspace
Rank-8 Answer-Subspace Overlap	$.944$ $\pm.080$	$.928$ $\pm.091$	$\hphantom{-}.889$ $\pm.086$	$.855$ $\pm.176$	$.924$ $\pm.086$
Cross-Ablation Drop, Pre-MLP	$.928$ $\pm.036$	$.923$ $\pm.046$	$\hphantom{-}.956$ $\pm.046$	$.947$ $\pm.047$	$.983$ $\pm.005$

Table 2: Identity alignment and shared causal answer structure across 20 bilingual models per condition. Cells report means

\pm

standard deviations across models. Broad identity scores average the 113 answer-specific scores within each model. Identity cross-readout uses held-out operand pairs and the other language’s 113 output-number tokens. Subspace bases are fitted on training examples, ablation drops use held-out examples. The focal task is addition except for sub-only , where it is subtraction.

Identity Cross-Readout
	Answer 0	Nonzero Ans.
add-only	$\hphantom{-}.016$ $\pm.068$	$\hphantom{-}.011$ $\pm.009$
add+add-copy	$\hphantom{-}.055$ $\pm.221$	$\hphantom{-}.012$ $\pm.007$
sub-only	$\hphantom{-}.775$ $\pm.248$	$\hphantom{-}.011$ $\pm.014$
add+sub	$\hphantom{-}.658$ $\pm.386$	$\hphantom{-}.006$ $\pm.006$
add+max	$\hphantom{-}.273$ $\pm.370$	$\hphantom{-}.123$ $\pm.075$

Table 3: Answer-level cross-lingual alignment across 20 bilingual models per condition. Cells report means

\pm

standard deviations across models. For each model, Nonzero Answers averages the 112 answer-specific scores for

c\in\{1,\ldots,112\}

; the table then reports the mean and standard deviation of these model-level averages. Identity cross-readout uses held-out operand pairs and the other language’s 113 output-number tokens.

Full Fine-Tuning
Pre-Training	Target 0	Target 1
add-only	$.494$ $\pm.085$	$.492$ $\pm.117$
add+add-copy	$.490$ $\pm.118$	$.545$ $\pm.119$
sub-only	$.852$ $\pm.065$	$.488$ $\pm.078$
add+sub	$.808$ $\pm.144$	$.521$ $\pm.102$
add+max	$.778$ $\pm.125$	$.772$ $\pm.107$

Table 4: Zero-shot cross-lingual transfer for two balanced downstream properties. Targets 0 and 1 denote

\mathrm{sum}=0/1

for addition-based conditions and

\mathrm{diff}=0/1

for sub-only . For each pre-trained model, we average five repetitions within each language direction and then average the resulting L0

\rightarrow

L1 and L1

\rightarrow

L0 target-language test accuracies. Cells report means

\pm

standard deviations over 20 models. Chance accuracy is

0.5

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Prime modulus $p$	113
Transformer layers	1
Attention heads	4
$d_{\mathrm{model}}$	128
$d_{\mathrm{head}}$	32
$d_{\mathrm{MLP}}$	512

Appendix

Table 5: Architecture and training hyperparameters used across conditions. Each update processes the complete training split for every active task-language combination.

Condition	Runs to 20 eligible models	Failed runs
add-only	20	0
add+add-copy	20	0
sub-only	51	31
add+sub	21	1
add+max	72	52

Appendix

Table 6: Condition-specific training stability. Runs are processed in seed order, and counting stops once the 20th eligible model is obtained. Eligibility requires at least

98\%

held-out accuracy in every active task–language combination.

Condition	Task	Full Accuracy	Retained Accuracy	Removal Drop
add-only	Add	$1.000\pm 0.000$	$1.000\pm 0.000$	$0.992\pm 0.004$
add+add-copy	Add 1	$1.000\pm 0.000$	$1.000\pm 0.000$	$0.994\pm 0.002$
add+add-copy	Add 2	$1.000\pm 0.000$	$1.000\pm 0.000$	$0.994\pm 0.002$
sub-only	Sub	$1.000\pm 0.000$	$1.000\pm 0.000$	$0.996\pm 0.003$
add+sub	Add	$0.999\pm 0.002$	$0.992\pm 0.023$	$0.993\pm 0.007$
add+sub	Sub	$0.999\pm 0.002$	$0.992\pm 0.023$	$0.993\pm 0.006$

Appendix

Table 7: Causal Fourier validation for modular addition and subtraction tasks (mean

\pm

standard deviation across 20 models; languages averaged within model). For each language and task, we select the five task-relevant frequencies carrying the most MLP-activation energy and either retain or remove their Fourier components. The

\max

task in add+max is exempt from retention/removal as the Fourier-mechanism prediction is not defined for ordinary numerical ordering.

Condition	Same Bilingual Model	Independent Bilingual Models	Random Subspaces
add-only	$.944\pm 0.080$	$0.216\pm 0.014$	$0.211\pm 0.020$
add+add-copy	$.928\pm 0.091$	$0.210\pm 0.019$	$0.210\pm 0.019$
sub-only	$.889\pm 0.086$	$0.215\pm 0.008$	$0.210\pm 0.019$
add+sub	$.855\pm 0.176$	$0.224\pm 0.020$	$0.210\pm 0.019$
add+max	$.924\pm 0.086$	$0.208\pm 0.010$	$0.210\pm 0.019$

Appendix

Table 8: Rank-8 answer-subspace overlap at the final residual stream (mean

\pm

standard deviation). Same-model values use the 20 frozen bilingual models in each condition. Independent-model controls use ten disjoint pairs of bilingual models; the two same-language comparisons are averaged within each pair. Random controls use 2,000 pairs of isotropically sampled rank-matched subspaces in

\mathbb{R}^{d_{\mathrm{model}}}

. All learned bases are estimated from arithmetic training examples only.

Pre-MLP Residual Stream
Accuracy Drop	add-only	add+add-copy	sub-only	add+sub	add+max
Own-Language Subspace	$0.958\pm 0.014$	$0.956\pm 0.026$	$0.985\pm 0.021$	$0.969\pm 0.020$	$0.984\pm 0.004$
Other-Language Subspace	$0.928\pm 0.036$	$0.923\pm 0.046$	$0.956\pm 0.046$	$0.947\pm 0.047$	$0.983\pm 0.005$
Independent-Model Subspace	$0.001\pm 0.001$	$0.001\pm 0.003$	$0.001\pm 0.002$	$0.007\pm 0.011$	$0.114\pm 0.122$
Random Subspace	$0.001\pm 0.001$	$0.001\pm 0.001$	$0.001\pm 0.002$	$0.005\pm 0.009$	$0.100\pm 0.101$
Final Residual Stream

Appendix

Table 9: Rank-8 answer-subspace ablations. We report the mean held-out accuracy drop

\pm

standard deviation across 20 models, after averaging over both language directions within each model. Random-subspace results are additionally averaged over five independently sampled rank-matched subspaces. Other-language subspaces are estimated from the other language in the same bilingual model; independent-model subspaces are taken from a separately initialized model.

Final-Residual Answer-Subspace Overlap
Condition	$r=2$	$r=4$	$r=8$	$r=16$	$r=32$
add-only	$0.577\pm 0.250$	$0.833\pm 0.125$	$0.944\pm 0.080$	$0.932\pm 0.038$	$0.931\pm 0.030$
add+add-copy	$0.636\pm 0.293$	$0.839\pm 0.153$	$0.928\pm 0.091$	$0.921\pm 0.060$	$0.933\pm 0.032$
sub-only	$0.671\pm 0.251$	$0.810\pm 0.153$	$0.889\pm 0.086$	$0.863\pm 0.063$	$0.940\pm 0.047$
add+sub	$0.833\pm 0.255$	$0.827\pm 0.211$	$0.855\pm 0.176$	$0.866\pm 0.063$	$0.956\pm 0.029$
add+max	$0.969\pm 0.113$	$0.878\pm 0.126$	$0.924\pm 0.086$	$0.947\pm 0.032$	$0.939\pm 0.029$

Appendix

Table 10: Rank sensitivity of answer-subspace overlap and causal cross-language ablation. Cells report means

\pm

standard deviations across the same 20 frozen models per condition. Subspace bases are fitted on training examples; ablation drops are evaluated on held-out examples and averaged over both language directions within each model. Rank 8 is the primary analysis. Larger ranks mechanically span and remove more directions, so magnitudes should not be compared across ranks as though rank were an experimental condition. The sweep tests whether the qualitative same-model overlap and cross-language causal dependence hinge on the primary rank choice. Rank-8 random- and independent-model controls are reported in Tabs. 8 and 9 .

Cross-Lingual Representation Alignment

Cross-lingual representation alignment studies how models map meaning across languages into comparable internal or embedding spaces.

See all 69 papers

Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
Aug 27, 2026Ej Zhou, Suchir Salhan, Catherine Arnett +1Cross-Lingual Representation Alignment Language Modeling
University of Cambridge · EleutherAI
An In-Vitro Study on Cross-Lingual Generalization in Language Models
May 26, 2026Adrian CosmaMultilingual Language Model Evaluation Cross-Lingual Transfer
Dalle Molle Institute for Artificial Intelligence (IDSIA)
A theory of platonic representations in language models
Oct 5, 2026Darshil Doshi, Wenjie Zhou, Corinna Elena Wegner +3Belief Propagation Multilingual Language Models
Johns Hopkins Univeristy · Johns Hopkins University & EPFL

Aug 27, 2026J/K move · Enter open · S save

Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

Ej Zhou, Suchir Salhan, Catherine Arnett, Anna Korhonen

University of Cambridge · EleutherAI

Cross-Lingual Representation Alignment Language Modeling Representation Geometry in Language Models

Open paper AskPDF arXiv

Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives. We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training. By testing on strictly monolingual language models, such as the Goldfish model families and independently developed models from different research labs, we find three results. Correlation: these models develop alignable representational geometry across layers, with alignment strengthening as data scale, model scale, or linguistic proximity increases. Construction: a single Procrustes rotation fit on parallel sentences maps hidden states between models. Causation: the same rotation transfers functional content; patching a rotated English residual into a German model on a factual cloze flips the prediction to the donor's capital in most cases. We confirm that cross-lingual alignment can emerge from the structure of language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular multilingual systems built from monolingual components.

Abstract

Cross-Lingual Representation Alignment

Explore similar work

Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

An In-Vitro Study on Cross-Lingual Generalization in Language Models

A theory of platonic representations in language models