Shared multilingual representations are essential for cross-lingual tasks and knowledge transfer across languages. This study looks at the impact of parallel data, i.e. translated sentences, in pretraining as a signal to trigger representations that are aligned across languages. We train reference models with different proportions of parallel data and show that parallel data seem to have only a minimal effect on the cross-lingual alignment. Based on multiple evaluation methods, we find that the effect is limited to potentially accelerating the representation sharing in the early phases of pretraining, and to decreasing the amount of language-specific neurons in the model. Cross-lingual alignment seems to emerge on similar levels even without the explicit signal from parallel data.
Figures & tables
Figure 1: English and Finnish translation sentence pair representations of each model projected to the first two principal components. Finnish sentences are in blue and English sentences in yellow. Each model is exposed to different amounts of parallel data from no parallel data in lm-0 (top row) to 5% parallel data in lm-5 (bottom row).
Model
English
Finnish
Parallel
lm-0
160×109
40×109
0
lm-1
159×109
39×109
2×109
lm-2
158×109
38×109
4×109
lm-5
155×109
35×109
10×109
Table 1: The distributions of the training data in tokens for the four language models.
Figure 2: The average cosine similarities across the layers for the fully trained models and the untrained model. Translation refers to the average cosine similarity between the Finnish-English translation pairs, whereas random refers to the average cosine similarity between random Finnish and English sentence pairs.
Figure 3: The PWCCA scores for the final checkpoints of all the models and the untrained model. Random refers to the scores obtained when shuffling the parallel dataset, thus breaking the possible correlation.
Figure 4: The distribution of the 1000 most language-specific neurons across the layers of each model.
Figure 5: The percentage of neurons on each layer that score lower than or equal to 0.55 average precision in predicting both of the languages.
Finnish
lm-0
lm-1
lm-2
lm-5
>= 0.75
4666
4485
4538
4531
>= 0.9
496
423
398
437
>= 0.95
134
106
92
108
>= 0.99
16
19
16
15
English
lm-0
lm-1
lm-2
lm-5
>= 0.75
3448
3208
3165
3201
Table 2: The total number of neurons in each model with varying average precision thresholds. Finnish refers to average precision predicting Finnish, whereas English to average precision predicting English. The total number of neurons in each model is 132,096.
Fluency
Coherence
lm-0
μ
σ
μ
σ
FI
3.818
0.221
2.899
0.479
EN
1.008
0.052
3.123
0.425
0.09
3.713
0.266
2.324
0.636
lm-1
μ
σ
μ
σ
FI
3.827
0.211
2.92
0.478
Table 3: The mean ( μ ) scores with standard deviations ( σ ) from the LLM-judge for fluency and coherence for the continuations to the 200 stories. 0.09 refers to the controlled continuations, FI to the Finnish baseline continuations without any control vectors applied, and EN to English baseline continuations.
Figure 6: PWCCA scores computed across layers for early checkpoints of each model.
Figure 7: Difference in mean fluency and coherence scores between the controlled and Finnish baseline completions for checkpoints during training.
Figure 8: The development of the number of neurons given four thresholds for average precisions.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Parameters
1,420,296,192
Non-embedding parameters
1,317,273,600
Number of layers
24
Model dimension
2048
Attention type
Multi-headed attention
Number of heads
16
Appendix
Table 4: Hyperparameters for the model architecture used in this study.
Input
Kia had always had a distaste for mushrooms. But then one day she ordered pasta at a restaurant. The sauce was wonderful, and she asked what was in it. To her surprise, the waiter said it was mostly mushrooms!
lm-0
Kia ei ollut ainoa, joka oli ihastunut tähän ruokaan. Pian muutkin asiakkaat alkoivat kertoa syövänsä sieniruokaa. Sitten Kia päätti Kia wasn’t the only one who was in love with this dish. Soon other customers started telling me they were eating mushroom food. Then Kia decided
lm-1
Kia oli täysin järkyttynyt. Hän ei ollut koskaan syönyt mitään niin hyvää. Mikä ihme on tämä? Mutta siinä vaiheessa hän tajusi, että Kia was completely shocked. She had never eaten anything so good. What the hell is this? But at that moment she realized that
lm-2
Kia ei ollut tästä moksiskaan, vaan ajatteli, että totta kai hän söi tuon. Mutta sitten hän huomasi, että ruoka oli Kia wasn’t at all surprised by this, and thought that of course he ate that. But then he noticed that the food was
lm-5
Kia ei ollut koskaan aiemmin ostanut ravintolasta mitään muuta kuin ruokaa, joten hän ei ollut kovin yllättynyt kuullessaan, että ateria oli tehty sienistä Kia had never bought anything other than food from a restaurant before, so she wasn’t too surprised to hear that the meal was made with mushrooms.
Appendix
Table 5: Example text completions from the language control vector experiment. English translation in italics.
Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives. We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training. By testing on strictly monolingual language models, such as the Goldfish model families and independently developed models from different research labs, we find three results. Correlation: these models develop alignable representational geometry across layers, with alignment strengthening as data scale, model scale, or linguistic proximity increases. Construction: a single Procrustes rotation fit on parallel sentences maps hidden states between models. Causation: the same rotation transfers functional content; patching a rotated English residual into a German model on a factual cloze flips the prediction to the donor's capital in most cases. We confirm that cross-lingual alignment can emerge from the structure of language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular multilingual systems built from monolingual components.
Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.
Barah Fazili, Koustava Goswami
IIT Bombay, Mumbai, India · Adobe Research, Bangalore, India
Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain 360M- and 7B-parameter LLMs and show that poor cross-lingual knowledge generalization emerges during pretraining and persists under standard interventions. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross-lingual knowledge generalization. Guided by this understanding, we suggest mapping languages into a shared token space by simple word-wise translation and find it substantially improves cross-lingual knowledge generalization, recovering up to 12.6% of native-language learning efficiency --- 14× the baseline.
Adam Gaber, Uriel Dolev, Elisabeth Fittschen +3
Weizmann Institute of Science · Bar-Ilan University · Johns Hopkins University +4