High-quality human-translated parallel corpora can improve cross-lingual transfer, but are typically too small for large-scale pretraining. Machine-translated web corpora help overcome this limitation, making parallel data available at pretraining scale. Standard multilingual pretraining, however, samples source documents and their translations independently, without explicitly using their parallel structure. We investigate how to use the parallel structure of the data to improve multilingual pretraining by concatenating each document with its translation in the same training sequence. We compare this parallel pretraining (PPT) approach with standard pretraining (SPT), training models from scratch at matched token budgets. Surprisingly, despite the benefits of parallel data reported in prior cross-lingual work, PPT alone does not consistently improve language understanding over SPT. Our further analysis suggests that although PPT builds a strong cross-lingual foundation, as shown by its better use of shared information across languages, the mismatch between parallel training contexts and single-language task inputs limits its gains in general language understanding. This finding motivates a two-stage approach following the pretraining-fine-tuning paradigm: we first use PPT to build a cross-lingual foundation, then continue with SPT to adapt it to standard language modeling. In both high-resource and low-resource settings, the two-stage approach consistently improves general language understanding and cross-lingual task performance over standard pretraining alone at matched training budgets.
Figures & tables
Figure 1: Sequence construction and the motivation for PPT followed by SPT. (a) SPT samples documents independently; PPT joins translation pairs before sequence packing. Both use the same bilingual corpus. Letters identify content and EN/FR identify languages. (b) Continuing PPT with SPT improves general language understanding and translation over SPT at matched training budgets. The book pages illustrate these tasks using two HellaSwag continuations and an English–French translation excerpt.
Figure 2: English–French pretraining with SPT, PPT, and their staged combinations. SPT and PPT curves contain all ten checkpoints from 20B to 200B tokens. Each staged run branches from its initial method at 60B, 100B or 160B and trains for another 20B tokens, with evaluations every 5B tokens in both panels. Each branch is a separate training run. Accuracy averages six tasks and both languages equally. BLEU equally weights FLORES-200 and WMT14, both translation directions and the two prompt formats specified in Section 3.1 .
Language understanding (%)
Translation (BLEU)
Tokens
SPT
PPT
PPT → SPT
SPT → PPT
SPT
PPT
PPT → SPT
SPT → PPT
80B
45.88
45.77
46.79
46.86
17.79
0.24
29.46
20.18
120B
47.74
46.18
47.79
47.59
19.25
14.23
34.03
22.34
180B
47.24
47.64
47.64
48.26
22.29
20.68
35.77
33.32
Table 1: English–French results at matched training-token budgets. Accuracy averages six tasks and both languages equally. BLEU averages both datasets, directions and prompt formats equally. Bold marks the highest value for each budget and metric. Staged models at different budgets are trained independently.
Understanding (%)
BLEU
Model
EN
SW
Mean
SPT
53.08
45.69
49.38
9.07
PPT
51.12
45.02
48.07
12.28
PPT → SPT
52.51
47.72
50.11
13.18
SPT → PPT
54.24
45.98
50.11
1.54
Table 2: English–Swahili results at 120B tokens. Understanding scores average four tasks per language; Mean weights English and Swahili equally. BLEU averages both directions and both prompt formats on FLORES-200 (Table 14 ). Bold marks the highest score in each column.
Figure 3: Fixed-target prediction on the English–French WMT14 news test set. The left pair shows target-only loss, measuring monolingual prediction; the right pair shows corresponding-source benefit, the unrelated-minus-corresponding source loss. Each pair compares English and French targets on a common scale. Points average 173 document pairs in bits per byte; staged runs are independent. Table 6 gives differences from the corresponding SPT or PPT baseline at matched budgets, with 95% document-bootstrap intervals.
Figure 4: Corresponding-content use on the English–French WMT14 news test set at 120B. Each pair compares French and English targets. (a) Loss reduction after extending source sentences 1–3 to 1–8, with the target fixed. Circles denote target sentences 2–3, whose source counterparts were already present; squares denote sentences 4–8, whose counterparts are added. (b) Loss increase on target sentences 4–8 after blocking attention to their source counterparts in the middle eight layers (9–16). Both effects subtract the same operation with an unrelated source. Points average 133 document pairs in nats per target token, with 95% document-bootstrap intervals. Appendix C.3 illustrates the operations and reports all three layer groups.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
SPT or PPT alone
Staged continuation
Parameters
1.3B
Same architecture
Layers / attention heads
24 / 16
24 / 16
Hidden / feed-forward width
2,048 / 5,504
2,048 / 5,504
Position encoding / activation / normalization
RoPE / SwiGLU / RMSNorm
Same
Embeddings / dropout
Untied / none
Same
Optimizer
AdamW, β₁=0.9, β₂=0.95
Same optimizer family
Appendix
Table 3: Model and training settings. Global batch size counts sequences; sequence length is in tokens.
Task
Language
Items
Split
Sampled examples
Metric
ARC-Challenge
English
1,172
test
5
acc_norm
mARC
French
1,169
test
5
acc_norm
HellaSwag
English
10,042
validation
5
acc_norm
mHellaSwag
French
9,338
val
5
acc_norm
TruthfulQA MC1
English
817
validation
0
acc
mTruthfulQA MC1
French
787
val
5
acc
Appendix
Table 4: Language-understanding evaluations. Task names link to the dataset releases. Item counts are per language. Sampled examples exclude the fixed question–answer prefix in the TruthfulQA templates.
Training order
Target language
Training tokens
Target-only loss reduction
Corresponding-source benefit
[95% CI]
Change [95% CI]
SPT/PPT alone
Combined training
SPT → PPT
FR
80B
0.011 [0.008, 0.015]
0.085 [0.077, 0.091]
0.354
0.439
SPT → PPT
FR
120B
0.015 [0.011, 0.019]
0.096 [0.088, 0.104]
0.345
0.442
SPT → PPT
FR
180B
0.019 [0.015, 0.023]
0.083 [0.076, 0.090]
0.347
0.431
SPT → PPT
EN
80B
0.015 [0.011, 0.019]
0.106 [0.096, 0.117]
0.359
0.465
SPT → PPT
EN
120B
0.023 [0.019, 0.027]
0.113 [0.105, 0.122]
0.346
0.459
Appendix
Table 6: Prediction comparisons on 173 English–French WMT14 documents at matched training budgets. Baselines are SPT for SPT → PPT and PPT for PPT → SPT. Positive differences denote lower target-only loss or a larger corresponding-source benefit. The last two columns give the baseline and combined-training benefit values. All values are in bits per byte; intervals are 95% document-bootstrap intervals.
Figure 5: Input and attention interventions for Figure 4 . Numbered boxes denote aligned sentences. (a) The target is fixed while source sentences 4–8 are added in blue. (b) The complete input remains present; the crossed connection denotes blocked attention from target prediction positions to source sentences 4–8. Both operations are repeated with an unrelated source.
Figure 6: Loss increase after blocking attention from positions predicting target sentences 4–8 to source sentences 4–8 in each layer group separately. Points average 133 English–French WMT14 documents at 120B, in nats per target token, with 95% document-bootstrap intervals. The effect of blocking an unrelated source is subtracted. Both target languages share a scale. Layers 9–16 also appear in Figure 4 (b).
Intervention
SPT
PPT
PPT → SPT
SPT → PPT
Swahili target, source extension
Target sentences 2–3
− 0.071
− 0.002
− 0.022
− 0.009
Target sentences 4–8
0.372
0.686
0.622
0.671
Swahili target, attention blocking
Layers 1–8
0.006
0.030
0.020
0.003
Layers 9–16
0.273
0.604
0.547
0.533
Appendix
Table 8: Corresponding-content interventions on 61 English–Swahili AfriDocMT documents at 120B. Source extension reports loss reduction for target sentences whose source counterparts were already present (2–3) or added (4–8). Attention blocking reports loss increase when prediction positions for target sentences 4–8 cannot attend to source sentences 4–8. Both effects subtract the unrelated-source effect and average documents equally, in nats per target token.
Figure 7: Answer changes from 100B to 120B, as percentages of all 10,042 English HellaSwag or 9,338 French mHellaSwag questions. Corrected minus lost gives the accuracy change. Connected pairs share a 100B starting model.
Language
120B model
Corrected answers
Lost answers
Retained correct
Starting from SPT at 100B
EN
SPT
3.51
3.90
60.62
SPT → PPT
3.38
2.45
62.08
SW
SPT
3.04
3.11
52.81
SPT → PPT
2.65
2.25
53.67
Mean
SPT
3.28
3.51
56.72
Appendix
Table 9: XStoryCloze answer changes on 1,511 paired English–Swahili stories. Adjacent rows compare continued and switched training from the same 100B model. Values are percentages of all questions; Mean averages the two languages equally.
Model
Func.
Visual
Other
Encycl.
Taxon.
Groups, n
3,399
4,072
767
3,240
857
SPT 120B
77.76
73.37
70.24
74.28
86.29
SPT→PPT 120B
79.76
74.83
71.77
76.38
87.25
PPT 120B
70.40
67.15
60.98
66.10
75.32
PPT→SPT 120B
77.90
73.37
70.53
73.46
83.17
Appendix
Table 10: COMPS-BASE accuracy (%) on English minimal pairs for the 120B English–French models. Columns cover functional, visual perceptual, other perceptual, encyclopedic and taxonomic properties. The Groups row counts concept–property groups; each contributes four negative-concept comparisons.
Table 13: English–Swahili task accuracy (%) before (100B) and after (120B) continuation. Mean averages four tasks per language.
Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain 360M- and 7B-parameter LLMs and show that poor cross-lingual knowledge generalization emerges during pretraining and persists under standard interventions. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross-lingual knowledge generalization. Guided by this understanding, we suggest mapping languages into a shared token space by simple word-wise translation and find it substantially improves cross-lingual knowledge generalization, recovering up to 12.6% of native-language learning efficiency --- 14× the baseline.
Adam Gaber, Uriel Dolev, Elisabeth Fittschen +3
Weizmann Institute of Science · Bar-Ilan University · Johns Hopkins University +4
Large language models exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We use word-level translation as a testbed, introducing a novel dataset to trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining.
Felicia Körner, Maria Matveev, Florian Eichin +3
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML) · Department of Mathematics, LMU Munich, Germany +2
Shared multilingual representations are essential for cross-lingual tasks and knowledge transfer across languages. This study looks at the impact of parallel data, i.e. translated sentences, in pretraining as a signal to trigger representations that are aligned across languages. We train reference models with different proportions of parallel data and show that parallel data seem to have only a minimal effect on the cross-lingual alignment. Based on multiple evaluation methods, we find that the effect is limited to potentially accelerating the representation sharing in the early phases of pretraining, and to decreasing the amount of language-specific neurons in the model. Cross-lingual alignment seems to emerge on similar levels even without the explicit signal from parallel data.