How to Use Parallel Data for Multilingual Pretraining

Zhihong Huang; Yao Lu; Derek F. Wong; Hamid Alinejad-Rokny; Min Yang

cs.CLOct 20, 2026

How to Use Parallel Data for Multilingual Pretraining

Authors: Zhihong Huang, Yao Lu, Derek F. Wong, Hamid Alinejad-Rokny, Min Yang

Organizations: University of Macau · University College London · UNSW Sydney · SIAT, CAS

Abstract

High-quality human-translated parallel corpora can improve cross-lingual transfer, but are typically too small for large-scale pretraining. Machine-translated web corpora help overcome this limitation, making parallel data available at pretraining scale. Standard multilingual pretraining, however, samples source documents and their translations independently, without explicitly using their parallel structure. We investigate how to use the parallel structure of the data to improve multilingual pretraining by concatenating each document with its translation in the same training sequence. We compare this parallel pretraining (PPT) approach with standard pretraining (SPT), training models from scratch at matched token budgets. Surprisingly, despite the benefits of parallel data reported in prior cross-lingual work, PPT alone does not consistently improve language understanding over SPT. Our further analysis suggests that although PPT builds a strong cross-lingual foundation, as shown by its better use of shared information across languages, the mismatch between parallel training contexts and single-language task inputs limits its gains in general language understanding. This finding motivates a two-stage approach following the pretraining-fine-tuning paradigm: we first use PPT to build a cross-lingual foundation, then continue with SPT to adapt it to standard language modeling. In both high-resource and low-resource settings, the two-stage approach consistently improves general language understanding and cross-lingual task performance over standard pretraining alone at matched training budgets.

Figures & tables

Figure 1: Sequence construction and the motivation for PPT followed by SPT. (a) SPT samples documents independently; PPT joins translation pairs before sequence packing. Both use the same bilingual corpus. Letters identify content and EN/FR identify languages. (b) Continuing PPT with SPT improves general language understanding and translation over SPT at matched training budgets. The book pages illustrate these tasks using two HellaSwag continuations and an English–French translation excerpt.

Figure 2: English–French pretraining with SPT, PPT, and their staged combinations. SPT and PPT curves contain all ten checkpoints from 20B to 200B tokens. Each staged run branches from its initial method at 60B, 100B or 160B and trains for another 20B tokens, with evaluations every 5B tokens in both panels. Each branch is a separate training run. Accuracy averages six tasks and both languages equally. BLEU equally weights FLORES-200 and WMT14, both translation directions and the two prompt formats specified in Section 3.1 .

	Language understanding (%)				Translation (BLEU)
Tokens	SPT	PPT	PPT $\to$ SPT	SPT $\to$ PPT	SPT	PPT	PPT $\to$ SPT	SPT $\to$ PPT
80B	45.88	45.77	46.79	46.86	17.79	0.24	29.46	20.18
120B	47.74	46.18	47.79	47.59	19.25	14.23	34.03	22.34
180B	47.24	47.64	47.64	48.26	22.29	20.68	35.77	33.32

Table 1: English–French results at matched training-token budgets. Accuracy averages six tasks and both languages equally. BLEU averages both datasets, directions and prompt formats equally. Bold marks the highest value for each budget and metric. Staged models at different budgets are trained independently.

	Understanding (%)			BLEU
Model	EN	SW	Mean
SPT	53.08	45.69	49.38	9.07
PPT	51.12	45.02	48.07	12.28
PPT $\to$ SPT	52.51	47.72	50.11	13.18
SPT $\to$ PPT	54.24	45.98	50.11	1.54

Table 2: English–Swahili results at 120B tokens. Understanding scores average four tasks per language; Mean weights English and Swahili equally. BLEU averages both directions and both prompt formats on FLORES-200 (Table 14 ). Bold marks the highest score in each column.

Figure 3: Fixed-target prediction on the English–French WMT14 news test set. The left pair shows target-only loss, measuring monolingual prediction; the right pair shows corresponding-source benefit, the unrelated-minus-corresponding source loss. Each pair compares English and French targets on a common scale. Points average 173 document pairs in bits per byte; staged runs are independent. Table 6 gives differences from the corresponding SPT or PPT baseline at matched budgets, with 95% document-bootstrap intervals.

Figure 4: Corresponding-content use on the English–French WMT14 news test set at 120B. Each pair compares French and English targets. (a) Loss reduction after extending source sentences 1–3 to 1–8, with the target fixed. Circles denote target sentences 2–3, whose source counterparts were already present; squares denote sentences 4–8, whose counterparts are added. (b) Loss increase on target sentences 4–8 after blocking attention to their source counterparts in the middle eight layers (9–16). Both effects subtract the same operation with an unrelated source. Points average 133 document pairs in nats per target token, with 95% document-bootstrap intervals. Appendix C.3 illustrates the operations and reports all three layer groups.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	SPT or PPT alone	Staged continuation
Parameters	1.3B	Same architecture
Layers / attention heads	24 / 16	24 / 16
Hidden / feed-forward width	2,048 / 5,504	2,048 / 5,504
Position encoding / activation / normalization	RoPE / SwiGLU / RMSNorm	Same
Embeddings / dropout	Untied / none	Same
Optimizer	AdamW, β₁=0.9, β₂=0.95	Same optimizer family

Appendix

Table 3: Model and training settings. Global batch size counts sequences; sequence length is in tokens.

Task	Language	Items	Split	Sampled examples	Metric
ARC-Challenge	English	1,172	test	5	acc_norm
mARC	French	1,169	test	5	acc_norm
HellaSwag	English	10,042	validation	5	acc_norm
mHellaSwag	French	9,338	val	5	acc_norm
TruthfulQA MC1	English	817	validation	0	acc
mTruthfulQA MC1	French	787	val	5	acc

Appendix

Table 4: Language-understanding evaluations. Task names link to the dataset releases. Item counts are per language. Sampled examples exclude the fixed question–answer prefix in the TruthfulQA templates.

Training order	Target language	Training tokens	Target-only loss reduction	Corresponding-source benefit
			[95% CI]	Change [95% CI]	SPT/PPT alone	Combined training
SPT $\to$ PPT	FR	80B	0.011 [0.008, 0.015]	0.085 [0.077, 0.091]	0.354	0.439
SPT $\to$ PPT	FR	120B	0.015 [0.011, 0.019]	0.096 [0.088, 0.104]	0.345	0.442
SPT $\to$ PPT	FR	180B	0.019 [0.015, 0.023]	0.083 [0.076, 0.090]	0.347	0.431
SPT $\to$ PPT	EN	80B	0.015 [0.011, 0.019]	0.106 [0.096, 0.117]	0.359	0.465
SPT $\to$ PPT	EN	120B	0.023 [0.019, 0.027]	0.113 [0.105, 0.122]	0.346	0.459

Appendix

Table 6: Prediction comparisons on 173 English–French WMT14 documents at matched training budgets. Baselines are SPT for SPT

\to

PPT and PPT for PPT

\to

SPT. Positive differences denote lower target-only loss or a larger corresponding-source benefit. The last two columns give the baseline and combined-training benefit values. All values are in bits per byte; intervals are 95% document-bootstrap intervals.

Figure 5: Input and attention interventions for Figure 4 . Numbered boxes denote aligned sentences. (a) The target is fixed while source sentences 4–8 are added in blue. (b) The complete input remains present; the crossed connection denotes blocked attention from target prediction positions to source sentences 4–8. Both operations are repeated with an unrelated source.

Figure 6: Loss increase after blocking attention from positions predicting target sentences 4–8 to source sentences 4–8 in each layer group separately. Points average 133 English–French WMT14 documents at 120B, in nats per target token, with 95% document-bootstrap intervals. The effect of blocking an unrelated source is subtracted. Both target languages share a scale. Layers 9–16 also appear in Figure 4 (b).

Swahili target, source extension
Intervention	SPT	PPT	PPT $\to$ SPT	SPT $\to$ PPT
Target sentences 2–3	$-$ 0.071	$-$ 0.002	$-$ 0.022	$-$ 0.009
Target sentences 4–8	0.372	0.686	0.622	0.671
Swahili target, attention blocking
Layers 1–8	0.006	0.030	0.020	0.003
Layers 9–16	0.273	0.604	0.547	0.533

Appendix

Table 8: Corresponding-content interventions on 61 English–Swahili AfriDocMT documents at 120B. Source extension reports loss reduction for target sentences whose source counterparts were already present (2–3) or added (4–8). Attention blocking reports loss increase when prediction positions for target sentences 4–8 cannot attend to source sentences 4–8. Both effects subtract the unrelated-source effect and average documents equally, in nats per target token.

Figure 7: Answer changes from 100B to 120B, as percentages of all 10,042 English HellaSwag or 9,338 French mHellaSwag questions. Corrected minus lost gives the accuracy change. Connected pairs share a 100B starting model.

Language	120B model	Corrected answers	Lost answers	Retained correct
Starting from SPT at 100B
EN	SPT	3.51	3.90	60.62
	SPT $\to$ PPT	3.38	2.45	62.08
SW	SPT	3.04	3.11	52.81
	SPT $\to$ PPT	2.65	2.25	53.67
Mean	SPT	3.28	3.51	56.72

Appendix

Table 9: XStoryCloze answer changes on 1,511 paired English–Swahili stories. Adjacent rows compare continued and switched training from the same 100B model. Values are percentages of all questions; Mean averages the two languages equally.

Model	Func.	Visual	Other	Encycl.	Taxon.
Groups, $n$	3,399	4,072	767	3,240	857
SPT 120B	77.76	73.37	70.24	74.28	86.29
SPT→PPT 120B	79.76	74.83	71.77	76.38	87.25
PPT 120B	70.40	67.15	60.98	66.10	75.32
PPT→SPT 120B	77.90	73.37	70.53	73.46	83.17

Appendix

Table 10: COMPS-BASE accuracy (%) on English minimal pairs for the 120B English–French models. Columns cover functional, visual perceptual, other perceptual, encyclopedic and taxonomic properties. The Groups row counts concept–property groups; each contributes four negative-concept comparisons.

Table 13: English–Swahili task accuracy (%) before (100B) and after (120B) continuation. Mean averages four tasks per language.

Language Model Pretraining

Language model pretraining teaches models language patterns and knowledge from large text corpora before they are adapted to specific tasks.

See all 441 papers

Why Pretraining Fails to Share Cross-Lingual Knowledge
Sep 16, 2026Adam Gaber, Uriel Dolev, Elisabeth Fittschen +3Language Model Pretraining Multilingual Language Models
Weizmann Institute of Science · Bar-Ilan University · Johns Hopkins University +4
Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
Apr 19, 2026Felicia Körner, Maria Matveev, Florian Eichin +3Language Model Pretraining Multilingual Language Models
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML) · Department of Mathematics, LMU Munich, Germany +2
On the limited utility of parallel data for learning shared multilingual representations
Mar 30, 2026Julius Leino, Jörg TiedemannCross-Lingual Representation Alignment Multilingual Language Models
University of Helsinki

Sep 16, 2026J/K move · Enter open · S save

Why Pretraining Fails to Share Cross-Lingual Knowledge

Adam Gaber, Uriel Dolev, Elisabeth Fittschen, Bobby Cheng+2

Weizmann Institute of Science · Bar-Ilan University · Johns Hopkins University+4

Language Model Pretraining Multilingual Language Models Cross-Lingual Transfer

Open paper AskPDF arXiv

Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exhibit surprisingly limited cross-lingual knowledge transfer. While this limitation is well documented, its origins during multilingual training remain unclear. We pretrain 360M- and 7B-parameter LLMs and show that poor cross-lingual knowledge generalization emerges during pretraining and persists under standard interventions. To isolate its cause, we employ a controlled bilingual pretraining setting using two copies of the same language, sharing identical text and token segmentation, but mapped to disjoint token spaces. We find that disjoint tokens alone are enough to induce knowledge compartmentalization, even between identical copies of the same language, establishing disjoint token spaces as a fundamental barrier to cross-lingual knowledge generalization. Guided by this understanding, we suggest mapping languages into a shared token space by simple word-wise translation and find it substantially improves cross-lingual knowledge generalization, recovering up to 12.6% of native-language learning efficiency --- 14

\times

the baseline.

Abstract

Language Model Pretraining

Explore similar work

Why Pretraining Fails to Share Cross-Lingual Knowledge

Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

On the limited utility of parallel data for learning shared multilingual representations