LLMs Get Smarter from Targeted Synthetic Multilingual Data

Ishika Agarwal; Arkajyoti Charaborty; Tanner Sorensen; Neha Gupta; Andreas Stolcke

cs.CLAug 16, 2026

LLMs Get Smarter from Targeted Synthetic Multilingual Data

Authors: Ishika Agarwal, Arkajyoti Charaborty, Tanner Sorensen, Neha Gupta, Andreas Stolcke

Organizations: UIUC · Uniphore

Abstract

Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt. In other words, a language model outputs different (and potentially incorrect) responses to the same semantic query when prompted in different languages. Prior work attributes this to an internal misalignment of semantic representation across languages. Currently, there are two main approaches to address LSC in the literature: (1) routing all queries through English, improving performance, but limiting language expressivity to English; or (2) training on language-balanced data, equalizing model performance across languages, but reducing overall performance. In this work, we take a data centric perspective and introduce HOTFIXR: Hardness Optimized Training data For Improving X-Lingual Reasoning. It is a data generation framework that uses models to probe and learn a student model's multilingual weaknesses, and generates data to mitigate them. HOTFIXR can generate multilingual synthetic training data that can improve multilingual performance. We evaluate on three in-distribution tasks, three out-of-distribution tasks, and four out-of-distribution languages. On average, HOTFIXR (1) improves in-distribution performance by 6.2%, (2) reduces catastrophic forgetting (induced by fine-tuning) on OOD tasks by 3.7%, and (3) on OOD languages by 7.1%. Overall, as many real-world applications requires multilingual LLMs, our work contributes to the efforts of making LLMs multilingually proficient. We will release code upon acceptance.

Figures & tables

	Performance		Language
Method	ID ( $\uparrow$ )	OOD ( $\uparrow$ )	Spread ( $\downarrow$ )
Base	51.9	65.6	9.6
EngReason	50.8	65.4	8.8
SelectionGT	49.2	60.9	9.8
SelectionGEN	50.2	55.8	9.5
Filtered	48.9	58.2	10.6

Table 1: H O T F I X R has the best in-distribution performance, the best out-of-distribution performance among the training-based baselines, and remains multilingually consistent. This table shows the cross-lingual consistency with respect to ID and OOD performance. Language spread is the standard deviation of performance across languages. Low spread indicates more consistent performance across languages. Note: the training-based baselines are SelectionGT , SelectionGEN , Filtered , Untrained , and DataEnvGym .

Figure 1: Motivating results for LSC: the performance of Aya (with a more language-balanced pretraining dataset) versus Qwen (with a more English-dominant pretraining dataset) on multilingual HotPotQA ( Li et al., 2026 ) . We see that although Aya performs more consistently across languages, Aya degrades in performance (see Qwen’s performance in English).

Figure 2: Intuition behind H O T F I X R . The “data space” represents the data samples generated by the question generator. Essentially, we want to hill climb through the data space to find samples that have high lingual deficit scores, as they will be most informative to the student model. By fine-tuning the question generation model to generate samples according to the student model’s lingual deficit, we can improve the information embedded within data, improving our models on downstream tasks.

Figure 3: An illustration of how to train the question generation model in H O T F I X R , along with the reward design.

Figure 4: Performance of data curation methods in-distribution .

Figure 5: Performance of data curation methods for factual (MMMLU) queries, an out-of-distribution task.

Figure 6: Performance of data curation methods for RAG (multilingual HotPotQA) queries, an out-of-distribution task.

Figure 7: Performance of data curation methods for translation (OPUS-100) queries, an out-of-distribution task.

	In-Distribution		Out-of-Distribution
Baseline	$\Delta$	Wins	$\Delta$	Wins
Base	+ 4.3	9/9	- 0.9	3/9
EngReason	+ 5.4	9/9	- 0.7	4/9
SelectionGT	+ 7.0	9/9	+ 3.8	9/9
SelectionGEN	+ 6.0	9/9	+ 8.9	8/9
Filtered	+ 7.3	9/9	+ 6.6	8/9

Table 2: Average performance difference of H O T F I X R versus each baseline. The average is over nine settings: three students (Qwen 7B, Qwen 14B, Llama 8B)

\times

three ID/OOD tasks. “Wins” counts the number of settings where H O T F I X R scores higher.

Method	De	Ja	Ru	Zh	Avg
SelectionGT	-3.3	-4.7	-9.5	-10.5	-7.0
SelectionGEN	-1.9	-4.0	-22.5	-21.5	-12.5
Filtered	-3.8	-4.8	-14.7	-16.3	-9.9
Untrained	-2.9	-2.0	-2.3	-5.3	-3.1
DataEnvGym	-6.8	-6.0	-12.2	-15.4	-10.1
H O T F I X R	-2.6	-0.7	-0.9	-1.3	-1.4

Table 3: OOD language generalization: performance differences relative to the Base model, averaged across all models and corresponding tasks. Averaging the difference in H O T F I X R and other methods, we can compute how much H O T F I X R avoids catastrophic forgetting. For example, averaged across all languages, H O T F I X R avoids catastrophic forgetting by 8.7% compared to DataEnvGym (10.1-1.4).

Method	Range	Trim. Range	Std	IQR	CV
Base	20.2	16.0	7.2	9.6	0.12
EngReason	20.7	15.4	7.2	8.8	0.12
SelectionGT	22.3	17.1	7.9	9.8	0.13
SelectionGEN	18.8	14.9	6.9	9.5	0.13
Filtered	22.6	17.1	8.0	10.6	0.14
Untrained	20.1	15.0	7.1	9.7	0.12

Table 4: Cross-lingual spread of student performance, averaged over three student models (Qwen2.5-7B, Qwen2.5-14B, Llama-3.1-8B) and three multilingual tasks (OPUS, MMMLU, mHotpotQA) evaluated with each task’s primary metric. Lower is more consistent for all dispersion columns; Bold marks the best value among data-generation methods. Range is the max

-

min performance across all languages. Trimmed Range is the best minus the 2nd worst language accuracy. Std is standard deviation of the performance. IQR is the Inter-Quartile Range (difference in the 25% and 75% quartile). CV is the coeffiecient of variation (the standard devation divided by the mean).

		ID	OOD
Model	Config	Nemo.	Trans.	Fact.	RAG
Qwen 7B	Format	52.0	57.6	64.0	75.8
	+ LAI	50.9	57.9	64.3	77.2
	+ LSI	50.8	57.8	63.6	77.7
	+ LAI + LSI	57.8	59.1	64.3	76.7
Llama 8B	Format	48.2	57.8	49.9	70.2

Table 5: Reward-signal ablation. All rows include the format reward. “Nemo.” is Nemotron, “Trans.” is Translation, “Fact.” is Factual. Highlights the impact each reward has—there is more effect of the reward functions combined, than individually. LAI is the “Language Agnostic Incompetency” and LSI is the “Language Specific Incompetency”, as described in Section 3 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	Qwen 7B		Qwen 14B		Llama 8B
Method	min	max	min	max	min	max
Base	0.30	1.50	0.10	0.40	0.20	2.00
EngReason	0.25	1.23	0.16	0.75	0.34	1.00
SelectionGT	0.20	3.40	0.10	1.00	0.40	2.20
SelectionGEN	0.30	3.10	0.00	0.90	0.10	2.20
Filtered	0.70	0.90	0.30	0.80	0.30	1.70

Appendix

Table 6: Minimum and maximum standard deviation (across 3 runs) for the Nemotron task (STEM, MATH, and CHAT), per model and method. This explains the small error bars in Figure 8 .

	Qwen 7B		Qwen 14B		Llama 8B
Method	min	max	min	max	min	max
Base	0.43	0.85	0.41	1.55	0.58	1.56
EngReason	0.40	0.83	0.43	1.05	0.55	1.56
SelectionGT	0.10	1.40	0.20	1.00	0.40	1.60
SelectionGEN	0.00	0.80	0.00	1.50	0.10	1.50
Filtered	0.10	1.60	0.10	0.80	0.10	1.60

Appendix

Table 7: Minimum and maximum standard deviation (across 3 runs) for the Translation task (OPUS-100), per model and method. This explains the small error bars in Figure 10 .

	Qwen 7B		Qwen 14B		Llama 8B
Method	min	max	min	max	min	max
Base	0.10	1.50	0.10	1.10	0.00	2.30
EngReason	0.09	1.27	0.28	1.09	0.33	1.73
SelectionGT	0.30	2.30	0.20	1.80	0.10	1.60
SelectionGEN	0.40	2.60	0.10	1.80	0.10	2.30
Filtered	0.10	1.30	0.20	1.20	0.00	1.20

Appendix

Table 8: Minimum and maximum standard deviation (across 3 runs) for the Factual task (MMMLU), per model and method. This explains the small error bars in Figure 9 .

	Qwen 7B		Qwen 14B		Llama 8B
Method	min	max	min	max	min	max
Base	0.10	0.90	0.10	0.80	0.80	2.50
EngReason	0.33	0.99	0.16	1.05	0.53	1.71
SelectionGT	0.30	1.20	0.10	0.80	0.30	1.80
SelectionGEN	0.80	1.40	0.30	1.50	0.00	2.50
Filtered	0.10	1.10	0.40	1.50	0.00	1.30

Appendix

Table 9: Minimum and maximum standard deviation (across 3 runs) for the RAG task (mHotPotQA), per model and method. This explains the small error bars in Figure 11 .

Figure 8: Performance of data curation methods for Nemotron queries, per task.

Figure 9: Performance of data curation methods for factual (MMMLU) queries, per language.

Figure 10: Performance of data curation methods for translation (OPUS-100) queries, per language.

Figure 11: Performance of data curation methods for RAG (multilingual HotPotQA) queries, per language.

Qwen2.5-7B-Instruct
Method	Nemotron	Factual	Translation	RAG
Base	52.3	67.0	59.2	79.0
EngReason	51.2	68.2	59.0	79.3
SelectionGT	48.5	63.9	53.8	61.5
SelectionGEN	50.3	59.9	56.8	41.4
Filtered	48.6	61.7	56.6	66.5

Appendix

Table 10: Average performance per model, with the added Distillation method. Here, we are able to show that a lot of H O T F I X R ’s empirical success comes from the targeted question generation, rather than having correct, distilled labels from a large model.

Panel of Figure 12 — Figure 12: The delta in performance of student models trained with two rounds of H O T F I X R versus one round. The improvement with two rounds is much more than the improvement with one round.

Panel of Figure 13 — Figure 13: Prompt used during H O T F I X R training and data generation, for the Nemotron MATH subset. As input, the prompt takes a question , answer , and reasoning from the Nemotron MATH training dataset, and a language that alternates between English, French, Spanish, Arabic, Portuguese, or Italian.

Panel of Figure 14 — Figure 14: Prompt used during H O T F I X R training and data generation, for the Nemotron STEM subset. As input, the prompt takes a question , answer , and reasoning from the Nemotron STEM training dataset, and a language that alternates between English, French, Spanish, Arabic, Portuguese, or Italian.

Panel of Figure 15 — Figure 15: Prompt used during H O T F I X R training and data generation, for the Nemotron CHAT subset. As input, the prompt takes a question , answer , and reasoning from the Nemotron CHAT training dataset, and a language that alternates between English, French, Spanish, Arabic, Portuguese, or Italian.

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Apr 13, 2026

Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. However, teacher model selection is often ad hoc, typically defaulting to the largest available option, even though such models may have significant capability gaps in non-English languages. This practice can result in poor-quality synthetic data and suboptimal student downstream performance. In this work, we systematically characterize what makes an effective multilingual teacher. We combine intrinsic measures of data quality with extrinsic student model performance in a metric we call Polyglot Score. We evaluate 10 LMs across 6 typologically diverse languages, generating over 1.4M SFT examples and training 240 student models. Our analyses reveal that model scale alone does not significantly predict teacher effectiveness: the most effective teachers we identify are consistently smaller than the largest models evaluated, and their ranking is stable across student base model families. Instead, data qualities such as prompt diversity, length, and response fluency capture 93.3% of the variance in intrinsic data quality and predict student performance. Finally, we provide practical recommendations, including matching the model families of teacher-student pairs and generating responses to existing prompts or translating them from English, which can yield improvements for less-resourced languages. We hope that our work advances data-centric research in multilingual synthetic data and LM development.

Lester James V. Miranda, Ivan Vulić, Anna Korhonen

Language Technology Lab, University of Cambridge

Supervised Fine-Tuning Multilingual Language Models Multilingual Language Model Evaluation Language Model Generation Evaluation Synthetic Data Generation Synthetic Data Generation for Language Models+1

Ask PDF

Apr 14, 2026

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Despite advances in multilingual capabilities, most large language models (LLMs) remain English-centric in their training and, crucially, in the reasoning traces they produce: even for non-English problems, they predominantly reason in English, leaving non-English users with intermediate reasoning they cannot follow. We address this disparity with two contributions. (a) We introduce

\texttt{ReasonXL}

, the first large-scale parallel corpus of cross-domain reasoning traces in five European languages (English, German, French, Italian, and Spanish): over 2.5M aligned samples (11B tokens) per language, each comprising prompt, reasoning trace, and final output, enabling direct supervision of language-specific reasoning. (b) Using

\texttt{ReasonXL}

, we demonstrate that LLMs can be adapted to reason entirely in a target language through a simple two-stage pipeline of supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), matching or exceeding the base models' English-reasoning performance in two of four languages and recovering most of it in the remaining two.

Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt +3

German Research Center for Artificial Intelligence (DFKI), Saarland University · Berliner Hochschule für Technik

Cross-Lingual Reasoning in Language Models Multilingual Language Models

Ask PDF

Mar 26, 2026

Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties

Recent strategies for low-resource machine translation rely on LLMs to generate synthetic data based on text in higher-resource languages. We revisit this idea for Romansh, a language with 6 distinct varieties. LLMs tend to confuse these varieties when translating into Romansh, but they are quite good at translating out of Romansh into a high-resource language such as German. Due to this asymmetry, the direction of data augmentation is a crucial choice. We find that contrary to recent strategies, creating synthetic translations into the higher-resource language is the superior approach, and only this approach allows us to surpass a Gemini 3 Pro baseline on German-Romansh translation (+23 BLEU over Gemini in the lowest-resource variety). A human evaluation confirms that our experiments yield the first model that generates fluent translations in the individual Romansh varieties.

Jannis Vamvas, Ignacio Pérez Prat, Angela Heldstab +3

University of Zurich · Lia Rumantscha

Machine Translation Low-Resource MT Large Language Model-Based Data Augmentation Synthetic Data Generation for Language Models

Ask PDF

Abstract

Multilingual Language Models

Explore similar work

Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties