LLMs Get Smarter from Targeted Synthetic Multilingual Data

Ishika Agarwal; Arkajyoti Charaborty; Tanner Sorensen; Neha Gupta; Andreas Stolcke

cs.CLAug 16, 2026

LLMs Get Smarter from Targeted Synthetic Multilingual Data

Authors: Ishika Agarwal, Arkajyoti Charaborty, Tanner Sorensen, Neha Gupta, Andreas Stolcke

Organizations: UIUC · Uniphore

Abstract

Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt. In other words, a language model outputs different (and potentially incorrect) responses to the same semantic query when prompted in different languages. Prior work attributes this to an internal misalignment of semantic representation across languages. Currently, there are two main approaches to address LSC in the literature: (1) routing all queries through English, improving performance, but limiting language expressivity to English; or (2) training on language-balanced data, equalizing model performance across languages, but reducing overall performance. In this work, we take a data centric perspective and introduce HOTFIXR: Hardness Optimized Training data For Improving X-Lingual Reasoning. It is a data generation framework that uses models to probe and learn a student model's multilingual weaknesses, and generates data to mitigate them. HOTFIXR can generate multilingual synthetic training data that can improve multilingual performance. We evaluate on three in-distribution tasks, three out-of-distribution tasks, and four out-of-distribution languages. On average, HOTFIXR (1) improves in-distribution performance by 6.2%, (2) reduces catastrophic forgetting (induced by fine-tuning) on OOD tasks by 3.7%, and (3) on OOD languages by 7.1%. Overall, as many real-world applications requires multilingual LLMs, our work contributes to the efforts of making LLMs multilingually proficient. We will release code upon acceptance.

Figures & tables

	Performance		Language
Method	ID ( $\uparrow$ )	OOD ( $\uparrow$ )	Spread ( $\downarrow$ )
Base	51.9	65.6	9.6
EngReason	50.8	65.4	8.8
SelectionGT	49.2	60.9	9.8
SelectionGEN	50.2	55.8	9.5
Filtered	48.9	58.2	10.6

Table 1: H O T F I X R has the best in-distribution performance, the best out-of-distribution performance among the training-based baselines, and remains multilingually consistent. This table shows the cross-lingual consistency with respect to ID and OOD performance. Language spread is the standard deviation of performance across languages. Low spread indicates more consistent performance across languages. Note: the training-based baselines are SelectionGT , SelectionGEN , Filtered , Untrained , and DataEnvGym .

Figure 1: Motivating results for LSC: the performance of Aya (with a more language-balanced pretraining dataset) versus Qwen (with a more English-dominant pretraining dataset) on multilingual HotPotQA ( Li et al., 2026 ) . We see that although Aya performs more consistently across languages, Aya degrades in performance (see Qwen’s performance in English).

Figure 2: Intuition behind H O T F I X R . The “data space” represents the data samples generated by the question generator. Essentially, we want to hill climb through the data space to find samples that have high lingual deficit scores, as they will be most informative to the student model. By fine-tuning the question generation model to generate samples according to the student model’s lingual deficit, we can improve the information embedded within data, improving our models on downstream tasks.

Figure 3: An illustration of how to train the question generation model in H O T F I X R , along with the reward design.

Figure 4: Performance of data curation methods in-distribution .

Figure 5: Performance of data curation methods for factual (MMMLU) queries, an out-of-distribution task.

Figure 6: Performance of data curation methods for RAG (multilingual HotPotQA) queries, an out-of-distribution task.

Figure 7: Performance of data curation methods for translation (OPUS-100) queries, an out-of-distribution task.

	In-Distribution		Out-of-Distribution
Baseline	$\Delta$	Wins	$\Delta$	Wins
Base	+ 4.3	9/9	- 0.9	3/9
EngReason	+ 5.4	9/9	- 0.7	4/9
SelectionGT	+ 7.0	9/9	+ 3.8	9/9
SelectionGEN	+ 6.0	9/9	+ 8.9	8/9
Filtered	+ 7.3	9/9	+ 6.6	8/9

Table 2: Average performance difference of H O T F I X R versus each baseline. The average is over nine settings: three students (Qwen 7B, Qwen 14B, Llama 8B)

\times

three ID/OOD tasks. “Wins” counts the number of settings where H O T F I X R scores higher.

Method	De	Ja	Ru	Zh	Avg
SelectionGT	-3.3	-4.7	-9.5	-10.5	-7.0
SelectionGEN	-1.9	-4.0	-22.5	-21.5	-12.5
Filtered	-3.8	-4.8	-14.7	-16.3	-9.9
Untrained	-2.9	-2.0	-2.3	-5.3	-3.1
DataEnvGym	-6.8	-6.0	-12.2	-15.4	-10.1
H O T F I X R	-2.6	-0.7	-0.9	-1.3	-1.4

Table 3: OOD language generalization: performance differences relative to the Base model, averaged across all models and corresponding tasks. Averaging the difference in H O T F I X R and other methods, we can compute how much H O T F I X R avoids catastrophic forgetting. For example, averaged across all languages, H O T F I X R avoids catastrophic forgetting by 8.7% compared to DataEnvGym (10.1-1.4).

Method	Range	Trim. Range	Std	IQR	CV
Base	20.2	16.0	7.2	9.6	0.12
EngReason	20.7	15.4	7.2	8.8	0.12
SelectionGT	22.3	17.1	7.9	9.8	0.13
SelectionGEN	18.8	14.9	6.9	9.5	0.13
Filtered	22.6	17.1	8.0	10.6	0.14
Untrained	20.1	15.0	7.1	9.7	0.12

Table 4: Cross-lingual spread of student performance, averaged over three student models (Qwen2.5-7B, Qwen2.5-14B, Llama-3.1-8B) and three multilingual tasks (OPUS, MMMLU, mHotpotQA) evaluated with each task’s primary metric. Lower is more consistent for all dispersion columns; Bold marks the best value among data-generation methods. Range is the max

-

min performance across all languages. Trimmed Range is the best minus the 2nd worst language accuracy. Std is standard deviation of the performance. IQR is the Inter-Quartile Range (difference in the 25% and 75% quartile). CV is the coeffiecient of variation (the standard devation divided by the mean).

		ID	OOD
Model	Config	Nemo.	Trans.	Fact.	RAG
Qwen 7B	Format	52.0	57.6	64.0	75.8
	+ LAI	50.9	57.9	64.3	77.2
	+ LSI	50.8	57.8	63.6	77.7
	+ LAI + LSI	57.8	59.1	64.3	76.7
Llama 8B	Format	48.2	57.8	49.9	70.2

Table 5: Reward-signal ablation. All rows include the format reward. “Nemo.” is Nemotron, “Trans.” is Translation, “Fact.” is Factual. Highlights the impact each reward has—there is more effect of the reward functions combined, than individually. LAI is the “Language Agnostic Incompetency” and LSI is the “Language Specific Incompetency”, as described in Section 3 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

	Qwen 7B		Qwen 14B		Llama 8B
Method	min	max	min	max	min	max
Base	0.30	1.50	0.10	0.40	0.20	2.00
EngReason	0.25	1.23	0.16	0.75	0.34	1.00
SelectionGT	0.20	3.40	0.10	1.00	0.40	2.20
SelectionGEN	0.30	3.10	0.00	0.90	0.10	2.20
Filtered	0.70	0.90	0.30	0.80	0.30	1.70

Appendix

Table 6: Minimum and maximum standard deviation (across 3 runs) for the Nemotron task (STEM, MATH, and CHAT), per model and method. This explains the small error bars in Figure 8 .

	Qwen 7B		Qwen 14B		Llama 8B
Method	min	max	min	max	min	max
Base	0.43	0.85	0.41	1.55	0.58	1.56
EngReason	0.40	0.83	0.43	1.05	0.55	1.56
SelectionGT	0.10	1.40	0.20	1.00	0.40	1.60
SelectionGEN	0.00	0.80	0.00	1.50	0.10	1.50
Filtered	0.10	1.60	0.10	0.80	0.10	1.60

Appendix

Table 7: Minimum and maximum standard deviation (across 3 runs) for the Translation task (OPUS-100), per model and method. This explains the small error bars in Figure 10 .

	Qwen 7B		Qwen 14B		Llama 8B
Method	min	max	min	max	min	max
Base	0.10	1.50	0.10	1.10	0.00	2.30
EngReason	0.09	1.27	0.28	1.09	0.33	1.73
SelectionGT	0.30	2.30	0.20	1.80	0.10	1.60
SelectionGEN	0.40	2.60	0.10	1.80	0.10	2.30
Filtered	0.10	1.30	0.20	1.20	0.00	1.20

Appendix

Table 8: Minimum and maximum standard deviation (across 3 runs) for the Factual task (MMMLU), per model and method. This explains the small error bars in Figure 9 .

	Qwen 7B		Qwen 14B		Llama 8B
Method	min	max	min	max	min	max
Base	0.10	0.90	0.10	0.80	0.80	2.50
EngReason	0.33	0.99	0.16	1.05	0.53	1.71
SelectionGT	0.30	1.20	0.10	0.80	0.30	1.80
SelectionGEN	0.80	1.40	0.30	1.50	0.00	2.50
Filtered	0.10	1.10	0.40	1.50	0.00	1.30

Appendix

Table 9: Minimum and maximum standard deviation (across 3 runs) for the RAG task (mHotPotQA), per model and method. This explains the small error bars in Figure 11 .

Figure 8: Performance of data curation methods for Nemotron queries, per task.

Figure 9: Performance of data curation methods for factual (MMMLU) queries, per language.

Figure 10: Performance of data curation methods for translation (OPUS-100) queries, per language.

Figure 11: Performance of data curation methods for RAG (multilingual HotPotQA) queries, per language.

Qwen2.5-7B-Instruct
Method	Nemotron	Factual	Translation	RAG
Base	52.3	67.0	59.2	79.0
EngReason	51.2	68.2	59.0	79.3
SelectionGT	48.5	63.9	53.8	61.5
SelectionGEN	50.3	59.9	56.8	41.4
Filtered	48.6	61.7	56.6	66.5

Appendix

Table 10: Average performance per model, with the added Distillation method. Here, we are able to show that a lot of H O T F I X R ’s empirical success comes from the targeted question generation, rather than having correct, distilled labels from a large model.

Panel of Figure 12 — Figure 12: The delta in performance of student models trained with two rounds of H O T F I X R versus one round. The improvement with two rounds is much more than the improvement with one round.

Panel of Figure 13 — Figure 13: Prompt used during H O T F I X R training and data generation, for the Nemotron MATH subset. As input, the prompt takes a question , answer , and reasoning from the Nemotron MATH training dataset, and a language that alternates between English, French, Spanish, Arabic, Portuguese, or Italian.

Panel of Figure 14 — Figure 14: Prompt used during H O T F I X R training and data generation, for the Nemotron STEM subset. As input, the prompt takes a question , answer , and reasoning from the Nemotron STEM training dataset, and a language that alternates between English, French, Spanish, Arabic, Portuguese, or Italian.

Panel of Figure 15 — Figure 15: Prompt used during H O T F I X R training and data generation, for the Nemotron CHAT subset. As input, the prompt takes a question , answer , and reasoning from the Nemotron CHAT training dataset, and a language that alternates between English, French, Spanish, Arabic, Portuguese, or Italian.

Multilingual Language Models

Multilingual language models are trained and adapted to understand and generate text across languages, scripts, and cultural contexts.

See all 265 papers

Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
Apr 13, 2026Lester James V. Miranda, Ivan Vulić, Anna KorhonenSupervised Fine-Tuning Multilingual Language Models
Language Technology Lab, University of Cambridge
ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs
Apr 14, 2026Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt +3Cross-Lingual Reasoning in Language Models Multilingual Language Models
German Research Center for Artificial Intelligence (DFKI), Saarland University · Berliner Hochschule für Technik
Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties
Mar 26, 2026Jannis Vamvas, Ignacio Pérez Prat, Angela Heldstab +3Machine Translation Low-Resource MT
University of Zurich · Lia Rumantscha

Apr 13, 2026J/K move · Enter open · S save

Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

Lester James V. Miranda, Ivan Vulić, Anna Korhonen

Language Technology Lab, University of Cambridge

Supervised Fine-Tuning Multilingual Language Models Multilingual Language Model Evaluation Language Model Generation Evaluation Synthetic Data Generation Synthetic Data Generation for Language Models+1

Open paper AskPDF arXiv

Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. However, teacher model selection is often ad hoc, typically defaulting to the largest available option, even though such models may have significant capability gaps in non-English languages. This practice can result in poor-quality synthetic data and suboptimal student downstream performance. In this work, we systematically characterize what makes an effective multilingual teacher. We combine intrinsic measures of data quality with extrinsic student model performance in a metric we call Polyglot Score. We evaluate 10 LMs across 6 typologically diverse languages, generating over 1.4M SFT examples and training 240 student models. Our analyses reveal that model scale alone does not significantly predict teacher effectiveness: the most effective teachers we identify are consistently smaller than the largest models evaluated, and their ranking is stable across student base model families. Instead, data qualities such as prompt diversity, length, and response fluency capture 93.3% of the variance in intrinsic data quality and predict student performance. Finally, we provide practical recommendations, including matching the model families of teacher-student pairs and generating responses to existing prompts or translating them from English, which can yield improvements for less-resourced languages. We hope that our work advances data-centric research in multilingual synthetic data and LM development.

Abstract

Multilingual Language Models

Explore similar work

Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties