How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?

Wen Yang; Junhong Wu; Chong Li; Chengqing Zong; Jiajun Zhang

cs.CLOct 2, 2025

How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?

Authors: Wen Yang, Junhong Wu, Chong Li, Chengqing Zong, Jiajun Zhang

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research

Abstract

Recent advances in Reinforcement Post-Training (RPT) have substantially improved the mathematical reasoning capabilities of large language models (LLMs), but it remains unclear how well such gains transfer across languages. In this work, we study cross-lingual mathematical reasoning transfer in English-centric LLMs under controlled multilingual evaluation. We first evaluate a range of English-centric reasoning models on multilingual mathematical reasoning benchmarks and find that transfer is highly variable across initial model choice, target language, and training paradigm. Through controlled comparative experiments, we further find that stronger English-centric initialization does not necessarily lead to stronger relative transfer efficiency, even when it yields higher multilingual reasoning accuracy. Finally, we show that the largest improvement comes from moving beyond English-only supervision: adding a single parallel language yields a substantial gain, while further gains from additional languages are smaller but consistent. Overall, our results highlight the need to evaluate multilingual mathematical reasoning directly, using both absolute performance and relative transfer-efficiency metrics.

Figures & tables

Figure 1: Cross-lingual transfer across open-source reasoning models. The left panel reports average MTI across four benchmarks and eleven languages. The right panel shows average transfer behavior on multilingual MATH500, comparing SFT-tuned and RL-tuned models.

Model	Acc	Off-target	MTI
Qwen2.5-7B-Base	12.00	11.41	-
$\looparrowright$ GRPO on En Data	22.45	3.12	1.95
Qwen2.5-7B-Instruct	22.45	1.43	-
$\looparrowright$ GRPO on En Data	23.51	0.94	1.23
Qwen2.5-Math-7B	12.25	22.59	-
$\looparrowright$ GRPO on En Data	19.37	9.50	2.12

Table 1: The Impact of Initial Model Type. Average Accuracy (%), Off-target rate (%), and MTI across eleven languages on four benchmarks.

Figure 2: Effect of model family under controlled post-training. Multilingual reasoning performance on MATH500. Base denotes the initial model, and +GRPO the model after English-only GRPO.

Figure 3: Effect of model size under controlled English-only post-training.

\Delta

Performance denotes the average accuracy gain over the initial model, averaged across English and unseen languages.

Figure 4: Transfer improves with parallel multilingual supervision. Each panel shows performance as a function of the number of training languages. The red star denotes the English-only baseline (

X=1

), shown for reference but excluded from the fit. Blue points denote multilingual configurations (

X\geq 2

). The solid red curve shows a power-law-like descriptive fit over

X\geq 2

, with the shaded region indicating the

95\%

confidence interval. The fit summarizes the bilingual-to-multilingual trend; Appendix E.3 provides the marginal-gain analysis.

Budget	Type	Acc	Off-target	MTI
$1\times$	$1\times$ En	54.24	0.49	1.16
$2\times$	$2\times$ En	55.82	0.58	1.92
$2\times$	$1\times$ En + $1\times$ Ru	57.87	0.20	2.50
$3\times$	$3\times$ En	57.13	0.42	2.38
$3\times$	$1\times$ En + $1\times$ Ru + $1\times$ Fr	58.38	0.24	2.65

Table 2: The Fixed-Budget ablation experiment based on Qwen2.5-7B-Instruct.

Figure 5: Scaling training data based on Qwen2.5-7B-Instruct.

Evaluation	Training	Acc	Off-target
No language control	English-only	57.82	75.76
No language control	English+Ru	58.43	0.36
Language-controlled	English-only	54.20	0.50
Language-controlled	English+Ru	57.90	0.20

Table 3: Prompt ablation. Average accuracy and off-target rate on multilingual MATH500 under two evaluation protocols.

Training Data	Acc	Off-target	MTI
Zh	55.00	0.38	0.86
Zh + Ru	58.13 (+3.13)	0.26 (-0.12)	1.69 (+0.83)
Zh + Ru + Fr	58.76 (+3.76)	0.23 (-0.15)	1.87 (+1.01)

Table 4: Chinese-centric parallel training results on multilingual MATH500.

Figure 6: Parallel vs. non-parallel multilingual supervision. Accuracy differences under parallel and non-parallel data training based on Qwen2.5-7B-Instruct.

Figure 7: The impact of the selected parallel language. Multilingual reasoning performance across different single-language parallel training settings based on Qwen2.5-7B-Instruct.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Initial Model	Size	Training Paradigm
DeepSeek-R1-Distill-Qwen-7B	Qwen2.5-Math-7B-Base	7B	SFT
Open-Reasoner-Zero-7B	Qwen2.5-7B-Base	7B	RL
OpenThinker2-7B	Qwen2.5-7B-Instruct	7B	SFT
OpenThinker3-7B	Qwen2.5-7B-Instruct	7B	SFT
Qwen-2.5-1.5B-SimpleRL-Zoo	Qwen2.5-1.5B-Base	1.5B	RL
Qwen-2.5-7B-SimpleRL-Zoo	Qwen2.5-7B-Base	7B	RL

Appendix

Table 5: The Overview of the Open-source LLMs Used in Observational Study , including their initial model, parameter size, and training paradigm.

Panel of Figure 8 — Figure 8: Distribution of question difficulty. (a) The 1,000 English questions were utilized in the controlled comparative study and for parallel training. (b) The 1,000 Russian questions for non-parallel training, comprising a separate and non-overlapping set from the questions in (a).

Method	En	Avg	Non-En	Low-res.	Off-tgt.
Base	74.80	50.91	48.52	27.40	0.18
SFT	76.80	50.80	48.20	22.13	4.40
GRPO	79.20	54.24	51.74	29.53	0.49

Appendix

Table 6: Matched SFT-vs.-GRPO comparison on multilingual MATH500. All values are percentages. All models use Qwen2.5-7B-Instruct as the backbone. SFT and GRPO are trained on the same 1K English MATH examples and evaluated under the same multilingual MATH500 protocol. Low-res. denotes average accuracy over bn , sw , and te ; Off-tgt. denotes the off-target language rate.

Setting	En	Non-En	Gap	Low-res.	MTI
English-only	79.20	51.74	27.46	29.53	1.163
+1 parallel	78.40	55.82	22.58	36.27	2.496
+2 parallels	79.00	56.32	22.68	34.67	2.650
+3 parallels	77.80	56.44	21.36	34.73	3.002
+4 parallels	77.20	56.76	20.44	37.13	3.282
+5 parallels	77.40	57.18	20.22	39.33	3.475

Appendix

Table 7: Absolute-transfer metrics for Qwen2.5-7B-Instruct under parallel training. All accuracy and gap values are percentages. Non-En denotes average accuracy over non-English languages; Gap denotes the En–Non-En accuracy gap; Low-res. denotes average accuracy over bn , sw , and te . MTI is reported as an auxiliary relative transfer-efficiency metric.

Qwen MTI
Fit	$\mathbf{R^{2}}$	MSE	AIC	BIC
Linear	0.951	0.0178	-29.523	-29.631
Log	0.974	0.0121	-33.950	-34.058
Power-law	0.977	0.0082	-34.653	-34.762
Saturation	0.982	0.0894	-34.368	-34.530
Llama MTI

Appendix

Table 8: Curve-fitting comparison for MTI in the multilingual regime. Fits are evaluated on configurations with

X=2,\ldots,8

. MSE denotes the leave-one-out cross-validation (LOOCV) MSE.

$\mathbf{X}$	Qwen MTI	Qwen Acc.	Llama MTI	Llama Acc.
1	2.005	4.503	3.684	3.341
2	0.414	0.512	0.495	0.862
3	0.015	0.099	0.067	0.004
4	0.019	0.001	0.005	0.015
5	0.016	0.007	0.009	0.000
6	0.009	0.009	0.008	0.000

Appendix

Table 9: Cook’s distance under all-point power-law fits. The power-law curve is fit to all observed points,

X=1,\ldots,8

Transition	Avg Acc	Non-En	Low-res.	MTI
$1\rightarrow 2$	+3.70	+4.08	+6.73	+1.333
$2\rightarrow 3$	+0.50	+0.50	-1.60	+0.154
$3\rightarrow 4$	+0.00	+0.12	+0.07	+0.352
$4\rightarrow 5$	+0.20	+0.32	+2.40	+0.280
$5\rightarrow 6$	+0.40	+0.42	+2.20	+0.193
$6\rightarrow 7$	+0.40	+0.54	+1.20	+0.059

Appendix

Table 10: Marginal-gain decomposition over consecutive training-language counts. Each row reports the gain from increasing the number of training languages from

X

X+1

in the Qwen2.5-7B-Instruct parallel-training experiments.

Metric	$\mathbf{1\rightarrow 2}$	$\mathbf{1\rightarrow 8}$	Fraction
Avg Acc.	+3.70	+5.30	69.8%
Non-En Acc.	+4.08	+6.07	67.2%
Low-res. Acc.	+6.73	+10.93	61.6%
MTI	+1.333	+2.468	54.0%

Appendix

Table 11: Fraction of total gain explained by the first added parallel language. The table compares the gain from English-only to bilingual training (

1\rightarrow 2

) with the total gain from English-only to eight-language training (

1\rightarrow 8

). Avg Acc. denotes average accuracy over all languages; Non-En Acc. denotes average accuracy over non-English languages; Low-res. Acc. denotes average accuracy over bn , sw , and te .

Group	$\mathbf{n}$	Mean Adv.	W/T/L
Overall	44	+0.995	28 / 8 / 8
Non-English	40	+1.101	28 / 5 / 7
Unseen Non-En	36	+1.213	27 / 5 / 4
Unseen low-res.	12	+1.632	9 / 3 / 0
Added lang. (ru)	4	+0.093	1 / 0 / 3
English	4	-0.063	0 / 3 / 1

Appendix

Table 12: Parallel vs. non-parallel multilingual supervision. The advantage is computed as

Acc_{\text{parallel}}-Acc_{\text{non-parallel}}

over benchmark-language pairs. W/T/L denotes wins, ties, and losses.

Scope	Pearson $r$	Spearman $\rho$
Overall	-0.956	-0.882
MATH500	-0.971	-0.946
AIME24	-0.901	-0.946
AIME25	-0.948	-0.936
GPQA-Diamond	-0.777	-0.727

Appendix

Table 13: Correlation between mean accuracy and relative variance across languages. For each benchmark and language, mean accuracy and CV are computed across open-source models. Negative correlations indicate that lower-accuracy languages tend to have higher relative variance.

Setting	MTI	NGR	RGR
En-only	1.163	0.396	0.732
+1 parallel	2.496	0.963	1.480
+2 parallels	2.650	1.061	1.601

Appendix

Table 14: Robustness of transfer metrics on the main parallel-training comparison. We compare the original MTI with normalized-gain ratio (NGR) and raw-gain ratio (RGR). All three metrics show a large gain from English-only to one-parallel training, followed by a smaller additional gain.

Setting	All	$\mathbf{Acc^{base}>20}$	$\mathbf{Acc^{base}>30}$
Relative-gain MTI
En-only	1.16	1.16	1.23
+1 parallel	2.50	2.42	1.88
+2 parallels	2.65	2.62	2.13
+3 parallels	3.00	3.16	2.58
+4 parallels	3.28	3.25	2.34

Appendix

Table 15: Sensitivity of transfer scores to low-base-accuracy cases. Scores are recomputed on multilingual MATH500 after excluding unseen-language cases with base accuracy below the specified thresholds.

Settings	Accuracy per language											Average
Settings	en	es	ru	de	fr	bn	th	sw	zh	ja	te	Acc	Off-target
Qwen2.5-7B-Base
Qwen-Math Template	49.2	31.8	25.2	28.2	30.0	5.8	23.0	2.4	27.0	15.2	1.4	21.7	16.6
Qwen-Instruct Template	50.6	38.0	30.0	33.2	38.4	10.4	26.8	2.4	30.0	27.8	4.4	26.5	15.7
No Template	44.4	38.2	28.2	28.4	35.6	6.0	17.0	0.2	29.2	19.8	1.2	22.6	18.5
Qwen2.5-Math-7B-Base

Appendix

Table 16: The Performance of Base Models with Different Template Settings. Accuracy (%) and Off-target rate (%) across languages for different template settings on multilingual MATH500 benchmark.

Settings	Accuracy per language											Average
Settings	en	es	ru	de	fr	bn	th	sw	zh	ja	te	Acc	Off-target
Multilingual MATH500
Qwen2.5-1.5B	19.60	10.60	7.80	1.00	9.80	1.00	3.00	0.00	8.60	2.20	0.00	5.78	21.02
Qwen2.5-Math-7B	56.60	46.40	11.20	33.40	36.80	31.40	28.20	4.20	44.40	25.20	3.20	29.18	17.96
Qwen2.5-7B-Instruct	74.80	69.00	59.60	56.60	62.60	37.60	49.80	18.00	53.00	52.40	26.60	50.91	0.18
Qwen2.5-7B	50.60	38.00	30.00	33.20	38.40	10.40	26.80	2.40	30.00	27.80	4.40	26.55	15.69

Appendix

Table 17: The Performance of Initial Models. Accuracy (%) and Off-target rate (%) across languages for different Initial models.

Setting	$(\lambda_{1},\lambda_{2},\lambda_{3})$	Accuracy per Language											Avg.	Off-tgt.	MTI
Setting	$(\lambda_{1},\lambda_{2},\lambda_{3})$	en	es	ru	de	fr	bn	th	sw	zh	ja	te	Avg.	Off-tgt.	MTI
C1 (Balanced)	(0.8, 0.1, 0.1)	78.4	73.4	66.0	65.6	67.2	48.8	57.4	26.2	57.8	62.0	33.8	57.87	0.20	2.50
C2 (w/o LCR)	(0.9, 0.1, 0.0)	77.8	71.4	66.2	63.4	68.0	44.2	56.4	26.2	59.6	57.8	31.8	56.62	1.24	2.10
C3 (Strong LCR)	(0.7, 0.1, 0.2)	77.2	70.2	65.8	65.8	67.2	43.6	55.4	25.6	60.2	56.2	32.6	56.35	0.38	2.25
C4 (w/o FR)	(0.9, 0.0, 0.1)	75.2	60.8	62.2	56.0	60.2	36.2	43.8	18.0	54.4	31.8	24.8	47.58	0.31	-3.45
C5 (Strong FR)	(0.7, 0.2, 0.1)	78.2	71.6	65.2	62.6	67.8	45.0	60.0	24.8	60.0	60.8	29.2	56.84	0.35	2.23

Appendix

Table 18: Sensitivity analysis of reward hyperparameters. Models are trained on English data with parallel Russian data.

(\lambda_{1},\lambda_{2},\lambda_{3})

denotes the weights of the three reward components.

Setting	M500	A24	A25	GPQA	Avg.	Off-tgt.	MTI
Base	26.55	1.23	0.42	19.79	12.00	11.41	–
$\looparrowright$ GRPO	52.16	7.10	3.35	27.18	22.45	3.12	1.95
Instruct	50.91	5.93	3.28	29.71	22.45	1.43	–
$\looparrowright$ GRPO	54.24	7.41	3.92	28.47	23.51	0.94	1.23
Math	29.18	4.11	1.88	13.82	12.25	22.59	–
$\looparrowright$ GRPO	45.73	8.84	3.96	18.96	19.37	9.50	2.12

Appendix

Table 19: Impact of model initialization on cross-lingual transfer. All models are initialized from Qwen2.5-7B variants and trained with GRPO on the same English data. We report average accuracy (%) across languages on MATH500 (M500), AIME24 (A24), AIME25 (A25), and GPQA-Diamond (GPQA), together with overall average accuracy, off-target rate (%), and MTI.

Panel of Figure 9 — Figure 9: The Impact of Different Model Families in Controlled Comparative Study. Multilingual reasoning performance across languages, comparing the influence of model family using Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct as initial models.

Model	bo	my	zh
Qwen2.5-7B-Instruct	10.40	9.20	53.00
+ GRPO (En)	13.60	11.60	58.60
$\Delta$	(+3.20)	(+2.40)	(+5.60)
Llama-3.1-8B-Instruct	0.80	2.40	28.40
+ GRPO (En)	15.00	8.40	33.80
$\Delta$	(+14.20)	(+6.00)	(+5.40)

Appendix

Table 20: Cross-lingual transfer to Sino-Tibetan languages. We report accuracy (%) before and after English-only RPT with GRPO. bo , my , and zh denote Tibetan, Myanmar, and Chinese, respectively.

\Delta

denotes the absolute accuracy gain after GRPO.

Settings	$\Delta$ Performance											Average across languages
Settings	en	es	ru	de	fr	bn	th	sw	zh	ja	te	Training	Untraining
Qwen2.5-1.5B-Instruct with GRPO on En Data
MATH500	20.40	27.20	17.40	14.40	17.40	6.20	7.80	4.40	16.80	14.20	7.40	20.40	13.32
AIME24	2.08	0.42	-0.21	1.25	0.42	0.21	0.21	0.00	-0.21	0.21	0.21	2.08	0.25
AIME25	1.04	0.21	0.00	0.21	0.00	0.00	0.00	-0.21	0.42	0.21	0.21	1.04	0.10
GPQA-Diamond	9.09	20.71	-0.51	5.05	12.12	1.52	-2.53	2.02	8.59	3.54	0.51	9.09	5.10

Appendix

Table 21: The Impact of Model Size in Interventional Study.

\Delta

Performance on various benchmarks across Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct .

Settings	Accuracy per language											Average
Settings	en	es	ru	de	fr	bn	th	sw	zh	ja	te	Acc	Off-target	MTI
Multilingual MATH500
Only English	79.2	70.0	61.0	62.2	68.2	41.8	55.4	19.4	53.4	58.6	27.4	54.2	0.5	1.163
w. One parallel	78.4	73.4	66.0	65.6	67.2	48.8	57.4	26.2	57.8	62.0	33.8	57.9	0.2	2.496
w. Two parallel	79.0	73.4	64.4	67.4	69.2	45.2	60.2	26.2	63.0	61.6	32.6	58.4	0.2	2.650
w. Three parallel	77.8	73.6	64.6	68.4	69.8	46.0	60.8	24.0	62.2	60.8	34.2	58.4	0.4	3.002

Appendix

Table 22: The Detailed Results in Parallel Scaling Law.

Settings	Training Parallel Languages
Only English	en
w. One parallel	en, ru
w. Two parallel	en, ru, fr
w. Three parallel	en, ru, fr, es
w. Four parallel	en, ru, fr, es, de
w. Five parallel	en, ru, fr, es, de, bn

Appendix

Table 23: The Language Settings in Parallel Scaling Law.

Panel of Figure 10 — Figure 10: Analysis across Selected Parallel Languages. The accuracy and relative gain across various benchmarks with different parallel languages. “Only en” denotes only fine-tuned on English data. “en& LANGUAGE ” indicates the model was fine-tuned on English and a parallel language, with LANGUAGE representing ru, bn, de, zh , respectively.

Lang.	Base	Zh	Zh+Ru	Zh+Ru+Fr
en	74.8	76.8	78.8	78.0
es	69.0	70.4	72.0	72.2
ru	59.6	61.8	64.4	63.6
de	56.6	62.2	65.0	66.4
fr	62.6	66.8	70.0	69.8
bn	37.6	43.0	44.4	45.4

Appendix

Table 24: Parallel scaling with Chinese as the source language. All experiments use Qwen2.5-7B-Instruct and are evaluated on multilingual MATH500. Adding the first parallel language yields the largest gain, while adding a second language provides a smaller additional improvement.

Settings	Accuracy per language											Average
Settings	en	es	ru	de	fr	bn	th	sw	zh	ja	te	Acc	Off-target
Multilingual MATH500
DeepSeek-R1-Distill-Qwen-7B	86.20	76.20	67.00	66.40	69.80	40.20	53.80	18.40	70.00	53.20	17.60	56.25	2.69
Open-Reasoner-Zero-7B	81.60	76.00	70.40	69.80	71.20	45.20	63.20	15.00	62.80	61.80	17.60	57.69	5.20
OpenThinker2-7B	86.00	74.80	64.80	63.00	73.00	34.40	58.80	12.60	65.80	70.00	8.40	55.60	36.13
OpenThinker3-7B	85.80	81.80	75.00	69.20	76.40	17.00	48.80	17.40	69.60	65.00	10.40	56.04	60.75

Appendix

Table 25: The Performance of Various Open-source Models. Accuracy (%) and Off-target rate (%) across languages for various open-source models.

Cross-Lingual Reasoning in Language Models

Cross-lingual reasoning in language models studies how models solve problems and transfer knowledge across languages, scripts, and cultures.

See all 69 papers

LLM Parameters for Math Across Languages: Shared or Separate?
Jun 16, 2026Behzad Shomali, Luisa Victor, Tim Selbach +5Multilingual Language Models Cross-Lingual Reasoning in Language Models
Lamarr Institute · University of Bonn · Fraunhofer IAIS
TAPO: Translation Augmented Policy Optimization for Multilingual Mathematical Reasoning
Mar 26, 2026Xu Huang, Zhejian Lai, Zixian Huang +2Cross-Lingual Reasoning in Language Models RL for Language Model Reasoning
National Key Laboratory for Novel Software Technology, Nanjing University · Shanghai Artificial Intelligence Laboratory
LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance
May 21, 2026Yuchun Fan, Bei Li, Peiguang Li +9Multilingual Language Models Reinforcement Learning
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Meituan Inc. 3NiuTrans Research, Shenyang, China

Jun 16, 2026J/K move · Enter open · S save

LLM Parameters for Math Across Languages: Shared or Separate?

Behzad Shomali, Luisa Victor, Tim Selbach, Ali Hamza Bashir+4

Lamarr Institute · University of Bonn · Fraunhofer IAIS

Multilingual Language Models Cross-Lingual Reasoning in Language Models LLM Interpretability LLM Reasoning Cross-Lingual Transfer

Open paper AskPDF arXiv

Large language models (LLMs) exhibit substantial cross-lingual variation in mathematical reasoning performance, but it remains unclear whether these differences reflect language-specific parameters or a shared mechanism that manifests differently by language. We present a cross-lingual mechanistic analysis of mathematical reasoning in LLMs, enabling us to localize and compare model parameters that support mathematical reasoning across languages. We find that the extracted math-associated parameters exhibit partial cross-lingual overlap, with the strongest overlap concentrated in intermediate model layers. We further observe that English consistently produces the largest set of math-relevant parameters, whereas lower-resource languages reveal smaller sets of relevant parameters. These results suggest that math-related behavior in multilingual LLMs is neither fully language-invariant nor fully language-specific, but instead exhibits partial cross-lingual parameter overlap with systematic language-dependent differences.

Abstract

Cross-Lingual Reasoning in Language Models

Explore similar work

LLM Parameters for Math Across Languages: Shared or Separate?

TAPO: Translation Augmented Policy Optimization for Multilingual Mathematical Reasoning

LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance