TAPO: Translation Augmented Policy Optimization for Multilingual Mathematical Reasoning

Xu Huang; Zhejian Lai; Zixian Huang; Jiajun Chen; Shujian Huang

cs.CLMar 26, 2026

TAPO: Translation Augmented Policy Optimization for Multilingual Mathematical Reasoning

Authors: Xu Huang, Zhejian Lai, Zixian Huang, Jiajun Chen, Shujian Huang

Organizations: National Key Laboratory for Novel Software Technology, Nanjing University · Shanghai Artificial Intelligence Laboratory

Abstract

Large Language Models (LLMs) have demonstrated remarkable proficiency in English mathematical reasoning, yet a significant performance disparity persists in multilingual contexts, largely attributed to deficiencies in language understanding. To bridge this gap, we introduce Translation-Augmented Policy Optimization (TAPO), a novel reinforcement learning framework built upon GRPO. TAPO enforces an explicit alignment strategy where the model leverages English as a pivot and follows an understand-then-reason paradigm. Crucially, we employ a step-level relative advantage mechanism that decouples understanding from reasoning, allowing the integration of translation quality rewards without introducing optimization conflicts. Extensive experiments reveal that TAPO effectively synergizes language understanding with reasoning capabilities and is compatible with various models. It outperforms baseline methods in both multilingual mathematical reasoning and translation tasks, while generalizing well to unseen languages and out-of-domain tasks.

Figures & tables

Figure 1: Problem-level relationship between translation quality and reasoning accuracy on MGSM. Each point represents a distinct non-English problem, for which the accuracy of the corresponding English problem is over 0.9. The results are generated by Qwen2.5-3B-Instruct with 100 runs. The x-axis indicates the mean ChrF++ score of the translation, while the y-axis shows the overall accuracy. The dashed line represents the line of best fit, with the Pearson correlation coefficient

r=0.583

p<1e-5

Figure 2: The performance of Qwen2.5-3B-Instruct on MGSM with different prompts. The full prompts are in Appendix A . The last two are the same as Translate-Test, but the model’s translation is replaced with Gemini2.5-Flash’s and the reference, respectively.

Figure 3: The illustration of the step-level relative advantage calculation in TAPO.

Model	Trained Languages					Avg	Untrained Languages						Total Avg
Model	Bn	Ja	Sw	Te	Th	Avg	De	En	Es	Fr	Ru	Zh	Total Avg
Qwen	56.5	67.2	13.2	19.9	67.5	44.8	77.7	85.5	78.0	75.0	79.0	73.6	44.8
+ SFT-TransTest	2.8	8.8	2.0	1.6	10.0	5.0	26.0	85.6	28.0	29.6	23.2	47.2	24.1
+ QAlign	32.0	46.4	6.4	10.8	50.0	29.1	61.2	70.8	62.4	60.0	60.4	63.6	47.6
+ RAFT	60.7	62.4	24.6	36.8	68.1	50.5	71.6	80.4	73.8	69.0	71.3	70.3	62.6
+ GRPO-EnCoT	75.2	78.7	39.0	51.0	82.2	65.2	84.0	89.4	84.7	82.7	85.8	81.4	75.8

Table 1: The performance of each model based on Qwen2.5-3B-Instruct and Llama3.2-3B-Instruct, respectively. The highest scores in each column are in bold. The x in TAPO-x is the translation metric function. CK is short for CometKiwi .

Model	Bn	De	Es	Fr	Ja	Ru	Sw	Te	Th	Zh	Avg
Qwen	58.60	84.07	87.30	85.89	81.10	87.28	7.72	17.82	77.68	86.72	67.42
+ GRPO-EnCoT	62.97	83.58	87.88	86.66	80.26	88.88	18.18	29.31	77.99	83.66	69.94
+ GRPO-TransTest	62.59	82.69	86.76	85.89	79.91	86.85	19.94	33.67	79.09	83.87	70.12
+ TAPO-CK	67.06	85.82	89.02	87.21	82.48	87.51	30.03	42.90	80.96	85.67	73.87
+ TAPO- xCOMET	70.88	51.49	51.52	54.85	84.47	87.07	0.00	41.64	82.71	85.72	61.04
+ TAPO-ChrF++	68.89	87.47	89.83	87.11	83.32	88.80	29.29	42.37	82.70	86.60	74.64

Table 2: The Gemini Scores of each model on the task of translating the MGSM problems from non-English to English. The score of each sample is averaged by eight random runs.

Model	MMATH	MSVAMP
Qwen	54.25	77.55
+ GRPO-EnCoT	55.85	77.28
+ GRPO-TransTest	57.63	76.39
+ TAPO-ChrF++	56.77	80.21
Llama	28.07	61.43
+ GRPO-EnCoT	42.22	73.75

Table 3: The performance of each model on OOD tasks. The scores are averaged across all the languages in the benchmarks.

Type	MGSM	MMATH	MSVAMP
Qwen
Traj-level	75.45	56.37	78.35
Step-level	77.08	56.77	80.21
Llama
Traj-level	74.11	42.13	75.87
Step-level	76.41	43.78	77.25

Table 4: The comparison between the trajectory-level reward by addition and our step-level reward. Scores are averaged across all languages.

Figure 4: The proportions of false positive and false negative advantages across each training step. The overall proportion is the sum of the two proportions.

Figure 5: The average number of response tokens on MGSM across all languages. Each bar represents total number of tokens of a response, and the hatched area represents the number of translation tokens.

Figure 6: Accuracy and Gemini Score trends for Qwen and Llama models trained by TAPO-ChrF++ across different

\alpha

values.

Figure 7: The validation scores of Qwen + TAPO-ChrF++. The bold lines are smoothed by Exponential Moving Average.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Bn	De	Es	Fr	Ja	Ru	Sw	Te	Th	Zh	Avg
Qwen	55.57	70.06	72.87	65.20	59.39	67.73	33.69	40.68	57.97	62.00	58.52
+ GRPO-EnCot	56.85	70.49	73.00	64.97	59.07	68.13	41.57	44.16	58.50	60.14	59.69
+ GRPO-TransTest	54.76	68.84	71.25	63.42	56.96	66.42	41.22	44.32	56.99	59.27	58.35
+ TAPO-CK	58.56	70.65	72.77	64.54	59.38	66.92	47.18	50.59	59.30	60.61	61.05
+ TAPO- xCOMET	58.11	48.54	49.98	47.86	61.22	67.41	15.68	49.36	60.12	60.52	51.88
+ TAPO-ChrF++	63.93	74.34	75.83	67.49	64.67	71.48	51.64	56.87	63.71	64.68	65.46

Appendix

Table 8: The ChrF++ scores of each model on the task of translating the MGSM problems from non-English to English. The score of each sample is averaged by eight random runs.

Model	Bn	De	Es	Fr	Ja	Ru	Sw	Te	Th	Zh	Avg
Qwen	33.75	52.92	57.28	46.55	37.92	50.01	14.18	18.26	34.33	41.19	38.64
+ GRPO-EnCot	35.04	53.48	57.67	46.16	37.77	50.51	20.61	22.00	34.10	39.37	39.67
+ GRPO-TransTest	34.28	51.78	55.49	44.32	35.81	48.81	21.54	21.70	35.31	38.29	38.73
+ TAPO-CK	38.57	53.49	57.58	45.51	38.47	48.63	27.70	29.54	38.05	39.83	41.74
+ TAPO- xCOMET	34.37	31.99	34.10	28.79	40.41	48.66	1.83	26.17	37.08	40.04	32.35
+ TAPO-ChrF++	42.08	58.41	60.88	49.49	43.98	54.33	30.71	33.84	40.51	44.65	45.89

Appendix

Table 9: The BLEU scores of each model on the task of translating the MGSM problems from non-English to English. The score of each sample is averaged by eight random runs.

Model	Ar	En	Es	Fr	Ja	Ko	Pt	Th	Vi	Zh	Avg
Qwen	52.07	56.78	56.32	56.32	52.01	52.71	56.35	51.97	52.67	55.31	54.25
+ GRPO-EnCoT	53.44	58.52	57.45	57.09	54.85	54.48	57.55	53.94	54.91	56.25	55.85
+ GRPO-TransTest	55.45	59.79	58.79	58.82	56.82	56.95	58.42	55.41	58.05	57.79	57.63
+ TAPO-ChrF++	55.48	59.39	58.05	58.46	55.15	54.88	57.55	55.51	56.02	57.22	56.77
Llama	24.30	37.77	32.22	31.68	24.80	21.56	30.58	24.10	24.23	29.45	28.07
+ GRPO-EnCoT	34.73	48.09	44.05	44.95	40.51	38.30	44.79	42.68	40.68	43.45	42.22

Appendix

Table 10: The detailed performance in each language on MMATH.

Model	Bn	De	En	Es	Fr	Ja	Ru	Sw	Th	Zh	Avg
Qwen	68.46	85.25	89.10	85.45	86.05	82.48	83.59	33.81	76.78	84.49	77.55
+ GRPO-EnCoT	69.36	81.49	85.61	81.18	81.98	80.13	79.78	56.73	75.54	80.99	77.28
+ GRPO-TransTest	68.78	80.23	84.06	79.24	80.70	78.11	79.49	59.03	74.43	79.86	76.39
+ TAPO-ChrF++	72.38	83.96	86.75	83.10	84.31	82.51	82.51	65.96	77.88	82.71	80.21
Llama	39.30	69.61	78.88	72.60	71.15	60.20	67.99	30.30	56.75	67.49	61.43
+ GRPO-EnCoT	62.85	77.11	83.91	77.55	77.81	73.26	75.68	67.16	67.91	74.21	73.75

Appendix

Table 11: The detailed performance in each language on MSVAMP.

Cross-Lingual Reasoning in Language Models

Cross-lingual reasoning in language models studies how models solve problems and transfer knowledge across languages, scripts, and cultures.

See all 69 papers

How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?
Oct 2, 2025Wen Yang, Junhong Wu, Chong Li +2Cross-Lingual Reasoning in Language Models Cross-Lingual Transfer
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research
AdaMame: A Training Recipe for Adaptive Multilingual Reasoning
Jun 13, 2026Dayeon Ki, Kevin Duh, Marine CarpuatMultilingual Language Models Cross-Lingual Reasoning in Language Models
University of Maryland · Johns Hopkins University
LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance
May 21, 2026Yuchun Fan, Bei Li, Peiguang Li +9Multilingual Language Models Reinforcement Learning
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China · Meituan Inc. 3NiuTrans Research, Shenyang, China

Oct 2, 2025J/K move · Enter open · S save

How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?

Wen Yang, Junhong Wu, Chong Li, Chengqing Zong+1

School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research

Cross-Lingual Reasoning in Language Models Cross-Lingual Transfer LLM Reasoning Multilingual Language Model Evaluation

Open paper AskPDF arXiv

Recent advances in Reinforcement Post-Training (RPT) have substantially improved the mathematical reasoning capabilities of large language models (LLMs), but it remains unclear how well such gains transfer across languages. In this work, we study cross-lingual mathematical reasoning transfer in English-centric LLMs under controlled multilingual evaluation. We first evaluate a range of English-centric reasoning models on multilingual mathematical reasoning benchmarks and find that transfer is highly variable across initial model choice, target language, and training paradigm. Through controlled comparative experiments, we further find that stronger English-centric initialization does not necessarily lead to stronger relative transfer efficiency, even when it yields higher multilingual reasoning accuracy. Finally, we show that the largest improvement comes from moving beyond English-only supervision: adding a single parallel language yields a substantial gain, while further gains from additional languages are smaller but consistent. Overall, our results highlight the need to evaluate multilingual mathematical reasoning directly, using both absolute performance and relative transfer-efficiency metrics.

Abstract

Cross-Lingual Reasoning in Language Models

Explore similar work

How Well Does Mathematical Reasoning Transfer Across Languages in English-Centric LLMs?

AdaMame: A Training Recipe for Adaptive Multilingual Reasoning

LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidance