GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings

Konstantin Dobler; Federico Scozzafava; Jonathan Janke; Mohamed Ali; Simon Lehnerer

cs.CLAug 13, 2026

GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings

Authors: Konstantin Dobler, Federico Scozzafava, Jonathan Janke, Mohamed Ali, Simon Lehnerer

Organizations: Apple · 1Hasso Plattner Institute & ELLIS Unit Potsdam (work done during Apple internship)

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR), often optimized with Group Relative Policy Optimization (GRPO), has become a central recipe for improving the reasoning capabilities of pretrained language models but current studies remain heavily English-centric. We conduct a large-scale empirical study of multilingual and non-English GRPO across a wide range of base models, training languages, and different reasoning language rewards. We find that training to reason in the native language often leaves only a small gap to training for English reasoning. We further observe strong crosslingual transfer: training in one language often improves performance in many others. However, specific trends are highly model- and language-dependent. In some cases, training in a particular language induces severe regressions on out-of-domain capabilities in other languages. Our analysis shows that RLVR beyond English can provide broad crosslingual gains, but also requires broad evaluation to detect language-specific regressions.

Figures & tables

Panel of Figure 1 — Figure 1: Reasoning languages detected during evaluation. We classify into four mutually exclusive buckets: (1) the rewarded language(s), (2) the prompt language (if different from rewarded language), (3) English (if different from both), and (4) all other languages. Results are averaged over all evaluated benchmarks and training languages.

Table 1: Avg@8 improvement over base models, in percentage points, on seen and unseen Multilingual Reasoning Gym tasks. Results average all nine base models under the English reasoning reward. Cells are colored within each evaluation-language column from light blue to dark blue . For multilingual training, Italian (it), Korean (ko), and Brazilian Portuguese (pt) are unseen. Full per-model results are available in Appendix D .

Panel of Figure 4 — Figure 4: In-domain (seen tasks) Multilingual Reasoning Gym performance under English reasoning rewards. Each language label on the x-axis represents a target evaluation language. Values show different training strategies’ performance as a percentage of direct target-language training. Labels on the best-other line show the best other source language for transfer (selected separately for each evaluation language as an oracle).

	Easy Math		Hard Math
Model	Seen language	Crosslingual Transfer	Seen language	Crosslingual Transfer
Qwen3-1.7B-Base	58.6 ( +52.5 )	57.6 ( +51.4 )	7.4 ( +6.5 )	7.4 ( +6.6 )
Qwen3-4B-Base	77.9 ( +61.0 )	76.8 ( +59.9 )	17.5 ( +14.1 )	17.4 ( +14.0 )
Qwen3-8B-Base	83.3 ( +49.9 )	82.7 ( +49.3 )	21.3 ( +17.2 )	21.2 ( +17.0 )
Qwen3-1.7B	71.4 ( +4.2 )	70.6 ( +3.5 )	13.7 ( -12.0 )	12.4 ( -14.3 )
Qwen3-4B	81.9 ( +1.0 )	81.5 ( +0.5 )	22.1 ( -15.7 )	18.1 ( -20.7 )

Table 2: Easy and Hard Math performance under English reasoning rewards. Cells report Avg@8 accuracy, with change from the original model in percentage points in parentheses, averaged over monolingual runs in the 11 training languages. “Seen” denotes evaluation in the training language. Crosslingual transfer denotes evaluation in any other language. Full per-model and per-language results are available in Appendix D .

Table 3: Language-dependent regressions on English-content Multilingual Reasoning Gym tasks (unseen during training) under English reasoning rewards. Panels excerpt the corresponding crosslingual transfer matrices: rows are training languages, columns are evaluation languages, and cells report avg@8 change from the original model in percentage points. Full results are available in the Appendix: Tables 33 , 41 and 49 .

Panel of Figure 5 — Figure 5: Hard Math language-dependent regressions under English reasoning rewards. Bars report avg@8 change from the original model in percentage points, averaged over all evaluation languages. Full results are available in the Appendix: Tables 35 , 43 , 51 and 75 .

Panel of Figure 6 — Figure 6: Reward-specific regression examples. Bars report avg@8 change from the original model in percentage points, averaged over all evaluation languages. Full results are available in the Appendix: Tables 59 , 67 and 75 .

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Table 9

Table 10

Table 11

Table 12

Table 13

Table 14

Table 15

Table 16

Table 17

Table 18

Table 19

Table 20

Table 21

Table 22

Table 23

Table 24

Table 25

Table 26

Table 27

Table 28

Table 29

Table 30

Table 31

Table 32

Table 33

Table 34

Table 35

Table 36

Table 37

Table 38

Table 39

Table 40

Table 41

Table 42

Table 43

Table 44

Base model	en	zh	de	fr	es	ru	ja	th	bn	te	sw	it	ko	pt	Avg
Qwen3-1.7B-Base	5.0	3.9	3.2	3.4	3.1	3.2	2.5	2.1	0.6	0.5	0.1	2.8	1.8	3.4	2.5
Qwen3-4B-Base	8.7	11.6	8.0	9.4	9.9	9.8	8.7	6.4	3.8	4.0	1.3	8.6	6.8	9.0	7.6
Qwen3-8B-Base	20.3	17.9	16.2	16.6	17.8	18.5	12.9	12.1	9.0	8.5	3.5	16.2	11.1	17.5	14.2
Qwen3-1.7B	51.3	49.4	52.9	54.2	53.6	50.5	53.0	48.9	40.9	37.4	16.6	52.5	44.5	51.2	46.9
Qwen3-4B	72.8	65.1	70.5	70.0	69.0	63.4	69.3	69.0	59.1	54.3	43.4	68.2	63.0	66.8	64.6
Qwen3-8B	71.8	67.2	74.2	72.3	70.1	67.6	72.9	67.3	60.6	59.5	52.1	70.5	67.2	69.5	67.3

Appendix

Table 76: Base model performance on tasks seen during training from Multilingual Reasoning Gym.

Base model	en	zh	de	fr	es	ru	ja	th	bn	te	sw	Avg
Qwen3-1.7B-Base	6.5	3.9	3.1	3.3	3.2	4.0	1.6	1.5	0.3	0.1	0.1	2.5
Qwen3-4B-Base	11.2	12.4	8.3	8.9	10.8	9.6	5.8	6.5	3.9	4.5	1.3	7.6
Qwen3-8B-Base	23.1	17.3	14.9	16.2	19.4	18.6	11.3	11.9	10.8	8.4	3.0	14.1
Qwen3-1.7B	47.4	41.4	41.9	43.9	45.3	43.5	41.2	40.0	31.5	26.4	7.8	37.3
Qwen3-4B	59.6	56.7	56.3	56.7	58.6	53.7	55.0	54.5	51.0	45.4	24.5	52.0
Qwen3-8B	61.6	58.2	58.3	58.4	59.6	56.6	57.6	56.5	55.8	50.6	40.5	55.8

Appendix

Table 77: Base model performance on a macro average over all unseen tasks.

Base model	en	zh	de	fr	es	ru	ja	th	bn	te	sw	it	ko	pt	Avg
Qwen3-1.7B-Base	0.9	0.9	0.7	0.6	0.7	0.9	0.7	0.5	0.3	0.1	0.0	0.6	0.3	0.6	0.6
Qwen3-4B-Base	2.7	3.6	2.8	3.3	3.9	3.0	2.6	2.2	1.3	1.9	0.3	2.1	2.1	2.9	2.5
Qwen3-8B-Base	7.8	7.0	4.8	4.9	6.2	5.6	4.5	4.3	4.0	2.5	1.4	4.1	4.5	5.6	4.8
Qwen3-1.7B	24.1	19.4	21.8	23.6	22.3	20.4	26.6	19.4	14.1	16.3	2.0	20.6	22.3	22.9	19.7
Qwen3-4B	44.1	41.1	41.2	41.3	41.6	32.8	41.9	41.5	34.4	32.2	17.5	39.1	42.0	41.6	38.0
Qwen3-8B	47.2	45.9	45.3	45.8	45.6	40.6	45.7	42.0	40.8	37.0	32.6	45.2	45.2	44.8	43.1

Appendix

Table 78: Base model performance on the average over all unseen tasks from Multilingual Reasoning Gym.

Base model	en	zh	de	fr	es	ru	ja	th	bn	te	sw	it	ko	pt	Avg
Qwen3-1.7B-Base	1.1	1.1	0.5	0.8	0.8	0.6	0.7	0.3	0.1	0.1	0.0	0.8	0.2	0.6	0.5
Qwen3-4B-Base	2.2	4.6	1.6	2.8	3.1	3.0	2.6	1.4	1.4	1.8	0.7	1.7	1.5	2.7	2.2
Qwen3-8B-Base	6.1	7.4	3.9	4.4	4.8	4.8	3.4	3.2	3.2	2.0	1.2	4.1	3.4	4.6	4.0
Qwen3-1.7B	26.1	21.7	22.6	26.2	23.1	24.8	28.8	20.4	15.1	16.4	2.9	25.4	21.2	24.0	21.3
Qwen3-4B	50.6	47.8	46.6	46.4	46.1	38.7	46.4	46.8	36.9	35.8	31.1	45.5	44.3	46.6	43.5
Qwen3-8B	54.1	54.7	52.4	54.1	54.4	45.8	50.9	46.2	42.3	40.2	40.9	53.1	48.6	53.3	49.4

Appendix

Table 79: Base model performance on the main heldout tasks from Multilingual Reasoning Gym.

Base model	en	zh	de	fr	es	ru	ja	th	bn	te	sw	it	ko	pt	Avg
Qwen3-1.7B-Base	1.8	1.6	1.6	1.2	1.2	2.0	1.3	1.1	0.8	0.1	0.1	1.0	0.7	1.3	1.1
Qwen3-4B-Base	5.8	6.3	6.8	7.2	8.4	5.9	5.1	5.1	2.5	4.1	0.1	4.5	4.7	6.1	5.2
Qwen3-8B-Base	17.1	13.6	10.5	10.2	13.6	12.0	9.9	9.6	8.8	5.4	2.9	8.2	10.1	12.2	10.3
Qwen3-1.7B	41.6	34.9	38.6	41.3	41.2	34.4	46.4	35.7	25.3	31.7	3.1	34.3	43.6	42.1	35.3
Qwen3-4B	65.4	61.8	61.8	62.2	62.6	49.2	63.2	61.7	51.0	48.8	16.4	55.2	63.5	61.8	56.0
Qwen3-8B	69.7	66.1	65.6	66.4	67.5	61.3	67.7	65.3	67.8	55.8	47.7	65.8	67.8	65.3	64.3

Appendix

Table 80: Base model performance on English-content tasks from Multilingual Reasoning Gym.

Base model	en	zh	de	fr	es	ru	ja	th	bn	te	sw	it	ko	pt	Avg
Qwen3-1.7B-Base	0.0	0.0	0.0	0.0	0.1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Qwen3-4B-Base	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.1	0.0	0.0	0.0
Qwen3-8B-Base	0.1	0.0	0.0	0.1	0.1	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Qwen3-1.7B	4.8	1.7	4.0	3.2	2.7	2.1	4.6	2.0	1.8	0.9	0.1	2.0	2.0	2.5	2.5
Qwen3-4B	16.3	13.8	15.2	15.1	16.0	10.5	16.0	16.0	15.4	11.9	5.0	16.6	18.1	16.4	14.5
Qwen3-8B	17.9	16.8	18.0	17.0	14.8	14.7	18.4	14.6	12.4	14.9	9.1	16.7	19.2	15.7	15.7

Appendix

Table 81: Base model performance on very difficult tasks from Multilingual Reasoning Gym.

Base model	en	zh	de	fr	es	ru	ja	th	bn	te	sw	it †	ko †	pt †	ar †	id †	ms †	vi †	Avg
Qwen3-1.7B-Base	17.0	9.7	7.7	7.9	7.8	9.7	3.4	3.4	0.5	0.1	0.1	7.7	2.8	9.2	3.2	6.9	6.6	8.3	6.2
Qwen3-4B-Base	27.0	28.8	18.9	18.6	23.6	21.4	12.2	15.0	8.5	9.0	2.6	20.0	13.8	20.8	10.7	13.9	15.4	16.6	16.5
Qwen3-8B-Base	54.2	39.4	36.1	38.7	46.5	44.9	25.9	28.7	26.2	20.2	6.2	43.3	26.5	46.4	30.9	35.3	32.9	42.7	34.7
Qwen3-1.7B	90.4	77.4	74.9	78.4	84.2	82.7	69.6	75.2	56.4	42.4	7.0	83.7	70.3	81.6	76.7	81.5	75.3	73.1	71.2
Qwen3-4B	95.6	88.6	86.6	86.9	92.9	90.5	83.7	84.0	81.2	69.0	31.7	91.1	89.3	88.9	88.9	88.7	88.5	87.7	84.1
Qwen3-8B	96.5	88.1	88.9	87.8	92.4	92.1	85.9	89.8	88.5	77.1	59.5	94.0	89.1	88.6	88.6	90.5	91.3	90.5	87.7

Appendix

Table 82: Base model performance on Easy Math (MGSM + PolyMath-low);

\dagger

: includes only PolyMath-low.

Base model	en	zh	de	fr	es	ru	ja	th	bn	te	sw	it	ko	pt	ar	id	ms	vi	Avg
Qwen3-1.7B-Base	1.6	1.2	1.0	1.3	1.2	1.3	0.8	0.8	0.0	0.2	0.2	0.9	0.7	0.8	0.3	0.9	0.9	1.0	0.8
Qwen3-4B-Base	4.0	4.7	3.2	4.8	4.8	4.4	2.8	2.4	2.0	2.6	1.0	3.9	2.5	4.7	2.9	3.7	3.6	4.1	3.4
Qwen3-8B-Base	7.3	5.5	3.7	5.1	5.5	5.3	3.5	2.8	2.4	2.4	1.3	5.5	2.7	5.7	3.4	4.0	3.8	4.9	4.2
Qwen3-1.7B	27.6	27.5	29.1	29.7	29.4	27.2	27.2	25.5	24.0	20.4	14.4	28.8	27.6	30.2	27.1	27.7	27.5	28.0	26.6
Qwen3-4B	39.1	40.2	40.9	41.9	41.5	37.8	39.5	37.9	37.5	35.2	24.4	41.9	39.7	40.9	40.3	40.2	39.8	39.6	38.8
Qwen3-8B	41.1	40.8	40.5	41.5	40.9	37.2	41.2	37.6	37.9	37.8	29.6	42.1	40.9	41.1	41.2	39.7	39.7	40.5	39.5

Appendix

Table 83: Base model performance on Hard Math (PolyMath medium, high, and top).

Explore similar work

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs
Apr 14, 2026Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt +3Cross-Lingual Reasoning in Language Models Multilingual Language Models
German Research Center for Artificial Intelligence (DFKI), Saarland University · Berliner Hochschule für Technik
ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection
Feb 25, 2026Changjiang Gao, Zixian Huang, Kaichen Yang +3RL for Language Model Reasoning Cross-Lingual Reasoning in Language Models
National Key Laboratory for Novel Software Technology, Nanjing University, Jiangsu, China · Department of Linguistics and Translation, City University of Hong Kong, Hong Kong, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China +1
Language as a Latent Variable for Reasoning Optimization
Apr 23, 2026Linjuan Wu, Haoran Wei, Jialong Tang +4Multilingual Language Models Cross-Lingual Reasoning in Language Models
Zhejiang University · Tongyi Lab, Alibaba Grou.

Apr 14, 2026J/K move · Enter open · S save

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt, Josef van Genabith+2

German Research Center for Artificial Intelligence (DFKI), Saarland University · Berliner Hochschule für Technik

Cross-Lingual Reasoning in Language Models Multilingual Language Models

Open paper AskPDF arXiv

Despite advances in multilingual capabilities, most large language models (LLMs) remain English-centric in their training and, crucially, in the reasoning traces they produce: even for non-English problems, they predominantly reason in English, leaving non-English users with intermediate reasoning they cannot follow. We address this disparity with two contributions. (a) We introduce

\texttt{ReasonXL}

, the first large-scale parallel corpus of cross-domain reasoning traces in five European languages (English, German, French, Italian, and Spanish): over 2.5M aligned samples (11B tokens) per language, each comprising prompt, reasoning trace, and final output, enabling direct supervision of language-specific reasoning. (b) Using

\texttt{ReasonXL}

, we demonstrate that LLMs can be adapted to reason entirely in a target language through a simple two-stage pipeline of supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), matching or exceeding the base models' English-reasoning performance in two of four languages and recovering most of it in the remaining two.

Abstract

Explore similar work

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection

Language as a Latent Variable for Reasoning Optimization