ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection

Changjiang Gao; Zixian Huang; Kaichen Yang; Jiajun Chen; Jixing Li; Shujian Huang

cs.CLFeb 25, 2026

ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection

Authors: Changjiang Gao, Zixian Huang, Kaichen Yang, Jiajun Chen, Jixing Li, Shujian Huang

Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, Jiangsu, China · Department of Linguistics and Translation, City University of Hong Kong, Hong Kong, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · School of Mathematical Sciences, Dalian University of Technology, Liaoning, China

Abstract

Current large reasoning models (LRMs) have shown strong ability on challenging tasks after reinforcement learning (RL) based post-training. However, previous work mainly focuses on English reasoning in expectation of the strongest performance, despite the demonstrated potential advantage of multilingual thinking, as well as the requirement for native thinking traces by global users. In this paper, we propose ExpLang, a novel LLM post-training pipeline that enables on-policy thinking language selection to improve exploration and exploitation during RL with the use of multiple languages. The results show that our method steadily outperforms English-only training with the same training budget, while showing high thinking language compliance for both seen and unseen languages. Analysis shows that, by enabling on-policy thinking language selection as an action during RL, ExpLang effectively extends the RL exploration space with diversified language preference and improves the RL exploitation outcome with leveraged non-English advantage. The method is orthogonal to most RL algorithms and opens up a new perspective on using multilinguality to improve LRMs.

Figures & tables

Figure 1: Overview of our work. a. Multilingual thinking shows larger diversity and higher Pass@ $k$ , suggesting better exploration space; b. The ExpLang pipeline with on-policy thinking language selection to leverage the advantage.

Size	Setting	Acc(%) $\uparrow$	Pass@k(%) $\uparrow$	Tokens $\downarrow$	Compl. $\uparrow$
MATH-500
4B	en	79.4	91.6	2900.7	100.0
4B	multi	75.7	93.4	2339.7	73.3
8B	en	77.8	90.6	3007.8	100.0
8B	multi	75.0	93.4	2505.7	72.7
32B	en	80.2	93.0	2796.3	100.0

Table 1: Performances of Qwen3 models across test sets under en and multi settings. “Compl.” denotes thinking language compliance.

MATH-500
Model	Acc(%) $\uparrow$	Pass@k(%) $\uparrow$	Tokens $\downarrow$
Qwen3-4B	78.1	90.0	2953.2
SFT (Ctrl)	78.6 (+0.5)	93.4	2764.1
SFT (Ours)	72.0 (-6.1)	92.6	2590.9
RLVR (Naive)	91.4 (+13.3)	96.6	1416.9
RLVR (Ctrl)	91.0 (+12.4)	96.6	1440.0

Table 2: Performance comparison between models, with automatic (default) thinking language selection across test sets. “Ctrl” stands for the controlled baseline, while “Naive” stands for the naive baseline and “Ours” stands for ExpLang. Parentheses denote performance change during the last training stage.

MATH-500
Model	$\mathrm{Acc}^{F}\uparrow$	$\mathrm{Acc}^{*}\uparrow$	Tokens $\downarrow$	Compl. $\uparrow$
Qwen3-4B	73.8	54.2	2339.7	73.3
SFT (Ctrl)	70.5 (-3.3)	57.3 (+3.1)	2298.4	80.9
SFT (Ours)	69.0 (-4.8)	68.6 (+14.4)	2298.1	99.3
RLVR (Naive)	87.2 (+13.4)	57.8 (+3.7)	1321.3	66.2
RLVR (Ctrl)	85.5 (+15.0)	67.6 (+10.3)	1317.8	78.8

Table 3: Results on seen languages: performance, tokens, and compliance with forced language selection.

\mathrm{Acc}^{F}

means accuracy filtered on compliant samples only, and

\mathrm{Acc}^{*}

means accuracy with incompliance counted as error. Other notations are consistent with previous tables.

MATH-500
Model	$\mathrm{Acc}^{F}\uparrow$	$\mathrm{Acc}^{*}\uparrow$	Tokens $\downarrow$	Compl. $\uparrow$
Qwen3-4B	80.4	54.2	2201.5	34.0
SFT (Ctrl)	70.0 (-10.4)	32.8 (-21.4)	2147.6	46.4
SFT (Ours)	64.5 (-15.9)	41.7 (-12.5)	2233.7	62.8
RLVR (Naive)	83.5 (+3.1)	24.6 (-29.6)	1208.1	28.6
RLVR (Ctrl)	81.1 (+11.1)	37.9 (+5.2)	1258.9	46.6

Table 4: Results on unseen languages: performance, tokens, and compliance with forced language selection. The notations are consistent with previous tables.

Figure 2: Trends of thinking language selection rates of our model at the end of different training stages, across the three test sets.

MATH-500
Model	Auto $\uparrow$ Acc	Forced $\mathrm{Acc}^{F}$ $\uparrow$	Compl. $\uparrow$
Qwen3-4B	78.1	73.8	73.3
LoRA	72.0	69.0	99.3
Fully	66.9	68.7	99.5
AIME-2025
Qwen3-4B	18.9	14.2	71.9

Table 5: Accuracy and language compliance of the original, LoRA-tuned and fully-tuned models. “Auto Acc” stands for the average accuracy with automatic thinking language selection, while “Forced

\mathrm{Acc}^{F}

” stands for the average accuracy filtered by compliance with forced thinking language selection of seen non-English languages.

MATH-500
RL Model	Auto Acc $\uparrow$	Forced $\mathrm{Acc}^{F}$ $\uparrow$	Compl. $\uparrow$
w/ explore	91.5	85.7	99.9
w/o explore	91.0	85.0	99.7
AIME-2025
w/ explore	31.9	27.3	99.7
w/o explore	28.9	23.9	100.0

Table 6: Accuracy and language compliance of training with or without the multilingual exploration stage. The notations are consistent with previous tables.

MATH-500
RL Model	Auto Acc $\uparrow$	Forced $\mathrm{Acc}^{F}$ $\uparrow$	Compl. $\uparrow$
w/ multi. exploit	91.5	85.7	99.9
w/ naive exploit	91.7	85.3	99.7
w/o exploit	88.0	87.5	99.9
AIME-2025
w/ multi. exploit	31.9	27.3	99.7

Table 7: Accuracy and language compliance of training with multilingual exploitation, naive GRPO exploration, and without exploration. The notations are consistent with previous tables.

Stage-Wise KL Use	Auto Acc $\uparrow$	Forced $\mathrm{Acc}^{F}$ $\uparrow$	Compl. $\uparrow$
MATH-500
(N, Y)	91.5	85.7	99.9
(N, N)	90.8	85.4	99.7
(Y, Y)	91.1	84.3	99.8
AIME-2025
(N, Y)	31.9	27.3	99.7

Table 8: Accuracy and language compliance of training with different KL loss settings during the exploration and exploitation stages, respectively. The notations are consistent with previous tables.

Figure 3: Policy entropy of ExpLang and the controlled baseline during the RLVR process.

Figure 4: Win-Tie-Lose rate of non-English vs. English thinking, in terms of the average accuracy of the models on 2000 training samples, run for 5 times per sample.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Comparison of Cluster Counts After Spectral Clustering of English and Multilingual Thinking.

Figure 6: Prefixes and accept rates used in multilingual thinking data generation and forced multilingual thinking for baseline models.

Explore similar work

Language as a Latent Variable for Reasoning Optimization
Apr 23, 2026Linjuan Wu, Haoran Wei, Jialong Tang +4Multilingual Language Models Cross-Lingual Reasoning in Language Models
Zhejiang University · Tongyi Lab, Alibaba Grou.
ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs
Apr 14, 2026Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt +3Cross-Lingual Reasoning in Language Models Multilingual Language Models
German Research Center for Artificial Intelligence (DFKI), Saarland University · Berliner Hochschule für Technik
Poly-EPO: Training Exploratory Reasoning Models
Apr 19, 2026Ifdita Hasan Orney, Jubayer Ibn Hamid, Shreya S Ramanujam +5Reinforcement Learning RL for Language Model Reasoning
Stanford University

Apr 23, 2026J/K move · Enter open · S save

Language as a Latent Variable for Reasoning Optimization

Linjuan Wu, Haoran Wei, Jialong Tang, Shuang Luo+3

Zhejiang University · Tongyi Lab, Alibaba Grou.

Multilingual Language Models Cross-Lingual Reasoning in Language Models RL for Language Model Reasoning Group Relative Policy Optimization

Open paper AskPDF arXiv

As LLMs reduce English-centric bias, a surprising trend emerges: non-English responses sometimes outperform English on reasoning tasks. We hypothesize that language functions as a latent variable that structurally modulates the model's internal inference pathways, rather than merely serving as an output medium. To test this, we conducted a Polyglot Thinking Experiment, in which models were prompted to solve identical problems under language-constrained and language-unconstrained conditions. Results show that non-English responses often achieve higher accuracy, and the best performance frequently occur when language is unconstrained, suggesting that multilinguality broadens the model's latent reasoning space. Based on this insight, we propose polyGRPO (Polyglot Group Relative Policy Optimization), an RL framework that treats language variation as an implicit exploration signal. It generates polyglot preference data online under language-constrained and unconstrained conditions, optimizing the policy with respect to both answer accuracy and reasoning structure. Trained on only 18.1K multilingual math problems without chain-of-thought annotations, polyGRPO improves the base model (Qwen2.5-7B-Instruct) by 6.72% absolute accuracy on four English reasoning testset and 6.89% in their multilingual benchmark. Remarkably, it is the only method that surpasses the base LLM on English commonsense reasoning task (4.9%), despite being trained solely on math data-highlighting its strong cross-task generalization. Further analysis reveals that treating language as a latent variable expands the model's latent reasoning space, yielding consistent and generalizable improvements in reasoning performance.

Abstract

Explore similar work

Language as a Latent Variable for Reasoning Optimization

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Poly-EPO: Training Exploratory Reasoning Models