Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models

Xin Xu; Clive Bai; Kai Yang; Tianhao Chen; Cai Yuchen; Yangkun Chen; Weijie Liu; Hao Chen

cs.CLFeb 12, 2026

Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models

Authors: Xin Xu, Clive Bai, Kai Yang, Tianhao Chen, Cai Yuchen, Yangkun Chen, Weijie Liu, Hao Chen,

Organizations: HY, Tencent · The Hong Kong University of Science and Technology · The University of Hong Kong

Abstract

Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand further. Recent studies focus on better exploiting limited training data by prioritizing hard prompts whose rollout pass rate is 0. However, easy prompts with a pass rate of 1 also become increasingly prevalent as training progresses, thereby reducing the effective data size. To mitigate this, we propose Composition-RL, an offline data-augmentation approach that composes existing prompts before RL to address the pass-rate-1 bottleneck without modifying the RL optimizer. The composed dataset is generated and verified once from the fixed source pool and then used for RL training. Extensive experiments across model sizes from 4B to 30B show that Composition-RL consistently improves reasoning capability over RL trained on the original dataset. Performance can be further boosted with a curriculum variant of Composition-RL that gradually increases compositional depth over training. Additionally, Composition-RL enables more effective cross-domain RL by composing prompts drawn from different domains. All our code, data, and models are publicly available to facilitate future research.

Figures & tables

Figure 1: Overview of Composition-RL . Top: an example of composing two math problems, illustrating the high-level idea of Composition-RL . Bottom left: pass@1 (%) on AIME24 versus training steps, summarizing key findings in Sections 5.2 and 5.3 . Bottom right: cross-topic results on MMLU-Pro subjects with the top-5 largest sample sizes, highlighting the main finding in Section 5.4 .

Figure 2: Visualization of meta-experiments. Left: solve_all ratio curve for RL of Qwen3-4B-Base with original prompts (MATH12K) versus compositional prompts. Right: avg@8 accuracy on a subset of MATH500 and its corresponding compositional test prompts.

Method / Data	Mathematics (In-Domain)					Multi-Task (Out-Of-Domain)			Overall
	AIME 24	AIME 25	Beyond AIME	IMOBench	Overall	GPQA	MMLU-Pro	Overall	Overall
	Avg@32	Avg@32	Avg@8	Avg@4	Avg.	Avg@8	Avg@1	Avg.	Avg.
Qwen3-4B-Base
GRPO	23.3	19.5	9.0	14.4	16.6	43.7	58.6	51.2	28.1
MetaMath ∗	12.7	9.7	3.9	8.1	8.6	43.9	58.5	51.2	22.8
SAND-Math ∗	25.6	20.5	8.9	15.4	17.6	45.4	57.9	51.7	29.0

Table 1: Main Results. “ Avg@k ” denotes the average accuracy (%) over

k

random generations (i.e., pass@1 ). The rows “Depth 1 + 2” and “+ Depth 3” are the results of curriculum Composition-RL in Section 5.3 . ∗ indicates that the corresponding results are obtained using the same training set size as MATH12K. For Composition-RL , the starred row uses a 12K subset of MATH-Composition-199K , while the unstarred row uses the full set.

Dataset	Mathematics					Multi-Task			Overall
	AIME 24	AIME 25	Beyond AIME	IMOBench	Overall	GPQA	MMLU-Pro	Overall	Overall
	Avg@32	Avg@32	Avg@8	Avg@4	Avg.	Avg@8	Avg@1	Avg.	Avg.
MATH12K	23.3	19.5	9.0	14.4	16.6	43.7	58.6	51.2	28.1
MATH12K + Physics	19.7	16.5	8.3	12.0	14.1	44.4	59.6	52.0	26.8
Physics after MATH12K	25.3	22.3	8.6	14.4	17.7	45.2	61.4	53.3	29.5
Physics-MATH-Composition-141K	32.4	25.5	10.6	17.8	21.6	46.6	62.7	54.7	32.6

Table 2: Results of cross-topic experiments across multiple benchmarks. “ Avg@k ” denotes the average accuracy (%) over

k

random generations (i.e., pass@1 ). “ MATH12K + Physics ” corresponds to the Mix Training baseline, and “ Physics after MATH12K ” corresponds to the Math-then-Physics baseline. The final block uses 10K multidisciplinary prompts from SCP-116K [ 64 ] .

$\mathcal{D}_{1}$	$\mathcal{D}_{2}$	Mathematics (In-Domain)					Multi-Task (Out-Of-Domain)			Overall
		AIME 24	AIME 25	Beyond AIME	IMOBench	Overall	GPQA	MMLU-Pro	Overall	Overall
		Avg@32	Avg@32	Avg@8	Avg@4	Avg.	Avg@8	Avg@1	Avg.	Avg.
Baseline		23.3	19.5	9.0	14.4	16.6	43.7	58.6	51.2	28.1
RAND	RAND	22.6	19.6	8.2	13.8	16.1	43.6	60.0	51.8	28.0
FULL	RAND	24.5	23.4	8.7	14.1	17.7	44.8	59.9	52.4	29.2
RAND	FULL	30.5	23.3	12.6	14.3	20.2	46.3	61.4	53.9	31.4

Table 3: Results of the ablation study.

\mathcal{D}_{1}

specifies the strategy for constructing the candidate set for

q_{1}

: “RAND” randomly samples 20 prompts, whereas “FULL” selects from the entire original prompt set

\mathcal{D}

. “Baseline” denotes RL training on the original prompts

\mathcal{D}

. “Direct Concat” denotes RL training on prompts that are directly concatenated from two prompts without composition. For results on the fully solvable subset, n is the number of rollouts. Additional blocks report the Feeding-

v_{1}

control, standalone training at each composition depth, and adaptive Depth-2 composition.

Figure 3: Left : avg@8 accuracy on Depth-3 compositional test prompts across different model sizes. The bars compare Original RL with Composition-RL, i.e., whether Depth-2 compositional prompts are used during RL training. Right : The fraction of prompts for which

q_{1:2}

is solved correctly, and the accuracy of recovering

v_{1}

at each training step.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: The Prompt for Verifying the Correctness of Finding $v_{1}$ in LLMs’ Response.

Source difficulty	Composition	pass@1	pass@8	solve_all	solve_none	Informative
Easy $\rightarrow$ Easy	N	99.3	100.0	94.5	0.0	5.5
Easy $\rightarrow$ Easy	Y	89.2	98.6	69.9	1.4	28.8
Easy $\rightarrow$ Hard	N	66.3	78.3	51.7	21.7	26.7
Easy $\rightarrow$ Hard	Y	57.7	83.3	26.7	16.7	56.7
Hard $\rightarrow$ Easy	N	99.8	100.0	98.3	0.0	1.7
Hard $\rightarrow$ Easy	Y	68.5	93.1	34.5	6.9	58.6

Appendix

Table 4: Difficulty-stratified evaluation (%). “Informative” denotes prompts that are neither solve_all nor solve_none .

Step	At least one source	Both sources
	is solve_all	are solve_all
50	84.8	36.4
100	91.2	48.7
200	95.7	62.1
250	97.3	69.0

Appendix

Table 5: Overlap of solve_all source prompts during training (%).

Metric (%)	Baseline	Ours
First-attempt correctness	42.8	51.7
$P(\text{correct}\mid\text{reflection})$	11.0	19.8
Self-correction success	9.1	26.1

Appendix

Table 6: Additional evidence of implicit process supervision on AIME24 with the 4B model.

Error Type	Original	Composition-RL
Linkage error	22	7
$q_{1}$ error	7	1
$q_{2}$ error	11	5

Appendix

Table 7: Error analysis on 40 cases where the model solves both constituent questions in isolation but fails on the compositional question.

Composition	Mathematics (In-Domain)					Multi-Task (Out-Of-Domain)			Overall
Composition	AIME 24	AIME 25	Beyond AIME	IMO-Answer	ID Avg	GPQA	MMLU-Pro	OOD Avg	Avg
Qwen3-4B-Base
✘	16.0	13.4	13.3	12.8	13.9	4.2	2.3	3.3	10.3
✔	14.7	13.2	12.3	12.9	13.3	4.0	2.0	3.0	9.9
Qwen3-8B-Base
✘	17.5	15.7	16.9	15.8	16.5	5.1	2.5	3.8	12.3

Appendix

Table 8: Average response length (in K tokens) across different benchmarks. “ID Avg” denotes the average response length over in-domain mathematics benchmarks, “OOD Avg” denotes the average response length over out-of-domain multi-task benchmarks, and “Avg” denotes the overall average.

Composition	Mathematics (In-Domain)					Multi-Task (Out-Of-Domain)			Overall
	AIME 24	AIME 25	Beyond AIME	IMO-Answer	ID Avg	GPQA	MMLU-Pro	OOD Avg	Avg
	Pass@32	Pass@32	Pass@8	Pass@4	Avg.	Pass@8	Pass@1	Avg.	Avg.
Qwen3-4B-Base
✘	56.7	46.7	23.0	25.5	38.0	79.3	58.6	69.0	48.3
✔	66.7	56.7	31.0	26.0	45.1	77.8	61.4	69.6	53.3
Qwen3-8B-Base

Appendix

Table 9: Pass@k (%) across different benchmarks. The value of

k

for each benchmark is consistent with that used in the corresponding Avg@k setting in the main results table. “ID Avg” denotes the average pass@k over in-domain mathematics benchmarks, “OOD Avg” denotes the average pass@k over out-of-domain multi-task benchmarks, and “Avg” denotes the overall average.

Benchmark	Baseline	Composition-RL
AIME24	1.58	1.65
AIME25	1.52	1.60
BeyondAIME	1.23	1.35
GPQA	1.08	1.14
IMOBench	1.27	1.33
MMLU-Pro	0.80	0.92

Appendix

Table 10: Standard deviation of pass@1 (percentage points) over three training seeds for Qwen3-30B-A3B-Base .

Benchmark	Baseline	Composition-RL
AIME24	3.44	5.94
AIME25	1.88	2.81
AMC	24.85	28.12
BeyondAIME	1.25	1.62
GPQA	25.88	29.23
IMO-Answer	5.31	5.44

Appendix

Table 11: pass@1 (%) with OctoThinker-3B .

Panel of Figure 5 — Figure 5: solve_all and solve_none ratio across model sizes.

Panel of Figure 6 — Figure 6: The Prompt for Generating Variable $v_{1}$ and Definition $d_{1}$ for $q_{1}$ .

Panel of Figure 7 — Figure 7: The Prompt for Verifying the Modification of $q_{1}$ .

RL for Language Model Reasoning

Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.

See all 1,094 papers

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
Jul 30, 2026Shuang Liang, Haoyang Zhou, Yifan Gong +2Numerical Reasoning in Language Models Prompt Selection for RL
Renmin University of China · AntGroup
Improving Generalization Robustness of Multimodal RLVR
Aug 9, 2026Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng +11Multimodal Robustness Multimodal Large Language Models
1National University of Singapore, HPC-AI Lab · 2DAMO Academy, Alibaba Group · 3Hupan Lab +7
Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward
May 11, 2026Xuexiang Wen, Hang Yu, Linchao Zhu +1Intrinsic Reward Methods Reinforcement Learning
1Zhejiang University · 2Ant Group

Jul 30, 2026J/K move · Enter open · S save

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

Shuang Liang, Haoyang Zhou, Yifan Gong, Guowei Wang+1

Renmin University of China · AntGroup

Numerical Reasoning in Language Models Prompt Selection for RL Reinforcement Learning with Verifiable Rewards Sample-Efficient RL

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models, but prompt groups with identical rollout rewards consume generation budget without effective learning signals. Pre-rollout prompt selection can reduce this waste by screening prompts before rollout generation. However, existing pre-rollout methods struggle to balance exploitation and exploration: repeatedly exploiting historically informative prompts can narrow training coverage, whereas broader exploration can lower the fraction of informative prompts. To address these limitations, we introduce LEEPS, a Latent-Guided Explore--Exploit Prompt Sampler that adaptively balances the reuse of previously observed informative prompts with continued exploration of uncertain ones. LEEPS partitions candidates into exploit and explore portfolios and adaptively allocates rollout budget according to their recent non-trivial ratios. It further uses representation-space neighbors and historical rollout outcomes to prioritize uncertain prompts likely to yield non-zero reward variance, thereby making exploration more targeted without additional rollouts. Across six mathematical reasoning benchmarks, LEEPS achieves the highest average score at both model scales, with relative gains of 2.6% and 3.7% over the strongest baseline for Qwen2.5-Math-1.5B and 7B, respectively, and generally improves faster during the training process. It also achieves the highest average score across the three evaluated OOD general-reasoning benchmarks at both model scales and adds only about 2 seconds of online sampling overhead per training step. Code is available at https://github.com/ShuangLiangX/LEEPS.

Abstract

RL for Language Model Reasoning

Explore similar work

LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models

Improving Generalization Robustness of Multimodal RLVR

Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward