When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

Zehao Wang; Hanzhu Chen; Runquan Gui; Zhe Wang; Lin Yang

cs.LGOct 27, 2026

When All Rollouts Fail: Policy-Compatible Joint Decoding for Hard-Problem Reasoning

Authors: Zehao Wang, Hanzhu Chen, Runquan Gui, Zhe Wang, Lin Yang

Organizations: University of Science and Technology of China · Tencent · Shanghai Artificial Intelligence Laboratory

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) derives its learning signal from binary rewards via on-policy exploration and becomes a powerful paradigm for improving the reasoning of large language models. Yet it faces a learning cliff on problems beyond model's current ability: when every rollout fails, the samples reveal nothing about a correct solution, and group-relative estimators (e.g., GRPO, RLOO) collapse the update to zero advantage. Expert trajectories offer a remedy, but they are incompatible with the current policy and often lie in the student's low-probability regions, so the surviving updates demand large parameter shifts and decay into surface-level imitation rather than transferable reasoning. We argue that a useful learning signal on hard problems must be both verifiably correct and sufficiently compatible with the current policy. We propose Policy-Compatible Reinforcement Learning (PICO), which performs distribution-constrained joint decoding: expert data guides decoding at the token level while preserving the student's relative token probabilities, and a dynamic top-k mechanism confines the sampling distribution under a finite divergence budget, yielding trajectories that are verified and policy-compatible by construction. These trajectories are optimized jointly with on-policy rollouts under a mixed-behavior objective that mitigates their distribution mismatch via importance sampling. Experiments across reasoning diverse datasets and agentic benchmarks demonstrate that PICO consistently outperforms standard RLVR and prior off-policy baselines.

Figures & tables

Figure 1: Overview of PICO . Given an all-incorrect rollout group sampled from the student policy, PICO generates a verified joint trajectory to replace one failed rollout for mixed-behavior policy optimization. The joint trajectory is generated token by token through distribution-constrained joint decoding, which intersects the teacher and student top- $k$ candidate sets.

Figure 2: Training dynamics of Qwen3-4B-Base and Qwen3-8B-Base. We report the training reward, solve-none question number, and average response length throughout training.

Qwen3-4B-Base
Method	In-domain (%)				Out-of-domain (%)		Avg.
Method	AIME25	AIME26	HMMT26	MATH500	GPQA-D	HLE
Base	6.9	8.1	3.6	67.8	23.5	3.0	18.8
GRPO	21.7	14.4	10.0	84.6	34.3	2.2	27.9
LUFFY	22.1	18.1	14.0	84.2	36.6	3.8	29.8
Prefix-RFT	15.4	12.1	7.9	79.8	34.3	3.4	25.5

Table 1: Overall performance on in-domain and out-of-domain benchmarks, reported as avg@16. Bold and underlined values indicate the best and second-best results, respectively.

\triangle

denotes the absolute performance gain of our method over GRPO. Pass@

k

results are reported in Table 4 .

Method	ALFWorld (%)		ScienceWorld (%)	Avg.
Method	Seen	Unseen	ScienceWorld (%)	Avg.
Qwen3-4B-Instruct
GRPO	80.7	71.6	50.3	67.5
LUFFY	81.4	74.6	52.1	69.4
Prefix-RFT	82.1	72.4	52.9	69.1
PICO	88.6	79.1	55.3	74.3

Table 2: Performance on agentic reasoning benchmarks based on Qwen3-4B-Instruct. Bold and underlined values indicate the best and second-best results, respectively.

\triangle

denotes the absolute performance gains, in percentage points, of our method over GRPO.

Qwen3-4B-Base
Method / Variant	AIME25	AIME26	HMMT26	MATH500	Avg.
GRPO	21.7	14.4	10.0	84.6	32.7
PICO	23.1	22.3	15.9	87.6	37.2
w/o Joint Decoding	20.6	16.9	14.1	83.6	33.8
w/o Dynamic Top- $k$
Fixed $k=5$	23.3	16.9	12.7	84.2	34.3

Table 3: Ablation study of PICO on four mathematical reasoning benchmarks using Qwen3-4B-Base. “w/o Joint Decoding” directly generates responses from the teacher policy. The fixed-

k

variants replace the dynamic top-

k

mechanism with a fixed teacher candidate-set size. “w/o IS Ratio” removes the importance-sampling ratio from policy optimization. Bold and underlined values indicate the best and second-best results, respectively.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Illustration of dynamic top- $k$ joint decoding. At each decoding step, the student and teacher policies produce their respective top- $k$ token sets, whose intersection defines the candidate set $F_{t}(k)$ . The teacher-side value of $k$ is progressively increased until the retained student probability mass $Z_{t}(k)$ reaches the prescribed threshold $Z_{\min}$ . The next token is then sampled from the student distribution restricted to the resulting candidate set.

Qwen3-4B-Base
Method	AIME25	AIME26	HMMT26	Avg.
Base	26.7	16.7	18.2	20.5
GRPO	33.3	40.0	30.3	34.5
LUFFY	40.0	40.0	27.3	35.8
Prefix-RFT	40.0	36.7	24.2	33.6
ReLIFT	40.0	23.3	24.2	29.2

Appendix

Table 4: Pass@16 performance on three mathematical reasoning benchmarks. Bold and underlined values indicate the best and second-best results, respectively.

\triangle

denotes the absolute performance gains, in percentage points, of our method over GRPO.

Figure 4: Dynamics of question rescue during training. The blue curves show the number of questions remaining unsolved through student self-sampling, while the orange curves show the number remaining unsolved after distribution-constrained joint decoding. The shaded regions represent questions successfully rescued by joint decoding. Results are reported for Qwen3-4B-Base and Qwen3-8B-Base on mathematical reasoning tasks, and Qwen3-4B-Instruct on ALFWorld.

Figure 5: Evolution of the normalization term

Z_{t}

and the corresponding KL divergence,

\mathrm{KL}=-\log Z_{t}

, during training with Qwen3-4B-Base. The light blue curve shows the raw measurements, while the dark blue curve shows the smoothed trend.

Hyperparameter	Value
Common (shared by all methods)
Optimizer	AdamW
Learning rate	1e-6
Global training batch size	128
PPO mini-batch size	64
Rollout group size	8

Appendix

Table 5: Implementation and hyperparameter configuration for mathematical reasoning experiments.

Hyperparameter	Value
Common optimization configuration
Optimizer	AdamW
Learning rate	1e-6
Global training batch size	16
PPO mini-batch size	16
Rollout group size	8

Appendix

Table 6: Implementation and hyperparameter configuration for agentic experiments.

Method	Time/step (min)	Rollout Utilization (%)	Utilization/time
GRPO	28.3	57.2	2.01
PICO	43.6	91.9	2.11

Appendix

Table 7: Comparison of training efficiency for Qwen3-4B-Base on DAPO-Math-17K. All experiments use Qwen3-4B-Base and four NVIDIA H20 GPUs. Time per step is averaged over all policy-update steps. Rollout utilization is the percentage of sampled rollout groups with non-zero group-relative advantages. Utilization/time is calculated as rollout utilization divided by average time per step and is reported in percentage points per minute (higher is better).

RL for Language Models

Reinforcement learning for language models uses feedback-driven policy updates to improve reasoning, alignment, and interactive behavior.

See all 435 papers

P^2O: Joint Policy and Prompt Optimization
Mar 23, 2026Xinyu Lu, Kaiqi Zhang, Jinglin Yang +6RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences +2
Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization
Oct 26, 2026Sohyun Son, Yerim Hong, Jaeyun Park +1Reinforcement Learning RL for Language Model Reasoning
Kyung Hee University, Yongin, Republic of Korea
CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR
Mar 10, 2026Sijia Cui, Pengyu Cheng, Jiajun Song +6RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Qwen Large Model Application Team, Alibaba · Institute of Automation, Chinese Academy of Sciences

Mar 23, 2026J/K move · Enter open · S save

P^2O: Joint Policy and Prompt Optimization

Xinyu Lu, Kaiqi Zhang, Jinglin Yang, Boxi Cao+5

Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences+2

RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards Prompt Optimization Sparse-Reward RL

Open paper AskPDF arXiv

Reinforcement Learning with Verifiable Rewards (RLVR) enhances Large Language Model (LLM) reasoning but is suffer from advantage collapse: when all rollouts of a query receive identical rewards, the group variance vanishes, most damagingly on hard samples, where scaling rollout budgets yields little. We introduce Joint Policy and Prompt Optimization (P O) to mitigate this collapse by alternating continuous policy updates with discrete prompt evolution. P O mines hard samples with a success-rate threshold, evolves reasoning prompts for them with GEPA, and internalizes the elicited trajectories via context distillation, which optimizes each trajectory under the original query and thus removes inference-time prompting, with a Context Ratio Mask (CRM) filtering out extreme likelihood ratios. P O restores critical advantage signals and surpasses the GRPO baseline by up to 8.2 points in average accuracy on six held-out benchmarks across all training datasets and backbones, while also outperforming DAPO and other baselines. The gains are especially pronounced on hard benchmarks, reaching up to 16.3 points above GRPO on average across AIME24 and AIME25. Our findings expose the limits of standard exploration in sparse-reward environments, illuminating the potential of unifying evolutionary algorithms with reinforcement learning. This integration of discrete semantic search and continuous parameter updates provides a self-reinforcing framework that facilitates more effective LLM alignment.

Abstract

RL for Language Models

Explore similar work

P^2O: Joint Policy and Prompt Optimization

Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization

CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR