cs.LGOct 2, 2026

Understanding Enrichment in Reinforcement Learning

Authors: Jinwoo Kim, Shraddha Barke

Organizations: University of California-San Diego La Jolla, CA, USA · Microsoft Research · Microsoft Research Redmond, WA, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When rewards are sparse, reinforcement learning with verifiable rewards (RLVR) often uses hints or intermediate guidance to generate more successful rollouts. This enrichment biases policy-gradient updates unless corrected via importance weights, but existing methods omit correction or truncate importance weights in order to avoid the high variance of correction. It thus remains unclear what exactly is gained or lost in RLVR by correcting enriched rollouts. We show, mathematically, that omitted or truncated correction implicitly reweights the defined reward, and we decompose the resulting gradient error into scale, rotation, and variance to explain their distinct effects on learning. To make correction practical, we develop a novel sequential Monte Carlo (SMC) weight correction mechanism that, under stability and particle-order assumptions, tempers the exponential compounding of standard correction variance over the length of a sample to an additive accumulation. We then apply our analysis of enrichment to interpreting the results of a fine-tune of Qwen3-1.7B on a sparse band of OpenMathReasoning, establishing a concrete mechanism of how enrichment, both corrected and uncorrected, helps avoid collapse in sparse domains. Our main contribution is to understand, in general, how enrichment and correction can affect training, as opposed to claiming that either mode of operation is superior to unenriched RL.

Figures & tables

Figure 1: Scale-modified (left, y-axis) and original objectives (right, y-axis) tracked across training steps kk (x-axis), for the scale-isolated simulated training runs, over different values of the scale coefficient α\alpha . We use θk\theta_{k} to denote the parameters at step kk .
Figure 2: Reward evolution across update steps for uncorrected (orange) and corrected enriched (green) training.
Figure 3: Percentage of reward <0.9<0.9 endpoints generated by ( i ) corrected enriched and ( ii ) base REINFORCE, mapped against batch size.
Figure 4: Results of training Qwen3-1.7B on the described OpenMathReasoning subset, using ( i ) uncorrected enrichment, ( ii ) SMC-corrected enrichment, and ( iii ) unenriched training. Panel (a) maps pass@1 percentages on the eval set across training steps. Panel (b) illustrates the advantage that uncorrected and SMC-corrected enrichment have compared to the unenriched baseline across training steps. Panel (c) maps training typicality, the average length-normalized likelihood of the training samples according to each policy being trained, across training steps. Panel (d) maps pass@ kk across kk after all 500 training steps completed, illustrating that the two enriched arms achieve an increase compared to the base untrained policy while unenriched training results in a decrease.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Normalized variance compared over enriched base correction (as described at the head of Section 3 , SMC correction, and unenriched estimators.
More in this topic

Importance Sampling for RL

See all 16 papers

Explore similar work

CardsList
Jul 13, 2026

SCOPE-RL: Optimizing Reasoning Paths Before and After Success

Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.

Xiaojian Liu, Han Xu, Jianqiang Xia +6

Baidu Inc. · Shandong University

Reward ShapingRL for Language Model ReasoningEfficient Language Model ReasoningReinforcement Learning with Verifiable RewardsReward Design for RL
AskPDF
May 14, 2026

Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models. However, RLVR training is often hindered by sparse binary rewards and weak credit assignment, resulting in ambiguous optimization signals and underutilization of the useful information embedded in failed trajectories. To address this challenge, we propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision, without relying on any external signals. By jointly optimizing correction samples derived from the model's own failed attempts together with the standard RLVR objective, CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors. Extensive experiments across 11 benchmarks spanning mathematical reasoning and code generation demonstrate that CIPO consistently and significantly outperforms strong baselines in both reasoning and correction performance. Moreover, CIPO yields stronger pass@K gains, indicating that it improves the model's intrinsic reasoning capacity rather than merely redistributing probability mass over existing correct answers.

Mengjie Ren, Jie Lou, Boxi Cao +6

Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.

RL for Code GenerationLLM Self-CorrectionRL for Language Model ReasoningReinforcement Learning with Verifiable Rewards
AskPDF
Oct 14, 2026

When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?

Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient ββ controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing ββ within a range that prevents overshoot strictly accelerates optimization. When gradients are estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate ηη decreases, the expected gain with fixed ββ vanishes faster than these fluctuations, whereas increasing ββ to keep η/(1−β)η/(1-β) within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing β1β_1 improves mean validation accuracy by nearly 22 percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale.

Kaifei Wang, Zikang Shan, Liwei Wang +2

Peking University · Shanghai Jiao Tong University, SIMIS, and Stanford University · Shanghai Jiao Tong University

Bandit LearningMomentum MethodsReinforcement Learning with Verifiable RewardsRL Post-Training
AskPDF