When rewards are sparse, reinforcement learning with verifiable rewards (RLVR) often uses hints or intermediate guidance to generate more successful rollouts. This enrichment biases policy-gradient updates unless corrected via importance weights, but existing methods omit correction or truncate importance weights in order to avoid the high variance of correction. It thus remains unclear what exactly is gained or lost in RLVR by correcting enriched rollouts. We show, mathematically, that omitted or truncated correction implicitly reweights the defined reward, and we decompose the resulting gradient error into scale, rotation, and variance to explain their distinct effects on learning. To make correction practical, we develop a novel sequential Monte Carlo (SMC) weight correction mechanism that, under stability and particle-order assumptions, tempers the exponential compounding of standard correction variance over the length of a sample to an additive accumulation. We then apply our analysis of enrichment to interpreting the results of a fine-tune of Qwen3-1.7B on a sparse band of OpenMathReasoning, establishing a concrete mechanism of how enrichment, both corrected and uncorrected, helps avoid collapse in sparse domains. Our main contribution is to understand, in general, how enrichment and correction can affect training, as opposed to claiming that either mode of operation is superior to unenriched RL.
Figures & tables
Figure 1: Scale-modified (left, y-axis) and original objectives (right, y-axis) tracked across training steps k (x-axis), for the scale-isolated simulated training runs, over different values of the scale coefficient α . We use θk to denote the parameters at step k .
Figure 2: Reward evolution across update steps for uncorrected (orange) and corrected enriched (green) training.
Figure 3: Percentage of reward <0.9 endpoints generated by ( i ) corrected enriched and ( ii ) base REINFORCE, mapped against batch size.
Figure 4: Results of training Qwen3-1.7B on the described OpenMathReasoning subset, using ( i ) uncorrected enrichment, ( ii ) SMC-corrected enrichment, and ( iii ) unenriched training. Panel (a) maps pass@1 percentages on the eval set across training steps. Panel (b) illustrates the advantage that uncorrected and SMC-corrected enrichment have compared to the unenriched baseline across training steps. Panel (c) maps training typicality, the average length-normalized likelihood of the training samples according to each policy being trained, across training steps. Panel (d) maps pass@ k across k after all 500 training steps completed, illustrating that the two enriched arms achieve an increase compared to the base untrained policy while unenriched training results in a decrease.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Normalized variance compared over enriched base correction (as described at the head of Section 3 , SMC correction, and unenriched estimators.
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.