cs.LGMar 20, 2026
SaveFIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization
Organizations: Qwen Pilot Team, Alibaba Group
Abstract
We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0%). This outperforms both DeepSeek-R1-Zero-Math-32B (around 47.0%) and o1-mini (approximately 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORM-based algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.
Figures & tables
Figure 1: FIPO vs. Baselines Performance Comparison on AIME2024. FIPO demonstrates that pure RL training alone is sufficient to not only outperform other pure RL baselines (the reproduced DAPO and Deepseek-R1-Zero-32B), but also surpass o1-mini. This performance gain is accompanied by the generation of significantly longer responses on average.
Figure 2: Training instability with vanilla FutureKL. Analysis of the unstable run observed around Step 70. (a) A sharp spike in the low-clip fraction (indicating a drastic shift in the policy distribution driven by negative samples) triggers (b) a sudden divergence in Policy KL. (c) an immediate explosion in gradient norm and These internal instabilities collectively precipitate (d) a catastrophic collapse in response length, confirming that unregulated negative signals destabilize the optimization.
| Method | AIME 2024 | AIME 2025 | ||||
| Avg@32 | Cons@32 | Pass@32 | Avg@32 | Cons@32 | Pass@32 | |
| DAPO (Baseline) | 50.0% | 60.0% | 80.0% | 38.0% | 47.0% | 63.0% |
| FIPO (Ours) | 56.0% | 73.0% | 83.0% | 43.0% | 50.0% | 67.0% |
Table 1: Comparison of reasoning performance on AIME benchmarks. All results are reported as percentages (%). We report the average Pass@1 across 32 samples (Avg@32), the majority vote (Cons@32), and the probability of at least one correct answer (Pass@32). To align with prior baseline reporting and reduce sensitivity to digit-level generation variance, final values are rounded to the nearest integer.
Figure 3: Dynamics of response length and performance scaling during training. Subplots (a)-(e) show the evolution of response length metrics (Min, Q25, Mean, Median, Q75) over global training steps. Compared to the DAPO baseline, FIPO significantly increases response length, effectively eliciting more extensive Chain-of-Thought reasoning. Subplot (f) demonstrates that this increased length correlates strongly with improved accuracy, suggesting that longer CoT is key to breaking performance barriers.
Figure 4: Analysis of training reward and length-weighted advantages. (a) Mean training rewards. DAPO achieves higher raw scores, which is expected as both methods incorporate an overlong penalty that suppresses the reward of longer responses. (b) Number of Sampled Batches. This metric indicates the sampling redundancy required to maintain a sufficient number of effective batches. A higher sampling need suggests the model frequently generates non-informative trajectories on the training set, serving as a potential indicator of overfitting. (c) Response length weighted mean advantages. FIPO exhibits a sustained upward trend, establishing a positive reinforcement cycle where longer responses increasingly yield positive advantages. In contrast, DAPO shows a declining trend, suggesting a failure to convert length into effective reasoning gains, which ultimately limits its performance.
Figure 5: Policy evolution and optimization dynamics. (a) Policy KL divergence. (b) Policy Entropy. (c) Gradient Norm. FIPO exhibits a more controlled policy drift and smoother update gradients than DAPO. Notably, FIPO’s rising entropy, paired with the weighted advantage trends in Fig. 4(b), indicates that the model is actively exploring a broader reasoning space where longer CoT paths increasingly correspond to correct solutions.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | DAPO (Baseline) | FIPO (Ours) |
| Shared Optimization Settings | ||
| Base Model | Qwen2.5-32B-Base | |
| Global Batch Size | 512 | |
| Group Size ( ) | 16 | |
| Learning Rate | 1e-6 | |
| LR Scheduler | Constant with 10 Warmup Steps | |
Table 2: Hyperparameter settings for Qwen2.5-32B-Base experiments. We compare the configuration of our proposed FIPO against the DAPO baseline. Most infrastructure and optimization settings remain identical to ensure a fair comparison.
| Hyperparameter | DAPO (Baseline) | FIPO (Ours) |
| Shared Optimization Settings | ||
| Base Model | Qwen2.5-7B-MATH | |
| Global Batch Size | 512 | |
| Learning Rate | 1e-6 | |
| LR Scheduler | Constant with 10 Warmup Steps | |
| Weight Decay | 0.1 | |
Table 3: Hyperparameter settings for Qwen2.5-7B-MATH experiments. We compare the configuration of our proposed FIPO against the DAPO baseline. Most infrastructure and optimization settings remain identical to ensure a fair comparison.
| Method | AIME 2024 (Pass@1) | AIME 2025 (Pass@1) |
| GRPO ( Guo et al., 2025 ) | 22.0% | 18.0% |
| DAPO | 36.0% | 18.0% |
| FIPO (Ours) | 40.0% | 19.0% |
Table 4: Qwen 2.5-7B-MATH Performance Comparison of Pass@1 Performance on AIME2024 and AIME2025. All results are reported as percentages (%). We report the peak average Pass@1 across 32 samples (Avg@32). To reduce sensitivity to digit-level generation variance, final values are rounded to the nearest integer.
Figure 6: Analysis of length-weighted advantages and mean response length of Qwen2.5-7B-MATH. (a) Subfigure (a) shows FIPO maintains a stable Length-Weighted Mean Advantage. (b) Subfigure (b) shows the mean response length remains steady around 1200 tokens. This suppressed length growth is likely caused by the 4K context window limit of the base model, and its inherent bias toward concise code-based reasoning, which anchors the generation depth.
Figure 7: Policy evolution and optimization dynamics of Qwen2.5-7B-MATH. (a) Policy KL and (b) Gradient Norm illustrate the optimization path. (c) Notably, FIPO results in lower entropy compared to DAPO, indicating that the model converges toward more deterministic reasoning traces during the RL process.
Figure 8: Ablation study on Response Length Control on Qwen2.5-32B-Base. We compare the baseline FIPO with FIPO under cliphigh 1.4 and FIPO with a max resonse length of 25K. (a-d) Increasing the maximum length limit triggers a significant early surge in response length, yet results in lower training efficiency and marginal performance gains. (e) A higher clip ratio causes a sharp escalation in policy entropy, indicating potential training instability. (f) Overall, while both modifications encourage longer responses, they are less effective in improving AIME24 performance compared to the balanced FIPO configuration.
| Method | AIME 2024 (Pass@1) | AIME 2025 (Pass@1) |
| FIPO ( , ) | 36.0% | 19.0% |
| FIPO (w/o filtering, , ) | 38.0 % | 21.0 % |
| FIPO ( , ) | 40.0% | 19.0% |
Table 5: Ablation on Extreme Value Filtering and Influence Weight Clipping Performance Comparison of Pass@1 Performance on AIME2024 and AIME2025 on Qwen2.5-7B-MATH. All results are reported as percentages (%). We report the peak average Pass@1 across 32 samples (Avg@32).
Figure 9: Ablation Study on Filtering Mechanism in Future-KL on Qwen2.5-7B-MATH. We compare the training stability of FIPO with and without filtering extreme importance sampling (IS) ratios. (a) Future-KL influence weights: removing extreme IS ratios results in a significantly more constrained and compact weight range, leading to more stable policy updates. (b-c) Clipping statistics: extreme IS ratios rise disproportionate influence weights that drive more aggressive updates. This is reflected in the increased dual clip and policy clip fractions, where a larger proportion of tokens are pushed beyond the trust region when filtering is absent.
Figure 10: Ablation Study on Influence Weight Clipping on Qwen-2.5-7B-MATH. (a) Similar to 32B, a influence weight with range resulted in a continuous growth in entropy, thus encouraging more explorations. (b) However, we do not observe any surge in response length as we discussed in the previous section.
Figure 11: Ablation Study on the Decay Rate Horizon ( ) on Qwen2.5-7B-MATH. (a) Mean Influence Weight: Extending the decay horizon (e.g., ) significantly increases weight fluctuations, potentially destabilizing policy updates. Conversely, a highly restrictive horizon (e.g., ) limits the weights too close to the baseline. (b) Influence Weight Clip Ratio: A larger decay horizon results in a higher clipping frequency. This reflects more drastic variations in the raw influence weights, directly corroborating the trends observed in (a). (c) Entropy: An intermediate horizon ( ) strikes a critical balance. It avoids the excessive volatility induced by larger decay rates, as well as the premature entropy collapse caused by the myopic guidance of shorter horizons, allowing the 7B model to maintain controlled and effective exploration.
| Method | AIME 2024 (Pass@1) | AIME 2025 (Pass@1) |
| FIPO ( =8) | 40.0% | 17.0% |
| FIPO ( =32) | 40.0% | 19.0% |
| FIPO ( =128) | 39.0% | 21.0% |
| FIPO ( =256) | 42.0% | 16.0% |
Table 6: Ablation on the Effective Horizon of Decay Rate Performance Comparison of Pass@1 Performance on AIME2024 and AIME2025 on Qwen2.5-7B-MATH. All results are reported as percentages (%). We report the peak average Pass@1 across 32 samples (Avg@32).
Figure 12: Training Dynamics and Stability across Mini-batch Sizes on Qwen2.5-32B-Base. (a) Response Length Mean: While a mini-batch size of 32 can occasionally yield strong performance, it suffers from severe reproducibility issues and frequently encounters length stagnation. Notably, simply removing the overlong penalty (w/o Overlong) fails to resolve this bottleneck. (b) Policy Clip Fraction: In the successful mini-batch 32 trial, fewer tokens are clipped compared to the failed cases, a phenomenon that aligns with our findings in the clip_high ablation study. Because a mini-batch size of 64 is closer to on-policy and suffers less from importance sampling (IS) weight fluctuations, it naturally results in fewer clipped tokens, thereby easing the scaling of response length and performance. (c) Gradient Norm: Similarly, the successful mini-batch 32 run maintains a noticeably stabler and slightly lower gradient norm than the failed ones, a critical stabilizing property inherently achieved by the mini-batch 64 baseline. (d) Policy Entropy: The successful case sustains higher entropy throughout the training, indicating healthier exploration capabilities. (e-f) AIME24 Performance: Benefiting from stable optimization and sustained exploration, the successful mini-batch 32 case reaches a peak Mean@32 of approximately 60% and a Cons@32 of 70% , which is highly comparable to the final performance of our mini-batch 64 baseline.
Figure 13: Stage 1: Superficial Planning (Initial Step). A baseline response from Qwen 2.5 32B. The model generates a "template-like" outline of the solution steps but fails to execute any actual mathematical derivation, leading to a hallucinated conclusion.
Figure 14: Stage 2: Linear Execution (DAPO Convergence, and FIPO early stage). While the model correctly executes a standard Chain-of-Thought (CoT) to reach the ground truth, its reasoning is strictly linear and terminates immediately upon finding the first result, lacking any self-verification mechanism.
Figure 15: Stage 3: Emergent Self-Reflection (FIPO Intermediate Stage). A distinct behavioral shift occurs as FIPO training progresses. The model begins to utilize the expanded token budget for spontaneous self-reflection , cross-validating its initial conclusion through alternative mathematical paths.
Figure 16: Stage 4: Systematic Deep Reasoning (FIPO Late Stage). In the late stages of training, the model converges on a "compute-heavy" strategy . It moves beyond simple reflection to perform rigorous multi-pass auditing —including symbolic re-derivation and granular arithmetic verification—to ensure better performance in complex reasoning tasks.
More in this topic
RL for Language Model Reasoning
Reinforcement learning for language model reasoning uses reward-driven training to improve multi-step problem solving, tool use, and reliable answers.
See all 1,094 papers