Constraints are essential for stabilizing reinforcement learning fine-tuning (RFT) and preventing degenerate outputs, yet they inherently conflict with the optimization objective because stronger constraints limit the ability of a fine-tuned model to discover better solutions. We propose dynamic constraints that resolve this tension by adapting to the evolving capabilities of the fine-tuned model based on the insight that constraints should only intervene when degenerate outputs occur. We implement this by using a reference model as an online refiner that takes the response from the fine-tuned model and generates a minimally corrected version which preserves correct content verbatim while fixing errors. A supervised fine-tuning loss then trains the fine-tuned model to produce the refined output. This mechanism yields a constraint that automatically strengthens or relaxes based on output quality. Experiments on dialogue and code generation show that dynamic constraints outperform both KL regularization and unconstrained baselines, achieving substantially higher task rewards while maintaining training stability.
Figures & tables
Figure 1: An illustration of the insight of dynamic constraint. (a) The left figure illustrates the conventional KL regularization, which constrains the fine-tuned policy πθ to remain close to the reference policy π0 . However, when the optimal policy lies far from π0 and requires πθ to deviate significantly, the KL regularization becomes an obstacle to policy optimization. (b) The right figure presents the insight of the dynamic constraint, where the constraint is derived from π0 ’s refined response based on πθ ’s response and context. When πθ deviates substantially from π0 , the dynamic constraint not only avoids hindering policy optimization but also provides effective guidance and correction for πθ .
Figure 2: The pipeline for calculating a dynamic constraint. After πθ generates a response a for the query s0 , a refiner LLM is employed to refine the response based on the necessary context (quesry s0 , response a ) and a predefined template. The refinement process is conservative. In most cases, the refiner simply repeats a when no obvious issues are detected, while in other cases it makes only minimal edits to a when necessary. The detailed template is provided in Appendix A . The dynamic constraint is computed as a cross entropy loss that treats the refined response a′ as the ground truth, which is equivalent to the SFT loss on the pair ⟨s0,a′⟩ .
Figure 3: Dynamic constraint from a dataset perspective. The dynamic constraint can be interpreted as an RFT-SFT hybrid training approach with a dynamically updated SFT dataset.
Figure 4: Training dynamics on Prompt-Collection-v0.1 (top) and APPS (bottom). (a, d) Dynamic (orange) achieves continuous reward growth, whereas Static (blue) saturates and w/o constraint (red) collapses. (b, e) The KL divergence of Dynamic rises steadily, indicating deep exploration beyond the initial policy π0 , while Static remains tethered. (c, f) The cross entropy remains low, confirming that the Refiner πrefiner successfully tracks the evolving policy πθ .
Figure 5: Training curves compared with DAPO on the APPS dataset. The right figure shows the proportion of rollouts improved by the Refiner LLM.
Model
HumanEval
HumanEval+
MBPP
MBPP+
Avg.
Qwen2.5-Coder-1.5B-Instruct
12.8
11.6
36.2
40.2
25.2
DAPO
18.0 (+40.6%)
15.9 (+37.1%)
39.4 (+8.8%)
43.9 (+9.2%)
29.3 (+16.3%)
Dynamic
20.7 (+61.7%)
19.9 (+71.6%)
44.4 (+22.7%)
46.8 (+16.4%)
33.0 (+30.8%)
Table 1: Pass@1 (%) on Python code generation benchmarks. Qwen2.5-Coder-1.5B-Instruct is fine-tuned using DAPO and our method, respectively, and evaluated on four code generation benchmarks with the bigcode-evaluation-harness .
Figure 6: Demonstration of how the dynamic constraint refines model responses. The two panels show Python backtracking solutions for the LeetCode problem "1415. The k-th Lexicographical String of All Happy Strings of Length n". (a) The left panel presents the original response, which contains an indexing error. (b) The right panel shows the refined response produced by the Refiner LLM, where the model enforces a more accurate condition ( 3*(2*(n-1)) ). The color on the left indicates the cross entropy values for each token.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Prompt-Collection-v0.1
APPS
Actor Learning Rate
1e-6
1e-6
Critic Learning Rate
9e-6
9e-6
Epochs
1
7
PPO Epoch
1
4
Batch Size
64
128
Mini Batch Size
16
16
Appendix
Table 2: Hyperparameters in Section 4.1 .
Hyperparameter
DAPO
Dynamic
Actor Learning Rate
1e-6
1e-6
Epochs
5
5
Batch Size
64
64
Mini Batch Size
8
8
Samples per Prompt
8
8
Iterations
150
150
Appendix
Table 3: Hyperparameters in Section 4.2 .
Figure 7: Ablation studies on the filtering mechanism and dynamic constraint coefficient η . The curves show training dynamics under different configurations: with/without the filter that excludes refined responses with lower rewards, and with different values of η that control the strength of the dynamic constraint.
Table 4: Examples of Prompt-Collection-v0.1.
Figure 8: The reward of the original response is −0.3 , while that of the refined response is 1.0 . The original response also misses the initial signal python , which is correctly restored in the refined version.
Figure 9: The reward of the original response is −1.0 , while that of the refined response is 1.0 . The refined response eliminates redundant operations and enforces a clearer logical structure.
Figure 10: The reward of the original response is −1.0 , while that of the refined response is 1.0 . The original response produces an incorrect computation branch, which the refined response corrects by revising the conditional structure.
Figure 11: The reward of the original response is −0.6 , while that of the refined response is 1.0 . The refined version improves loop control and resolves syntax inconsistencies present in the original code.
Supervised Fine-Tuning (SFT) is the predominant paradigm for aligning large language models (LLMs), yet it suffers from optimization instability and limited generalization. Recent work attributes this issue to pathological gradient scaling and proposes Dynamic Fine-Tuning (DFT) to correct it at the token level. However, DFT assumes all demonstrations are equally suitable learning targets, an assumption violated by the strong heterogeneity of large-scale instruction data, where demonstration-policy mismatch induces high-variance updates at the sample level. We introduce Compatibility-Aware Dynamic Fine-Tuning (CADFT), a principled extension of DFT that controls sample-level optimization variance. CADFT derives a dynamic, policy-dependent compatibility signal from model likelihoods to modulate supervised updates, suppressing high-variance gradients from incompatible demonstrations. We further propose a delayed, low-frequency compatibility-guided rewriting strategy to transform persistently incompatible demonstrations into learnable targets. We show that CADFT can be interpreted as a variance-controlled estimator that generalizes token-level stabilization in DFT to the sample level. Extensive experiments demonstrate improved stability, generalization, and cold-start reinforcement learning initialization, while remaining fully supervised and independent of explicit reward modeling.
Yucheng Zhou, Junwei Sheng, Qianning Wang +1
SKL-IOTSC, CIS, University of Macau · Auckland University of Technology
Reinforcement fine-tuning (RFT) is increasingly used to strengthen the reasoning abilities of large models, yet its effectiveness is bound by how training data are selected and used. Most data-centric RFT methods rely on static or heuristic sample selection, implicitly assuming a sample's value is fixed over training. This overlooks the non-stationary dynamics of policy learning and can lead to suboptimal updates. We propose Dynamic Important Example Mining (DIEM), a principled and fully automated framework that makes data utilization adaptive throughout RFT. DIEM integrates two components into each optimization step: (i) a gradient-alignment importance estimator that efficiently approximates each sample's marginal contribution to policy improvement; and (ii) a constrained batch reweighting scheme that maximizes aggregate utility while preserving the update's gradient magnitude to stabilize optimization. Across several reasoning benchmarks, DIEM consistently outperforms strong static and dynamic baselines. The code will be released via https://github.com/hrtan/DIEM.
Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT often improves pass@16 performance on math and coding evaluations, with gains up to 17% absolute, while incurring minimal computational overhead. These higher-coverage checkpoints consistently translate to up to 4% absolute pass@1 gains in subsequent GRPO runs, demonstrating that TailSFT checkpoints serve as better initializations for RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.
Sadhika Malladi, Samy Jelassi, Dylan Foster +2
University of California San Diego · Microsoft Research NYC