Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner

Hao Ma; Zhiqiang Pu; Yang Liu; Xiaolin Ai

cs.LGMar 18, 2026

Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner

Authors: Hao Ma, Zhiqiang Pu, Yang Liu, Xiaolin Ai

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences

Abstract

Constraints are essential for stabilizing reinforcement learning fine-tuning (RFT) and preventing degenerate outputs, yet they inherently conflict with the optimization objective because stronger constraints limit the ability of a fine-tuned model to discover better solutions. We propose dynamic constraints that resolve this tension by adapting to the evolving capabilities of the fine-tuned model based on the insight that constraints should only intervene when degenerate outputs occur. We implement this by using a reference model as an online refiner that takes the response from the fine-tuned model and generates a minimally corrected version which preserves correct content verbatim while fixing errors. A supervised fine-tuning loss then trains the fine-tuned model to produce the refined output. This mechanism yields a constraint that automatically strengthens or relaxes based on output quality. Experiments on dialogue and code generation show that dynamic constraints outperform both KL regularization and unconstrained baselines, achieving substantially higher task rewards while maintaining training stability.

Figures & tables

Panel of Figure 1 — Figure 1: An illustration of the insight of dynamic constraint. (a) The left figure illustrates the conventional KL regularization, which constrains the fine-tuned policy $\pi_{\theta}$ to remain close to the reference policy $\pi_{0}$ . However, when the optimal policy lies far from $\pi_{0}$ and requires $\pi_{\theta}$ to deviate significantly, the KL regularization becomes an obstacle to policy optimization. (b) The right figure presents the insight of the dynamic constraint, where the constraint is derived from $\pi_{0}$ ’s refined response based on $\pi_{\theta}$ ’s response and context. When $\pi_{\theta}$ deviates substantially from $\pi_{0}$ , the dynamic constraint not only avoids hindering policy optimization but also provides effective guidance and correction for $\pi_{\theta}$ .

Figure 2: The pipeline for calculating a dynamic constraint. After

\pi_{\theta}

generates a response

a

for the query

s_{0}

, a refiner LLM is employed to refine the response based on the necessary context (quesry

s_{0}

, response

a

) and a predefined template. The refinement process is conservative. In most cases, the refiner simply repeats

a

when no obvious issues are detected, while in other cases it makes only minimal edits to

a

when necessary. The detailed template is provided in Appendix A . The dynamic constraint is computed as a cross entropy loss that treats the refined response

a^{\prime}

as the ground truth, which is equivalent to the SFT loss on the pair

\langle s_{0},a^{\prime}\rangle

Figure 3: Dynamic constraint from a dataset perspective. The dynamic constraint can be interpreted as an RFT-SFT hybrid training approach with a dynamically updated SFT dataset.

Panel of Figure 4 — Figure 4: Training dynamics on Prompt-Collection-v0.1 (top) and APPS (bottom). (a, d) Dynamic (orange) achieves continuous reward growth, whereas Static (blue) saturates and w/o constraint (red) collapses. (b, e) The KL divergence of Dynamic rises steadily, indicating deep exploration beyond the initial policy $\pi_{0}$ , while Static remains tethered. (c, f) The cross entropy remains low, confirming that the Refiner $\pi_{\text{refiner}}$ successfully tracks the evolving policy $\pi_{\theta}$ .

Panel of Figure 5 — Figure 5: Training curves compared with DAPO on the APPS dataset. The right figure shows the proportion of rollouts improved by the Refiner LLM.

Model	HumanEval	HumanEval+	MBPP	MBPP+	Avg.
Qwen2.5-Coder-1.5B-Instruct	12.8	11.6	36.2	40.2	25.2
DAPO	18.0 (+40.6%)	15.9 (+37.1%)	39.4 (+8.8%)	43.9 (+9.2%)	29.3 (+16.3%)
Dynamic	20.7 (+61.7%)	19.9 (+71.6%)	44.4 (+22.7%)	46.8 (+16.4%)	33.0 (+30.8%)

Table 1: Pass@1 (%) on Python code generation benchmarks. Qwen2.5-Coder-1.5B-Instruct is fine-tuned using DAPO and our method, respectively, and evaluated on four code generation benchmarks with the bigcode-evaluation-harness .

Panel of Figure 6 — Figure 6: Demonstration of how the dynamic constraint refines model responses. The two panels show Python backtracking solutions for the LeetCode problem "1415. The k-th Lexicographical String of All Happy Strings of Length n". (a) The left panel presents the original response, which contains an indexing error. (b) The right panel shows the refined response produced by the Refiner LLM, where the model enforces a more accurate condition ( 3*(2*(n-1)) ). The color on the left indicates the cross entropy values for each token.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Prompt-Collection-v0.1	APPS
Actor Learning Rate	1e-6	1e-6
Critic Learning Rate	9e-6	9e-6
Epochs	1	7
PPO Epoch	1	4
Batch Size	64	128
Mini Batch Size	16	16

Appendix

Table 2: Hyperparameters in Section 4.1 .

Hyperparameter	DAPO	Dynamic
Actor Learning Rate	1e-6	1e-6
Epochs	5	5
Batch Size	64	64
Mini Batch Size	8	8
Samples per Prompt	8	8
Iterations	150	150

Appendix

Table 3: Hyperparameters in Section 4.2 .

Panel of Figure 7 — Figure 7: Ablation studies on the filtering mechanism and dynamic constraint coefficient $\eta$ . The curves show training dynamics under different configurations: with/without the filter that excludes refined responses with lower rewards, and with different values of $\eta$ that control the strength of the dynamic constraint.

Panel of Table 4 — Table 4: Examples of Prompt-Collection-v0.1.

Panel of Figure 8 — Figure 8: The reward of the original response is $-0.3$ , while that of the refined response is $1.0$ . The original response also misses the initial signal python , which is correctly restored in the refined version.

Panel of Figure 9 — Figure 9: The reward of the original response is $-1.0$ , while that of the refined response is $1.0$ . The refined response eliminates redundant operations and enforces a clearer logical structure.

Panel of Figure 10 — Figure 10: The reward of the original response is $-1.0$ , while that of the refined response is $1.0$ . The original response produces an incorrect computation branch, which the refined response corrects by revising the conditional structure.

Panel of Figure 11 — Figure 11: The reward of the original response is $-0.6$ , while that of the refined response is $1.0$ . The refined version improves loop control and resolves syntax inconsistencies present in the original code.

RL Fine-Tuning

Reinforcement learning fine-tuning adapts pretrained models and policies through task rewards, preference feedback, or environment interaction.

See all 230 papers

Compatibility-Aware Dynamic Fine-Tuning for Large Language Models
Apr 22, 2026Yucheng Zhou, Junwei Sheng, Qianning Wang +1Supervised Fine-Tuning Fine-Tuning
SKL-IOTSC, CIS, University of Macau · Auckland University of Technology
Dynamic Important Example Mining for Reinforcement Finetuning
Aug 29, 2026Haoru Tan, Sitong Wu, Yanfeng Chen +9RL Fine-Tuning Training Data Selection
HKU · Hunyuan Team, Tencent · CUHK +1
TailSFT: Filtered Fine-Tuning Improves Post-Training Performance
Aug 26, 2026Sadhika Malladi, Samy Jelassi, Dylan Foster +2Supervised Fine-Tuning Instruction-Tuning Data Selection
University of California San Diego · Microsoft Research NYC

Apr 22, 2026J/K move · Enter open · S save

Compatibility-Aware Dynamic Fine-Tuning for Large Language Models

Yucheng Zhou, Junwei Sheng, Qianning Wang, Jianbing Shen

SKL-IOTSC, CIS, University of Macau · Auckland University of Technology

Supervised Fine-Tuning Fine-Tuning LLM Fine-Tuning Instruction Tuning

Open paper AskPDF arXiv

Supervised Fine-Tuning (SFT) is the predominant paradigm for aligning large language models (LLMs), yet it suffers from optimization instability and limited generalization. Recent work attributes this issue to pathological gradient scaling and proposes Dynamic Fine-Tuning (DFT) to correct it at the token level. However, DFT assumes all demonstrations are equally suitable learning targets, an assumption violated by the strong heterogeneity of large-scale instruction data, where demonstration-policy mismatch induces high-variance updates at the sample level. We introduce Compatibility-Aware Dynamic Fine-Tuning (CADFT), a principled extension of DFT that controls sample-level optimization variance. CADFT derives a dynamic, policy-dependent compatibility signal from model likelihoods to modulate supervised updates, suppressing high-variance gradients from incompatible demonstrations. We further propose a delayed, low-frequency compatibility-guided rewriting strategy to transform persistently incompatible demonstrations into learnable targets. We show that CADFT can be interpreted as a variance-controlled estimator that generalizes token-level stabilization in DFT to the sample level. Extensive experiments demonstrate improved stability, generalization, and cold-start reinforcement learning initialization, while remaining fully supervised and independent of explicit reward modeling.

Abstract

RL Fine-Tuning

Explore similar work

Compatibility-Aware Dynamic Fine-Tuning for Large Language Models

Dynamic Important Example Mining for Reinforcement Finetuning

TailSFT: Filtered Fine-Tuning Improves Post-Training Performance