GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification

Wangjie Gan; Miao Pan; Linbo Xi; Wenqi Zhang; Jintao Chen; Jianwei Yin; Xuhong Zhang

cs.AIApr 15, 2026

GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification

Authors: Wangjie Gan, Miao Pan, Linbo Xi, Wenqi Zhang, Jintao Chen, Jianwei Yin, Xuhong Zhang

Organizations: School of Software Technolog,Zhejiang University

Abstract

Large language models are typically post-trained using supervised fine-tuning (SFT) and reinforcement learning (RL), yet effectively unifying efficient knowledge injection with robust generalization remains challenging. In this work, we provide a training-dynamics analysis showing that SFT can be interpreted as a special case of policy gradient optimization with an extremely sparse implicit reward and unstable inverse-probability weighting, which together lead to single-path dependency, entropy collapse, and gradient explosion. Motivated by this diagnosis, we propose Group Fine-Tuning (GFT), a unified post-training framework that addresses these intrinsic limitations through two mechanisms: Group Advantage Learning, which constructs diverse response groups and derives normalized contrastive supervision to alleviate reward sparsity, and Dynamic Coefficient Rectification, which adaptively bounds inverse-probability weights to stabilize optimization while preserving efficient knowledge injection. Experiments demonstrate that GFT consistently surpasses SFT-based methods and yields policies that integrate more smoothly with subsequent RL training.

Figures & tables

Figure 1: Performance of Qwen2.5-Math-1.5B on Numina-Math. (a) Accuracy changes relative to the base model: SFT consistently degrades performance, highlighting catastrophic forgetting. (b) Accuracy across different training pipelines: the SFT+GRPO pipeline exhibits poor synergy, underperforming GRPO alone.

Figure 2: GFT comprises two components: (1) Group Advantage Learning , which computes standardized relative advantages ( $A_{k}$ ) from hybrid response groups (expert demonstrations, teacher outputs, and rollout samples); and (2) Dynamic Coefficient Rectification , which bounds importance weights via per-token gradient clipping.

Model	Method	AMC23	College Math	Gaokao2023En	Math	Minerva Math	TabMWP
Qwen2.5-Math-1.5B	Base Model	30.16	24.30	34.81	46.54	10.51	24.55
	+ SFT	31.25 (+1.09)	36.45 (+12.15)	48.86 (+14.05)	60.66 (+14.12)	23.99 (+13.48)	79.34 (+54.79)
	+ SFT(mix)	32.70 (+2.54)	36.35 (+12.05)	50.82 (+16.01)	60.41 (+13.87)	25.76 (+15.25)	80.15 (+55.60)
	+ GRPO	44.84 (+14.68)	35.58 (+11.28)	51.80 (+16.99)	65.97 (+19.43)	21.17 (+10.66)	76.94 (+52.39)
	+ ASFT	43.12 (+12.96)	29.40 (+5.10)	47.99 (+13.18)	60.35 (+13.81)	15.55 (+5.04)	65.06 (+40.51)
	+ PSFT	31.56 (+1.40)	33.77 (+9.47)	47.66 (+12.85)	59.51 (+12.97)	19.13 (+8.62)	71.61 (+47.06)

Table 1: Main results on seven math benchmarks. SFT(mix) indicates that the dataset is a mixture of expert datasets and distilled teacher datasets, while GFT(no mix) represents using only expert datasets without distilled data. Bold and blue denote the best intra-group and overall performance, respectively. Overall, GFT achieves the best average performance across diverse model scales.

Method	AMC23	MATH	Olympiad
Base Model	30.16	46.54	23.39
GFT w/o (GAL + DCR)	31.25	60.66	24.58
GFT w/o GAL	35.78	63.91	26.63
GFT w/o DCR	42.81	65.97	27.82
GFT (Ours)	46.09	70.50	30.52

Table 2: Ablation on Qwen2.5-Math-1.5B . GAL is important for complex reasoning (e.g., Olympiad) and DCR enhances performance by ensuring optimization stability, their synergy yields optimal results.

Figure 3: Learning dynamics on MATH-lighteval. Removing DCR causes severe volatility, while removing GAL results in slow convergence and a lower ceiling.

Figure 4: Performance comparison on Qwen2.5-Math-1.5B (Pass@16). Bottom-right: Sat-Math training dynamics. SFT+GFT+GRPO achieves top performance via stable optimization, demonstrating GFT’s high compatibility and effective synergy between SFT and GRPO.

Method	Mawps	Svamp	Mmlu stem
Base Model	96.06	86.36	41.03
+SFT	91.97 (- 4.09)	78.73 (- 7.63)	35.05 (-5.98)
+GRPO	94.60 (- 0 1.46)	88.11 (+ 0 1.75)	39.48 (-1.55)
+GFT (Ours)	95.79 (- 0 0.27)	84.65 (- 0 1.71)	43.89 (+2.86)

Table 3: Performance of LLaMA-3.2-3B-Instruct on general reasoning benchmarks. While SFT induces substantial catastrophic forgetting, GFT largely preserves base performance.

Figure 5: KL divergence quantifies distributional drift from the base model. SFT exhibits the highest divergence, while GFT maintains a significantly lower level, effectively mitigating catastrophic forgetting.

Metric	Method	SAT Math	Minerva	TabMWP	Avg.
Pass@128	Base Model	39.69	9.71	24.17	24.52
	Distillation	66.67	22.98	79.32	56.32
	GRPO	52.95	19.89	76.77	49.87
	GFT	72.58	28.59	85.31	62.16
Pass@256	Base Model	38.76	9.25	24.36	24.12
	Distillation	67.20	21.84	79.28	56.11

Table 4: Comparison of Pass@k (

k=128,256

) performance between Distillation, GRPO and GFT. GFT consistently achieves the highest Pass@k scores, effectively enhancing response diversity.

Ratio	Minerva Math	Olympiad	Sat Math	Avg.
8 : 0	15.11	22.48	36.92	24.84
6 : 2	29.53	29.60	71.68	43.60
4 : 4	28.93	30.52	69.93	43.13
2 : 6	31.01	32.73	73.04	45.59
0 : 8	23.31	28.61	40.60	30.84

Table 5: Impact of group composition ratio (

N_{demo}:N_{sample}

); 2:6 achieves the best accuracy, indicating richer contrast from self-samples with demo samples.

Figure 6: Effect of the clipping threshold $\tau$ : larger $\tau$ rectifies more tokens. Accuracy follows an inverted U-shape; insufficient clipping is unstable, while excessive clipping reduces learning efficiency.

Explore similar work

Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
Apr 10, 2026Taojie Zhu, Dongyang Xu, Ding Zou +4LLM Fine-Tuning RL for Language Model Reasoning
Shenzhen International Graduate School, Tsinghua University · Intelligent System Department, Zhongxing Telecom Equipment (ZTE) · Institute of Advanced Interdisciplinary Studies, Chongqing University of Posts and Telecommunications
PriFT: Prior-Support Guided Supervised Fine-Tuning
Jun 8, 2026Ke Wang, Shuangqi Li, Mathieu Salzmann +1Supervised Fine-Tuning Fine-Tuning
EPFL, Lausanne, Switzerland
Compatibility-Aware Dynamic Fine-Tuning for Large Language Models
Apr 22, 2026Yucheng Zhou, Junwei Sheng, Qianning Wang +1Supervised Fine-Tuning Fine-Tuning
SKL-IOTSC, CIS, University of Macau · Auckland University of Technology

Apr 10, 2026J/K move · Enter open · S save

Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning

Taojie Zhu, Dongyang Xu, Ding Zou, Sen Zhao+3

Shenzhen International Graduate School, Tsinghua University · Intelligent System Department, Zhongxing Telecom Equipment (ZTE) · Institute of Advanced Interdisciplinary Studies, Chongqing University of Posts and Telecommunications

LLM Fine-Tuning RL for Language Model Reasoning RL Fine-Tuning Language Model Robustness

Open paper AskPDF arXiv

Post-training paradigms for Large Language Models (LLMs), primarily Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), face a fundamental dilemma: SFT provides stability (low variance) but suffers from high fitting bias, while RL enables exploration (low bias) but grapples with high gradient variance. Existing unified optimization strategies often employ naive loss weighting, overlooking the statistical conflict between these distinct gradient signals. In this paper, we provide a rigorous theoretical analysis of this bias-variance trade-off and propose DYPO (Dynamic Policy Optimization), a unified framework designed to structurally mitigate this conflict. DYPO integrates three core components: (1) a Group Alignment Loss (GAL) that leverages intrinsic group dynamics to significantly reduce RL gradient variance; (2) a Multi-Teacher Distillation mechanism that corrects SFT fitting bias via diverse reasoning paths; and (3) a Dynamic Exploitation-Exploration Gating mechanism that adaptively arbitrates between stable SFT and exploratory RL based on reward feedback. Theoretical analysis confirms that DYPO linearly reduces fitting bias and minimizes overall variance. Extensive experiments demonstrate that DYPO significantly outperforms traditional sequential pipelines, achieving an average improvement of 4.8% on complex reasoning benchmarks and 13.3% on out-of-distribution tasks. Our code is publicly available at https://github.com/Tocci-Zhu/DYPO.

Abstract

Explore similar work

Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning

PriFT: Prior-Support Guided Supervised Fine-Tuning

Compatibility-Aware Dynamic Fine-Tuning for Large Language Models