An Empirical Study of SFT-DPO Interaction and Parameterization in Small Language Models

Yuming Feng; Christy Yang

cs.CLMar 20, 2026

An Empirical Study of SFT-DPO Interaction and Parameterization in Small Language Models

Authors: Yuming Feng, Christy Yang

Organizations: Department of Computer Science, Stanford University

Abstract

Direct Preference Optimization (DPO) is widely used after supervised fine-tuning (SFT) to align language models, yet empirical behavior under small backbones and modest data is under-specified. We systematically compare SFT-only, DPO-only, and staged SFT-to-DPO training alongside full fine-tuning (FFT) versus LoRA on a GPT-2-scale decoder, evaluating paraphrase detection and Shakespearean sonnet continuation. DPO yields small, task-dependent gains over strong SFT and can match competitive SFT accuracy without a warm start when the preference construction closely parallels the supervised objective. In contrast, parameterization dominates: FFT consistently outperforms LoRA at matched training depth, and LoRA does not reduce wall-clock time on our hardware. These findings indicate that, in this small-scale regime, supervised full-parameter adaptation remains the primary performance lever, while preference optimization and low-rank adaptation provide limited marginal returns.

Figures & tables

Data size	Epochs	Time/epoch	Total time	Train loss	Dev loss	Acc.	Prec.	Rec.	F1
2.83k	40	30.25s	20.2 min	0.033	0.770	78.7	77.1	77.3	77.2
28.3k	15	86.16s	21.5 min	0.123	0.403	83.9	82.9	82.3	82.6
283k	8	435s	58.0 min	0.200	0.280	88.7	87.5	89.3	88.2

Table 1: Effect of training data size on paraphrase detection. Larger datasets improve development performance and reduce the train–dev loss gap.

Setting	Training Time	Configuration	Dev F1	Dev Accuracy
$\sim$ 15 min	$\sim$ 14.5 min	283k $\times$ 2 epochs	86.20	86.81
$\sim$ 15 min	$\sim$ 14 min	28.3k $\times$ 10 epochs	82.34	83.25
$\sim$ 30 min	$\sim$ 29 min	283k $\times$ 4 epochs	87.11	87.94
$\sim$ 30 min	$\sim$ 29 min	28.3k $\times$ 20 epochs	$\sim$ 82.9	$\sim$ 83.9

Table 2: Same-time training comparison between large and small datasets. Under a fixed training-time budget, fewer epochs on a larger dataset outperform many epochs on a smaller dataset.

Method	Config	Epochs	Acc.	Prec.	Rec.	F1
FFT (SFT)	full	3	89.24	88.34	88.65	88.49
LoRA	$r=16$	3	85.15	83.99	85.84	84.54
LoRA	$r=8$	3	86.34	85.12	86.18	85.56
LoRA	$r=4$	3	85.67	84.54	86.48	85.09

Table 3: Comparison of FFT and LoRA on paraphrase detection at epoch 3.

Figure 1: Training curves for FFT and LoRA ( $r=8$ ) on paraphrase detection. Plot train and dev accuracy/F1 across epochs.

Table 4: DPO hyperparameter sweeps on paraphrase detection.

Strategy	SFT ep	DPO ep	Best	Acc	Prec	Rec	F1
SFT only	10	0	SFT@9	89.87	88.89	89.59	89.21
DPO only	0	10	DPO@7	89.46	88.38	89.43	88.83
SFT@3 $\rightarrow$ DPO	3	10	DPO@9	89.77	88.78	89.54	89.12
SFT@9 $\rightarrow$ DPO	9	6	DPO@5	90.05	89.05	89.91	89.43

Table 5: Effect of SFT

\rightarrow

DPO handoff timing on paraphrase detection.

Acc / F1	SFT	DPO
FFT	89.87 / 89.21	90.05 / 89.43
LoRA	87.70 / 87.00	88.48 / 87.76

Table 6: Best dev results for training objective

\times

parameterization on paraphrase detection.

Temp	Mean	Std	Min	Max
0.5	41.29	0.255	40.94	42.10
1.0	41.54	0.362	40.44	42.05
1.2	41.70	0.308	41.20	42.32
1.5	41.86	0.280	41.20	42.42
2.0	41.78	0.573	40.09	42.45

Table 7: Effect of sampling temperature on sonnet-generation chrF across 20 seeds.

Config	Pairs	Best chrF	Epoch	$\Delta$ vs SFT
DPO V1	97	41.94	2	+0.19
DPO V2	65	41.74	4	+0.28
DPO V3	325	41.46	0	0

Table 8: Comparison of DPO preference-pair construction strategies for sonnet generation.

Case	Sentence 1	Sentence 2	Gold	Model	Result
Success	Is it biologically good or bad to marry other caste?	Is it good or bad to marry other caste?	Yes	Yes	Correct
Failure	Why do Quorans answer questions that are already answered?	Why do Quorans downvote questions they cannot answer?	No	Yes	Incorrect (FP)

Table 9: Paraphrase detection Case Study

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	2.83k	28.3k	283k (chosen)
Optimizer	AdamW	AdamW	AdamW
Learning rate	$1\times 10^{-5}$	$1\times 10^{-5}$	$1\times 10^{-5}$
Batch size	128	128	128
Epochs	40	15	8
Weight decay	0	0	0
Max words/sent.	64	64	64

Appendix

Table 10: Hyperparameters for the dataset scale experiment.

Hyperparameter	FFT (SFT)	LoRA
Optimizer	AdamW	AdamW
Learning rate	$5\times 10^{-5}$	$2\times 10^{-4}$
Batch size	128	128
Epochs	3	3
Weight decay	0	0
Max words/sent.	64	64

Appendix

Table 11: Hyperparameters for the FFT vs. LoRA rank experiment.

Table 12: DPO hyperparameters: search space (left) and hand-off phase (right).

Table 13: Hyperparameters for the Sonnet Generation task (SFT and DPO phases).

Explore similar work

Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study
Jun 11, 2026Dezhi Yu, Yvonne Qiu, ShuoJia FuLLM Alignment Direct Preference Optimization
LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
Oct 6, 2026Yang Qu, Yusheng Han, Chengjia Feng +1Direct Preference Optimization LLM Alignment
Northeastern University
AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
May 27, 2026Shaolong Chen, Madalina Ciobanu, Qingqing Mao +1LLM Alignment Direct Preference Optimization
Incept Labs Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA

Abstract

Explore similar work

Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization

AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates