Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions

Xiaoyi Li

cs.LGMar 19, 2026

Do Post-Training Algorithms Actually Differ? A Controlled Study Across Model Scales Uncovers Scale-Dependent Ranking Inversions

Authors: Xiaoyi Li

Abstract

Post-training alignment has produced dozens of competing algorithms -- DPO, SimPO, KTO, GRPO, and others -- yet practitioners lack controlled comparisons to guide algorithm selection. We present OXRL, a unified framework implementing 51 post-training algorithms with identical infrastructure, enabling the first large-scale apples-to-apples evaluation. Our study spans 8 algorithms across 4 model scales (0.5B--7B), 3 evaluation domains, and a 20-variant DPO taxonomy (100 runs at 1.5B, 5 seeds each), totaling $\sim$ 240 training runs on H100 GPUs. Three headline findings emerge. (1)~~Algorithm rankings are unstable across scale: at 1.5B, online RL (SGRPO) tops all methods at 58.0%~~ $\pm$ 0.57 on GSM8K; by 7B, the worst small-scale method (SimPO) becomes the best (85.8%), a complete ranking inversion driven by model scale rather than LoRA regularization (confirmed via 2 $\times$ 2 factorial). (2)~~Loss function modifications yield negligible gains: none of 20 DPO variants significantly outperform vanilla DPO after Bonferroni correction; the sole significant outlier, SimPO, is worse ( $-$ 11.5~~pp, $p < 10^{-4}$ ). (3)~~Algorithm leverage is task-specific: the 19.3~~pp GSM8K spread collapses to 0.54~~pp on MATH ( $36\times$ ) and 0.47~~pp on general-domain benchmarks ( $41\times$ ), confirming that algorithm choice matters primarily within the training distribution. These findings yield a hierarchy of leverage for practitioners: model scale ( ${\sim}$ 50~~pp) $\gg$ training paradigm ( ${\sim}$ 10~~pp) $\gg$ online vs.\ offline ( ${\sim}$ 9~~pp) $\gg$ loss function ( ${\sim}$ 1~~pp). We release all code, configs, and evaluation data as a living community benchmark.

Figures & tables

Variant	Primary Modification	Ref.	Year
DPO	Baseline logistic loss	✓	2023
IPO	Squared loss regularization	✓	2024
SimPO	Length-normalized, no ref	$\times$	2024
KTO	Unpaired binary feedback	✓	2024
Hinge	Hinge loss replacement	✓	2024
CPO	Contrastive preference	$\times$	2024

Table 1: Taxonomy of 20 DPO variants evaluated in this study, organized by primary modification to the DPO objective (Eq. 1 ). All evaluated at 1.5B on GSM8K with 5 seeds.

		3B	7B
Full FT	Base	10.69	75.82
	SFT	18.35 ‡ ( +7.7 )	—
	DPO	14.44 ±2.01 ( +3.8 )	—
	SimPO	6.90 ‡ ( $-$ 3.8 )	—
LoRA	Base	10.69	75.82
	SFT	18.04 ( +7.4 )	76.42 ( +0.6 )

Table 2: The 2

\times

2 factorial disentangling scale from LoRA. GSM8K strict-match accuracy (%). Full FT at 3B uses seed-fix reruns: DPO reports mean

\pm\sigma

over 2 seeds; ‡ single seed available. At 3B, the full FT/LoRA gap is negligible for all methods (SFT: 0.31 pp, DPO: 1.56 pp—within DPO’s seed variance of

\sigma=2.01

, SimPO: 0.00 pp). The 3B

\to

7B gains (DPO: +67.9 pp, SimPO: +78.9 pp) are attributable to scale . “—” = not run (7B full FT).

Figure 1: GSM8K accuracy across model scales. Dashed gray: base model. Error bars:

\pm

\sigma

where multi-seed data is available. The V-shaped trajectories from 3B to 7B visualize the scale-driven ranking inversion; DPO’s error bar at 3B reflects genuine seed variance (

\sigma=2.01

pp). At 1.5B, SGRPO achieves the highest accuracy (58.0%). Note: 0.5B–3B use full FT; 7B uses LoRA.

Algorithm	0.5B	1.5B	3B (full FT)	7B (LoRA)
Base Model	19.56	31.16	10.69 §	75.82
SFT	33.97 ±0.0 ( +14.4 )	54.36 ±0.59 ( +23.2 )	18.35 ‡ ( +7.7 )	76.42 ±0.0 ( +0.6 )
DPO	33.97 ±0.0 ( +14.4 )	49.08 ±0.61 ( +17.9 )	14.44 ±2.01 ( +3.8 )	83.85 ±0.0 ( +8.0 )
IPO	34.50 ±0.3 ( +14.9 )	52.24 ±0.22 ( +21.1 )	—	81.35 ( +5.5 )
KTO	33.81 ±0.0 ( +14.3 )	51.15 ±1.77 ( +20.0 )	15.31 ‡ ( +4.6 )	80.21 ( +4.4 )
SimPO	26.08 ±0.0 ( +6.5 )	38.67 ±1.78 ( +7.5 )	6.90 ‡ ( $-$ 3.8 )	85.82 ±0.0 ( +10.0 )

Table 3: GSM8K exact-match accuracy (%) across scales. 1.5B values are seed-fix means with

\pm\sigma

across 3 seeds (Appendix G ). 3B uses full FT with seed-fixed reruns; DPO reports mean

\pm\sigma

over 2 seeds, ‡ single seed available. LoRA results at 3B appear in Table 2 . Best per scale in bold . † Epoch-1 checkpoint. ∗ Pathological failure. § 62.9% with flexible extraction but 10.69% strict-match, indicating format non-compliance. CISPO = Clipped Importance-Sampled Policy Optimization, a sentence-level GRPO variant with per-token ratio clipping.

Algorithm	MATH (%)	GSM8K (%)	$\Delta$ Rank
SFT	27.04	54.36 ±0.59	$\uparrow$ (2nd $\to$ 1st)
SimPO	26.72	38.67 ±1.78	$\uparrow$ (5th $\to$ 2nd)
KTO	26.64	51.15 ±1.77	$=$ (3rd $\to$ 3rd)
SGRPO	26.58	58.00 ±0.57	$\downarrow$ (1st $\to$ 4th)
DPO	26.50	49.08 ±0.61	$\downarrow$ (4th $\to$ 5th)
Spread	0.54 pp	19.33 pp	$36\times$ compression

Table 4: Cross-task comparison at 1.5B: GSM8K (seed-fix mean) vs. MATH accuracy (%). The 19.3 pp GSM8K spread collapses to 0.54 pp on MATH (

36\times

compression). Rankings invert: SimPO rises from last to 2nd, SGRPO drops from 1st to 4th.

Algorithm	ARC-C (%)	HellaSwag (%)	WinoGrande (%)	Avg (%)
Base Model	46.76	68.23	62.90	59.30
SFT	46.16 ( $-$ 0.60 )	68.16 ( $-$ 0.07 )	62.90 (+0.00)	59.08 ( $-$ 0.22 )
DPO	46.67 ( $-$ 0.09 )	68.38 ( +0.15 )	62.67 ( $-$ 0.24 )	59.24 ( $-$ 0.06 )
IPO	46.76 (+0.00)	68.36 ( +0.13 )	62.67 ( $-$ 0.24 )	59.26 ( $-$ 0.04 )
KTO	47.01 ( +0.26 )	68.25 ( +0.02 )	63.06 ( +0.16 )	59.44 ( +0.14 )
SimPO	46.08 ( $-$ 0.68 )	68.37 ( +0.14 )	62.59 ( $-$ 0.32 )	59.01 ( $-$ 0.29 )

Table 5: General-domain evaluation at 1.5B: ARC-Challenge (acc_norm), HellaSwag (acc_norm), WinoGrande (acc). All models trained on GSM8K. The 19.3 pp GSM8K spread collapses to 0.47 pp on general reasoning (

41\times

compression). No method meaningfully deviates from the base model.

Figure 2: GSM8K accuracy across 5 seeds for 20 DPO variants at 1.5B (100 runs total). Dashed line: vanilla DPO mean (49.76%). Error bars:

\pm

1 std. † SimPO is the only variant significantly different from DPO (

p<0.0026

after Bonferroni correction)—and it is worse .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Category	Mean (%)	Std	$\Delta$ DPO	$p$	$d$
ORPO	Reference-free	53.89	0.60	+4.12	0.0134	2.48
DPOP	Weighting/calibration	53.62	1.80	+3.85	0.0189	1.88
SPPO	Data augmentation	52.60	1.45	+2.84	0.0521	1.49
IPO	Alternative divergence	52.36	0.75	+2.59	0.0614	1.53
CalDPO	Weighting/calibration	52.16	1.58	+2.40	0.0934	1.22
CPO	Reference-free	51.43	1.57	+1.67	0.2187	0.85

Appendix

Table 6: Full DPO variant results at 1.5B scale on GSM8K (5 seeds).

\Delta

is the accuracy difference from vanilla DPO. † indicates statistical significance after Bonferroni correction (

p<0.0026

). Cohen’s

d

measures practical effect size.

Parameter	Value
Optimizer	AdamW ( $\beta_{1}\!=\!0.9$ , $\beta_{2}\!=\!0.95$ , $\epsilon\!=\!10^{-8}$ )
Weight decay	0.01
LR schedule	Cosine with 10% linear warmup
Effective batch size	8 prompts (1 $\times$ 8 grad. accum.)
Training epochs	3
Precision	BF16 mixed precision

Appendix

Table 7: Shared training configuration across all experiments.

Algorithm	Learning Rate	Key Hyperparameters	GPUs/Run
SFT	$1\!\times\!10^{-6}$	–	1
DPO	$1\!\times\!10^{-6}$	$\beta=0.1$	1
SimPO	$1\!\times\!10^{-6}$	$\beta=0.1$ , $\gamma=0.5$	1
IPO	$1\!\times\!10^{-6}$	$\beta=0.1$	1
KTO	$1\!\times\!10^{-6}$	$\beta=0.1$	1
SGRPO	$1\!\times\!10^{-6}$	$\lambda_{\text{KL}}=0$ , $n=4$	2

Appendix

Table 8: Algorithm-specific hyperparameters.

Figure 3: Training dynamics comparison at 0.5B scale. Left: training loss over steps (smoothed). Right: validation loss over epochs. DPO converges to lower loss; SimPO maintains higher, noisier loss with flat validation loss.

Figure 4: Training determinism at 0.5B: per-seed GSM8K accuracy for each algorithm across 3 seeds. Five of six algorithms produce identical accuracy (

\sigma=0

). Only IPO shows variance (

\sigma=0.27

Scale	Algorithm	Strict (%)	Flexible (%)	Gap (pp)
1.5B	SFT	54.66	55.57	$-0.9$
	DPO	51.18	49.81	$+1.4$
	IPO	52.39	52.31	$+0.1$
	KTO	51.71	51.18	$+0.5$
	SimPO	43.44	43.06	$+0.4$
3B (full FT)	Base	10.69	62.93	$-52.2$

Appendix

Table 9: Strict-match vs. flexible-extract accuracy on GSM8K. 3B full FT values are from original single runs (pre-seed-fix). At 7B, SimPO has the best format compliance (+4.9 pp) while SFT has the worst (

-

3.9 pp).

Algorithm	ARC-C (%)	HellaSwag (%)	WinoGrande (%)	Avg (%)
Base Model	55.20	80.48	71.11	68.93
SFT	55.29 ( +0.09 )	80.45 ( $-$ 0.03 )	70.72 ( $-$ 0.39 )	68.82 ( $-$ 0.11 )
DPO	54.86 ( $-$ 0.34 )	80.58 ( +0.10 )	70.72 ( $-$ 0.39 )	68.72 ( $-$ 0.21 )
IPO	55.55 ( +0.35 )	80.37 ( $-$ 0.11 )	71.35 ( +0.24 )	69.09 ( +0.16 )
KTO	55.03 ( $-$ 0.17 )	80.39 ( $-$ 0.09 )	70.72 ( $-$ 0.39 )	68.71 ( $-$ 0.22 )
SimPO	54.44 ( $-$ 0.77 )	80.53 ( +0.05 )	70.17 ( $-$ 0.95 )	68.38 ( $-$ 0.56 )

Appendix

Table 10: General-domain evaluation at 7B (LoRA): ARC-Challenge (acc_norm), HellaSwag (acc_norm), WinoGrande (acc). Despite a 9.4 pp GSM8K spread at 7B, the general-domain spread is just 0.71 pp. Rankings bear no resemblance to GSM8K rankings.

Category	# Runs	Time/Run	GPUs	GPU-h
Core 0.5B SL (5 algs $\times$ 3 seeds)	15	4–8 min	1	1.5
Core 0.5B RL (3 algs $\times$ 3 seeds)	9	$\sim$ 3 h	2	54
Core 1.5B SL (5 algs $\times$ 3 seeds)	15	14–27 min	1	5.0
Core 3B SL full FT (3 algs)	3	12–13 min	1	0.6
3B LoRA factorial (3 algs)	3	12–13 min	1	0.6
Core 7B SL (5 algs $\times$ 2 seeds)	10	335–362 min	1	58

Appendix

Table 11: Compute budget breakdown on H100 80GB GPUs.

Algorithm	0.5B	1.5B	3B	7B (LoRA)
SFT	4 min	14 min	12 min	$\sim$ 330 min
DPO	4 min	20 min	13 min	$\sim$ 360 min
SimPO	4 min	14 min	12 min	$\sim$ 330 min
IPO	4 min	26 min	–	$\sim$ 335 min
KTO	4 min	23 min	–	$\sim$ 460 min

Appendix

Table 12: Wall-clock training time (3 epochs) for SL methods on a single H100.

Scale	Method	GPUs	Time	Gain (pp)	GPU-h	GPU-h/pp
0.5B	SFT/DPO	1	4 min	+14.4	0.07	0.005
	IPO	1	4 min	+14.9	0.07	0.004
	SGRPO	2	183 min	+12.9	6.1	0.47
1.5B	SGRPO	2	$\sim$ 84 min	+26.8	2.8	0.10
	SFT	1	14 min	+23.2	0.23	0.010
	DPO	1	20 min	+17.9	0.33	0.018

Appendix

Table 13: Compute-performance Pareto frontier. GPU-h/pp = GPU-hours per percentage point gained over base. At 1.5B, SGRPO achieves highest accuracy at

10\times

SFT’s cost per point.

Method	Original	Mean	$\sigma$	s42	s123	s456
SFT	54.66	54.36	0.59	53.90	55.19	53.98
IPO	52.39	52.24	0.22	52.54	52.16	52.01
KTO	51.71	51.15	1.77	51.33	53.22	48.90
DPO	51.18	49.08	0.61	49.58	49.43	48.22
SimPO	43.44	38.67	1.78	36.85	38.06	41.09

Appendix

Table 14: Seed-fix results at 1.5B on GSM8K with corrected DistributedSampler (3 seeds each, 15 runs total). Rankings preserved; SFT remains the strongest offline method. IPO shows the lowest variance.

Method	Original	Mean	$\sigma$	s42	s123
SFT	18.04	18.35 ‡	—	—	18.35
KTO	—	15.31 ‡	—	—	15.31
DPO	16.45	14.44	2.01	16.45	12.43
SimPO	6.90	6.90 ‡	—	6.90	—

Appendix

Table 15: Seed-fix results at 3B (full FT) on GSM8K. ‡ Single seed available. DPO shows substantially higher seed variance at 3B (

\sigma=2.01

) than at 1.5B (

\sigma=0.61

), suggesting scale-dependent training instability.

Language Model Post-Training

Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.

See all 193 papers

The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It
Oct 20, 2026Hannan Mahadik, David Salinas, Jacek Golebiowski +1Direct Preference Optimization Hyperparameter Optimization
ELLIS Institute Tübingen Germany · ELLIS Institute Tübingen & Prior Labs Germany · distil labs Germany
Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3LLM Alignment Direct Preference Optimization
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
May 27, 2026Shaolong Chen, Madalina Ciobanu, Qingqing Mao +1LLM Alignment Direct Preference Optimization
Incept Labs Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA

Oct 20, 2026J/K move · Enter open · S save

The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

Hannan Mahadik, David Salinas, Jacek Golebiowski, Aaron Klein

ELLIS Institute Tübingen Germany · ELLIS Institute Tübingen & Prior Labs Germany · distil labs Germany

Direct Preference Optimization Hyperparameter Optimization LLM Post-Training Hyperparameter Transfer

Open paper AskPDF arXiv

Direct preference optimisation (DPO) is widely used for language model post-training, but its performance depends strongly on the choice of its hyperparameters, such as the learning rate and the reference-policy regularisation parameter

β

. To better understand the DPO hyperparameter landscape, we conduct extensive experiments using a dense hyperparameter grid across different model scales from the same family. This allows us to analyse how the optimal DPO hyperparameters evolve with model size and the number of update steps, with intermediate checkpoints providing an inexpensive signal to accelerate the hyperparameter optimisation process. Furthermore, by evaluating our models on both static and dynamic benchmarks, we identify which benchmarks provide the strongest signal for hyperparameter selection. Finally, we investigate how different state-of-the-art hyperparameter optimisation strategies perform in this setting. Our experiments show that the optimal learning rate varies with model size and that hyperparameters are not transferable across training datasets. We also find that validation loss is a poor proxy for hyperparameter optimisation, as hyperparameters that minimize validation loss only weakly correlate with optimal hyperparameters for downstream benchmarks. Therefore, DPO hyperparameters need to be tuned specifically for the target model, dataset, and evaluation objective. We also show that short candidate training runs can substantially reduce the cost of this selection. We release all our training configurations and evaluation results to support reproducible analysis of selection quality and training cost.

Abstract

Language Model Post-Training

Explore similar work

The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates