The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

Hannan Mahadik; David Salinas; Jacek Golebiowski; Aaron Klein

cs.LGOct 20, 2026

The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

Authors: Hannan Mahadik, David Salinas, Jacek Golebiowski, Aaron Klein

Organizations: ELLIS Institute Tübingen, Germany · ELLIS Institute Tübingen & Prior Labs, Germany · distil labs, Germany

Abstract

Direct preference optimisation (DPO) is widely used for language model post-training, but its performance depends strongly on the choice of its hyperparameters, such as the learning rate and the reference-policy regularisation parameter $β$ . To better understand the DPO hyperparameter landscape, we conduct extensive experiments using a dense hyperparameter grid across different model scales from the same family. This allows us to analyse how the optimal DPO hyperparameters evolve with model size and the number of update steps, with intermediate checkpoints providing an inexpensive signal to accelerate the hyperparameter optimisation process. Furthermore, by evaluating our models on both static and dynamic benchmarks, we identify which benchmarks provide the strongest signal for hyperparameter selection. Finally, we investigate how different state-of-the-art hyperparameter optimisation strategies perform in this setting. Our experiments show that the optimal learning rate varies with model size and that hyperparameters are not transferable across training datasets. We also find that validation loss is a poor proxy for hyperparameter optimisation, as hyperparameters that minimize validation loss only weakly correlate with optimal hyperparameters for downstream benchmarks. Therefore, DPO hyperparameters need to be tuned specifically for the target model, dataset, and evaluation objective. We also show that short candidate training runs can substantially reduce the cost of this selection. We release all our training configurations and evaluation results to support reproducible analysis of selection quality and training cost.

Figures & tables

Panel of Figure 1 — Figure 1: DPO performance is sensitive to the jointly tuned learning rate and $\beta$ . Each cell is one configuration in the displayed $4\times 4$ Qwen3-4B subgrid, scored by Elo ( left ) and HellaSwag ( right ); higher is better. Elo spans roughly 295 points, and the two evaluations select different configurations.

Figure 2: Validation loss is an unreliable DPO selection signal. Left: Normalised final-performance regret from selecting the configuration with lowest final validation loss at each Qwen3 size; zero denotes the best and one the worst configuration under the corresponding aggregate. Right: Spearman correlation between negative final validation loss and final

Z_{\mathrm{Static}}

Z_{\mathrm{Dynamic}}

, and

Z_{\mathrm{All}}

scores. Positive values indicate that lower loss is associated with better evaluation performance. The minimum-loss choice is never near-optimal, and correlations are weak and inconsistent across scales.

Qwen3 size	Chat ( $\bar{\rho}\pm\sigma$ )	Static ( $\bar{\rho}\pm\sigma$ )	Val. loss vs chat	Val. loss vs static
0.6B	$0.47\pm 0.20$	$0.03\pm 0.54$	$-0.19\pm 0.10$	$-0.18\pm 0.36$
1.7B	$0.84\pm 0.10$	$0.27\pm 0.55$	$0.50\pm 0.17$	$0.38\pm 0.37$
4B	$0.78\pm 0.14$	$0.18\pm 0.37$	$0.37\pm 0.06$	$0.13\pm 0.08$
8B	$0.57\pm 0.17$	$0.01\pm 0.64$	$0.16\pm 0.15$	$0.01\pm 0.34$
14B	$0.63\pm 0.18$	$-0.10\pm 0.58$	$0.47\pm 0.20$	$-0.08\pm 0.40$

Table 1: Chat benchmarks produce more consistent DPO configuration rankings than static benchmarks on Qwen3. Mean pairwise Spearman correlation and population standard deviation across within-group benchmark pairs, computed separately at each size over jointly scored configurations. Validation-loss columns are the mean

\rho

(-\text{eval loss})

vs. each benchmark in that group

Figure 3: Dynamic and static evaluations favour different Qwen3-4B DPO learning rates. Left: Final

Z_{\mathrm{Dynamic}}

and

Z_{\mathrm{Static}}

scores. The red curve marks the Pareto frontier, and squares and circles identify configurations that maximize the

Z_{\mathrm{Static}}

and

Z_{\mathrm{Dynamic}}

scores. Right: Best score at each learning rate after maximizing over

\beta

Figure 4: Learning-rate trend and configuration-ranking stability under $Z_{\mathrm{Dynamic}}$ across Qwen3 sizes. Left: Per-size response-surface estimates of the preferred learning rate and a descriptive fit across all five sizes. Right: Spearman correlations between rankings of the nine configurations in the shared ( $3\times 3$ ) grid. Positive values indicate similar configuration rankings and hence more plausible transfer, whereas negative values indicate ranking reversals. Rankings agree within the 0.6B–1.7B and 4B–14B groups but reverse across them.

Figure 5: HPO recommendation regret over time under the GP surrogate. Each panel shows final-fidelity

Z_{\mathrm{Dynamic}}

recommendation regret for one Qwen3 size as a function of normalised wall-clock budget; lower is better. Lines show the mean and the shaded region shows the 95%

t

-interval for random search across 25 seeds. Results for other surrogates appear in Appendix D.4 .

Surrogate	TPE	CQR	ASHA	BOHB
Gaussian process	$+0.05^{\dagger}$	$+0.08^{\dagger}$	$+0.08^{\dagger}$	$\mathbf{+0.10}^{\dagger}$
$1$ -NN	$+0.04$	$\mathbf{+0.09}^{\dagger}$	$+0.01$	$+0.03$
$3$ -NN	$+0.06^{\dagger}$	$\mathbf{+0.11}^{\dagger}$	$+0.04^{\dagger}$	$+0.05^{\dagger}$
$5$ -NN	$+0.09^{\dagger}$	$\mathbf{+0.16}^{\dagger}$	$+0.02$	$+0.06^{\dagger}$

Table 2: Final

Z_{\mathrm{Dynamic}}

improvement over random search under the

8\times

wall-clock budget. Differences are paired by seed after averaging the five Qwen3 sizes within each of 25 seeds. Bold marks the largest mean improvement for each surrogate;

\dagger

denotes a paired 95% interval excluding zero.

Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Hub split	Training	Validation
Dolci-Instruct-SFT	2,152,112	1,944,341	—
Dolci-Instruct-DPO	259,922	258,622	1,300
UltraFeedback	61,135	60,000	1,135

Appendix

Table 3: Datasets used for post-training. Hub-split sizes and the numbers retained for training and validation after preprocessing. Repository names and configurations are given in the text.

Setting	SFT	Primary DPO
Epochs	2	1
Learning rate	$3\times 10^{-5}$	Swept
$\beta$	—	Swept
Per-device batch size	1	1
Global batch size	128	128
Optimiser	adamw_torch	adamw_torch

Appendix

Table 4: Shared SFT and DPO optimisation settings. Learning rate and

\beta

are the variables swept during DPO; their per-model search spaces are given in Table 7 . UltraFeedback uses the primary DPO values except where stated in the text.

Series	Size	Training	Evaluation	Total
Qwen3/Dolci	0.6B	234	410	644
	1.7B	315	356	671
	4B	1,787	723	2,510
	8B	1,578	500	2,078
	14B	2,809	614	3,423
	Subtotal	6,723	2,603	9,326

Appendix

Table 5: H100 GPU-hours for the reported DPO experiments. Training and evaluation columns include completed jobs only. ’Dolci’ denotes Dolci-Instruct-DPO and ’UF’ denotes UltraFeedback Binarized.

Models	SFT GPUs	SFT strategy	DPO GPUs	DPO strategy
Qwen3-0.6B and Qwen3-1.7B	8	DDP	8	DDP
Qwen3-4B	8	DDP	8	ZeRO-3
Qwen3-8B	8	ZeRO-2	8	ZeRO-3
Qwen3-14B	16	ZeRO-3	8	ZeRO-3
Llama-3.2-1B and Llama-3.2-3B	8	DDP	8	DDP
Llama-3.1-8B	8	ZeRO-3	8	ZeRO-3

Appendix

Table 6: Distributed training configuration. Each node contributes four GPUs. DDP denotes distributed data parallel training ( Li et al., 2020b ) ; ZeRO stages refer to DeepSpeed parameter and optimiser sharding ( Rasley et al., 2020 ) .

Figure 6: SFT learning-rate sensitivity on dynamic evaluations. Mean $Z_{\mathrm{Dynamic}}$ score after SFT, standardized separately within each Qwen3 size across the five SFT learning rates. Rankings vary by model size, but $5\times 10^{-4}$ is consistently poor.

Model	DPO search space	# Configurations
Qwen3-0.6B	Core $+\,(8\times 10^{-6},0.01)$	10
Qwen3-1.7B	Core $+\,(8\times 10^{-6},0.01)$	10
Qwen3-4B	Core + size-specific extensions	18
Qwen3-8B	$\{0.4,0.8,1,2,4\}\times 10^{-6}\times\{0.01,0.02,0.04\}$	15
Qwen3-14B	Core + size-specific extensions	17
Llama-3.2-1B	Core	9

Appendix

Table 7: DPO search spaces. “Core” denotes the shared

3\times 3

Cartesian product defined above. Added configurations extend this support for a particular model or dataset.

Benchmark	Split ( $n$ )	Shots	Metric
ARC-Challenge	test (1,172)	25	Normalised accuracy
GPQA-Diamond	train (198)	0	Normalised accuracy
GSM8K	test (1,319)	5	Strict exact match
HellaSwag	validation (10,042)	10	Normalised accuracy
PIQA	validation (1,838)	0	Normalised accuracy
TruthfulQA-MC2	validation (817)	0	Accuracy

Appendix

Table 8: Static evaluation settings. Shots are the few-shot examples supplied by the evaluation harness.

Evaluation	Comparator	Reported score
Arena-Hard v2.0	Matched SFT checkpoint	Bootstrapped win rate
MT-Bench	Matched SFT checkpoint	Pairwise win rate
AlpacaEval 2.0	GPT-4 reference; 805 prompts	Length-controlled win probability
Elo	LMArena pool; 500 prompts/opponent	Bradley–Terry rating

Appendix

Table 9: Chat-evaluation protocols. The SFT comparator is the checkpoint used to initialize the corresponding DPO run.

Evaluation	Candidate temperature	Judge temperature	Token limit
Arena-Hard v2.0	0	0	4,096 / 4,096
MT-Bench	Category-specific	0	4,096 / 2,048
AlpacaEval 2.0	0	1 (one token)	2,048 / 1
Elo	0.6 ( $p=0.95$ )	0.6 ( $p=0.95$ )	32,768 / 32,768

Appendix

Table 10: Decoding settings for chat evaluation. Limits are candidate/judge output tokens.

Figure 7: Benchmark-level configuration-ranking correlations and negative validation loss across Qwen3 sizes. Each cell gives the Spearman correlation between two final configuration rankings within one model-size grid. The final row and column use negative validation loss, so positive values indicate that lower validation loss is associated with stronger downstream performance. Benchmark agreement and alignment with validation loss both vary substantially by metric and model size.

Figure 8: Agreement of final DPO signals with

Z_{\mathrm{All}}

across Qwen3 sizes. Dashed lines rank configurations across the full grid; solid lines rank learning rates separately within each

\beta

before pooling the ranks. Blue denotes negative validation loss, so positive correlation means lower loss agrees with higher

Z_{\mathrm{All}}

; green denotes reward accuracy.

Figure 9: Preferred Qwen3 DPO learning rates differ by selection signal. Each point is the learning-rate component of a two-dimensional quadratic response-surface optimum over

(\log_{10}\mathrm{LR},\log_{10}\beta)

at one Qwen3 size. The three downstream aggregates are maximized, whereas validation loss is minimized. The estimates describe preferences within the measured grids and are not extrapolated scaling laws.

Figure 10: Objective-specific Qwen3 learning-rate trends. Left: $Z_{\mathrm{All}}$ . Right: $Z_{\mathrm{Static}}$ . Each point is the learning-rate estimate from a two-dimensional response surface at one Qwen3 size; dashed lines are descriptive log-linear fits across the five estimates. These within-family fits are not held-out predictions.

Model	$Z_{\mathrm{All}}$	$Z_{\mathrm{Macro}}$	$Z_{\mathrm{Dynamic}}$	$Z_{\mathrm{Static}}$
Qwen3-0.6B	$(4\times 10^{-6},0.01)$	$(4\times 10^{-6},0.01)$	$(8\times 10^{-6},0.01)^{\uparrow}$	$(4\times 10^{-6},0.04)$
Qwen3-1.7B	$(8\times 10^{-6},0.01)^{\uparrow}$	$(8\times 10^{-6},0.01)^{\uparrow}$	$(4\times 10^{-6},0.02)$	$(8\times 10^{-6},0.01)^{\uparrow}$
Qwen3-4B	$(2\times 10^{-6},0.01)$	$(1\times 10^{-6},0.01)$	$(4\times 10^{-7},0.01)^{\downarrow}$	$(2\times 10^{-6},0.01)$
Qwen3-8B	$(1\times 10^{-6},0.01)$	$(4\times 10^{-7},0.01)^{\downarrow}$	$(4\times 10^{-7},0.01)^{\downarrow}$	$(4\times 10^{-6},0.01)^{\uparrow}$
Qwen3-14B	$(4\times 10^{-7},0.01)$	$(4\times 10^{-7},0.01)$	$(4\times 10^{-7},0.01)$	$(4\times 10^{-6},0.02)^{\uparrow}$

Appendix

Table 11: Best observed final Qwen3 configuration under each aggregate. Each entry is

(\mathrm{learning\ rate},\beta)

and is selected from the full grid available at that model size.

\downarrow

and

\uparrow

mark the lowest and highest tested learning rates, respectively. Wherever selected,

\beta=0.01

is the lower tested bound; the 0.6B

Z_{\mathrm{Static}}

winner uses the upper bound

\beta=0.04

. Boundary selections do not locate unconstrained optima.

Figure 11: Dynamic–static trade-offs at the remaining Qwen3 sizes. Each point is one final DPO configuration, with

Z_{\mathrm{Dynamic}}

and

Z_{\mathrm{Static}}

standardized within its model-size grid. Red segments mark the per-size Pareto frontier; squares and circles identify the configurations maximizing

Z_{\mathrm{Static}}

and

Z_{\mathrm{Dynamic}}

, respectively. Point colour denotes DPO learning rate on a common logarithmic scale.

Figure 12: Final $Z_{\mathrm{All}}$ response surfaces across Qwen3 sizes. Each cell gives the final all-benchmark aggregate score of one evaluated DPO configuration; lighter cells indicate stronger performance. The evaluated grids differ by size, so blank cells denote unobserved configurations and colour scales are normalised separately within each panel. The surfaces show a shift toward lower preferred learning rates at larger scales, but also substantial size-specific variation in the relative ordering of configurations.

Figure 13: Validation loss remains an unreliable selection signal under equal group weighting. The $Z_{\mathrm{Macro}}$ aggregate gives chat and static benchmarks equal total weight. Selecting the minimum-validation-loss configuration remains far from the best $Z_{\mathrm{Macro}}$ configuration at every Qwen3 size, and validation-loss rankings agree only weakly with $Z_{\mathrm{Macro}}$ .

Figure 14: Cross-scale ranking instability persists under equal group weighting. Each cell gives Spearman’s $\rho$ between rankings of the nine configurations in the shared $3\times 3$ grid. $Z_{\mathrm{All}}$ weights all eleven benchmarks equally, whereas $Z_{\mathrm{Macro}}$ assigns equal total weight to chat and static benchmarks. Neither aggregate shows a consistent relationship between model-size distance and ranking agreement.

Figure 15: Winner recovery under benchmark bootstrap resampling. Bars show how often resampling the benchmarks within an aggregate selects the same configuration as the original aggregate on the shared nine-configuration core. The dotted line gives the $1/9$ uniform reference. $Z_{\mathrm{Dynamic}}$ has the most stable winner at every Qwen3 size.

Figure 16: Pairwise agreement between Qwen3 configuration rankings. Each coloured line gives the Spearman correlation between one pair of benchmarks at each model size. Black diamonds denote the mean across the six chat pairs (left) or 21 static pairs (right), and the shaded regions show one population standard deviation across pairs. “Dynamic” in the figure denotes the four chat-based evaluations.

Panel of Figure 17 — Figure 17: Intermediate Qwen3 rankings increasingly resemble the final ranking, but convergence is scale- and objective-dependent. Each panel compares the ranking of all available $(\mathrm{learning\ rate},\beta)$ configurations at an intermediate checkpoint with their ranking after the full DPO run. Lines report Spearman correlation for $Z_{\mathrm{All}}$ , $Z_{\mathrm{Dynamic}}$ , and $Z_{\mathrm{Static}}$ .

Figure 18: Early target-model selection can reduce tuning cost, but its reliability varies by scale. Final normalised

Z_{\mathrm{All}}

regret when the configuration is selected using only the scores at each intermediate checkpoint. Regret is zero for the best final configuration and one for the worst.

Size	Dolci winner	UltraFeedback winner	Dolci $\rightarrow$ UF	UF $\rightarrow$ Dolci
0.6B	$(4\times 10^{-6},0.01)$	$(4\times 10^{-6},0.01)$	$0.00$	$0.00$
1.7B	$(4\times 10^{-6},0.01)$	$(4\times 10^{-6},0.04)$	$0.25$	$0.11$
4B	$(2\times 10^{-6},0.01)$	$(4\times 10^{-6},0.04)$	$0.67$	$0.92$
8B	$(1\times 10^{-6},0.01)$	$(4\times 10^{-6},0.04)$	$1.00$	$1.00$
14B	$(1\times 10^{-6},0.01)$	$(4\times 10^{-6},0.04)$	$1.00$	$1.00$

Appendix

Table 12: Preference-data sensitivity across Qwen3 sizes.

Z_{\mathrm{All}}

winners are selected from the shared

3\times 3

grid. Transfer regret evaluates each dataset’s winner on the other dataset and normalizes using the target dataset’s best and worst shared-grid configurations. All Dolci winners use the lower tested

\beta

boundary, while all UltraFeedback winners use the upper tested learning-rate boundary; the latter also select the upper

\beta

boundary at 1.7B–14B.

Panel of Figure 19 — Figure 19: Intermediate-ranking stability across UltraFeedback model sizes. Spearman correlation between each intermediate configuration ranking and the final ranking on the shared nine-configuration grid. Dynamic rankings usually become informative by roughly $30\%$ of training, whereas static rankings fluctuate more across checkpoints and sizes.

Qwen3 size	Dynamic ( $\bar{\rho}\pm\sigma$ )	Static ( $\bar{\rho}\pm\sigma$ )
0.6B	$0.49\pm 0.21$	$0.09\pm 0.35$
1.7B	$0.62\pm 0.14$	$0.40\pm 0.34$
4B	$0.39\pm 0.18$	$0.16\pm 0.48$
8B	$0.57\pm 0.12$	$0.53\pm 0.31$
14B	$0.57\pm 0.19$	$-0.01\pm 0.57$

Appendix

Table 13: Benchmark-ranking agreement under UltraFeedback. Mean pairwise Spearman correlation and population standard deviation across within-group benchmark pairs, computed separately at each size over the shared nine configurations.

Model	$Z_{\mathrm{All}}$	$Z_{\mathrm{Dynamic}}$	$Z_{\mathrm{Static}}$
Llama 3.2-1B	$(2\times 10^{-6},0.01)$	$(4\times 10^{-6},0.01)^{\uparrow}$	$(2\times 10^{-6},0.01)$
Llama 3.2-3B	$(4\times 10^{-6},0.01)^{\uparrow}$	$(2\times 10^{-6},0.01)$	$(4\times 10^{-6},0.01)^{\uparrow}$
Llama 3.1-8B	$(4\times 10^{-6},0.02)^{\uparrow}$	$(1\times 10^{-6},0.02)^{\downarrow}$	$(4\times 10^{-6},0.02)^{\uparrow}$

Appendix

Table 14: Best observed final Llama configuration under each aggregate. Each entry is

(\mathrm{learning\ rate},\beta)

, selected from the nine-configuration grid.

\downarrow

and

\uparrow

mark the lowest and highest tested learning rates, respectively;

\beta=0.01

is the lower tested bound wherever selected. Boundary selections do not locate unconstrained optima.

Model	Chat	Chat without AE	Static
Llama 3.2-1B	$0.65\pm 0.18$	$0.54\pm 0.13$	$0.05\pm 0.62$
Llama 3.2-3B	$0.84\pm 0.09$	$0.87\pm 0.07$	$0.37\pm 0.57$
Llama 3.1-8B	$0.32\pm 0.61$	$0.92\pm 0.04$	$0.01\pm 0.66$

Appendix

Table 15: Agreement between Llama configuration rankings. Entries are the mean pairwise Spearman correlation

\rho

and population standard deviation

\sigma

across benchmark pairs. “Chat without AE” excludes AlpacaEval 2.0 as a sensitivity analysis.

Benchmark pair	Llama 3.2-1B	Llama 3.2-3B	Llama 3.1-8B
Arena-Hard–MT-Bench	$0.63$	$0.85$	$0.95$
Arena-Hard–Elo	$0.63$	$0.97$	$0.95$
MT-Bench–Elo	$0.35$	$0.80$	$0.87$
Arena-Hard–AlpacaEval	$0.95$	$0.88$	$-0.28$
MT-Bench–AlpacaEval	$0.60$	$0.68$	$-0.42$
AlpacaEval–Elo	$0.73$	$0.88$	$-0.17$

Appendix

Table 16: Pairwise Spearman correlations between Llama chat-benchmark rankings. Each correlation is computed over the nine DPO configurations at that model size. The upper block contains the three pairs retained by the analysis without AlpacaEval 2.0.

Panel of Figure 20 — Figure 20: Intermediate-to-final ranking stability for the Llama grids. The top row reports Llama 3.2-1B and 3B. The bottom row reports Llama 3.1-8B using the complete evaluation suite (left) and with AlpacaEval 2.0 excluded from $Z_{\mathrm{All}}$ and $Z_{\mathrm{Dynamic}}$ (right). Lines give the Spearman correlation between intermediate and final configuration rankings.

Figure 21: Early target-model selection on the Llama grids. Final normalised

Z_{\mathrm{All}}

regret of the configuration selected at each intermediate checkpoint. Regret is zero for the best final configuration and one for the worst.

Qwen3 size	$Z_{\mathrm{Dynamic}}$	Arena-Hard	MT-Bench	AlpacaEval	Elo
0.6B	$-0.27$	$0.53$	$-0.17$	$0.27$	$-0.73$
1.7B	$0.90$	$0.98$	$0.87$	$0.95$	$0.73$
4B	$0.98$	$0.85$	$0.93$	$0.97$	$0.93$
8B	$0.90$	$0.78$	$0.52$	$0.83$	$0.57$
14B	$0.95$	$0.83$	$0.45$	$0.88$	$0.77$

Appendix

Table 17: Agreement between configuration rankings produced by the two LLM judges on the shared Qwen3 core. Entries are Spearman correlations over the nine learning-rate–

\beta

configurations shared by every Qwen3 size.

Z_{\mathrm{Dynamic}}

averages separately standardized Arena-Hard, MT-Bench, AlpacaEval 2.0, and Elo scores.

Surrogate	RS score	TPE $\Delta$	CQR $\Delta$	ASHA $\Delta$	BOHB $\Delta$
GP	$1.07$ $[1.04,1.10]$	$+0.052$ $[0.005,0.100]$	$+0.077$ $[0.034,0.120]$	$+0.083$ $[0.050,0.116]$	$\mathbf{+0.098}$ $[0.058,0.138]$
$1$ -NN	$1.22$ $[1.18,1.27]$	$+0.04$ $[-0.02,0.11]$	$\mathbf{+0.09}$ $[0.04,0.15]$	$+0.01$ $[-0.05,0.07]$	$+0.03$ $[-0.02,0.09]$
$3$ -NN	$1.07$ $[1.04,1.11]$	$+0.06$ $[0.01,0.12]$	$\mathbf{+0.11}$ $[0.07,0.16]$	$+0.04$ $[0.01,0.08]$	$+0.05$ $[0.02,0.09]$
$5$ -NN	$0.95$ $[0.93,0.98]$	$+0.09$ $[0.03,0.14]$	$\mathbf{+0.16}$ $[0.11,0.20]$	$+0.02$ $[-0.01,0.05]$	$+0.06$ $[0.03,0.10]$

Appendix

Table 18: Final

Z_{\mathrm{Dynamic}}

scores and paired improvements under the

8\times

budget. The random-search column gives its mean score and 95%

t

-interval. Other columns give the paired difference from random search, after averaging the five sizes within each seed. Intervals are over 25 seed-level differences. Higher values are better; bold marks the largest mean improvement in each row. Comparisons use unrounded interval endpoints.

Qwen3 size	TPE	CQR	ASHA	BOHB
0.6B	$\mathbf{+0.06}$	$\mathbf{+0.05}$	$\mathbf{+0.06}$	$\mathbf{+0.08}$
1.7B	$0.00$	$+0.01$	$+0.02$	$+0.02$
4B	$+0.02$	$\mathbf{+0.08}$	$+0.04$	$+0.05$
8B	$+0.10$	$\mathbf{+0.20}$	$0.00$	$+0.10$
14B	$+0.08$	$+0.06$	$\mathbf{+0.29}$	$\mathbf{+0.24}$

Appendix

Table 19: Gaussian-process HPO improvement over random search by model size. Entries are paired mean differences in final

Z_{\mathrm{Dynamic}}

over 25 seeds. Bold entries have paired 95% intervals excluding zero.

Figure 22: Recommendation regret under the $8\times$ budget with the Gaussian-process surrogate. All methods select on $Z_{\mathrm{Dynamic}}$ . The left panel evaluates the recommendation on $Z_{\mathrm{Dynamic}}$ ; the right evaluates the same recommendation on $Z_{\mathrm{All}}$ . Curves average the five Qwen3 sizes within each seed and show the mean and 95% $t$ -interval across 25 seeds.

Figure 23: Recommendation regret under the $8\times$ budget with the one-nearest-neighbour surrogate. All methods select on $Z_{\mathrm{Dynamic}}$ . The left panel evaluates the recommendation on $Z_{\mathrm{Dynamic}}$ ; the right evaluates the same recommendation on $Z_{\mathrm{All}}$ . Curves average the five Qwen3 sizes within each seed and show the mean and 95% $t$ -interval across 25 seeds. The one-nearest-neighbour surface is piecewise constant between measured configurations.

Figure 24: Recommendation regret under the $8\times$ budget with distance-weighted three-nearest-neighbour interpolation. All methods select on $Z_{\mathrm{Dynamic}}$ . The left panel evaluates the recommendation on $Z_{\mathrm{Dynamic}}$ ; the right evaluates the same recommendation on $Z_{\mathrm{All}}$ . Curves average the five Qwen3 sizes within each seed and show the mean and 95% $t$ -interval across 25 seeds.

Figure 25: Recommendation regret under the $8\times$ budget with distance-weighted five-nearest-neighbour interpolation. All methods select on $Z_{\mathrm{Dynamic}}$ . The left panel evaluates the recommendation on $Z_{\mathrm{Dynamic}}$ ; the right evaluates the same recommendation on $Z_{\mathrm{All}}$ . Curves average the five Qwen3 sizes within each seed and show the mean and 95% $t$ -interval across 25 seeds.

Figure 26: Cross-scale configuration-ranking agreement by evaluation aggregate. Each cell reports Spearman’s $\rho$ between rankings of the nine configurations in the shared $3\times 3$ learning-rate– $\beta$ grid. The left panel uses $Z_{\mathrm{Dynamic}}$ and the right panel uses $Z_{\mathrm{Static}}$ . Neither aggregate exhibits a consistent decline in agreement with model-size distance.

Panel of Figure 27 — Figure 27: Cross-scale transfer regret is neither determined by model-size distance nor symmetric. Left: Normalised $Z_{\mathrm{All}}$ regret for smaller-to-larger transfers as a function of the log 2 parameter-count ratio. The shaded region gives the range of expected regret from choosing a shared-core configuration uniformly at random across target sizes. Right: The complete matrix of directed transfers. Each cell applies the source model’s best $Z_{\mathrm{All}}$ configuration from the shared $3\times 3$ grid to the target model. Regret of zero denotes the target’s best shared-core configuration and one its worst. Low and near-maximal regret occur at similar scale distances, while reversing the transfer direction can substantially change regret.

Figure 28: Configuration rankings do not transfer consistently across model families. Each panel compares final

Z_{\mathrm{All}}

scores for the nine shared

(\mathrm{learning\ rate},\beta)

configurations. Points are coloured by

\beta

and annotated with learning rate; the dotted diagonal is a visual reference. Agreement is positive for the smallest pair but negative for the two larger comparisons.

Setting	Qwen3-0.6B	Qwen3-1.7B–14B
Learning rate	$[10^{-6},5\!\times\!10^{-5}]$ , log-uniform
$\beta$	$[0.05,1]$ , uniform	$[0.01,1]$ , log-uniform
Sampler	Optuna TPE; seed 8
Objective	minimize held-out DPO loss
Startup / total trials	5 / 20 per model size
Stopping and pruning	fixed trial budget; no pruner

Appendix

Table 20: Settings for the preliminary validation loss directed searches. The different

\beta

domain at 0.6B reflects the bounds used in these preliminary runs.

Model	$n$	Learning rate	$\beta$	$Z_{\mathrm{All}}$	$Z_{\mathrm{Dynamic}}$	$Z_{\mathrm{Static}}$
Qwen3-0.6B	20	$1.83\times 10^{-5}$	0.0526	0.20 (4)	0.26 (11)	0.12 (5)
Qwen3-1.7B	18	$4.69\times 10^{-5}$	0.0109	0.63 (15)	0.89 (16)	0.36 (10)
Qwen3-4B	19	$2.60\times 10^{-5}$	0.0164	0.13 (5)	0.48 (16)	0.00 (1)
Qwen3-8B	16	$5.39\times 10^{-6}$	0.0638	0.11 (8)	0.15 (3)	0.29 (9)
Qwen3-14B	20	$4.38\times 10^{-6}$	0.0165	0.00 (1)	0.17 (5)	0.00 (1)
Mean regret	—	—	—	0.21	0.39	0.16

Appendix

Table 21: Downstream quality of the minimum validation loss TPE trial.

n

is the number of trials with complete loss and downstream results. Aggregate columns report normalised regret followed by the selected trial’s downstream rank in parentheses; lower values are better for both quantities.

	Chat benchmarks		Static benchmarks
Model	$\rho\pm\sigma$	$P$	$\rho\pm\sigma$	$P$
Qwen3-0.6B	$0.49\pm 0.19$	6	$0.15\pm 0.35$	15
Qwen3-1.7B	$0.52\pm 0.42$	6	$0.17\pm 0.31$	15
Qwen3-4B	$0.64\pm 0.33$	6	$0.30\pm 0.53$	15
Qwen3-8B	$0.82\pm 0.08$	3	$0.36\pm 0.34$	15
Qwen3-14B	$0.85\pm 0.05$	6	$0.08\pm 0.45$	15

Appendix

Table 22: Benchmark-ranking agreement within the preliminary TPE runs. Entries give the mean pairwise Spearman correlation

\rho

and its population standard deviation across benchmark pairs.

P

is the number of benchmark pairs contributing to each summary.

Direct Preference Optimization

Direct Preference Optimization trains generative models from preferred and rejected outputs without fitting an explicit reward model.

See all 162 papers

Aug 27, 2026

Disentangling Optimization Scale from Preference Scale in DPO

Direct Preference Optimization (DPO) is a widely used objective for aligning language models from preference data, with the coefficient

β

commonly interpreted as controlling the KL constraint to a reference policy. We show that

β

entangles two distinct roles: it governs the effective inverse preference-noise scale and simultaneously rescales the optimization dynamics, coupling this scale with the effective step size. As a consequence, at a fixed learning rate the achieved policy deviation is non-monotone in

β

: it vanishes in a dead zone at small

β

, reaches a peak at an intermediate value, and decreases again for larger

β

. Moreover, standard DPO loss values are not comparable across

β

: runs with nearly identical loss curves can differ several-fold in KL divergence from the reference model. This entanglement obscures the role of

β

, increases sensitivity to hyperparameter choices, and complicates learning-rate scheduling. We propose a centered-softplus reformulation that is argmin-equivalent to DPO for

β>0

, while making the inverse preference-noise-scale and learning-rate effects explicit and independently tunable. The normalized centered-softplus objective also admits a continuous

β\to0

endpoint that reduces to a linear preference-margin objective.

Ivan Kruzhilov

Direct Preference Optimization Preference Learning Deep Learning Optimization RL for Language Models

Ask PDF

Oct 6, 2026

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization

Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further find that different coefficient initializations induce distinct transient learning-signal trajectories even when their later signal levels become similar. Based on this observation, we derive a signal-budget compensation rule that adjusts the target learning signal to compensate for these transient differences. The resulting compensation substantially reduces performance variation across coefficient initializations.

Yang Qu, Yusheng Han, Chengjia Feng +1

Northeastern University

Direct Preference Optimization LLM Alignment Preference Learning

Ask PDF

Jun 11, 2026

Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study

We present an approach to fine-tuning large language models using Direct Preference Optimization (DPO), a reinforcement learning technique. Our experimental results demonstrate that DPO simplifies the training pipeline, improves computational efficiency, and achieves competitive performance. The evaluation using BLEU, ROUGE, and cosine similarity metrics indicates effective learning and convergence, though further investigation is needed to address observed training instability.

Dezhi Yu, Yvonne Qiu, ShuoJia Fu

LLM Alignment Direct Preference Optimization LLM Fine-Tuning LLM Post-Training Fine-Tuning Language Models

Ask PDF

Abstract

Direct Preference Optimization

Explore similar work

Disentangling Optimization Scale from Preference Scale in DPO

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization

Direct Preference Optimization for Chatbot Fine-Tuning: An Empirical Study