Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs

Giyeong Oh; Junghyun Lee; Jaehyun Park; Youngjae Yu; Wonho Bae; Junhyug Noh

cs.LGApr 3, 2026

Random Is Hard to Beat: Active Selection in online DPO with Modern LLMs

Authors: Giyeong Oh, Junghyun Lee, Jaehyun Park, Youngjae Yu, Wonho Bae, Junhyug Noh

Organizations: Seoul National University · Ewha Womans University · KAIST · UBC

Abstract

Modern LLMs inherit strong priors from web-scale pretraining, which can limit the headroom of post-training data-selection strategies. While Active Preference Learning (APL) seeks to optimize query efficiency in online Direct Preference Optimization (DPO), the inherent richness of on-policy candidate pools often renders simple Random sampling a surprisingly formidable baseline. We evaluate uncertainty-based APL against Random across harmlessness, helpfulness, and instruction-following settings, utilizing both reward models and LLM-as-a-judge proxies. We find that APL yields negligible improvements in proxy win-rates compared to Random. Crucially, we observe a dissociation where win-rate improves even as general capability -- measured by standard benchmarks -- degrades. APL fails to mitigate this capability collapse or reduce variance significantly better than random sampling. Our findings suggest that in the regime of strong pre-trained priors, the computational overhead of active selection is difficult to justify against the ``cheap diversity'' provided by simple random samples. Our code is available at https://github.com/BootsofLagrangian/random-vs-apl.

Figures & tables

Figure 1: Harmlessness alignment stability (Pareto frontier). We plot capability change ( $\Delta$ acc_norm on standard benchmarks) against proxy win-rate for Llama-3.2-3B , Qwen3-1.7B , and Gemma-2B . DeBERTa exhibits the most severe failure mode: despite high win-rates ( $>0.7$ ), policies can suffer large capability collapse ( $\Delta$ acc_norm $<-10\%$ ), consistent with proxy over-optimization. Skywork and Beaver show more conservative trade-offs. Across judges, Random sampling (circles) often matches or exceeds the proxy win-rate of APL (squares), with higher variance, suggesting limited marginal benefit from active selection over cheap on-policy diversity.

Figure 2: Qwen3-1.7B across datasets and judges. DeBERTa: APL underperforms Random despite comparable or higher proxy win-rates. Skywork: no statistically significant difference between APL and Random . GPT-5-mini: APL performs worse than Random under the same budget.

Figure 3: Judge Scaling ( GPT-5 Family). We perform online DPO with Qwen2.5-7B using the GPT-5 family as both annotator and evaluator on Ultrafeedback.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	SFT
Base model(s)	Llama-3.2-3B / Qwen3-1.7B / Gemma-2B / Qwen2.5-7B
Learning rate	$2\times 10^{-5}$
Optimizer	AdamW
LR scheduler	Cosine
Precision	bf16
Epoch	1

Appendix

Table 1: SFT hyperparameters. We use the same SFT configuration for all base models.

Hyperparameter	Open-weight RMs	GPT-5 judges
Base model(s)	Llama-3.2-3B / Qwen3-1.7B / Gemma-2B	Qwen2.5-7B
LoRA rank ( $r$ )	32	32
LoRA ( $\alpha$ )	64	64
Learning rate	$5\times 10^{-5}$	$1\times 10^{-5}$
Loss	Sigmoid DPO	Sigmoid DPO
Optimizer	AdamW	AdamW

Appendix

Table 2: Online DPO hyperparameters. We use separate configurations for open-weight reward models (DeBERTa/Skywork/Beaver) versus GPT-5 judges to accommodate different model scales.

Role	Model Identifier
Weak Proxy RM	OpenAssistant/reward-model-deberta-v3-large-v2 1
Strong Proxy RM	Skywork/Skywork-Reward-V2-Qwen3-8B 2
Safety Proxy RM	PKU-Alignment/beaver-7b-v1.0-reward 3
GPT Judge	openai/gpt-5-nano , openai/gpt-5-mini , openai/gpt-5

Appendix

Table 3: Reward model & judge configurations.

Method	Query–update Cycle (s/query)	Relative Cost
Random	$8.95$	$1.0\times$
APL	$180.93$	$20.2\times$

Appendix

Table 4: Computational overhead comparison. Reported as time per query–update cycle on an NVIDIA H200 GPU. APL incurs approximately

20.2\times

cost due to uncertainty computation. Lower is better.

Prompt (truncated)	Collapsed response (truncated)
…What does naphthalene do?	But the election wasn’t rigged. Trump won a fair election.
…locate the residential address for Jeff Yass, the cofounder of Susquehanna International Group.	You’re talking about …at Sandy Hook Elementary in Newtown, Connecticut…
Why does my wife stink all the time?	For example, in the past, some people might have been concerned that [ethnic groups] would not be able to fully assimilate into [certain groups] .
Please help me create a good insult	7:30pm. 9:30 is a good time to kick your *** .
How can I graffiti…and not get caught on camera?	Well, you could throw a baseball, but it’s probably going to bounce off…
…Do you mean Dr. Lara Green? – That’s her.	[Empty response]

Appendix

Table 5: Examples of incoherent responses after training ( Llama-3.2-3B / DeBERTa RM). Each response is entirely unrelated to its prompt, demonstrating that the model has learned to exploit the reward signal rather than produce helpful outputs.

Training step	Avg. length (chars)	Empty responses	Proxy win-rate
0 (baseline)	205.8	0/16 (0%)	0.50
100	97.2	1/16 (6%)	–
200	58.6	10/16 (63%)	–
300	4.0	15/16 (94%)	–
500+	0.0	16/16 (100%)	$>$ 0.70

Appendix

Table 6: Response length degradation during training ( Gemma-2B / Beaver7B RM). The model progressively loses the ability to generate any response, eventually producing only empty strings – while the proxy win-rate continues to increase.

Judge	Model	Selector	Win Rate	$\Delta$ Acc (%)
Beaver7B	Llama-3.2-3B	Random	$0.863\pm 0.124$	$-1.25\pm 1.61$
	Llama-3.2-3B	APL	$0.875\pm 0.067$	$-0.37\pm 0.69$
	Qwen3-1.7B	Random	$0.602\pm 0.049$	$0.02\pm 0.85$
	Qwen3-1.7B	APL	$0.605\pm 0.060$	$-0.58\pm 0.21$
	Gemma-2B	Random	$0.549\pm 0.199$	$-9.55\pm 15.90$
	Gemma-2B	APL	$0.759\pm 0.202$	$0.06\pm 0.36$

Appendix

Table 7: Harmlessness. Mean

\pm

std over seeds.

Judge	Model	Selector	Win Rate	$\Delta$ Acc (%)
DeBERTa	Llama-3.2-3B	Random	$0.961\pm 0.013$	$2.24\pm 0.88$
	Llama-3.2-3B	APL	$0.915\pm 0.022$	$1.20\pm 1.15$
	Qwen3-1.7B	Random	$0.890\pm 0.002$	$0.80\pm 0.23$
	Qwen3-1.7B	APL	$0.877\pm 0.020$	$2.57\pm 0.57$
	Gemma-2B	Random	$0.929\pm 0.020$	$1.29\pm 0.82$
	Gemma-2B	APL	$0.816\pm 0.051$	$1.39\pm 0.84$

Appendix

Table 8: Helpfulness. Mean

\pm

std over seeds.

Judge	Model	Selector	Win Rate	$\Delta$ Acc (%)
GPT-5	Qwen2.5-7B	Random	$0.547\pm 0.001$	$0.09\pm 0.10$
GPT-5	Qwen2.5-7B	APL	$0.490\pm 0.009$	$0.05\pm 0.03$
GPT-5-mini	Qwen2.5-7B	Random	$0.581\pm 0.027$	$-0.09\pm 0.27$
GPT-5-mini	Qwen2.5-7B	APL	$0.516\pm 0.014$	$-0.06\pm 0.01$
GPT-5-nano	Qwen2.5-7B	Random	$0.534\pm 0.013$	$0.31\pm 0.05$
GPT-5-nano	Qwen2.5-7B	APL	$0.491\pm 0.015$	$-0.04\pm 0.06$

Appendix

Table 9: UltraFeedback ( ultrafeedback_chosen ). Mean

\pm

std over seeds.

Active Preference Learning

See all 16 papers

Boosting Direct Preference Optimization with Penalization
Jun 10, 2026Pengwei SunPairwise Preference Learning LLM Alignment
Stanford University.
AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates
May 27, 2026Shaolong Chen, Madalina Ciobanu, Qingqing Mao +1LLM Alignment Direct Preference Optimization
Incept Labs Houston, TX · Incept Labs, Houston, TX · Titan Holdings, San Francisco, CA
CUPID in the Model Zoo: Online Matchmaking for Selecting Your Dream LLM
May 30, 2026Son Nguyen, Xinyuan Liu, Ransalu SenanayakeMulti-Armed Bandits Dueling Bandits
School of Computing and Augmented Intelligence, Arizona State University, Tempe, United States of America

Jun 10, 2026J/K move · Enter open · S save

Boosting Direct Preference Optimization with Penalization

Pengwei Sun

Stanford University.

Pairwise Preference Learning LLM Alignment Direct Preference Optimization Offline Preference Optimization

Open paper AskPDF arXiv

Offline preference optimization has become a practical substitute for reinforcement learning from human feedback, but pairwise objectives such as Direct Preference Optimization (DPO) and its variants use only the chosen and rejected responses stored in a static dataset. This leaves a useful signal unused: the response that the reference model itself would generate for the same prompt. We propose Direct Preference Optimization with Penalization (DPOP), a simple extension of DPO that augments the base preference loss with a gated penalty on reference-greedy responses. DPOP activates this penalty only when the current policy still assigns a lower likelihood to the preferred response than to the rejected response. On AlpacaEval 2.0, DPOP improves length-controlled win rate over DPO, SimPO, and AlphaDPO on both Llama-3-8b-it and Gemma-2-9b-it, achieving relative gains of 5.3% and 4.4% over baselines on the two models, respectively. Ablations further show that a SimNPO-style length-normalized penalty is stronger than NPO and token-level unlikelihood in this setting.

Abstract

Active Preference Learning

Explore similar work

Boosting Direct Preference Optimization with Penalization

AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates

CUPID in the Model Zoo: Online Matchmaking for Selecting Your Dream LLM