When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

Amirabbas Afzali; Myeongho Jeon; Maria Brbic

cs.CLMar 5, 2026

When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

Authors: Amirabbas Afzali, Myeongho Jeon, Maria Brbic

Organizations: EPFL · Sharif University of Technology

Abstract

Preference alignment is an essential step in adapting large language models (LLMs) to human values, but existing approaches typically depend on costly human annotations or large-scale API-based models. We explore whether a weak LLM can instead act as an effective annotator. We surprisingly find that selecting only a subset of a weak LLM's highly confident samples leads to substantially better performance than using full human annotations. Building on this insight, we propose Confidence-Weighted Preference Optimization (CW-PO), a general framework that re-weights training samples by a weak LLM's confidence and can be applied across different preference optimization objectives. Notably, the model aligned by CW-PO with just 20% of human annotations outperforms the model trained with 100% of annotations under standard DPO. These results suggest that weak LLMs, when paired with confidence weighting, can dramatically reduce the cost of preference alignment while even outperforming methods trained on fully human-labeled data.

Figures & tables

Panel of Figure 1 — Figure 1: Overall pipeline of our setting. Top : Conventional DPO ( Rafailov et al., 2023 ) . For each triplet consisting of a prompt $x$ and two candidate responses $(y_{1},y_{2})$ , human annotators provide preference labels, and the policy model is aligned with these labels using DPO. Bottom : CW-DPO framework. A weak LLM is first trained as a preference annotator using a subset of human-labeled triplets. It is then applied to annotate the remaining large-scale data, which is subsequently trained with CW-DPO. The bars on top right report Gold Reward Accuracy for standard DPO with human-labeled data (red) and for CW-DPO (blue) on the Anthropic HH-RLHF . CW-DPO uses only $30\%$ compared to DPO, which uses fully human-annotated dataset. OPT-125M and OPT-1.3B are used as the weak and strong models, respectively.

Figure 2: Alignment with the top-N% most confident samples. Gold reward accuracy (GRA) is reported for the trained strong models. We consider (OPT-125M

\rightarrow

OPT-1.3B) and (Qwen-0.5B

\rightarrow

Qwen-7B) as weak–strong model pairs. The graph shows the average GRA for two models. Here, 100% denotes using the weak LLM directly for annotation. Further details of the results are provided in Appendix H.1 .

OPT-125M $\rightarrow$ OPT-13B
	DPO			IPO			rDPO
Dataset	Human	WS-DPO	CW-DPO	Human	WS-DPO	CW-IPO	Human	WS-DPO	CW-rDPO
HH-RLHF	56.9	56.7	61.3	58.2	62.8	63.5	55.9	57.6	63.0
TL;DR	57.0	53.5	56.6	53.3	49.7	54.6	54.2	47.7	61.4
UFB	61.3	63.4	63.1	63.4	61.3	66.4	58.9	61.2	63.7
Avg.	58.4	57.9	60.3	58.3	57.9	61.5	56.3	55.5	62.7

Table 1: Results across different preference alignment methods. The reported values are GRA (%). Weak models in WS-DPO and CW-DPO are trained with

30\%

of human annotated data. Alignment data for the strong model is fixed across all experiments. CW-PO columns are highlighted in blue.

Dataset	Strong	OPT			Strong	Qwen
Dataset	Strong	Human	WS-DPO	CW-DPO	Strong	Human	WS-DPO	CW-DPO
HH-RLHF	1.3B	71.5	66.7	69.9	1.5B	53.4	55.8	63.3
	2.7B	55.1	58.5	60.3	3B	66.0	63.3	73.3
	6.7B	56.1	62.8	67.6	7B	71.1	72.0	75.2
	13B	56.9	56.7	61.3	14B	78.8	81.4	80.6
	Avg.	59.9	61.2	64.8	Avg.	67.3	68.1	73.1

Table 2: Performance across different student models measured as GRA (

\%

). We use OPT-125M and Qwen2.5-0.5B as the weak models for the OPT and Qwen families, respectively. GRA measures improvement over a model’s SFT baseline; thus larger models may not score higher GRA, since stronger baselines leave less room to improve even if absolute performance is higher.

Dataset	OPT-125M $\rightarrow$ OPT-1.3B		Qwen2.5-0.5B $\rightarrow$ Qwen2.5-7B
Dataset	Human (100%)	CW-DPO	Human (100%)	CW-DPO
Harmless	69.2	72.9 (+3.7)	65.7	72.0 (+6.3)
Helpful	70.2	72.7 (+2.5)	58.5	70.8 (+12.3)
HH-RLHF	71.9	69.9 ( $-$ 2.0)	72.7	75.2 (+2.5)
TL;DR	54.2	59.5 (+5.3)	63.4	64.4 (+1.0)
Avg.	66.4	68.8 (+2.4)	65.1	70.6 (+5.5)

Table 3: Comparison between DPO using the fully human-annotated dataset (

{\cal D}_{\text{labeled}}\cup D_{\text{unlabeled}}

) and CW-DPO . Parentheses show the relative change from the Human baseline.

Figure 3: Left : GRA when adjusting the proportion of

{\cal D}_{\text{labeled}}

used to fine-tune the weak LLM, while retaining 50% of the data as training for the strong LLM. Right : GRA across varying proportions of

{\cal D}_{\text{labeled}}

. As the split ratio decreases, the size of

{\cal D}_{\text{labeled}}

decreases and

{\cal D}_{\text{unlabeled}}

increases because the total dataset (

{\cal D}_{\text{labeled}}\cup{\cal D}_{\text{unlabeled}}

) is fixed.

Dataset	$\mathcal{C}_{1}$	$\mathcal{C}_{2}$	$\mathcal{C}_{3}$	$\mathcal{C}_{4}$
Harmless	72.0	70.3	68.6	69.1
Helpful	70.8	67.8	67.4	68.7
HH-RLHF	75.2	70.1	69.2	72.5
Avg.	72.7	69.4	68.4	70.1

Table 5: Performance comparison of different confidence weighting functions using Qwen models.

Model	Dataset	DPO	SFT+DPO	BT
OPT-125M	Harmless	55.2	56.3	69.1
	Helpful	54.1	55.4	64.2
	HH-RLHF	50.8	52.1	63.8
Qwen-0.5B	Harmless	56.1	57.1	65.3
	Helpful	55.2	56.0	63.1
	HH-RLHF	51.4	52.6	63.2

Table 6: Accuracy and efficiency of weak models.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Model(s)	OPT-125M; Qwen2.5-0.5B
Training epochs	5
Optimizer	Adam
Learning rate	$1\times 10^{-5}$
Per-device train batch size	32
Gradient accumulation steps	1

Appendix

Table 7: Training hyperparameters for weak models .

Parameter	Value
Training epochs	5
Learning rate	$5\times 10^{-6}$
LR scheduler	cosine
Warmup steps	100
Weight decay	0.05
Optimizer	paged_adamw_32bit

Appendix

Table 8: Training hyperparameters for strong models with DPO, IPO, rDPO and their confidence-weighted variants.

Parameter	Value
Training epochs	3
Learning rate	$1\times 10^{-5}$
LR scheduler	cosine
Warmup steps	100
Weight decay	0.05
Optimizer	paged_adamw_32bit

Appendix

Table 9: Training hyperparameters for supervised fine-tuning (SFT) .

Setting	30%	50%	75%	100%	Human
Harmless	72.3	67.9	67.7	67.4	66.2
Helpful	70.1	68.0	67.2	66.8	66.4
HH-RLHF	67.7	67.5	66.9	65.9	64.4
Avg.	70.03	67.80	67.27	66.70	65.67

Appendix

Table 10: Strong models’ Gold Reward Accuracy (GRA) for OPT-125M

\rightarrow

OPT-1.3B.

Setting	30%	50%	75%	100%	Human
Harmless	70.6	68.4	66.7	66.9	65.1
Helpful	58.7	55.2	51.5	52.3	54.4
HH-RLHF	71.3	68.4	64.9	63.2	61.8
Avg.	66.87	64.00	61.03	60.80	60.43

Appendix

Table 11: Strong models’ Gold Reward Accuracy (GRA) for Qwen2.5-0.5B

\rightarrow

Qwen2.5-7B.

Weak Model Size	0.5B	1.5B	3B	7B
Harmlessness	63.5	65.9	66.6	67.1
Helpfulness	63.2	64.7	65.3	67.2
TL;DR	60.7	61.0	61.6	62.9
Avg.	62.5	63.9	64.5	65.7

Appendix

Table 12: Weak Prediction Accuracy (%), across weak model sizes for the Qwen2.5 family. Accuracy is measured based on human annotations.

Dataset %	10%	20%	30%	40%	50%
Harmlessness	62.6	65.5	67.2	67.8	67.1
Helpfulness	61.9	63.4	65.3	66.4	65.3
Both	56.7	60.2	61.9	62.6	62.2
Avg.	60.4	63.1	64.8	65.6	64.9

Appendix

Table 13: Weak prediction accuracy (%) for OPT-125M across training-set portions (10–50%) of the weak model, measured based on human annotations.

Panel of Table 14 — Table 14: Comparative examples from the HH-RLHF dataset. For each prompt, we show the responses from the CW-DPO–optimized LLM and the SFT LLM.

Panel of Table 15 — Table 15: Comparative examples from the TL;DR dataset. For each prompt, we show the responses from the CW-DPO–optimized LLM and the SFT LLM.

	Human	WS-DPO	CW-DPO
Dataset	OPT-125M $\rightarrow$ Qwen-2.5-14B
Harmless	67.3	66.4	72.2
Helpful	54.1	55.9	57.2
HH-RLHF	78.8	77.7	79.7
	Qwen-2.5-0.5B $\rightarrow$ OPT-13B
Harmless	74.3	72.4	75.4

Appendix

Table 16: Weak–Strong model pairs drawn from different model families.

OPT-125M $\rightarrow$ OPT-13B
	DPO			IPO			rDPO
Dataset	Human	WS-DPO	CW-DPO	Human	WS-DPO	CW-IPO	Human	WS-DPO	CW-rDPO
HH-RLHF	43.9	44.6	51.9	43.7	33.5	45.8	46.4	42.1	55.1
TL;DR	55.7	52.8	52.8	59.5	53.4	63.2	56.6	54.5	61.0
UFB	59.8	64.7	64.3	62.1	60.2	63.9	55.3	56.8	59.4
Avg.	53.13	54.03	56.33	55.10	49.03	57.63	52.77	51.13	58.50

Appendix

Table 17: Comparison of preference alignment methods. Reported values are WR (%) evaluated by GPT-4 as a judge, comparing aligned models against the SFT baseline.

Panel of Table 18 — Table 18: Qualitative examples of preference pairs where the weak LLM assigns low confidence ( $\mathcal{C}(x,y^{+},y^{-})<0.5$ ) and disagrees with the human-annotated preference. In each sample, the preference annotation (which response is preferred) and confidence score are produced by the weak LLM.

Panel of Table 19 — Table 19: Qualitative examples of preference pairs where the weak LLM assigns high confidence ( $\mathcal{C}(x,y^{+},y^{-})>0.5$ ) and disagrees with the human-annotated preference. In each sample, the preference annotation (which response is preferred) and confidence score are produced by the weak LLM.

OPT-125M $\rightarrow$ OPT-13B
	DPO			IPO			rDPO
Dataset	Human	WS-DPO	CW-DPO	Human	WS-DPO	CW-IPO	Human	WS-DPO	CW-rDPO
HH-RLHF	56.88 ± 4.43	55.01 ± 3.58	60.79 ± 2.98	58.87 ± 3.18	61.79 ± 1.75	64.30 ± 2.24	55.92 ± 4.90	57.63 ± 3.59	62.95 ± 1.96
TL;DR	57.79 ± 3.47	53.45 ± 2.39	53.85 ± 5.33	51.54 ± 1.64	49.30 ± 2.14	54.80 ± 2.33	55.04 ± 0.28	48.14 ± 3.97	62.12 ± 4.80
UFB	62.12 ± 2.18	63.72 ± 2.93	63.61 ± 3.17	62.63 ± 1.07	60.96 ± 1.15	65.17 ± 2.32	57.11 ± 2.18	63.41 ± 2.35	63.95 ± 2.33
Avg.	58.9 ± 2.29	57.4 ± 4.52	59.4 ± 4.10	57.7 ± 4.61	57.4 ± 5.70	61.4 ± 4.70	56.0 ± 0.85	56.4 ± 6.29	63.0 ± 0.75

Appendix

Table 20: Results across different preference alignment methods. The setup is same as Table 1 .

Panel of Figure 5 — Figure 5: Gold Reward gap plots demonstrated as $\mathrm{R}^{*}_{\text{CW-DPO}}-\mathrm{R}^{*}_{\text{GT}}$ for responses generated by Qwen2.5-7B models optimized with CW-DPO and standard DPO (using human-preference annotations). The Win-Rate (WR), defined as the fraction of samples for which CW-DPO achieves a higher reward than the Human-trained model.

Dataset	Online DPO	Online CW-DPO	Online WS-DPO
HH-RLHF	85.3	55.1	56.3
TL;DR	66.9	46.2	44.1

Appendix

Table 21: Comparison of Online DPO, Online CW-DPO, and Online WS-DPO applied to Qwen2.5-3B as the strong model. Online DPO uses trl-lib/Qwen2-0.5B-Reward as the external reward model

\mathrm{R}

. Online CW-DPO and Online WS-DPO use a fine-tuned Qwen2.5-0.5B weak teacher as the reward model.

Dataset	CW-DPO (online)	CW-DPO (offline)
HH-RLHF	55.1	61.3
TL;DR	46.2	56.6

Appendix

Table 22: Comparison of offline and online CW-DPO applied to Qwen2.5-3B. Offline CW-DPO uses weak-labeled preference data from

\pi_{w}

; online CW-DPO uses on-policy pairs.

Panel of Figure 6 — Figure 6: Visualization of distribution shift in embedding space. We compute BERT-base CLS embeddings and project them into 2D via PCA. (a) Model-generated responses (online) occupy a region distinct from offline chosen responses. (b) Chosen and rejected responses from the offline dataset cluster differently from the model-generated responses in (a), illustrating the distributional mismatch between on-policy generations and the offline data used to train $\pi_{w}$ .

Domain	Strong	Human	WS-DPO	CW-DPO
askacademia	7B	53.6	55.1	56.4
askacademia	14B	50.6	53.3	55.1
askbaking	7B	49.1	50.9	51.4
askbaking	14B	56.6	54.5	58.4
askengineers	7B	56.9	58.0	59.1
askengineers	14B	54.9	57.2	55.4

Appendix

Table 23: Comparison of alignment performance on Stanford Human Preference (SHP) dataset ( Ethayarajh et al., 2022 ) , a more complex preference dataset. The reported values are GRA (%) calculated by the stanfordnlp/SteamSHP-flan-t5-large preference model, comparing aligned models against the SFT baseline.

Figure 7: Distribution of confidence scores

\mathcal{C}(x,y^{+},y^{-})

produced by weak LLMs. Left: Each weak model is trained separately on its own dataset, and confidence values are computed on the corresponding unlabeled training set of the strong model for that dataset, highlighting how confidence behavior shifts across heterogeneous tasks and domains. Right: Both weak models are trained on the HH-RLHF dataset, and confidence values are evaluated on different subsets of its test samples, illustrating how harmlessness and helpfulness signals vary within the same training domain.

${\cal D}_{\text{labeled}}$ mixture	Harmless		Helpful
${\cal D}_{\text{labeled}}$ mixture	CW-DPO	WS-DPO	CW-DPO	WS-DPO
50% + 50%	60.5	57.1	63.6	58.9
80% + 20%	61.6 (+1.1)	59.3 (+2.2)	60.5 (-3.1)	54.8 (-4.1)
20% + 80%	56.1 (-4.4)	52.2 (-4.9)	67.1 (+3.5)	62.4 (+3.5)

Appendix

Table 24: Harmlessness and Helpfulness performance across varying imbalance settings for Qwen2.5-0.5B

\rightarrow

Qwen2.5-7B.

	Human	WS-DPO	PairRM	CW-DPO
HH-RLHF	71.1	72.0	71.6	75.2
TL;DR	61.2	60.1	62.2	64.4
Avg.	66.15	66.05	66.90	69.80

Appendix

Table 26: Evaluation of CW-DPO when using a PairRM-style weak annotator. The weak–strong model pair is Qwen2.5-0.5B

\rightarrow

Qwen2.5-7B. In the PairRM column, the weak model is trained with the PairRM-style pairwise objective and then integrated into the same CW-DPO alignment pipeline. By contrast, CW-DPO refers to our original setting, where the weak annotator is trained using the BT-based objective (Eq. 5 ). PairRM-based weak annotation does not improve performance over CW-DPO, but still provides consistent gains over Human.

Preference Alignment

Preference alignment studies how AI systems are trained or steered to produce outputs that match human or user priorities.

See all 109 papers

Apr 14, 2026

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

Recent advances in large language models (LLMs) have significantly improved the alignment of models with general human preferences. However, a major challenge remains in adapting LLMs to individual preferences, which are not only diverse but also dynamic. In this paper, we introduce a novel framework, Preference-Paired Fine-Tuning (PFT), designed to align models with contradictory and evolving individual preferences. We present a new dataset, Value Conflict Dilemma (VCD), which includes scenarios that involve conflicting human preferences, facilitating the evaluation of our approach. Our experiments demonstrate that PFT outperforms single-preference training methods, achieving up to 96.6% accuracy in multi-choice classification tasks and the highest open-ended generation score of 8.69. PFT also shows significant improvements over DPO, SFT and some traditional training methods, especially when handling conflicting preferences. Additionally, with limited user history data, models can inferring preference vector rapidly, achieving a 44.76% improvement in user-specific preference alignment in comparison to single-preference models.

Shanyong Wang, Shuhang Lin, Yining Zhao +2

Rutgers University—New Brunswick · University of Illinois at Urbana-Champaign

LLM Fine-Tuning Personalized Language Model Alignment Preference Alignment Value Alignment User Preference Modeling Multi-Objective Language Model Alignment

Ask PDF

Sep 16, 2026

A Zeroth-Order Paradigm for LLM Preference Alignment

Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.

Peter Chen, Xi Chen, Wotao Yin +1

Department of Electrical Engineering and Computer Sciences (EECS) University of California, Berkeley Berkeley, CA 94720, USA · Stern School of Business New York University New York, NY 10012, USA · Decision Intelligence Lab (Seattle) DAMO Academy, Alibaba Group U.S. Bellevue, WA 98004, USA +1

Pairwise Preference Learning LLM Alignment Preference Optimization

Ask PDF

Apr 26, 2026

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.

Imranul Ashrafi, Inigo Jauregi Unanue, Massimo Piccardi

University of Technology Sydney, Australia

LLM Alignment Representation Engineering Multi-Objective Language Model Alignment Preference Alignment Inference-Time Language Model Alignment Preference Learning

Ask PDF

Abstract

Preference Alignment

Explore similar work

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

A Zeroth-Order Paradigm for LLM Preference Alignment

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing