DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding

Mingxi Zou; Jiaxiang Chen; Junfan Li; Langzhang Liang; Qifan Wang; Xu Yinghui; Zenglin Xu

cs.LGMar 9, 2026

DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding

Authors: Mingxi Zou, Jiaxiang Chen, Junfan Li, Langzhang Liang, Qifan Wang, Xu Yinghui, Zenglin Xu

Organizations: Fudan University, Shanghai, China · Independent Researcher · Meta AI · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China

Abstract

Preference-based alignment methods (e.g., RLHF, DPO) typically optimize a single scalar objective, implicitly averaging over heterogeneous human preferences. In practice, systematic annotator and user-group disagreement makes mean-reward maximization brittle and susceptible to proxy over-optimization. We propose Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC), a retraining-free inference-time method that frames response selection as distributionally robust, risk-sensitive decision making. Given multiple preference samples or scalable disagreement proxies, DARC reranks candidates by maximizing a KL-robust (entropic) satisfaction objective, and provides simple deployment controls that cap or penalize the corresponding entropic risk premium relative to the mean, enabling explicit risk budgets without retraining. We provide theoretical characterization linking this decoding rule to principled pessimism and KL-based distributionally robust optimization. Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality under noisy, heterogeneous feedback.

Figures & tables

Figure 1: Score Distribution shift. Ridge plot showing human score densities on the high-disagreement subset. DARC variants (blue) shift the distribution to the right (higher mean

\mu

) compared to the baseline (grey), with reduced spread (lower

\sigma

), indicating both increased satisfaction and reduced disagreement.

Type	Method	Llama-3.1-8B-Instruct								Qwen2.5-7B-Instruct
		MT-Bench				AlpacaEval 2.0				MT-Bench				AlpacaEval 2.0
		Reward	Risk $\downarrow$	Tradeoff $\uparrow$	Len(tok)	Reward	Risk $\downarrow$	Tradeoff $\uparrow$	Len(tok)	Reward	Risk $\downarrow$	Tradeoff $\uparrow$	Len(tok)	Reward	Risk $\downarrow$	Tradeoff $\uparrow$	Len(tok)
Dataset: Overall
Inference	Base (Best-of- $K$ )	4.53	6.50	-8.47	315	4.52	7.46	-10.40	331	4.09	3.54	-2.99	244	3.29	3.11	-2.93	256
	BoP(HedgeTune)	4.18	5.86	-7.54	316	4.12	6.24	-8.36	330	3.84	3.10	-2.36	231	3.14	2.68	-2.22	261
	Caution	4.09	5.43	-6.77	317	4.27	5.91	-7.55	320	3.89	3.21	-2.53	243	3.09	2.62	-2.15	259

Table 1: Evaluation on MT-Bench and AlpacaEval 2.0. Results across two generators families . We report mean reward , proxy risk (Risk), risk–reward tradeoff score (Tradeoff), and average length (Len(tok)).Blue/green highlight best/runner-up.

Dataset: Overall
Method	Human Score $\uparrow$	Risk (Avg $\sigma$ ) $\downarrow$	Tradeoff $\uparrow$	CVaR 10% $\uparrow$
Base (Best-of- $K$ )	7.56	0.67	6.22	6.73
BoP(HedgeTune)	7.79	0.63	6.53	7.12
Caution	7.82	0.61	6.60	7.15
DeAL	7.83	0.72	6.39	7.08
MC-Dropout	7.73	0.64	6.45	7.19

Table 2: Human Evaluation Results. We report Human Mean Score, Disagreement Risk, Tradeoff, and CVaR 10% (worst

10\%

prompt outcomes).

Figure 2: Gains concentrate on high-disagreement prompts. Mean improvement in lower-tail satisfaction (

\Delta

Tradeoff vs. base) across five prompt buckets ranked by baseline human disagreement

\hat{\sigma}

(low

\rightarrow

high). Error bars denote 95% CIs.

Bucket	Human Score $\Delta$ (DARC- $\epsilon$ – Base)	W / T / L
Q1	0.125	11 / 80 / 9
Q2	0.236	17 / 71 / 12
Q3	0.451	34 / 55 / 11
Q4	0.755	42 / 50 / 8
Q5	1.037	51 / 45 / 4

Table 3: Bucketed human evaluation versus Base by baseline human disagreement. DARC-

\epsilon

remains close to mean-based selection in low-disagreement prompts and becomes increasingly beneficial as disagreement grows.

Panel of Figure 3 — Figure 3: Proxy validity diagnostics. (Top) Rank correlation between proxy and human disagreement, with top-20% overlap. (Bottom) Top- $q$ overlap (Left) and proxy vs. human disagreement scatter (Right).

Figure 4: Ablation Studies. Impact of key hyperparameters on risk mitigation performance. (a) Candidate pool size

K

. (b) Risk sensitivity coefficient

\beta

. (c) Constraint threshold

\epsilon

. (d) Perturbation budget

N_{\text{aug}}

	Overall		High- $\hat{\sigma}$ Subset
Method	W/T/L	Mean $\Delta$	W/T/L	Mean $\Delta$
Across scorers (aggregated)
DARC	278 / 173 / 49	0.124	32 / 63 / 5	0.303
DARC- $\tau$	257 / 192 / 51	0.128	36 / 60 / 4	0.259
DARC- $\epsilon$	296 / 168 / 36	0.161	44 / 54 / 2	0.386

Table 4: Multi-scorer robust decoding. DARC variants vs. Base (mean Best-of-

K

\Delta

: per-prompt improvement over Base, aggregated across scorers (mean / min / mean

-\gamma

std); W/T/L is on aggregated

\Delta

. High-

\hat{\sigma}

uses the baseline disagreement proxy.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$\mathbb{E}_{s}[D(s)]$	$Q_{0.8}(D)$	$Q_{0.9}(D)$	$\Pr[D(s)\geq Q_{0.8}(D)]$
AlpacaEval 2.0	0.74	0.98	1.65	0.20
MT-Bench	0.65	0.84	1.32	0.20

Appendix

Table 5: Human disagreement on the raw Top-

K

pool (

K{=}5

) measured by the maximum across-candidate human rating variance

D(s)

. High-disagreement prompts are defined as the top 20% by

D(s)

Component	Hyperparameters (default)
Base model	meta-llama/Llama-3.1-8B-Instruct
Candidate generation	$K{=}16$ , temperature $0.8$ , top- $p{=}0.98$ , max_new_tokens $320$
Reward model scoring	Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 ( Liu et al., 2024a ) , max length $1024$ , batch size $16$
Proxy disagreement	$N_{\text{aug}}{=}8$ perturbations; samples $\mathcal{S}=\{r_{\mathrm{orig}}\}\cup\{r_{\mathrm{aug}}^{(j)}\}_{j=1}^{N_{\text{aug}}}.$ ; $\hat{\sigma}=\mathrm{Std}(\mathcal{S})$
Proxy tail metric	$\mathrm{CVaR}_{0.1}$ on $\mathcal{S}$
DARC entropic	$\beta{=}1.0$ (robust value), $q_{\mathrm{RP}}{=}0.25$ (DARC- $\tau$ ), $\epsilon_{V}{=}0.25$ (DARC- $\epsilon$ )

Appendix

Table 6: Default hyperparameters used in cDPO/rDPO training and inference-time evaluation.

Figure 5: Conservative metric exhibits the same bucketed trend. Bucketed improvements (vs. base) for a conservative cvar-style score (e.g.,

\Delta\mathrm{CVaR}_{10}

), , using the same human-disagreement buckets as Fig. 5 . Bars show mean; error bars denote 95% CIs.

Overall (valid prompts)
Method	Scalarization	Win/Tie/Loss	Mean $\Delta$
raw	S0 (absolute ratings)	247 / 179 / 74	+0.096
raw	S1 (pairwise vs. base)	241 / 198 / 61	+0.089
eps	S0 (absolute ratings)	259 / 166 / 75	+0.130
eps	S1 (pairwise vs. base)	249 / 173 / 78	+0.102
High-disagreement subset (top 20% by base human $\sigma$ )

Appendix

Table 7: Scalarization robustness (absolute vs. pairwise). S0 uses raw scalar satisfaction ratings. S1 converts ratings into pairwise preferences against the base response (win = 1, tie = 0.5, loss = 0) per annotator, then averages across annotators. Trends are consistent under both scalarizations, especially on high-disagreement prompts. (Additional prompt-level stability: Spearman correlation of per-prompt

\Delta

between S0 and S1 is 0.595/0.665 overall and 0.688/0.764 on the high-disagreement subset for raw/eps, respectively.)

$N_{\text{aug}}$	Total (s)	Overhead vs. $0$	Gen. share	Aug. scoring share
0	9.4906	–	99.7%	0.0%
4	9.6372	+1.54%	98.1%	1.6%
8	9.6787	+1.98%	97.6%	2.0%
16	9.8231	+3.12%	96.5%	3.2%

Appendix

Table 8: End-to-end latency on a single 32GB vGPU with Llama-3.1-8B candidate generation and Skywork-Reward scoring (50 prompts, batch

N=16

). Generation dominates latency; disagreement estimation adds

<2\%

overhead for

N_{\text{aug}}\leq 8

Figure 6: Bucketed predictive validity of the disagreement proxy. We partition prompts into quintiles (Q1–Q5) by disagreement of the baseline candidate, using either human disagreement (left) or proxy disagreement (right). We then report the improvement in

\mathrm{Tradeoff}

of DARC-

\epsilon

over the mean-only baseline, within each bucket. Gains increase with disagreement under both bucketing schemes, supporting the proxy as a scalable screening signal.

Error bucket	$n$	$\mathbb{E}[e]\!\pm\!\mathrm{Std}$	TO (Base)	$\Delta$ DeAL	$\Delta$ RBoN	$\Delta$ DARC- $\epsilon$
Q0–20%	101	$0.1459\pm 0.1355$	$7.8832\pm 2.0509$	$+0.1923$	$+0.1235$	$+0.8528$
Q20–40%	75	$0.4241\pm 0.0107$	$6.6059\pm 0.9990$	$+0.2185$	$+0.3564$	$+0.6639$
Q40–60%	132	$0.4952\pm 0.0657$	$6.2997\pm 0.8025$	$+0.1413$	$+0.2545$	$+0.7640$
Q60–80%	106	$0.7902\pm 0.0592$	$5.7252\pm 1.6313$	$+0.3413$	$+0.3464$	$+0.3809$
Q80–100% (worst)	86	$1.3385\pm 0.7033$	$5.8356\pm 2.0437$	$+0.4524$	$+0.5634$	$+0.6498$

Appendix

Table 9: Error-bucket analysis by proxy–human disagreement mismatch. TO denotes Tradeoff, computed from human-loop statistics as

\hat{\mu}-\lambda\hat{\sigma}

\Delta

denotes mean improvement in TO over the baseline within each bucket. Bucket edges:

0.0255,0.4061,0.4472,0.6737,0.8457,3.1948

	Overall						High- $\hat{\sigma}$
Method	RM1: Skywork-reward-llama-3.1-8b		RM2: nicholasKluge/RewardModel		RM3: OpenAssistant (DeBERTa-v3-Large-v2)		RM1: Skywork-reward-llama-3.1-8b		RM2: nicholasKluge/RewardModel		RM3: OpenAssistant (DeBERTa-v3-Large-v2)
	W/T/L	Mean $\Delta$	W/T/L	Mean $\Delta$	W/T/L	Mean $\Delta$	W/T/L	Mean $\Delta$	W/T/L	Mean $\Delta$	W/T/L	Mean $\Delta$
DARC	228 / 212 / 60	0.098	235 / 223 / 42	0.214	174 / 253 / 73	0.124	38 / 57 / 5	0.308	37 / 58 / 5	0.317	33 / 64 / 3	0.204
DARC- $\tau$	216 / 228 / 56	0.085	221 / 241 / 38	0.193	187 / 242 / 71	0.128	36 / 60 / 4	0.265	35 / 57 / 8	0.288	35 / 61 / 4	0.244
DARC- $\epsilon$	248 / 195 / 57	0.137	257 / 194 / 49	0.285	191 / 228 / 81	0.141	46 / 51 / 3	0.392	41 / 56 / 3	0.332	35 / 63 / 2	0.286

Appendix

Table 10: Per-scorer breakdown. W/T/L and mean score difference (

\Delta

) against Base (mean Best-of-

K

), reported separately under each evaluator reward model, on the overall set (

N=500

) and the high-disagreement subset (

N=100

) defined by the baseline.

Method	Scores	$\hat{\mu}\ \uparrow\ \ (\hat{\sigma}\ \downarrow)$
Base	$[9,\,1,\,6,\,5,\,4]$	$5.0\ \ (2.92)$
DARC- $\epsilon$ (ours)	$[7,\,8,\,8,\,6,\,7]$	$7.6\ \ (0.84)$

Appendix

Table 11: Same prompt, two responses: DARC increases average ratings and reduces disagreement by using a calmer, institutional framing, though some preference heterogeneity remains.

Method	Scores	$\hat{\mu}\ \uparrow\ \ (\hat{\sigma}\ \downarrow)$
Base	$[3,\,8,\,6,\,7,\,6]$	$6.0\ \ (1.87)$
DARC- $\epsilon$ (ours)	$[7,\,7,\,8,\,8,\,7]$	$7.4\ \ (0.55)$

Appendix

Table 12: Same prompt, two responses: DARC improves average ratings and substantially reduces cross-rater disagreement.

Method	Scores	$\hat{\mu}\ \uparrow\ \ (\hat{\sigma}\ \downarrow)$
Base	$[3,\,5,\,6,\,3,\,6]$	$4.6\ \ (1.52)$
DARC- $\epsilon$ (ours)	$[8,\,8,\,7,\,8,\,7]$	$7.6\ \ (0.55)$

Appendix

Table 13: A case where Base is penalized for hallucination/verbosity and potential copyright issues, while DARC yields a truthful, concise and safety response with higher consistency.

Type	$\hat{\sigma}_{\text{proxy}}$	$\sigma_{\text{human}}$	Prompt (abridged)	Selected response (abridged)	Mismatch note
FP	1.195	0.033	Race riddle: overtake the 2nd person $\rightarrow$ your position?	“You are now 2nd; the overtaken runner is 3rd.”	RM surface-form sensitivity; humans largely unanimous
FP	0.199	0.025	Startup invests $8000 then half next year; total?	Step-by-step arithmetic to $12,000 (with L a T e X formatting).	Formatting/verbosity shifts RM; human scores stable
FP	0.166	0.033	Extract (main character, book, author, year) for 3 blurbs.	Correct 3-line extraction: Harry / Frodo / Zylo with books+years.	Extraction task: humans agree; RM reacts to templates
FP	0.032	0.012	Lesson plan (3 $\times$ 45min) integrating drama/mime: Opium Wars (Gr 9–10).	Structured lesson plan with objectives, activities, materials (long).	Template/style perturbations move RM; humans consistent
FP	0.026	0.000	Compute $f(2)$ for $f(x)=4x^{3}-9x-14$ .	Step-by-step: $f(2)=0$ .	Deterministic math; RM sensitive to presentation
FN	0.187	1.643	Probability: like neither blue nor green given overlaps.	Inclusion–exclusion; response appears truncated mid-computation.	Completeness/correctness (orthogonal) disputed by raters

Appendix

Table 14: Representative mismatch cases between proxy disagreement

\hat{\sigma}_{\text{proxy}}

(reward-model score sensitivity under style-preserving perturbations) and human disagreement

\sigma_{\text{human}}

(std. over multiple independent judge ratings). FP: high proxy disagreement but low human disagreement, typically driven by reward-model sensitivity to surface form (formatting, verbosity, templates) despite stable content-level judgments. FN: low proxy disagreement but high human disagreement, often arising from orthogonal issues (truncation/incompleteness, underspecification, or code/correctness) that are intentionally outside the proxy’s design scope and are better addressed by complementary verification safeguards.

Dataset: Overall
Method	Reward (Avg $\mu$ )	Risk (Avg $\hat{\sigma}$ ) $\downarrow$	Tradeoff $\uparrow$	CVaR 10% $\uparrow$
Base (Best-of- $K$ )	6.31	3.14	0.03	5.11
CVaR (Best-of- $K$ )	5.73	3.01	-0.29	5.16
2nd-Moment (LCB)	5.81	2.83	0.15	5.23
DARC	5.92	2.73	0.46	5.38
DARC- $\tau$	6.11	2.71	0.69	5.43

Appendix

Table 15: Additional results on Qwen2.5-14B-Instruct . We report mean reward (Avg

\mu

), disagreement risk (Avg

\hat{\sigma}

), Tradeoff (computed with the same

\lambda

as in the main experiments), and prompt-level tail robustness (CVaR 10% ). “High-Variance” denotes the top 20% prompts ranked by baseline disagreement under mean Best-of-

K

Proxy variant	Overall $\rho$	Overall partial- $\rho$	Overall Jaccard@20	Stress $\rho$	Stress Jaccard@20
Style-only	0.6509	0.4084	0.47	0.4615	0.34
Targeted-only	0.6120	0.3751	0.44	0.5417	0.42
Hybrid $(4{+}4)$	0.6725	0.4210	0.49	0.5650	0.46

Appendix

Table 16: Proxy–human alignment under different perturbation families. Hybrid perturbations improve alignment on the artifact-focused stress-test slice while preserving strong overall alignment.

Method	Overall Mean	Overall Risk $\downarrow$	Overall Tradeoff $\uparrow$	Overall CVaR ${}_{10}\uparrow$	Stress Tradeoff $\uparrow$	Stress CVaR ${}_{10}\uparrow$
Base (Best-of- $K$ )	7.56	0.67	6.22	6.73	5.35	5.83
DARC- $\epsilon$ + Style-only	8.08	0.55	6.98	7.62	6.36	6.87
DARC- $\epsilon$ + Targeted	7.84	0.53	6.78	7.45	6.75	7.20
DARC- $\epsilon$ + Hybrid	7.96	0.51	6.94	7.71	6.82	7.35

Appendix

Table 17: Downstream robustness under different perturbation families. Each proxy is plugged into DARC-

\epsilon

using the same fixed candidate pools and held-out evaluation protocol. Hybrid perturbations improve stress-test tail robustness while maintaining strong overall performance.

Method	Rank@ $0.5\lambda_{0}$	Rank@ $\lambda_{0}$	Rank@ $2\lambda_{0}$
DARC- $\epsilon$	1	1	1
DARC	2	2	2
Base (Best-of- $K$ )	4	5	5
Caution	3	3	4

Appendix

Table 18: Sensitivity of the Tradeoff metric to the evaluation weight

\lambda_{\text{eval}}

. We report the method’s rank (lower is better) under different scalarization weights (

\lambda_{0}=1.99

). DARC variants consistently outperform baselines across a wide range of risk preferences.

Preference Alignment

Preference alignment studies how AI systems are trained or steered to produce outputs that match human or user priorities.

See all 109 papers

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment
May 20, 2026Zhiqin Yang, Yonggang Zhang, Wei Xue +3LLM Alignment Direct Preference Optimization
1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.
Safe Inference-Time Alignment via Lagrangian Reward Augmentation
Jul 2, 2026Yaswanth Chittepu, Ativ Joshi, Sohini Chintala +1LLM Alignment LLM Safety Alignment
University of Massachusetts Amherst · Independent Researcher
Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner
Sep 11, 2026Kazusato Oko, Annie Ulichney, Nika Haghtalab +1LLM Alignment Preference Alignment
University of California, Berkeley · Center for Advanced Intelligence Project, RIKEN · The Institute of Statistical Mathematics +1

May 20, 2026J/K move · Enter open · S save

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

Zhiqin Yang, Yonggang Zhang, Wei Xue, Dong Fang+2

1The Hong Kong University of Science and Technology · 2LIGHTSPEED · 3Hong Kong Baptist University.

LLM Alignment Direct Preference Optimization Preference Optimization Preference Alignment RL from Human Feedback

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) has emerged as a popular alternative to Reinforcement Learning from Human Feedback (RLHF), offering theoretical equivalence with simpler implementation. We prove this equivalence is conditional rather than universal, depending on an implicit assumption frequently violated in practice: the RLHF-optimal policy must prefer human-preferred responses. When this assumption fails, DPO optimizes relative advantage over the reference policy rather than absolute alignment with human preferences, leading to pathological convergence where policies decrease DPO loss while preferring dispreferred responses. We characterize when this assumption is violated, show the existence of an undesirable solution space, and prove that DPO and RLHF optimize fundamentally different objectives in such cases. To address this, we introduce Constrained Preference Optimization (CPO), augmenting RLHF with constraints for provable alignment. We further provide a geometric interpretation through soft margin ranking, revealing that DPO implements margin ranking with potentially negative targets. Our theoretical analysis establishes when DPOs' guarantees hold and provides solutions preserving simplicity with provable alignment. Comprehensive experiments on standard benchmarks demonstrate that CPO achieves state-of-the-art performance. Code is available at: https://github.com/visitworld123/CPO.

Abstract

Preference Alignment

Explore similar work

Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment

Safe Inference-Time Alignment via Lagrangian Reward Augmentation

Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner