Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment

Ruomeng Ding; Tianwei Gao; Lianrui Geng; Mohit Bansal; Zhun Deng

cs.LGOct 13, 2026

Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment

Authors: Ruomeng Ding, Tianwei Gao, Lianrui Geng, Mohit Bansal, Zhun Deng

Organizations: University of North Carolina at Chapel Hill

Abstract

Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.

Figures & tables

Figure 1: Welfare-aware preference acquisition for pluralistic alignment.

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
No queries	81.87	77.05	80.96	81.25	77.37	82.31	81.81	77.57	81.72	81.50	77.91	80.11
Welfare-agnostic baselines (shared across settings)
APL	82.86	78.50	81.94	82.36	79.16	83.68	82.73	78.99	82.64	82.50	78.94	81.25
BAL-PM	87.02	83.36	86.44	86.40	83.24	87.33	86.78	82.88	86.54	86.58	83.36	85.43
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$

Table 1: Acquisition utility (%) at 20% query budget with welfare-agnostic and welfare-weighted methods. Bold marks the best budgeted method for each welfare setting.

Figure 2: Acquisition utility versus query budget under

(\gamma,\rho)=(2,-5)

. All methods are evaluated at 1% budget increments. Summaries below each subplot show the fraction of the initial-to-oracle utility gap recovered at 20% budget; the oracle uses full-label access.

Panel of Figure 3 — Figure 3: Parameter effects (a) and outcome tradeoffs (b) at 20% budget with a random 10% Target. Changes from $(1,1)$ are in percentage points. In (a), red shows Target gains and blue shows Tail losses. In (b), paths hold $\gamma$ fixed and follow decreasing $\rho$ ; color shows Mean change.

Figure 4: Seen personas: reference-gap recovery at 20% query budget under

(2,-5)

. For every method and metric, 0% denotes All pseudo-labels and 100% denotes All ground-truth labels.

Figure 5: Held-out personas: reference-gap recovery at 20% query budget under

(2,-5)

. For every method and metric, 0% denotes All pseudo-labels and 100% denotes All ground-truth labels.

Method	NA ( $\downarrow$ )	Africa ( $\downarrow$ )	Overall ( $\downarrow$ )	First (%, $\uparrow$ )
Random	2.823	2.674	2.749	14.7
Entropy	2.628	2.661	2.645	20.1
WAPA	2.229	2.367	2.298	27.3
Full-GT	2.320	2.297	2.309	37.9

Table 2: Human evaluation: mean rank (1 = best) and first-place rate.

\downarrow

: lower is better;

\uparrow

: higher is better.

Appendix figures & tables46 assets

Supplementary material from the paper’s appendix.

Appendix

Research direction	Main question in related work	Relation to our work
Active preference acquisition	Which feedback should be collected to improve learning under a limited budget?	We make the distribution of supervision gains across people an explicit part of acquisition.
Personalized / pluralistic alignment	How can models learn or adapt to different individuals’ or groups’ preferences?	Provides the alignment goal our acquisition serves. We allocate a shared feedback budget to construct supervision for persona-conditioned policy training.
Human–AI feedback allocation	Which queries should receive human feedback, and which can use AI feedback?	Provides the labeling decision we study. We account for persona-level supervision quality and priorities when making that decision.
Social welfare and fair acquisition	How should benefits and data-collection resources be distributed across people or groups?	Provides welfare formulations and allocation precedents. We formulate and optimize welfare over preference-supervision quality for alignment.

Appendix

Table 3: Four related research directions and their relation to our work.

Setting	Feedback and its use	Decision studied	Evaluation focus
Active acquisition for policy alignment	Acquire preference labels under a limited budget to train a policy	Which queries improve learning most effectively?	Aggregate policy win rate or reward, without persona-level breakdown
Active elicitation for personalization	Query a user to infer personal preferences and guide responses	What should be asked next to understand this user?	Individual preference inference and personalization quality
Personalized / pluralistic alignment given preference data	Use available preferences to train or adapt models for individuals or groups	How should the model accommodate diverse preferences?	Individual- or group-level alignment quality
Our welfare-aware acquisition for pluralistic alignment	Combine selected human feedback with AI feedback for persona-conditioned policy training	How should a shared budget be allocated given query gains and priorities across personas?	Mean, lower-tail, and designated-group supervision utility

Appendix

Table 4: Feedback acquisition, personalization, and policy alignment: feedback use, decisions, and evaluation focus in representative settings and in our work.

Collection	Acquisition $N\times M$	Total	Training	Seen $N\times M$	Total	Held-out $N\times M$	Total
Africa	77 $\times$ 612	47,124	47,124	77 $\times$ 154	11,858	43 $\times$ 182	7,826
Asia	67 $\times$ 617	41,339	41,339	67 $\times$ 154	10,318	53 $\times$ 182	9,646
North America	75 $\times$ 616	46,200	46,200	75 $\times$ 154	11,550	45 $\times$ 182	8,190
South America	72 $\times$ 617	44,424	—	72 $\times$ 154	11,088	48 $\times$ 182	8,736
Europe	69 $\times$ 617	42,573	42,573	69 $\times$ 154	10,626	51 $\times$ 182	9,282
Heterogeneous	60 $\times$ 610	36,600	—	60 $\times$ 154	9,240	40 $\times$ 182	7,280

Appendix

Table S1 : Roleplay split supports (

N\times M

) and comparison totals. Training uses the full acquisition support for the four main collections; a dash indicates no downstream training.

Region	$N$	$M$	$N\times M$
Midwest	895	30	26,850
Northeast	603	30	18,090
South	735	30	22,050
West	897	30	26,910

Appendix

Table S2 : OpinionQA support: respondents

N

, questions per respondent

M

, and total observations.

Method	OpinionQA-Midwest			OpinionQA-Northeast			OpinionQA-South			OpinionQA-West
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
No queries	42.70	23.88	43.70	44.30	25.98	43.93	42.01	23.43	42.02	44.22	24.23	44.24
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$
Random	54.33	35.78	54.18	55.33	37.64	55.02	53.62	35.37	53.33	55.37	35.53	55.66
Entropy	53.43	34.66	55.15	55.35	37.44	54.77	52.65	33.66	52.40	55.15	35.78	55.32
Reference Disagreement	54.88	33.93	54.93	56.32	35.38	56.52	54.29	34.38	55.93	56.33	34.70	56.96

Appendix

Table S3 : Acquisition utility (%) at 20% query budget under matched welfare specifications. Bold marks the best budgeted method within each panel.

Method	Roleplay-South America			Roleplay-Heterogeneous
	Mean	Tail	Target	Mean	Tail	Target
No queries	81.67	77.14	82.76	81.21	77.84	80.49
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$
Random	85.31	81.87	85.98	85.09	81.75	84.49
Entropy	86.59	81.76	87.64	86.12	81.91	85.20
Reference Disagreement	87.24	83.67	87.72	86.82	84.15	86.71

Appendix

Table S4 : Acquisition utility (%) at 20% query budget under matched welfare specifications. Bold marks the best budgeted method within each panel.

Figure S1 : Acquisition budget curves under equal-weight mean welfare

(1,1)

, using the same metrics as Figure 2 . Summaries below each subplot report recovery at 20% budget with the same definition. All methods are evaluated at 1% budget increments. APL and BAL-PM use the same welfare-agnostic selections across both specifications.

Score	Africa			Asia			North America			Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
$(\gamma,\rho)=(1,1)$
WAPA	90.384	87.529	90.387	89.838	87.061	89.544	90.221	87.814	90.450	90.075	87.062	89.552
Discrepancy only	90.384	87.565	90.350	89.857	86.962	89.502	90.228	87.797	90.487	90.092	87.056	89.587
Reference uncertainty only	88.081	84.593	87.543	87.492	84.199	88.051	87.855	84.474	87.929	87.536	84.673	86.062
$(\gamma,\rho)=(2,-5)$

Appendix

Table S5 : Acquisition Brier utility (%) for score ablations at 20% query budget. All scores use exact marginal allocation and target-model completion. Bold marks the highest value within each setting.

Dataset	Overlap (%)	$\Delta$ Mean	$\Delta$ Tail	$\Delta$ Target
Roleplay-Africa	99.9894	+0.0021	+0.0000	+0.0000
Roleplay-Asia	99.9879	+0.0000	+0.0000	+0.0000
Roleplay-North America	99.9892	+0.0000	+0.0000	+0.0000
Roleplay-Europe	100.0000	+0.0000	+0.0000	+0.0000

Appendix

Table S6 : First-order versus exact WAPA at 20% budget under

(2,-5)

. Utility differences are first-order minus exact, in %. Overlap is the fraction of queried comparisons shared by the two sets.

References	Africa			Asia			North America			Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
$(\gamma,\rho)=(1,1)$
WAPA	90.384	87.529	90.387	89.838	87.061	89.544	90.221	87.814	90.450	90.075	87.062	89.552
Qwen only	89.899	86.990	89.612	89.399	86.705	89.165	90.006	87.883	90.272	89.606	86.717	89.122
GLM only	90.057	87.666	89.953	89.406	86.615	89.082	89.869	87.505	89.999	89.693	86.887	89.034
DeepSeek only	89.362	86.825	89.776	88.933	86.330	88.750	88.578	85.743	88.681	89.320	86.228	88.694

Appendix

Table S7 : Acquisition Brier utility (%) for reference-composition ablations at 20% query budget. All variants use exact marginal allocation and target-model completion. Bold marks the highest value within each setting.

Panel of Figure 18 — Figure S2 : Parameter effects and outcome tradeoffs under an alternative designated subgroup: the 10% of personas with the lowest initial reference-estimated supervision utility. Membership is fixed across settings. Changes are relative to $(1,1)$ with this same subgroup, in %.

Method	Africa			Asia			North America			Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
$\varepsilon=0.1$
Random	85.528	81.806	84.866	85.037	81.652	85.532	85.535	82.292	85.229	85.228	82.254	84.567
Entropy	86.760	82.083	87.086	86.168	82.092	89.013	86.750	82.346	87.758	86.435	82.278	86.545
Reference Disagreement	87.354	83.223	90.889	86.834	83.195	91.625	86.955	82.677	90.507	87.088	83.613	90.070
WAPA	90.286	87.296	93.847	89.750	86.860	92.833	90.077	87.430	93.602	90.003	87.254	93.032

Appendix

Table S8 : Acquisition Brier utility (%) across shifts

\varepsilon

(\gamma,\rho)=(2,-5)

and 20% budget. Each panel uses the same shift for all welfare-weighted methods. Random is shared. Bold marks the highest value within each panel.

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
All pseudo-labels	83.36	78.40	82.25	82.89	78.41	83.41	83.38	78.45	84.06	83.15	78.20	81.67
Welfare-agnostic baselines (shared across settings)
APL	83.68	79.06	82.75	83.32	79.20	83.64	83.65	79.03	84.26	83.66	78.87	82.41
BAL-PM	84.25	79.89	83.19	83.84	79.55	84.04	84.14	79.47	84.88	84.26	79.85	83.18
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$

Appendix

Table S9 : Seen-persona utility after soft-label DPO (%) under matched welfare specifications.

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
All pseudo-labels	83.05	78.70	85.31	83.52	79.38	85.49	82.90	78.23	84.76	83.18	79.33	83.82
Welfare-agnostic baselines (shared across settings)
APL	83.41	79.58	85.41	83.86	80.13	85.72	83.25	78.80	84.94	83.53	80.15	84.13
BAL-PM	84.07	80.28	85.99	84.32	80.67	86.02	83.70	79.06	85.52	84.09	81.04	84.48
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$

Appendix

Table S10 : Held-out-persona utility after soft-label DPO (%) under matched welfare specifications.

Figure S3 : Soft-label DPO: Seen-persona reference-gap recovery under

(1,1)

at 20% budget. All pseudo-labels defines 0%; dashed lines mark All ground-truth labels at 100%. Recovery is uncapped for every method.

Figure S4 : Soft-label DPO: Held-out-persona reference-gap recovery under

(1,1)

at 20% budget. All pseudo-labels defines 0%; dashed lines mark All ground-truth labels at 100%. Recovery is uncapped for every method.

Method	Seen personas			Held-out personas
	Mean	Tail	Target	Mean	Tail	Target
Qwen3-4B
All pseudo-labels	83.38	78.45	84.06	82.90	78.23	84.76
Random	83.94	79.18	84.59	83.49	78.82	85.39
Entropy	83.96	78.98	84.74	83.54	78.66	85.53
APL	83.65	79.03	84.26	83.25	78.80	84.94

Appendix

Table S11 : Roleplay-North America: absolute Brier utility (%) across backbones. Budgeted methods query 20% of training comparisons; Entropy and WAPA use

(\gamma,\rho)=(2,-5)

. Bold marks the best budgeted value within each backbone and column.

Method	Seen personas			Held-out personas
	Mean	Tail	Target	Mean	Tail	Target
Qwen3-4B
All pseudo-labels	75.00	65.97	75.88	74.30	67.09	77.88
Random	75.83	67.27	77.55	75.29	67.34	78.71
Entropy	76.30	67.62	78.01	75.53	68.13	78.85
APL	75.82	67.53	77.09	74.87	66.73	78.85

Appendix

Table S12 : Roleplay-North America: absolute decision accuracy (%) across backbones. Budgeted methods query 20% of training comparisons; Entropy and WAPA use

(\gamma,\rho)=(2,-5)

. Bold marks the best budgeted value within each backbone and column. Tail reranks personas by accuracy.

Method	Seen personas			Held-out personas
	Mean	Tail	Target	Mean	Tail	Target
Qwen3-4B
All pseudo-labels	83.36	78.40	82.25	83.05	78.70	85.31
Random	84.07	79.64	83.06	83.68	79.78	85.82
Entropy	84.09	79.74	82.98	83.81	79.47	85.82
APL	83.68	79.06	82.75	83.41	79.58	85.41

Appendix

Table S13 : Roleplay-Africa: absolute Brier utility (%) across backbones. Budgeted methods query 20% of training comparisons; Entropy and WAPA use

(\gamma,\rho)=(2,-5)

. Bold marks the best budgeted value within each backbone and column.

Method	Seen personas			Held-out personas
	Mean	Tail	Target	Mean	Tail	Target
Qwen3-4B
All pseudo-labels	75.20	67.26	74.58	74.56	67.06	78.02
Random	76.24	68.25	75.32	75.58	68.20	78.98
Entropy	76.70	68.56	74.95	75.94	68.89	79.95
APL	75.55	67.60	74.49	75.29	68.46	78.30

Appendix

Table S14 : Roleplay-Africa: absolute decision accuracy (%) across backbones. Budgeted methods query 20% of training comparisons; Entropy and WAPA use

(\gamma,\rho)=(2,-5)

. Bold marks the best budgeted value within each backbone and column. Tail reranks personas by accuracy.

Backbone	Split	Gain (pp)			Recovery (%)
		Mean	Tail	Target	Mean	Tail	Target
Roleplay-North America
Qwen3-4B	Seen	1.067	1.804	1.057	74.8	79.6	89.5
Qwen3-4B	Held-out	1.191	1.926	1.137	74.9	104.7	66.5
Llama-3.1-8B	Seen	1.823	2.705	1.819	28.5	51.7	25.8
Llama-3.1-8B	Held-out	1.853	2.857	1.574	30.4	45.7	28.6

Appendix

Table S15 : WAPA utility gains over All pseudo-labels (percentage points, pp) and reference-gap recovery (%) at 20% budget. Each backbone uses its own All ground-truth labels reference. Recovery uses unrounded inputs and is uncapped.

Figure S5 : Downstream lower-tail gains over Random and matched Entropy, in %. Solid/dashed curves indicate seen/held-out personas.

		Roleplay-North America			Roleplay-Africa
$\gamma$	$\rho$	Mean	Tail	Target	Mean	Tail	Target
1	1	84.4665	80.2619	85.1777	84.7394	81.1228	83.9953
2	1	84.4739	80.2681	85.0778	84.7235	81.2481	83.8969
1	-5	84.4821	80.3644	85.0693	84.7745	81.2085	83.9798
2	-5	84.4467	80.2528	85.1182	84.7729	81.2272	83.9800

Appendix

Table S16 : Seen-persona Brier utilities (%) after soft-label DPO across four welfare settings on Roleplay-North America and Roleplay-Africa. All settings use a 20% query budget and the same training protocol.

Acquisition	Completion	Africa			Asia			North America			Europe
	source	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
No queries	Target	81.873	77.054	80.962	81.250	77.370	82.310	81.809	77.574	81.718	81.500	77.907	80.112
No queries	Reference	85.270	83.333	85.365	84.922	82.640	84.860	85.117	83.375	85.181	85.158	83.416	84.291
(a) $(\gamma,\rho)=(1,1)$
Random	Target	85.528	81.806	84.866	85.037	81.652	85.532	85.535	82.292	85.229	85.228	82.254	84.567
Entropy	Target	86.755	81.832	85.701	86.181	82.256	87.693	86.762	82.474	86.253	86.427	82.207	84.788

Appendix

Table S17 : Acquisition Brier utility (%) with target or reference-mean probability completion. Budgeted methods query 20% of the same pool; each method keeps its query set fixed across completion sources. Bold marks the best budgeted method within each setting and completion source. No-query rows use no acquired labels.

Acquisition	Completion	Africa			Asia			North America			Europe
	source	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
No queries	Target	77.563	71.948	76.447	76.736	72.019	77.823	77.123	72.294	77.435	77.002	72.452	75.932
No queries	Reference	81.606	77.209	81.956	80.795	75.786	80.146	80.597	76.374	80.821	81.244	77.063	79.146
(a) $(\gamma,\rho)=(1,1)$
Random	Target	82.071	77.602	81.256	81.465	77.348	81.902	81.801	77.944	81.633	81.620	77.965	81.226
Entropy	Target	85.727	80.449	84.594	85.070	81.086	86.683	85.703	81.266	85.204	85.289	80.626	83.522

Appendix

Table S18 : Completed-label accuracy (%) with target thresholding or reference majority voting. Budgeted methods query 20% of the same pool; each method keeps its query set fixed across completion sources. Bold marks the best budgeted method within each setting and completion source. No-query rows use no acquired labels.

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
All pseudo-labels	83.04	78.04	81.92	82.74	78.05	83.07	83.15	77.92	83.91	82.89	77.46	81.29
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$
Random	83.91	79.26	82.71	83.48	79.09	83.71	83.77	78.87	84.43	83.76	78.89	82.43
Entropy	83.97	79.38	82.96	83.67	79.21	83.84	83.88	78.80	84.60	83.91	79.01	82.70
WAPA	84.64	81.00	83.76	84.25	80.25	84.20	84.31	79.93	85.07	84.43	80.43	83.46

Appendix

Table S19 : Seen-persona hard-label DPO utility (%) under matched welfare specifications.

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
All pseudo-labels	82.69	78.05	84.76	83.50	79.12	85.36	82.64	77.71	84.55	82.90	78.68	83.35
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$
Random	83.56	79.18	85.70	84.06	80.20	85.83	83.18	78.27	85.13	83.62	79.87	84.04
Entropy	83.77	79.35	85.82	84.28	80.34	86.02	83.50	78.38	85.46	83.89	80.17	84.30
WAPA	84.33	80.36	86.54	84.75	81.29	86.20	83.92	79.71	85.72	84.25	81.40	84.50

Appendix

Table S20 : Held-out-persona hard-label DPO utility (%) under matched welfare specifications.

Figure S6 : Hard-label DPO: Seen-persona reference-gap recovery under both welfare specifications. All pseudo-labels defines 0%; dashed lines mark All ground-truth labels at 100%. All methods use unrounded hard-label utilities, with no display cap.

Figure S7 : Hard-label DPO: Held-out-persona reference-gap recovery under both welfare specifications. All pseudo-labels defines 0%; dashed lines mark All ground-truth labels at 100%. Recovery is uncapped for every method.

Figure S8 : Hard-label DPO: lower-tail utility gains over Random and matched Entropy under both welfare specifications. Solid/dashed curves indicate Seen/Held-out personas.

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
All pseudo-labels	74.96	67.03	73.84	74.39	65.69	74.46	75.07	66.97	76.72	74.20	65.00	70.89
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$
Random	76.16	68.31	74.40	75.49	67.32	75.54	75.56	67.23	77.18	75.54	66.57	72.40
Entropy	76.49	68.76	75.23	75.73	67.71	75.76	76.21	66.84	78.66	75.93	67.41	72.94
WAPA	77.29	71.04	75.60	76.84	69.65	76.84	76.66	69.05	78.29	76.67	69.65	74.68

Appendix

Table S21 : Seen-persona hard-label DPO decision accuracy (%) under matched welfare specifications. Companion to Table S19 . Tail reranks personas by accuracy; bold marks the best budgeted method in each panel.

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
All pseudo-labels	74.39	67.57	76.92	75.27	67.81	78.35	74.14	66.24	78.71	74.54	66.97	75.38
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$
Random	75.43	67.35	78.16	75.70	69.32	78.79	75.02	66.18	78.43	75.58	68.07	75.60
Entropy	76.02	68.72	79.40	76.31	69.32	78.90	75.70	67.77	78.71	76.06	69.58	77.36
WAPA	76.64	68.49	81.18	77.40	71.21	79.78	76.14	68.50	79.12	76.60	70.56	77.47

Appendix

Table S22 : Held-out-persona hard-label DPO decision accuracy (%) under matched welfare specifications. Target uses the shared similarity-defined cohort (Appendix E.1 ). Companion to Table S20 . Tail reranks personas by accuracy; bold marks the best budgeted method in each panel.

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
No queries	77.55	72.00	76.41	76.74	71.99	77.86	77.14	72.23	77.47	77.00	72.45	75.93
Welfare-agnostic baselines (shared across settings)
APL	78.60	73.48	77.45	77.93	73.86	79.38	78.11	73.72	78.42	78.04	73.60	77.09
BAL-PM	84.02	79.56	83.03	83.21	78.99	84.55	83.36	78.72	83.56	83.33	79.19	82.50
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$

Appendix

Table S23 : Acquisition hard accuracy (%) at the same 20% query budget and frozen selections as Table 1 . Tail reranks personas by accuracy. Bold marks the best budgeted method for each welfare setting; Oracle retains its Brier-welfare objective. APL and BAL-PM share selections across settings; adaptations are described in Appendix F.3 .

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
All pseudo-labels	75.20	67.26	74.58	74.56	66.73	75.97	75.00	65.97	75.88	74.30	64.99	70.67
Welfare-agnostic baselines (shared across settings)
APL	75.55	67.60	74.49	75.07	66.41	74.78	75.82	67.53	77.09	75.32	65.82	72.40
BAL-PM	76.61	68.70	75.32	75.95	68.33	75.43	76.02	67.58	77.46	76.39	68.99	74.13
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$

Appendix

Table S24 : Seen-persona decision accuracy after soft-label DPO (%). Tail reranks personas by accuracy. Bold marks the best budgeted values in each welfare setting.

Method	Roleplay-Africa			Roleplay-Asia			Roleplay-North America			Roleplay-Europe
	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target	Mean	Tail	Target
All pseudo-labels	74.56	67.06	78.02	74.99	67.54	78.68	74.30	67.09	77.88	74.52	66.75	75.38
Welfare-agnostic baselines (shared across settings)
APL	75.29	68.46	78.30	75.78	69.19	79.67	74.87	66.73	78.85	75.08	68.62	76.26
BAL-PM	76.04	69.90	79.67	76.58	69.01	78.90	75.68	68.19	79.40	76.08	68.72	76.81
(a) Equal-weight mean welfare: $(\gamma,\rho)=(1,1)$

Appendix

Table S25 : Held-out-persona decision accuracy after soft-label DPO (%). Tail reranks personas by accuracy. Target uses the fixed similarity-defined cohort. Bold marks the best budgeted values in each welfare setting.

Dataset	$(\gamma,\rho)$	WAPA Brier	WAPA + PPCI Brier	Reduction (%)	$\alpha$
Africa	$(1,1)$	0.120199	0.106008	11.807	0
Africa	$(2,-5)$	0.122033	0.107696	11.749	0
Asia	$(1,1)$	0.127026	0.112180	11.687	0
Asia	$(2,-5)$	0.128574	0.113112	12.026	0
North America	$(1,1)$	0.122235	0.108740	11.040	0
North America	$(2,-5)$	0.124773	0.110633	11.333	0

Appendix

Table S26 : Prediction refinement after WAPA acquisition on Roleplay. Lower Brier loss is better. Reduction is relative to the raw WAPA completion;

\alpha

is selected using acquired labels before evaluation .

		Mean			Tail			Target
Dataset	$(\gamma,\rho)$	W	P	$\Delta$	W	P	$\Delta$	W	P	$\Delta$
Seen personas
Africa	$(1,1)$	84.74	84.81	$+0.066$	81.12	81.71	$+0.584$	84.00	84.17	$+0.171$
Africa	$(2,-5)$	84.77	84.82	$+0.043$	81.23	81.65	$+0.427$	83.98	84.20	$+0.224$
Asia	$(1,1)$	84.40	84.34	$-0.060$	80.64	80.92	$+0.284$	84.22	84.15	$-0.067$
Asia	$(2,-5)$	84.33	84.41	$+0.079$	80.54	81.02	$+0.471$	84.41	84.29	$-0.120$

Appendix

Table S27 : Downstream sensitivity to proxy-label refinement at the 20% query budget. W denotes WAPA; P denotes WAPA + PPCI. Utilities use the percentage scale;

\Delta

is P minus W in percentage points, computed before rounding. Tail is the lower-tail utility, reranked for each policy; Target membership is fixed across policies.

Prompt	How can I balance work and personal life better?
Profile excerpt	[…] She prefers traditional approaches over technology . […] She values personal growth through reflection and intrinsic motivation. […] She values adaptability and spontaneity. […]
Random	[…] So, go ahead and try this: each week, pick one day to simply be —no work, no plans. Just sit, watch, listen. That day will teach you more than any schedule ever could. […]
Entropy	[…] I’ve seen elders in my community who worked hard, yes, but they didn’t burn out because they made space for daily rituals—like sitting under a tree to reflect, or helping a neighbor with a small task. These small acts of care kept their spirit strong. So, consider setting a daily " stillness time "—even five minutes—where you don’t check messages or plan tasks. Just breathe, observe, and remember who you are. […]
WAPA	[…] You’ve expressed a preference for traditional, tangible methods —so instead of relying on digital calendars or apps, use a simple notebook or a woven journal to track your days. Write down your daily activities , and at the end of each week , review them with a calm, reflective mindset . This helps you see patterns and make intentional choices. […]
All ground-truth labels	[…] 6. Choose Low-Impact, Practical Solutions When managing time, avoid over-scheduling. Use traditional methods—like a simple notebook or calendar—to plan your days. Avoid digital overload. Instead, rely on your memory, intuition, and the wisdom of your experiences. […]
Reading the example	The persona prefers traditional tools, reflection, and flexibility. Random and Entropy address rest and reflection, but their full responses do not propose a concrete non-digital time-management tool. WAPA explicitly connects the preference for traditional methods to a notebook, then combines daily recording with weekly review. The all-ground-truth-label policy also recommends a notebook or calendar. The difference is a more explicit link from a stated tool preference to a reflective practice.

Appendix

Table S28 : Turning a non-digital preference into an actionable routine. Roleplay-Africa; Seen personas. Response excerpts retain the original wording; omissions are marked. Blue links non-digital preferences to tools; brown marks reflection and routine.

Prompt	Which smartphone should I buy?
Profile excerpt	[…] A 50-year-old North American female entrepreneur who runs a successful online business focused on sustainable products . […] She is budget-conscious and prefers financially prudent choices . […] She values adaptable and versatile solutions in her workspace. […]
Random	[…] For someone who values sustainability , adaptability, and a strong tech experience—especially in capturing food and creative content—the Google Pixel 7 or 7 Pro stands out. It offers excellent performance, strong camera capabilities, and a focus on user experience and innovation—all while being more budget-conscious than premium Apple or Samsung models. […] If you’re open to a more versatile device for work and lifestyle use, the Samsung Galaxy Z Flip 4 is a great alternative, especially if you value portability and dynamic experiences. […]
Entropy	[…] Given your interest in sustainability , financial prudence, and a balanced, dynamic experience, the Google Pixel 7 or 7 Pro stands out. It offers strong performance, excellent camera capabilities (perfect for food and nature), and a focus on user experience and innovation—all while being more affordable and environmentally conscious than some premium options. […] If you’re looking for a more versatile, adaptable device for work and lifestyle, consider the Samsung Galaxy Z Flip 4 . […]
WAPA	[…] - Samsung Galaxy Z Fold 4 or Z Flip 4 – Offers excellent build quality, long battery life, and a versatile form factor. However, it may have a higher environmental footprint due to materials and manufacturing. Consider this only if you value portability and durability above all else. […] Before purchasing, check for refurbished or trade-in options through certified sellers to reduce cost and environmental impact . Also, consider the device’s lifecycle—opt for one that supports long-term updates and repairability. […]
All ground-truth labels	[…] If you’re mindful of cost, look for devices with strong value over time. The Pixel line often offers better long-term value due to software updates and durability, even if the initial price is higher. […] If sustainability is your top priority, pair it with a device from Fairphone or a refurbished option from a certified seller to reduce environmental impact. […]
Reading the example	The persona values sustainability, financial prudence, and versatility. Random and Entropy acknowledge these priorities and recommend Pixel phones with foldable alternatives. WAPA also recommends Pixel phones, but conditions its foldable alternative on a stated environmental tradeoff and proposes refurbished or trade-in purchases to reduce cost and impact. The baselines do not propose these purchasing routes in their full responses. Refurbished purchasing also appears in the all-ground-truth-label response, although its final recommendation is Pixel 8 rather than WAPA’s Pixel 7. The analysis compares links between preferences and advice; the quoted product claims remain unverified.

Appendix

Table S29 : Translating sustainability and financial prudence into purchasing choices. Roleplay-North America; Held-out personas. Response excerpts retain the original wording; omissions are marked. Blue marks sustainability; brown marks financial prudence and purchasing options.

Prompt	Which musical instrument should I learn to play?
Profile excerpt	[…] An 82-year-old man from Africa who is a well-respected elder in his tribe, often called upon to perform traditional ceremonies and rituals that connect the community to their ancestors. […] He prefers practical and affordable options. […] He is interested in ethical consumption and supporting local economies . […]
Random	[…] A drum would be an excellent choice. […] It’s also practical: drums are relatively affordable, easy to learn, and require minimal maintenance. You can use it to lead or participate in community events, rituals, or storytelling sessions , making it a powerful tool for cultural expression and unity. […]
Entropy	[…] A drum would be an excellent choice. […] It’s also practical: drums are relatively affordable, easy to learn, and require minimal equipment. You can use it to lead or participate in community events, rituals, or storytelling sessions , reinforcing your role as a respected elder. […]
WAPA	[…] Given your appreciation for classic storytelling, community involvement, and practical, affordable options, a traditional African drum might be a meaningful and culturally resonant instrument to learn. […] - Participate in and lead traditional ceremonies . - Share stories through rhythm and sound, enhancing community engagement. - Support local economies by learning from or teaching within your community . - Use a practical, accessible instrument that doesn’t require expensive equipment. […]
All ground-truth labels	[…] Given your appreciation for classic storytelling, community involvement, and practical, affordable options, a traditional African drum might be a meaningful and culturally resonant instrument to learn. […] - Participate in and lead traditional ceremonies . - Share stories through rhythm and sound, enhancing community engagement. - Support local economies by purchasing instruments from local artisans . - Use the instrument in a practical, accessible way—many traditional drums are affordable and easy to maintain. […]
Reading the example	All four policies recommend drums and connect them to ceremonies, storytelling, community participation, and affordability. WAPA additionally addresses the explicit preference for supporting local economies, which Random and Entropy do not discuss in their full responses. It proposes learning or teaching within the community; the all-ground-truth-label policy instead proposes purchasing from local artisans. Both attend to the economic dimension, but the latter gives a clearer purchasing action: WAPA does not explain how learning or teaching provides financial support. The contrast is broader coverage of a stated preference.

Appendix

Table S30 : Addressing local economic support alongside cultural and community interests. Roleplay-Africa; Held-out personas. Response excerpts retain the original wording; omissions are marked. Blue marks shared cultural and practical preferences; brown marks local economic support.

Method	North America	Africa	Overall	First place (%)
Random	2.823	2.674	2.749	14.7
Entropy	2.628	2.661	2.645	20.1
WAPA	2.229	2.367	2.298	27.3
Full-GT	2.320	2.297	2.309	37.9

Appendix

Table S31 : Human evaluation on 96 instances (48 per region). Lower mean rank is better; first-place rates are instance-weighted. Two instances have eight raters, 18 have two, and 76 have one.

Comparison	$N$	Pairwise agreement (95% CI)	Same winner	Exact ranking
DeepSeek–human	96	53.7 [49.6, 57.8]	24.4	5.2
DeepSeek–human	32	54.6 [49.0, 60.3]	24.8	3.1
Qwen–human	32	55.6 [50.4, 60.9]	25.4	4.9
GLM–human	32	55.2 [50.3, 60.7]	26.2	7.2
Human–human	20	48.4	33.0	6.4

Appendix

Table S32 : Agreement (%);

N

is the number of instances. Pairwise agreement covers all six response pairs. We average agreement over raters and display orders within instances, then across instances. Human–human agreement uses the 20 instances with multiple raters. Instance-bootstrap CIs are conditional on the observed raters and do not account for rater reuse across instances.

Evaluator	$N$	Random	Entropy	WAPA	Full-GT
Human	96	2.749	2.645	2.298	2.309
DeepSeek	96	2.422	2.557	2.469	2.552
DeepSeek	32	2.344	2.453	2.563	2.641
Qwen	32	2.563	2.547	2.563	2.328
GLM	32	2.438	2.500	2.672	2.391
DeepSeek	200	2.420	2.533	2.528	2.520

Appendix

Table S33 : Mean ranks; lower is better. The first block compares humans and DeepSeek on the same 96 instances; the second compares judges on the same 32-instance subset. The 200-instance pool contains 100 personas and shares 64 instances with the human evaluation set.

Judge	$N$	Pairwise consistency	Same winner	Same complete ranking
DeepSeek	96	75.3	56.3	27.1
DeepSeek	32	75.5	50.0	18.8
Qwen	32	65.6	56.3	15.6
GLM	32	54.2	34.4	15.6
DeepSeek	200	72.8	56.5	24.0
Independent uniform rankings	–	50.0	25.0	4.2

Appendix

Table S34 : Consistency across reversed display orders (%), with rankings mapped back to response identities. Sample coverage matches Table S33 . The random row reports theoretical expectations.

DeepSeek prompt	$N$	Token limit	Pairwise agreement	Same winner
v1: shared rubric	32	8,192	54.6	24.8
v2: structured diagnostics	32	16,384	53.1	23.2
v1: matched subset	8	8,192	54.2	21.9
v3: compact holistic rubric	8	16,384	42.7	21.9
v1: matched complete subset	29	8,192	55.7	27.4
Transferred persona prompt	29	8,192	52.6	25.0

Appendix

Table S35 : Agreement with humans (%) on diagnostic subsets; compare within blocks. v3 uses eight development cases. The transferred prompt adapts the binary persona-preference prompt and yielded 61/64 valid judgments, covering 29 cases with both orders complete. v2 and v3 change the output format and permit random tie-breaking for indistinguishable and identical answers, respectively.

DeepSeek binary-choice diagnostic	Pairs	Result (%; 95% CI)
Oracle agreement: overall	192	79.2 [73.4, 84.4]
Oracle agreement: North America	96	76.6 [68.2, 84.4]
Oracle agreement: Africa	96	81.8 [74.0, 89.1]
Cross-order content consistency	192	89.6 [85.4, 93.8]
Same-order repeat consistency	48	95.8
Reversed-order consistency: same 48	48	87.5

Appendix

Table S36 : Separate binary diagnostic using synthetic oracle preferences: 192 original dataset pairs, 64 personas, and 48 pairs per region-by-split stratum. Presenting each pair in both orders yields 304/384 choices that match the oracle. Upper-block CIs resample personas; the control difference uses a paired, stratified bootstrap over response pairs.

Social Welfare Maximization

See all 20 papers

Oct 17, 2026

Pluralistic Alignment in Language Models: Objectives, Human References, and Preference Representations

Pluralistic alignment aims to represent meaningful variation in human preferences rather than collapse heterogeneous judgments into a single target. Yet existing work often mixes three distinct questions: what pluralistic behavior a model should exhibit, whose preferences define the reference, and what evidence is sufficient to demonstrate success. We organize the literature around a two-layer taxonomy. Layer 1 distinguishes representing a range of human-supported viewpoints (Overton Pluralism), adapting faithfully to a specified perspective or preference condition (Steerable Pluralism), and matching the response distribution observed in a target population (Distributional Pluralism). Layer 2 distinguishes aggregate populations, countries and regions, and subgroups as human references. Preference representation, implementation mechanism, modality, and evidence quality are treated as cross-cutting design choices rather than additional forms of pluralism. We synthesize a protocol-bounded corpus of 70 works on pluralistic alignment and neighboring areas, using the objective-by-human-reference matrix to map existing approaches and identify underexplored combinations. The coded literature concentrates on Steerable Pluralism and Distributional Pluralism, while eligible Overton Pluralism work appears only at the aggregate-population level under our stated protocol. We then connect each objective to the evidence needed to support it, including perspective validity, conditional fidelity, distributional fidelity, support preservation, and uncertainty in subgroup estimates. Together, the taxonomy and evaluation framework clarify what current methods establish, where their claims exceed their evidence, and how future systems can represent disagreement without mistaking variation for faithful representation.

Meng-Chen Wu, Umesh Bodhwani

Amazon Science

Pluralistic Alignment Preference Modeling

Ask PDF

May 2, 2026

Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracy

Prevailing alignment methods target a fixed set of preferences and therefore risk forcing value lock-in as societal norms evolve over time. We introduce Adaptive Pluralistic Alignment (APA), a modular pipeline for updating pluralistically aligned AI systems to track evolving values and avoid value lock-in without repeating costly pretraining or large-scale data collection. APA has three stages: (1) learning compact personalized reward models via low-rank reward basis decomposition, (2) using these models as a jury that collectively selects among candidate outputs through social-choice-theoretic voting, and (3) efficiently adapting the jury over time by fitting new annotator weights over the fixed reward bases as values shift. The resulting system is efficient, explainable, steerable, and modular. We implement a proof-of-concept instantiation using the PRISM multi-user alignment dataset and simulated historical annotators, and provide preliminary analysis showing that jury composition and the choice of voting rule can substantially affect outcomes, particularly when jury preferences are heterogeneous. We provide full code and resulting preference datasets at https://github.com/RachelFreedman/apa.

Rachel Freedman

Department of Electrical Engineering and Computer Science, University of California, Berkeley, USA.

Reward Modeling LLM Alignment Pluralistic Alignment

Ask PDF

Jun 9, 2026

Hidden Consensus:Preference-Validity Compression in Human Feedback

Standard RLHF pipelines often reduce heterogeneous human judgments into a single scalar reward target. We argue that this reduction can mis-measure alignment in structurally plural societies, where disagreement may reflect culturally, historically, linguistically, regionally, or normatively grounded interpretations rather than annotation noise. We call this failure Preference-Validity Compression, the collapse of multiple plural-valid response options into a single optimization target. Using Malaysia as a diagnostic setting, we analyze RLHF-style feedback aggregation through preference events linking prompts, responses, and acceptability judgments across interpretive frames. Across 321 preference events from 20 participants and 107 trio-annotated prompts, 79% of prompts contain more than one majority-supported response that single-winner aggregation would discard, and apparent dominance gaps between top responses diminish when all majority-supported options are considered. Participants frequently select multiple acceptable responses, and discarded responses demonstrably reflect coherent local, practical, or cultural frames. These findings show that majority aggregation in this corpus measures argmax acceptability rather than plural alignment. We treat this as a measurement-validity issue and argue that future alignment methods should satisfy Validity-Preserving Consistency, remaining stable across plural-valid interpretive frames rather than collapsing them into a single reward target.

Dorcas Chia Ern Chua, Karen Myn Hui Lee, Jia Yue Tan +9

1YTL AI Labs · 2Universiti Malaya

Human Preference Modeling Preference Alignment Pluralistic Alignment RL from Human Feedback

Ask PDF

Abstract

Social Welfare Maximization

Explore similar work

Pluralistic Alignment in Language Models: Objectives, Human References, and Preference Representations

Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracy

Hidden Consensus:Preference-Validity Compression in Human Feedback