Steerable Pluralistic Policy Alignment from Anonymized Crowd Feedback

Akansha Kalra; Ellen R. Novoseller; Vernon J. Lawhern; Daniel S. Brown

cs.LGOct 17, 2026

Steerable Pluralistic Policy Alignment from Anonymized Crowd Feedback

Authors: Akansha Kalra, Ellen R. Novoseller, Vernon J. Lawhern, Daniel S. Brown

Organizations: Kahlert School of Computing, University of Utah, Salt Lake City, USA · U.S. Army DEVCOM Army Research Laboratory, Aberdeen Proving Ground, MD, USA

Abstract

Pluralistic alignment requires AI systems to learn from diverse populations whose preferences may reflect heterogeneous or even conflicting objectives, and to personalize their behavior to individual users. Existing approaches in pluralistic alignment often assume access to annotator identities and rely on powerful pretrained representations to model individual preference variations. We propose Preference-PCA Pluralistic Learning (P3L), a novel framework for pluralistic alignment that jointly learns trajectory embeddings and the shared basis of preference directions via PCA decomposition from heterogeneous crowd feedback without requiring annotator identities. At test time, our approach enables rapid and accurate personalized reward learning and downstream policy optimization. Experiments across multiple embodied control tasks demonstrate that P3L enables steerable personal alignment that generalizes beyond the offline dataset to out-of-distribution new users and attains competitive RL and test performance, while substantially reducing computational cost and supporting variable adaptation query budgets.

Figures & tables

Figure 1: Preference-PCA Pluralistic Learning (P3L)

Figure 2: Environments MO-Highway, MO-Reacher, MO-Humanoid and MO-Kitchen.

	Baselines		P3L
Environment	SingleRM	VPL	Privileged-Traj-MaskedDP	Anonymized-CLS-MaskedDP
MO-Highway	70.222 ± 3.326	66.667 ± 3.143	84.889 ± 5.145	78.667 ± 4.576
MO-Reacher	54.667 ± 0.667	52.00 ±3.055	93.000 ± 1.453	93.500 ± 1.434
MO-Humanoid	76.280 ± 1.294	74.920 ± 2.018	88.320 ± 0.515	77.880 ± 1.017
MO-Kitchen-2d	50.000 ± 0.000	50.400 ± 0.800	100.000 ± 0.0	90.400 ± 4.079
MO-Kitchen-3d	53.185 ± 1.582	51.407 ± 1.525	84.148 ± 2.227	82.667 ± 3.647

Table 1: Average user accuracy of held out test-set preference queries on different environments reported as Mean and Std across 5 seeds with number of different users possible on different objectives reflected next to each environment . Both of our approaches, Privileged-Traj-MaskedDP which assumes privileged access to annotator id (as does VPL), as well as Anonymized-CLS-MaskedDP which does not assume any access to annotator identity, outperform both the baselines Single Reward Model and VPL on average across all users. The blue background indicates the best performance, while the gray background denotes the second-best performance.

	Baselines			P3L
Environment	SingleRM-RL	VPL-OfflineRL	VPL-RL	Privileged-RL	Anonymized-RL
MO-Highway	4.303 ± 9.059	8.409 ± 15.063	4.354 ± 18.164	7.721 ± 16.435	5.126 ± 20.505
MO-Reacher	16.704 ± 1.999	16.090 ± 12.372	20.427 ± 1.616	31.708 ± 4.520	28.603 ± 1.996
MO-Humanoid	-48.433 ± 171.307	-51.301 ± 171.186	-46.240 ± 172.485	-10.509 ± 169.372	-38.794 ± 173.861
MO-Kitchen-2d	0.274 ± 0.274	0.363 ± 0.037	–	0.834 ± 0.154	0.334 ± 0.182
MO-Kitchen-3d	0.181 ± 0.140	0.134 ± 0.188	–	0.799 ± 0.194	0.243 ± 0.325

Table 2: Average RL performance of personalized policies evaluated for 100 episodes across 5 seeds reported in terms of mean +/- std over rewards for all environments except MO-Kitchen, in which task success rate is used as metric . For this table only, we abbreviate Privileged Traj-MaskedDP to Privileged and Anonymized CLS-MaskedDP to Anonymized consistent with the rest of the paper. Privileged-RL achieves the best performance across most of the environments, consistently demonstrating strong performance relative to the baselines, while Anonymized-RL remains competitive with or surpasses the baselines without access to annotator identity, suggesting that P3L generates policies that are better aligned with individual user preferences. Blue denotes the best result and gray the second-best.

Panel of Figure 3 — Figure 3: (a) Training time in seconds reported as mean and standard deviation across 5 seeds: the lower time it takes, the faster the framework is. (b) Test-time accuracy vs. the number of queries used for adaptation demonstrates that both our frameworks Traj-MaskedDP and CLS-MaskedDP achieve high accuracy around 90% with smaller number of preference queries as with bigger adaptation query set size whereas VPL fails to achieve a comparable performance even when fed 100x adaptation queries per user.

Environment	Eval Metric	Baseline	P3L
		VPL	Privileged-Traj-MaskedDP	Anonymized-CLS-MaskedDP
MO-Highway	Test	42.667 ± 30.346	81.333 ± 9.978	78.000 ± 8.641
MO-Highway	RL	-5.270 ± 12.064	4.866 ± 20.605	7.119 ± 17.783
MO-Reacher	Test	42.333 ± 36.189	$100.00\pm 0.00$	$94.333\pm 9.815$
MO-Reacher	RL	19.598 ± 2.251	$32.552\pm 1.585$	$22.717\pm 5.334$

Table 3: Out-of-distribution adaptation to new users reported as mean ± std over 5 seeds for test preference accuracy and RL performance (100 evaluation rollouts) . P3L generalizes better to unseen users, achieving higher preference accuracy and stronger downstream RL performance than VPL, both with and without access to annotator identity.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

	Baselines		MaskedDP
	SingleRM	VPL	Traj-MaskedDP	CLS-MaskedDP
User 1	68.000 ± 13.597	73.333 ± 22.311	92.000 ± 7.775	85.333 ± 11.470
User 2	77.333 ± 9.97	77.333 ± 9.978	78.667 ± 13.597	77.333 ± 9.043
User 3	65.333 ± 13.597	49.333 ± 26.866	84.000 ± 5.333	73.333 ± 4.216
Average User Accuracy	70.222 ± 3.326	66.667 ± 3.143	84.889 ± 5.145	78.667 ± 4.576

Appendix

Table 4: Accuracy of Held out Test-Set Preference queries on MO-Highway across reported as mean and standard deviation across 5 seeds with 30 adaptive preference queries and 15 held out preference test queries. Both of our approaches, TrajMaskedDP which assumes access to privileged annotator ids (as does VPL), as well as CLSMaskedDP which does not assume any access to annotator identity, outperform VPL, for individual users, as well as on average. Single Reward Model achieves competitive performance for one user as our approaches but on average fails to align with each user.

	Baselines		MaskedDP
	SingleRM	VPL	Traj-MaskedDP	CLS-MaskedDP
User 1	44.000 ± 2.494	27.333 ± 21.848	96.000 ± 6.464	87.333 ± 3.887
User 2	65.333 ± 4.522	71.333 ± 18.086	93.333 ± 2.981	92.667 ± 2.494
User 3	24.000 ± 3.887	52.667 ± 32.069	87.333 ± 6.464	99.333 ± 1.333
User 4	85.333 ± 4.989	56.667 ± 35.777	95.333 ± 3.399	94.667 ± 2.667
Average User Accuracy	54.667 ± 0.667	52.000 ± 3.055	93.000 ± 1.453	93.500 ± 1.434

Appendix

Table 5: Accuracy of Held out Test-Set Preference queries on MO-Reacher across 5 seeds with 60 adaptive preference queries and 30 held out preference test queries. Both of our approaches, Traj-MaskedDP which assumes access to privileged annotator ids (as does VPL), as well as CLS-MaskedDP which does not assume any access to annotator identity, outperform both the baselines Single Reward Model and VPL, for individual users, as well as on average. The blue background indicates the best performance, while the gray background denotes the second-best performance.

	Baselines		MaskedDP
	SingleRM	VPL	Traj-MaskedDP	CLS-MaskedDP
User 1	81.000 ± 5.099	76.600 ± 7.864	90.400 ± 2.939	81.000 ± 2.191
User 2	72.400 ± 5.678	74.800 ± 5.741	85.800 ± 3.059	82.400 ± 0.800
User 3	73.200 ± 5.706	68.800 ± 6.046	88.000 ± 1.789	70.000 ± 2.191
User 4	71.400 ± 4.800	74.600 ± 6.343	89.000 ± 3.033	75.400 ± 2.653
User 5	83.400 ± 7.940	79.800 ± 6.524	88.400 ± 3.499	80.600 ± 2.653

Appendix

Table 6: Accuracy of Held out Test-Set Preference queries on MO-Humanoid across 5 seeds with 250 adaptive preference queries and 100 held out preference test queries. Both of our approaches, Traj-MaskedDP which assumes access to privileged annotator ids (as does VPL), as well as CLS-MaskedDP which does not assume any access to annotator identity, outperform both the baselines Single Reward Model and VPL, for most of individual users, as well as on average . The blue background indicates the best performance, while the gray background denotes the second-best performance.

	Baselines		MaskedDP
	SingleRM	VPL	Traj-MaskedDP	CLS-MaskedDP
User 1	48.800 ± 6.400	63.200 ± 39.387	100.000 ± 0.000	90.400 ± 4.079
User 2	51.200 ± 6.400	37.600 ± 38.831	100.000 ± 0.000	90.400 ± 4.079
Average User Accuracy	50.000 ± 0.000	50.400 ± 0.800	100.000 ± 0.0	90.400 ± 4.079

Appendix

Table 7: Accuracy of Held out Test-Set Preference queries on MO-Kitchen-2d across 5 seeds with 15 adaptive preference queries and 25 held out preference test queries. Both of our approaches, Traj-MaskedDP which assumes access to privileged annotator ids (as does VPL), as well as CLS-MaskedDP which does not assume any access to annotator identity, outperform both the baselines Single Reward Model and VPL, for most of individual users, as well as on average. The blue background indicates the best performance, while the gray background denotes the second-best performance.

	Baselines		MaskedDP
	SingleRM	VPL	Traj-MaskedDP	CLS-MaskedDP
User 1	72.000 ± 12.696	47.111 ± 16.354	84.444 ± 4.869	84.444 ± 3.718
User 2	36.889 ± 12.927	37.778 ± 9.428	82.667 ± 2.948	79.556 ± 7.221
User 3	50.667 ± 6.029	69.333 ± 17.855	85.333 ± 3.875	84.000 ± 3.266
Average User Accuracy	53.185 ± 1.582	51.407 ± 1.525	84.148 ± 2.227	82.667 ± 3.647

Appendix

Table 8: Accuracy of Held out Test-Set Preference queries on MO-Kitchen-3d across 5 seeds with 90 adaptive preference queries and 45 held out preference test queries. Both of our approaches, Traj-MaskedDP which assumes access to privileged annotator ids (as does VPL), as well as CLS-MaskedDP which does not assume any access to annotator identity, outperform both the baselines Single Reward Model and VPL, for most of individual users, as well as on average. The blue background indicates the best performance, while the gray background denotes the second-best performance.

	Baselines		MaskedDP
	SingleRM	VPL	Traj-MaskedDP	CLS-MaskedDP
User 1	65.778 ± 10.242	54.667 ± 17.104	79.111 ± 4.467	75.556 ± 4.332
User 2	51.333 ± 8.025	48.444 ± 10.390	73.778 ± 3.823	69.333 ± 5.287
User 3	42.889 ± 5.287	40.222 ± 9.671	77.556 ± 2.931	67.333 ± 3.414
User 4	61.111 ± 10.352	63.556 ± 4.838	83.333 ± 2.534	76.444 ± 3.326
Average User Accuracy	55.278 ± 2.159	51.722 ± 2.905	78.444 ± 0.572	72.167 ± 2.021

Appendix

Table 9: Accuracy of Held out Test-Set Preference queries on MO-Kitchen-4d across 5 seeds with 180 adaptive preference queries and 90 held out preference test queries. Both of our approaches, Traj-MaskedDP which assumes access to privileged annotator ids (as does VPL), as well as CLS-MaskedDP which does not assume any access to annotator identity, outperform both the baselines Single Reward Model and VPL, for most of individual users, as well as on average. The blue background indicates the best performance, while the gray background denotes the second-best performance.

	Baselines			MaskedDP
	SingleRM-PPO	VPL-OfflineRL-IQL	VPL-OnlineRL-PPO	Traj-MaskedDP-PPO	CLS-MaskedDP-PPO
User 1	3.335 ± 4.459	2.247 ± 3.455	4.828 ± 4.125	5.120 ± 3.651	6.105 ± 3.984
User 2	15.850 ± 12.777	29.150 ± 3.603	26.359 ± 8.281	29.024 ± 3.604	29.736 ± 1.043
User 3	-6.276 ± 9.282	-6.170 ± 10.090	-18.126 ± 10.030	-10.980 ± 10.782	-20.462 ± 7.263
Average RL performance	4.303 ± 9.059	8.409 ± 15.063	4.354 ± 18.164	7.721 ± 16.435	5.126 ± 20.505

Appendix

Table 10: RL performance MO-Highway presented in terms of mean and standard deviation across 5 seeds with 100 rollouts per seed. We achieve competitive performance as compared to baselines on individual users and on average except User 3 which prefers strictly conservative driving.

	Baselines			MaskedDP
	SingleRM-ARS	VPL-OfflineRL-IQL	VPL-OnlineRL-ARS	Traj-MaskedDP-ARS	CLS-MaskedDP-ARS
User 1	13.871 ± 7.598	1.491 ± 7.190	19.721 ± 2.142	37.845 ± 6.244	29.024 ± 7.352
User 2	19.369 ± 8.004	35.682 ± 13.034	21.106 ± 1.262	31.247 ± 4.934	31.216 ± 1.413
User 3	17.436 ± 6.164	12.407 ± 5.575	22.618 ± 0.453	31.031 ± 3.827	26.740 ± 6.324
User 4	16.140 ± 6.729	14.780 ± 12.437	18.262 ± 1.990	26.708 ± 1.444	27.432 ± 4.787
Average	16.704 ± 1.999	16.090 ± 12.372	20.427 ± 1.616	31.708 ± 4.520	28.603 ± 1.996

Appendix

Table 11: RL Performance on MO-Reacher when evaluated over 100 episodes across 5 seeds reported in terms of mean +/- std. Both our frameworks, Traj-MaskedDP and CLS-MaskedDP achieve higher average RL performance than the baselines, demonstrating improved policy alignment.

	Baselines			MaskedDP
	SingleRM-ARS	VPL-OfflineRL-IQL	VPL-OnlineRL-ARS	Traj-MaskedDP-ARS	CLS-MaskedDP-ARS
User 1	28.995 ± 14.759	46.603 ± 49.040	8.323 ± 20.652	44.243 ± 31.841	21.670 ± 12.302
User 2	5.452 ± 30.842	-9.016 ± 27.408	20.551 ± 36.566	74.810 ± 70.151	21.670 ± 12.302
User 3	55.512 ± 21.987	65.261 ± 47.616	63.180 ± 16.044	93.514 ± 57.554	60.124 ± 18.740
User 4	56.822 ± 26.341	30.802 ± 9.216	64.996 ± 44.143	82.556 ± 44.696	85.607 ± 12.131
User 5	-388.945 ± 6.111	-390.156 ± 8.367	-388.251 ± 8.060	-347.668 ± 42.936	-383.115 ± 4.540

Appendix

Table 12: RL Performance on MO-Humanoid when evaluated over 100 episodes across 5 seeds reported in terms of mean +/- std. Our framework outperforms all baselines across majority of users and on average too.

	Baselines		MaskedDP
	SingleRM-IQL	VPL-OfflineRL-IQL	Traj-MaskedDP-IQL	CLS-MaskedDP-IQL
User 1 (Microwave)	0. 548 ± 0.498	0.326 ± 0.469	0.680 ± 0.466	0.516 ± 0.500
User 2 (Kettle)	0.00 ± 0.000	0.400 ± 0.490	0.988 ± 0.109	0.152 ± 0.359
Average	0.274 ± 0.274	0.363 ± 0.037	0.834 ± 0.154	0.334 ± 0.182

Appendix

Table 13: RL Performance on MO-Kitchen-2d when evaluated over 100 episodes across 5 seeds reported in terms of mean +/- std for Task Success Rate. Since Franka-Kitchen is an offline environment, we run only offline RL IQL.

	Baselines		MaskedDP
	SingleRM-IQL	VPL-OfflineRL-IQL	Traj-MaskedDP-IQL	CLS-MaskedDP-IQL
User 1 (Microwave)	0.342 ± 0.474	0.400 ± 0.490	0.528 ± 0.499	0.702 ± 0.457
User 2 (Kettle)	0.000 ± 0.000	0.002 ± 0.045	0.900 ± 0.300	0.028 ± 0.165
User 3 (Light Switch)	0.200 ± 0.400	0.000 ± 0.000	0.970 ± 0.171	0.00 ± 0.000
Average	0.181 ± 0.140	0.134 ± 0.188	0.799 ± 0.194	0.243 ± 0.325

Appendix

Table 14: RL Performance on MO-Kitchen-3d when evaluated over 100 episodes across 5 seeds reported in terms of mean +/- std for Task Success Rate. Since Franka-Kitchen is an offline environment, we run only offline RL IQL.

	Baselines		MaskedDP
	SingleRM-IQL	VPL-OfflineRL-IQL	Traj-MaskedDP-IQL	CLS-MaskedDP-IQL
User 1 (Microwave)	0.000 ± 0.000	0.414 ± 0.493	0.526 ± 0.499	0.178 ± 0.383
User 2 (Kettle)	0.000 ± 0.000	0.000 ± 0.000	0.738 ± 0.440	0.004 ± 0.063
User 3 (Light Switch)	0.000 ± 0.000	0.000 ± 0.000	0.998 ± 0.045	0.002 ± 0.045
User 4 (Top Burner)	0.000 ± 0.000	0.000 ± 0.000	0.384 ± 0.486	0.000 ± 0.000
Average	0.000 ± 0.000	0.103 ± 0.179	0.661 ± 0.232	0.046 ± 0.076

Appendix

Table 15: RL Performance on MO-Kitchen-4d when evaluated over 100 episodes across 5 seeds reported in terms of mean +/- std for Task Success Rate. Since Franka-Kitchen is an offline environment, we run only offline RL IQL.

Figure 4: Per User Test Accuracy vs Number of PCA basis eigenvectors calculated per user across 10 seeds for each PCA size with standard deviation plotted as error bars.

Setup			Baselines	MaskedDP
OOD New User	Trained on Users	Eval Metric	VPL	Traj-MaskedDP	CLS-MaskedDP
1	2-3-4	Test	36.000±3.266	100 ± 0.00	100 ± 0.00
1	2-3-4	RL	22.048 ± 1.975	32.841 ± 3.033	29.787 ± 3.328
2	1-3-4	Test	33.333±0.000	100 ±0.00	100 ±0.00
2	1-3-4	RL	17.255 ± 2.414	31.249 ± 1.178	21.401 ± 4.460
3	1-2-4	Test	0.000±0.000	100.00±0.00	77.333±3.266

Appendix

Table 16: Out-of-distribution adaptation to new users on MO-Reacher . Results report mean ± std over 5 seeds for test preference accuracy (15 queries) and RL performance (100 evaluation rollouts). Our methods generalize better to unseen users, achieving higher preference accuracy and stronger downstream RL performance than VPL.

Setup			Baselines	MaskedDP
OOD New User	Trained on Users	Eval Metric	VPL	Traj-MaskedDP	CLS-MaskedDP
1	2-3	Test	0.000±0.000	68.000±7.483	74.000±21.541
1	2-3	RL	5.603 ± 3.621	5.389 ± 3.788	5.109 ± 3.490
2	1-3	Test	68.000±9.798	92.000±7.483	90.000±10.954
2	1-3	RL	0.680 ± 2.125	29.836 ± 0.235	29.833 ± 0.242
3	1-2	Test	60.000±16.733	84.000±10.198	70.000±23.664

Appendix

Table 17: Out-of-distribution adaptation to new users on MO-Highway . Results report mean ± std over 5 seeds for test preference accuracy (10 queries) and RL performance (100 evaluation rollouts). Our methods generalize better to unseen users, achieving higher preference accuracy and stronger downstream RL performance than VPL.

Figure 5: Average Test Accuracy vs Number of PCA basis eigenvectors calculated across all users over 10 seeds for each PCA size with standard deviation plotted as error bars.

Figure 6: Per-User Test Time Accuracy vs Number of adaptation queries where each user favors a distinct goal in the environment.

Pluralistic Alignment

Pluralistic alignment develops AI systems that represent and navigate diverse human values rather than optimize for a single preference.

See all 37 papers

Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment
Oct 13, 2026Ruomeng Ding, Tianwei Gao, Lianrui Geng +2Social Welfare Maximization Preference Alignment
University of North Carolina at Chapel Hill
Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences
Jul 14, 2026Taehyung Kim, Gwangmo Lee, Minjun Chang +2Clustering Preference Alignment
Yonsei University, Seoul, South Korea · Georgia Institute of Technology, United States of America · Hyundai Motor Company, Seoul, South Korea
Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracy
May 2, 2026Rachel FreedmanReward Modeling LLM Alignment
Department of Electrical Engineering and Computer Science, University of California, Berkeley, USA.

Oct 13, 2026J/K move · Enter open · S save

Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment

Ruomeng Ding, Tianwei Gao, Lianrui Geng, Mohit Bansal+1

University of North Carolina at Chapel Hill

Social Welfare Maximization Preference Alignment Pluralistic Alignment Preference Elicitation

Open paper AskPDF arXiv

Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.

Abstract

Pluralistic Alignment

Explore similar work

Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment

Deployable Human Preference Alignment in Robotics: Learning Representative Rewards from Diverse Human Preferences

Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracy