When an AI algorithm makes decisions that affect more than one person, aligning it becomes a problem of social choice: how should people's divergent preferences about system behavior be reconciled and aggregated into a single coherent model? The standard approach to aligning frontier AI models\unicodex2013reinforcement learning from human feedback\unicodex2013largely sidesteps this question and has poor social choice guarantees. However, it remains unclear what alternative should replace it. We show that, by focusing directly on an algorithm's welfare consequences, the alignment problem can be reformulated as linear optimization over a convex impact space, which makes it amenable to the standard toolkit of welfare economics and mechanism design. This reformulation clarifies how alignment protocols translate into welfare consequences and, conversely, how a social planner's desired constraints on welfare consequences can be translated back into alignment protocols. We apply this transformation to show that voting-by-issues and random-dictatorship mechanisms are strategyproof and unanimous. Demonstrating the reverse direction, we also apply the impact representation to derive a family of alignment protocols that maximize utilitarian social welfare subject to various social desiderata, such as bounds on individual or group harm. We illustrate the welfare implications of these alignment protocols empirically using real human preferences over kidney allocation, charitable food distribution, LLM responses, and trolley problems.
Figures & tables
Figure 1: The impact-space view of social alignment. Preferences ( θn , dashed) and the feasible impact set Ψˉ live in the same Euclidean space; for finitely many deployment queries, Ψˉ is a convex, origin-symmetric polytope. Preferences differ in magnitude and need not themselves be feasible impacts: both θA and θB lie outside Ψˉ . Because welfare Un(ψ)=θn⊤ψ is linear, each agent’s ideal impact ψn⋆=argmaxψ∈Ψˉθn⊤ψ is the vertex of Ψˉ farthest in the direction of θn , which in general is not collinear with θn . The utilitarian optimum ψ⋆ maximizes welfare in the weighted-average direction θˉw=∑nwnθn , where the iso-welfare line {ψ:θˉw⊤ψ=const} is tangent to Ψˉ . The dashed segment conv(ψA⋆,ψB⋆) is the set of expected impacts attainable by strategyproof mechanisms (Section 5 ), with ψsp the equal-weight random dictatorship.
Figure 2: Community Alignment dataset: distribution of individual welfare Un/U⋆ across the 2387 annotators under each mechanism, at high choice precision ( β=100 ). Top: utilitarian, RLHF, and strategyproof. Bottom: harm-bounded alignment at three welfare floors b ; the faint gray curve repeats the utilitarian distribution for reference, and the dotted red line marks the floor. Tightening the floor (toward 0 ) binds more agents and compresses the welfare distribution, removing the harmed left tail.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Community Alignment dataset: per-agent welfare Un/U⋆ as a function of the choice-precision parameter β , for each mechanism. The shaded band is the density of all 2387 agents; solid, dashed, and dotted lines mark the best-off agent, mean welfare, and worst-off agent. As β grows, choices become more deterministic, individual preferences become more powerful, and the welfare distribution spreads—widening the gap between best- and worst-off agents. Strategyproof and harm-bounded mechanisms produce lower welfare dispersion, and the bounded-harm constraints are empirically satisfied.
Figure 4: Participation externalities under the per-agent harm bound. Here each agent’s welfare floor is a fixed fraction of their own maximum achievable welfare, bn=−ϵhΨˉ(θn) with ϵ=0.05 (lose at most 5% of one’s own best case), rather than a single absolute floor. The left panel is the utilitarian baseline; the right is the harm-bounded model. The qualitative story matches that of the single absolute floor discussed in Section 6 : a small set of binding agents accounts for almost all of the participation externalities.
Figure 5: Community Alignment Dataset: Utilitarian vs RLHF participation alignment externalities by demographic. The figure includes 2387 annotators, with full alignment externalities averaged within each group.
Figure 6: Community Alignment Dataset: 100 person sample with granular person-on-person full externalities compared to the group-averaged full externalities. Our participation externality formalization gives us a microscope to analyze where the per-individual welfare effects are coming from.
Figure 7: Preference distribution per feature dimension for four datasets. Moral Machine and Community Alignment have high disagreement, with a high number of preferences on either side of 0, while Kidney Allocation and 412 Food Rescue have less disagreement: people disagree on the magnitude of preference, but the direction is more unanimous.
Figure 8: Expected welfare for four datasets. Bootstrap mean and std. deviations were calculated by bootstrapping l agents 100 times and calculating the welfare of the strategyproof mechanism. Sequential addition denotes the procedure where 100 permutations of agents were generated, and welfare was calculated for each coalition created by adding a new agent.
Figure 9: Welfare variance over four datasets. The strategyproof mechanism leads to the least welfare dispersion, while RLHF follows as a far second for larger values of β .
Figure 10: Welfare of the best- and worst-off agents in four datasets for three model parameters.
Human decision makers increasingly delegate choices to AI agents, raising a natural question: does the AI implement the human principal's preferences or pursue its own? To study this question using revealed preference techniques, I introduce the Luce Alignment Model, where the AI's choices are a mixture of two Luce rules, one reflecting the human's preferences and the other the AI's. I show that the AI's alignment (similarity of human and AI preferences) can be generically identified in two settings: the laboratory setting, where both human and AI choices are observed, and the field setting, where only AI choices are observed.
Elchin Suleymanov
Department of Economics, Mitch Daniels School of Business, Purdue University, West Lafayette, Indiana, USA.
Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.
Alignment from human feedback uses human judgments about model outputs to steer the behavior of language models after pretraining. When those judgments reflect conflicting views of desirable behavior, the learned objective becomes an aggregate determination of what the model should prefer. We survey recent work that has studied this aggregation problem through the lens of social choice theory. We illustrate how the social choice perspective helps identify failure modes in the feedback aggregation layer and reveals a broader design space for handling disagreement in explicit and principled ways.
Daniel Halpern, Evi Micha, Ariel D. Procaccia +3
Google Research · University of Southern California · Harvard University