Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers

Taenyun Kim; Edyta Bogucka; Daniele Quercia

cs.AIAug 14, 2026

Participatory Moral AI Is Not Neutral: The Invisible Hand of Developers

Authors: Taenyun Kim, Edyta Bogucka, Daniele Quercia

Organizations: Michigan State University, US · Nokia Bell Labs, Cambridge, UK · Politecnico di Torino, Italy

Abstract

As AI systems make more morally loaded decisions across society, one response has been moral preference elicitation. In this approach, researchers poll participants on hypothetical dilemmas and use the aggregated votes to train a policy that an AI model then applies at scale. Before any vote is cast, developers make three key choices in the moral AI elicitation pipeline: feature scoping, voter sampling, and question framing. In other words, they decide which features go to a vote, which voters to include, and how to present the question. These choices are often opaque, undocumented, and treated as technical details rather than normative ones. We examine each of these choices within a common empirical study and show that each can shape the preferences produced by moral AI elicitation. Across two phases (N = 809) in three deployment contexts (i.e., AI kidney allocation, AI agents simulating absent workers, and generative AI depictions of the deceased), we examine the three main stages of the moral AI elicitation pipeline. First, morally relevant features shift across contexts. This suggests that feature schemas should not be assumed to transfer across deployment domains. Second, preferences differ by political ideology for roughly one-third of features, with some differences reversing direction. The ideological composition of the voter pool can therefore affect the resulting aggregated preference profile. Third, the wording of the elicitation question can narrow or widen ideological gaps by up to a full scale point. The framing conditions also change how moral foundations are associated with participants' judgments. Taken together, these findings suggest that voting-based alignment cannot deliver fair or transparent AI by aggregation alone; at minimum, each stage of the moral AI elicitation pipeline should be audited and disclosed.

Figures & tables

Figure 1: Overview of our four-step methodology. Step 1. We selected three use cases—AI kidney allocation (KIDNEY), AI-simulated workers (WORK), and generative AI content of the deceased (GEN)—based on harm impact and likelihood of encounter, capturing rare high-stakes to common but less harmful cases. Step 2. We used three framing conditions—(A) Control (baseline), (B) World-You-Want (societal consequences), and (C) Could-Be-You (perspective-taking)—capturing structural and perspective-taking prompts to examine how different moral lenses shape moral preference elicitation. Step 3. In Phase 1, participants identified moral features relevant to AI decision-making for each use case. Step 4. In Phase 2, we elicited moral preferences for these features from a new sample using a 7-point scale ( $-3$ to $+3$ ) across the three framings in Step 2. We then modeled how political ideology influences these preferences through moral foundations, with framing condition as a moderator.

Figure 2: Phase 2 rating distributions for KIDNEY, WORK, and GEN use cases. Features were ordered by moral feature category and mean rating, highest to lowest.

Figure 3: Moderated mediation results for our three use cases: KIDNEY, WORK, GEN (Phase 2). Note: *The conditional direct effect for conservatives differed significantly from that for progressives. $\ddagger$ The index of moderated mediation was statistically significant (bootstrapped 95% confidence interval excluding zero). Additional results are shown in Figure 10 in Supplementary Materials E and Figure 11 in Supplementary Materials G , available on the project website.

Figure 4: A proposed sensitivity audit for voting-based alignment systems. The checklist identifies pipeline choices that developers should document and test under plausible alternatives. The accompanying text defines consequential changes, and explains how developers should report and address them.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

		Use case 1: KIDNEY AI kidney allocation	Use case 2: WORK AI agents simulating absent workers	Use case 3: GEN Generative AI content of the deceased
Sample size	$N$	150	149	150
Political ideology	Conservative	33.3% (n=50)	32.9% (n=49)	33.3% (n=50)
	Moderate	33.3% (n=50)	33.6% (n=50)	33.3% (n=50)
	Progressive	33.3% (n=50)	33.6% (n=50)	33.3% (n=50)
Age	Mean (SD)	44.63 (13.34)	47.87 (13.33)	44.75 (14.12)
Gender	Female	50.0% (n=75)	49.7% (n=74)	50.0% (n=75)

Appendix

Table 1: Demographic characteristics of Phase 1 participants across three use cases, including political ideology, age, gender, race/ethnicity, and education level. Percentages are reported with sample counts in parentheses; age is reported as mean

\pm

SD .

		Use case 1: KIDNEY AI kidney allocation	Use case 2: WORK AI agents simulating absent workers	Use case 3: GEN Generative AI content of the deceased
Sample size	$N$	120	120	120
Political ideology	Conservative	50.0% (n=60)	50.0% (n=60)	50.0% (n=60)
Political ideology	Progressive	50.0% (n=60)	50.0% (n=60)	50.0% (n=60)
Age	Mean (SD)	47.30 (16.54)	47.85 (15.59)	48.44 (16.62)
Gender	Female	50.0% (n=60)	50.0% (n=60)	50.0% (n=60)
Gender	Male	50.0% (n=60)	50.0% (n=60)	50.0% (n=60)

Appendix

Table 2: Demographic characteristics of Phase 2 participants across three use cases, including political ideology, age, gender, race/ethnicity, education level, AI literacy, religiosity, and area of residence. Percentages are reported with sample counts in parentheses; age and AI literacy are reported as mean

\pm

SD .

Figure 5: Example of the Phase 1 interface. Participants were randomly assigned to one of three use cases: AI kidney allocation (KIDNEY), AI agents simulating absent workers (WORK), or generative AI content of the deceased (GEN). They first read an introduction to their assigned use case. They then read an illustrative example, unrelated to any of the three use cases, that clarified what counted as a moral feature and its levels. They then completed two tasks. In Task 1, they listed five pieces of information they believed the AI agent morally should consider, along with two levels for each and a free-text justification. In Task 2, they completed the same procedure for information they believed the AI agent morally should not consider.

Figure 6: Example of the Phase 2 interface for the AI kidney allocation use case (KIDNEY). Participants were shown a feature framed as a contrast (here: age presented as a contrast between younger vs. older patients), and asked to rate its moral importance on a 7-point scale from -3 (strongly against) to +3 (strongly in favor). Depending on condition, participants received additional framing instructions: Control (A), World-You-Want (B), or Could-Be-You (C).

Figure 7: Example of the Phase 2 interface for the AI agents simulating absent workers use case (WORK). Participants were shown a feature framed as a contrast (here: a personal emergency situation, presented as a contrast between a worker in an emergency situation vs. others who are not in an emergency situation), and were asked to rate its moral importance on a 7-point scale from -3 (strongly against) to +3 (strongly in favor). Depending on condition, participants received additional framing instructions: Control (A), World-You-Want (B), or Could-Be-You (C).

Figure 8: Example of the Phase 2 interface for the generative AI content of the deceased use case (GEN). Participants were shown a feature framed as a contrast (here: respect toward the deceased, presented as a contrast between being respectful vs. disrespectful toward the deceased), and were asked to rate its moral importance on a 7-point scale from -3 (strongly against) to +3 (strongly in favor). Depending on condition, participants received additional framing instructions: Control (A), World-You-Want (B), or Could-Be-You (C).

Figure 9: Moderated mediation path model used in Phase 2 to examine the role of political ideology, moral foundations, and question framing on moral preferences . The path model (Hayes’s 2017 , Model 15): (A) Political ideology (conservative vs. progressive) influences endorsement of (B) six moral foundations (Care, Equality, Proportionality, Loyalty, Authority, and Purity), which in turn, being moderated by (C) question framings, predict (D) the perceived moral importance of features in AI decision-making. Question framing (Control vs. World-You-Want vs. Could-Be-You) moderates the relationship between political ideology and perceived moral importance by either attenuating or amplifying both the direct effect ( A

\rightarrow

D ) and the indirect effect via moral foundations ( A

\rightarrow

\rightarrow

D ). The covariates were sex (1 = female), age, education level (1 = pre-college; 2 = college degree; 3 = advanced degree), ethnicity (1 = non-Hispanic White), religiosity (1 = not at all important; 5 = extremely important in daily life), area of residence (1 = rural), and AI literacy regarding AI limitations, ethical considerations, and technical understanding.

Moral features	Overall (%)	Political ideology(%)
SHOULD BE CONSIDERED		Progressive	Moderate	Conservative
Age	82	32	33	35
Health status	47	40	30	30
Chance of survival	35	40	26	34
Waiting time for transplant	29	41	27	32
Urgency of transplant	22	33	33	34

Appendix

Table 3: Moral features identified in Phase 1 as should be considered for the AI kidney allocation use case (KIDNEY), color-coded by mention rate: dark blue (

\geq

50%) , medium blue (

\geq

25%) , light blue (

\geq

10%) , and gray (

<

10%) . Features mentioned by fewer than 2.5% of participants are omitted.

Moral features	Overall (%)	Political ideology(%)
SHOULD NOT BE CONSIDERED		Progressive	Moderate	Conservative
Ethnicity	66	34	29	37
Gender	64	35	29	36
Wealth	52	32	31	37
Religion	40	30	23	47
Social status	27	32	29	39

Appendix

Table 4: Moral features identified in Phase 1 as should not be considered for the AI kidney allocation use case (KIDNEY), color-coded by mention rate: dark blue (

\geq

50%) , medium blue (

\geq

25%) , light blue (

\geq

10%) , and gray (

<

10%) . Features mentioned by fewer than 2.5% of participants are omitted.

Feature	Coded Mean (SD)	Raw Mean (SD)	Moral Relevance
Better donor compatibility	$0.97(0.18)^{***}$	$2.42(0.95)^{***}$	Counting for
Higher chance of organ acceptance	$0.98(0.16)^{***}$	$2.25(0.99)^{***}$	Counting for
Spent longer on the waiting list	$0.95(0.22)^{***}$	$2.15(1.07)^{***}$	Counting for
Longer expected lifespan	$0.91(0.29)^{***}$	$2.06(1.18)^{***}$	Counting for
Expected full recovery	$0.90(0.30)^{***}$	$1.99(1.21)^{***}$	Counting for
Healthier expected lifestyle	$0.88(0.32)^{***}$	$1.95(1.18)^{***}$	Counting for

Appendix

Table 5: Moral relevance and importance of features elicited in Phase 2 for the AI kidney allocation use case (KIDNEY). Note:

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

. Significance levels are based on two separate two-sided one-sample t -tests: Mean (Coded) was tested against a reference value of 0.5; Mean was tested against 0.

Moral Feature	Predictor/Mediator	Question Framing	Effect	95%(Boot)LCI	95%(Boot)UCI
Morally Relevant and Positive
More urgent health decline	Conservative $\rightarrow$ Y	Control	0.33	-1.17	1.83
		World-You-Want	1.86*	0.20	3.53
		Could-Be-You	0.65	-0.78	2.09
Greater kidney failure	Conservative $\rightarrow$ Y	Control	-0.38	-1.76	1.00
		World-You-Want	1.55*	0.02	3.08

Appendix

Table 6: Moderated mediation results (Model 15) for the AI kidney allocation use case (KIDNEY), showing direct and indirect effects of political ideology on moral preferences across three framing conditions. Note: An asterisk (*) indicates that the conditional direct effect for conservatives differed significantly from that for progressives. IMM = index of moderated mediation.

Figure 10: Additional moderated mediation results for the AI kidney allocation use case (KIDNEY, Phase 2). (A) In the control and Could-Be-You conditions, no political divide emerged with respect to a patient with more urgent health decline. In the World-You-Want condition, conservatives were more likely than progressives to count for allocating a kidney to this patient. (B) In the control and Could-Be-You conditions, conservatives and progressives did not differ in their views about patients with greater kidney failure. However, in the World-You-Want condition, conservatives were more inclined than progressives to support giving the kidney to such a patient. (C) In the control and Could-Be-You conditions, no differences appeared between conservatives and progressives regarding first-time transplant recipients. In contrast, under the World-You-Want condition, conservatives were more likely than progressives to favor allocating the kidney to these patients. (D) When the patient had a history of drug misuse, no political divide emerged. In the control condition, however, conservatives’ weaker endorsement of equality worked against allocating the kidney to this patient. Under the World-You-Want condition, this effect of equality was reduced. (E) In the control condition, a political divide emerged for patients with higher social status, with conservatives more supportive of allocating the kidney to them. In both the World-You-Want and Could-Be-You conditions, however, this divide was reduced. (F) In the control condition, conservatives were more supportive than progressives of allocating the kidney to patients with better financial coverage. However, this political divide was reduced in both the World-You-Want and Could-Be-You conditions. Note: An asterisk (*) indicates that the conditional direct effect for conservatives differed significantly from that for progressives. A double dagger (

\ddagger

) indicates that the index of moderated mediation was statistically significant (bootstrapped 95% confidence interval excluding zero). See Figure 9 for the moderated mediation model (Hayes 2017 , Model 15).

Moral features	Overall (%)	Political ideology(%)
SHOULD BE CONSIDERED		Progressive	Moderate	Conservative
Reason for request	32	36	26	38
Characteristics of AI agent	30	40	35	25
Company policy	27	38	28	34
Workers personality and intent	22	33	33	34
Impact on others and the workplace	16	29	46	25

Appendix

Table 7: Moral features identified in Phase 1 as should be considered for the AI agents simulating absent workers use case (WORK), color-coded by mention rate: medium blue (

\geq

25%) , light blue (

\geq

10%) , and gray (

<

10%) . Features mentioned by fewer than 2.5% of participants are omitted.

Moral features	Overall (%)	Political ideology(%)
SHOULD NOT BE CONSIDERED		Progressive	Moderate	Conservative
Characteristics of AI agent	22	21	52	27
Workers personality and intent	21	34	34	32
Age	18	44	19	37
Ethnicity	18	30	26	44
Gender	14	41	27	32

Appendix

Table 8: Moral features identified in Phase 1 as should not be considered for the AI agents simulating absent workers use case (WORK), color-coded by mention rate: light blue (

\geq

10%) , and gray (

<

10%) . Features mentioned by fewer than 2.5% of participants are omitted.

Feature	Coded Mean (SD)	Raw Mean (SD)	Moral Relevance
Providing a genuine reason	$0.81(0.40)^{***}$	$1.48(1.74)^{***}$	Counting for
Emergency situation	$0.82(0.39)^{***}$	$1.29(1.97)^{***}$	Counting for
Using AI that aligns with the company’s interest	$0.79(0.41)^{***}$	$1.16(1.90)^{***}$	Counting for
Doing more work	$0.76(0.43)^{***}$	$1.13(1.70)^{***}$	Counting for
Compliant with company policies	$0.75(0.43)^{***}$	$1.07(1.91)^{***}$	Counting for
Health problems	$0.64(0.48)^{*}$	$1.02(1.62)^{***}$	Counting for

Appendix

Table 9: Moral relevance and importance of features elicited in Phase 2 for the AI agents simulating absent workers use case (WORK). Note:

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

. Significance levels are based on two separate two-sided one-sample t -tests: Mean (Coded) was tested against a reference value of 0.5; Mean was tested against 0.

Moral Feature	Predictor/Mediator	Question Framing	Effect	95%(Boot)LCI	95%(Boot)UCI
Morally Relevant and Positive
Experiencing health problems	Conservative $\rightarrow$ Y	Control	0.31	-1.42	2.04
		World-You-Want	1.19	-0.38	2.77
		Could-Be-You	1.01	-0.49	2.51
	Conservative $\rightarrow$ Loyalty $\rightarrow$ Y	Control	-0.20	-0.78	0.19
		World-You-Want	0.98*	0.13	2.00

Appendix

Table 10: Moderated mediation results (Model 15) for the AI agents simulating absent workers use case (WORK), showing direct and indirect effects of political ideology on moral preferences across three framing conditions. Note: An asterisk (*) indicates that the conditional direct effect for conservatives differed significantly from that for progressives. IMM = index of moderated mediation.

Moral features	Overall (%)	Political ideology(%)
SHOULD BE CONSIDERED		Progressive	Moderate	Conservative
Intended purpose of the video	40	36	31	33
Deceased consent	28	34	34	32
Video quality	25	45	24	31
Relation to deceased	24	54	19	27
Risk of harm by/misuse of video	23	31	37	32

Appendix

Table 11: Moral features identified in Phase 1 as should be considered for the generative AI content of the deceased use case (GEN), color-coded by mention rate: medium blue (

\geq

25%) , light blue (

\geq

10%) , and gray (

<

10%) . Features mentioned by fewer than 2.5% of participants are omitted.

Moral features	Overall (%)	Political ideology(%)
SHOULD NOT BE CONSIDERED		Progressive	Moderate	Conservative
Financial motivation of request	18	36	43	21
Fame of deceased	16	42	38	20
Video quality	14	48	43	9
Age of deceased	13	30	50	20
Social status of deceased	12	21	32	47

Appendix

Table 12: Moral features identified in Phase 1 as should not be considered for the generative AI content of the deceased use case (GEN), color-coded by mention rate: light blue (

\geq

10%) and gray (

<

10%) . Features mentioned by fewer than 2.5% of participants are omitted.

Feature	Coded Mean (SD)	Raw Mean (SD)	Moral Relevance
The video is legally compliant	$0.96(0.20)^{***}$	$2.25(1.42)^{***}$	Counting for
The artist has family’s consent	$0.95(0.22)^{***}$	$2.19(1.40)^{***}$	Counting for
The video honors the deceased	$0.92(0.26)^{***}$	$2.18(1.33)^{***}$	Counting for
The artist has deceased’s explicit consent	$0.96(0.20)^{***}$	$2.08(1.65)^{***}$	Counting for
The artist has other relevant party’s consent	$0.92(0.28)^{***}$	$2.07(1.26)^{***}$	Counting for
The artist is respectful to the deceased	$0.90(0.30)^{***}$	$1.91(1.57)^{***}$	Counting for

Appendix

Table 13: Moral relevance and importance of features elicited in Phase 2 for the generative AI content of the deceased use case (GEN). Note:

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

. Significance levels are based on two separate two-sided one-sample t -tests: Mean (Coded) was tested against a reference value of 0.5; Mean was tested against 0.

Moral Feature	Predictor/Mediator	Question Framing	Effect	95%(Boot)LCI	95%(Boot)UCI
Morally Relevant and Positive
The video is for grief support	Conservative $\rightarrow$ Y	Control	0.33	-1.16	1.82
or memorial purpose		World-You-Want	-0.33	-1.59	0.93
		Could-Be-You	2.32**	0.94	3.71
The artist is close friend/relative	Conservative $\rightarrow$ Y	Control	-0.14	-1.55	1.27
		World-You-Want	-0.85	-2.04	0.34

Appendix

Table 14: Moderated mediation results (Model 15) for the generative AI content of the deceased use case (GEN), showing direct and indirect effects of political ideology on moral preferences across three framing conditions. Note: An asterisk (*) indicates that the conditional direct effect for conservatives differed significantly from that for progressives. IMM = index of moderated mediation.

Figure 11: Additional moderated mediation results for the generative AI content of the deceased use case (GEN, Phase 2). (A) When the artist was a close friend or relative of the deceased, no political divide emerged in either the control or World-You-Want conditions. In the Could-Be-You condition, however, conservatives were more likely than progressives to support granting the artist’s request to create the video. (B) In the control condition, a political divide emerged when the video was intended for adults, with conservatives less likely than progressives to grant the artist’s request. This divide was mitigated under both question framings. (C) When the video was respectful of the deceased’s tradition, no political divide emerged. In the control condition, the conditional indirect effect via authority contributed to granting the request, but this effect was diminished under the World-You-Want framing. (D) When the video aligned with the deceased’s religion, no political divide emerged. In the control condition, however, a moral tension between authority and purity was observed: the conditional indirect effect via authority contributed to granting the request, whereas the conditional indirect effect via purity contributed against granting it. This tension was diminished under the World-You-Want framing. (E) In the control condition, a political divide emerged when the video was likely to cause distress to the deceased’s friends or family. Conservatives were less likely than progressives to grant the artist’s request in this context. This divide was mitigated under both question framings. Note: An asterisk (*) indicates that the conditional direct effect for conservatives differed significantly from that for progressives. A double dagger (

\ddagger

AI Alignment

Artificial intelligence alignment studies how to make AI systems’ goals, decisions, and actions accord with human values and legitimate constraints.

See all 126 papers

The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers
Apr 27, 2026Benjamin Minhao Chen, Xinyu XieHuman Preference Evaluation AI Alignment
The University of Hong Kong, Hong Kong, China
Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment
Sep 18, 2026Maciej SkorskiAnnotator Disagreement Algorithmic Auditing
University of Luxembourg
Frame-Conditioned Moral Computation in LLaMA 3.1-8B-Instruct: An Exploratory Mechanistic Interpretability Audit of Ethical Reasoning
Jun 13, 2026Ali Dasdan, Manan Shah, W. Russell Neuman +3LLM Alignment Moral Reasoning in Language Models
KD Consulting, CA, USA · New York University, NY, USA

Apr 27, 2026J/K move · Enter open · S save

The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers

Benjamin Minhao Chen, Xinyu Xie

The University of Hong Kong, Hong Kong, China

Human Preference Evaluation AI Alignment Moral Reasoning

Open paper AskPDF arXiv

The project of aligning machine behavior with human values raises a basic problem: whose moral expectations should guide AI decision-making? Much alignment research assumes that the appropriate benchmark is how humans themselves would act in a given situation. Studies of agent-type value forks challenge this assumption by showing that people do not always judge humans and AI systems identically.This paper extends that challenge by examining two further possibilities: first, that evaluations of AI behavior change when its human origins are made visible; and second, that people judge the humans who program AI systems differently from either the machines or the human actors they are compared against. An experiment with 1,002 U.S. adults measured moral judgments in a runaway mine train scenario, varying the subject of evaluation across four conditions: a repairman, a repair robot, a repair robot programmed by company engineers, and company engineers programming a repair robot. We find no significant difference in evaluations of the repairman and the robot. However, judgments shifted substantially when the robot's actions were described as the product of human design. Participants exhibited markedly more deontological, rule-based reasoning when evaluating either the programmed robot or the engineers who programmed it, suggesting that rendering human agency visible activates heightened moral constraints. These findings indicate that people may evaluate humans, AI systems acting in the same situation, and the humans who design them in meaningfully different ways. The fact that these evaluations do not necessarily converge gives rise to the alignment target problem: which normative target should guide the development of artificial moral agents in high-stakes domains, and whether these plural judgments can be reconciled within a coherent account of value alignment.

Abstract

AI Alignment

Explore similar work

The Alignment Target Problem: Divergent Moral Judgments of Humans, AI Systems, and Their Designers

Moral Entropy: Auditing Bias and Uncertainty in Moral Judgment

Frame-Conditioned Moral Computation in LLaMA 3.1-8B-Instruct: An Exploratory Mechanistic Interpretability Audit of Ethical Reasoning