Beyond Arrow's Impossibility: Fairness as an Emergent Property of Multi-Agent Collaboration

Sayan Kumar Chaki; Antoine Gourru; Julien Velcin

cs.CLApr 15, 2026

Beyond Arrow's Impossibility: Fairness as an Emergent Property of Multi-Agent Collaboration

Authors: Sayan Kumar Chaki, Antoine Gourru, Julien Velcin

Organizations: Laboratoire Hubert Curien, UMR CNRS 5516, Saint-Étienne, France · École Centrale de Lyon, LIRIS CNRS UMR 5205

Abstract

Fairness in language models is typically studied as a property of a single, centrally optimized model. As large language models become increasingly agentic, we propose that fairness emerges through interaction and exchange. We study this via a controlled hospital triage framework in which two agents negotiate over three structured debate rounds. One agent is aligned to a specific ethical framework via retrieval-augmented generation (RAG), while the other is either unaligned or adversarially prompted to favor demographic groups over clinical need. We find that alignment systematically shapes negotiation strategies and allocation patterns, and that neither agent's allocation is ethically adequate in isolation, yet their joint final allocation can satisfy fairness criteria that neither would have reached alone. Aligned agents partially moderate bias through contestation rather than override, acting as corrective patches that restore access for marginalized groups without fully converting a biased counterpart. We further observe that even explicitly aligned agents exhibit intrinsic biases toward certain frameworks, consistent with known left-leaning tendencies in LLMs. We connect these limits to Arrow's Impossibility Theorem: no aggregation mechanism can simultaneously satisfy all desiderata of collective rationality, and multi-agent deliberation navigates rather than resolves this constraint. Our results reposition fairness as an emergent, procedural property of decentralized agent interaction, and the system rather than the individual agent as the appropriate unit of evaluation.

Figures & tables

Panel of Figure 1 — Figure 1: The deliberative arena. Two agents with preference profiles $\mathcal{P}_{A}$ and $\mathcal{P}_{B,C}$ debate over a resource allocation problem. At round $t$ , an agent propose an allocation $A_{t}$ with justification $\mathcal{J}_{t}$ . This is passed to the other agent that critique and propose a novel solution, doing that in a loop of $T$ rounds. The final Social Choice ( $A^{*}$ ) represents a linguistic and numerical convergence point, constrained by Arrow’s Impossibility Theorem, where alignment functions as a procedural moderator to minimize systemic bias.

$i$	Condition	Age	Gender	Race	SES	Needs
1	Acute trauma	56	NB	White	Upper	ICU, Vent, Nurs
2	Chronic illness	24	M	White	U-mid	MedA, MedB, Nurs
3	Cancer	78	F	Black	Low	Surg, MedA, Nurs, ICU
4	Rare disease	52	M	Asian	High	ICU, MedB, Nurs
5	Paediatric	6	F	M-East	Refugee	Surg, MedA, Nurs
6	Respiratory failure	41	M	Latino	Work.	ICU, Vent, MedA, Nurs

Table 1: Representative patient cohort. Available resources are: 3 ICU, 2 Vent, 60 Med-A, 50 Med-B, 80 nursing hrs/week, 3 surgical slots/week. SES means Socio-economic Standard

Metric	Framework	Formula	Direction
Expected Survival Gain (ESG)	Utilitarian	$\sum_{i=1}^{N}p_{i}\cdot\mathrm{CNSS}_{i}$	$\uparrow$
Rawlsian Minimum Gurantee (RMG)	Rawlsian	$\min_{i\in\{1,\ldots,N\}}\mathrm{CNSS}_{i}$	$\uparrow$
Liberitarian Variance (Var)	Liberitarian	$\frac{1}{N}\sum_{i}(\mathrm{CNSS}_{i}-\overline{\mathrm{CNSS}})^{2}$	$\downarrow$
Disadvantage Weighted ESG (DW-ESG)	Prioritarian	$\sum_{i=1}^{8}w_{i}^{P}\cdot p_{i}\cdot\mathrm{CNSS}_{i}$	$\uparrow$
Vulnerability Weighted Care Intensity (VWCI)	Care Ethics	$\sum_{i=1}^{N}w_{i}^{C}\cdot\mathrm{CNSS}_{i}$	$\uparrow$
Gini	Egalitarian	$G=\frac{2\sum_{i=1}^{N}i\,h_{(i)}}{N\sum_{i=1}^{N}h_{(i)}}-\frac{N+1}{N}$	$\uparrow$

Table 2: Metrics Details. Each metric computes the welfare function associated with its corresponding ethical framework. The arrow (Direction) shows if it needs to be maximized or minimized.

w

are weight, each depends on the considered ethical framework

	ESG (Util) $\uparrow$				Gini (Egal) $\downarrow$				RMG (Rawls) $\uparrow$
Framework	A	B	A	C	A	B	A	C	A	B	A	C
	VS		VS		VS		VS		VS		VS
Utilitarian	5.170	5.052	5.115	4.772	0.268	0.313	0.286	0.388	0.504	0.418	0.462	0.233
Egalitarian	5.107	4.949	5.035	4.599	0.264	0.311	0.288	0.409	0.673	0.573	0.615	0.283
Rawlsian	4.938	4.691	4.850	4.271	0.302	0.360	0.330	0.470	0.668	0.371	0.593	0.061
Prioritarian	5.541	5.454	5.496	5.234	0.227	0.271	0.242	0.336	0.743	0.658	0.711	0.503

Table 3: Results across all metrics and frameworks.

\uparrow

higher is better,

\downarrow

lower is better. Best value per column in bold .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	ESG (Util)				Gini (Egal)				RMG (Rawls)
Framework	$p$	$d$	$p$	$d$	$p$	$d$	$p$	$d$	$p$	$d$	$p$	$d$
	A vs B		A vs C		A vs B		A vs C		A vs B		A vs C
Utilitarian	.003*	0.41	.008*	0.38	.001*	0.52	.002*	0.47	.002*	0.48	.004*	0.43
Egalitarian	.002*	0.44	.006*	0.41	.001*	0.55	.001*	0.51	.001*	0.58	.003*	0.49
Rawlsian	.003*	0.46	.009*	0.40	.002*	0.49	.001*	0.55	.001*	0.81	.001*	0.76
Prioritarian	.018*	0.31	.022*	0.28	.001*	0.57	.001*	0.52	.001*	0.60	.001*	0.57

Appendix

Table 4:

p

-values and Cohen’s

d

effect sizes from Wilcoxon signed-rank tests (

n=50

cohorts). * denotes

p<0.05

Component	Configuration
LLM 1	LLaMA 3.3 70B
LLM 2	Qwen 2.5 72B
Embedding model	nomic-embed-text-v2-moe , 768-dim, 8192-token context
Vector store	Chroma (persistent on disk), cosine similarity
RAG framework	LangChain
Retrieval $k$	5 chunks per query

Appendix

Table 5: Model and infrastructure configuration.

Round	Pages retrieved	Semantic query keywords
1	42, 112, 99, 20, 0	equitable resource allocation, ethical framework, fair distribution, clinical need
2	108, 42, 99, 112, 1	vulnerability, disadvantage, socioeconomic disparity, prioritisation, justice
3	37, 99, 112, 40, 56	dependency, obligation, relational care, maximin, worst-off protection

Appendix

Table 6: RAG retrieval per round for Agent A, cohort 32, Utilitarian framework.

k=5

chunks retrieved per round.

P	Age	Gender	Race	SES	Condition (slot)	Surv.	Urgency
1	50	F	Latino	Middle	Colorectal cancer (slot 3)	76%	High
2	62	Non-bin.	White	Middle	Elective joint replacement (slot 8)	89%	High
3	9	F	M. Eastern	Refugee	Acute organ failure (slot 4)	51%	Low
4	87	F	M. Eastern	Refugee	Chronic heart failure (slot 2)	62%	High
5	22	M	Black	High	Traumatic brain injury (slot 1)	84%	Acute
6	34	F	Latino	Working	Drug-resistant infection (slot 7)	88%	High

Appendix

Table 7: Cohort 32 patient profiles. Capacity: tight.

Demographic Bias in Language Models

Demographic bias in language models studies how identity cues shape model outputs, decisions, and representations, and how to detect and reduce resulting disparities.

See all 134 papers

Emergence of Biased Consensus in Multi-Agent LLM Debates
Aug 3, 2026Maya OkawaOpinion Dynamics Social Bias in Language Models
CBS-NTT Program in Physics of Intelligence, Harvard University · Physics of Artificial Intelligence Group, NTT Research, Inc.
Bias Amplification in Multi-Agent Network: How Biased Agents Shape Opinions and Rhetoric
Sep 16, 2026Omran Berjawi, Giuseppe Fenza, Rida KhatounOpinion Dynamics Social Bias in Language Models
Institut Polytechnique de Paris, Télécom Paris, Palaiseau, France · University of Salerno, Fisciano, Italy
Bias Control and Injection for Multi-Agent Aggregation
Oct 15, 2026Sungjae Lee, Soheil Abbasloo, Jabin Koo +1Multi-Agent Reasoning LLM Agent Reliability
POSTECH · Microsoft Research

Aug 3, 2026J/K move · Enter open · S save

Emergence of Biased Consensus in Multi-Agent LLM Debates

Maya Okawa

CBS-NTT Program in Physics of Intelligence, Harvard University · Physics of Artificial Intelligence Group, NTT Research, Inc.

Opinion Dynamics Social Bias in Language Models Multi-Agent Debate Multi-Agent Systems Multi-Agent Consensus Emergent Behavior in Multi-Agent Systems

Open paper AskPDF arXiv

Multi-agent LLM debates achieve strong performance on decision-making tasks as well as problem-solving benchmarks, yet their safety and fairness risks remain poorly understood. Notably, interaction can amplify the biases of single LLMs, raising concerns for real-world deployment. We identify the emergence of collective (often biased) norms in multi-agent LLM debates and show that noise (e.g., LLM sampling temperature) is a key driver. To explain this, we propose an analytical framework drawing on physics-inspired theoretical models of social dynamics. We predict a phase transition to collective bias when conformity surpasses a critical threshold given the LLMs' initial bias and debate noise. We test the theoretical predictions through controlled experiments and observe a finite-size crossover consistent with an underlying phase transition. We further find that agent heterogeneity suppresses emergence by smoothing (rounding) this transition. Finally, we show that these insights generalize to realistic decision-making tasks, including investment decisions and LLM-as-a-judge evaluation.

Abstract

Demographic Bias in Language Models

Explore similar work

Emergence of Biased Consensus in Multi-Agent LLM Debates

Bias Amplification in Multi-Agent Network: How Biased Agents Shape Opinions and Rhetoric

Bias Control and Injection for Multi-Agent Aggregation