Fairness in language models is typically studied as a property of a single, centrally optimized model. As large language models become increasingly agentic, we propose that fairness emerges through interaction and exchange. We study this via a controlled hospital triage framework in which two agents negotiate over three structured debate rounds. One agent is aligned to a specific ethical framework via retrieval-augmented generation (RAG), while the other is either unaligned or adversarially prompted to favor demographic groups over clinical need. We find that alignment systematically shapes negotiation strategies and allocation patterns, and that neither agent's allocation is ethically adequate in isolation, yet their joint final allocation can satisfy fairness criteria that neither would have reached alone. Aligned agents partially moderate bias through contestation rather than override, acting as corrective patches that restore access for marginalized groups without fully converting a biased counterpart. We further observe that even explicitly aligned agents exhibit intrinsic biases toward certain frameworks, consistent with known left-leaning tendencies in LLMs. We connect these limits to Arrow's Impossibility Theorem: no aggregation mechanism can simultaneously satisfy all desiderata of collective rationality, and multi-agent deliberation navigates rather than resolves this constraint. Our results reposition fairness as an emergent, procedural property of decentralized agent interaction, and the system rather than the individual agent as the appropriate unit of evaluation.
Figures & tables
Figure 1: The deliberative arena. Two agents with preference profiles PA and PB,C debate over a resource allocation problem. At round t , an agent propose an allocation At with justification Jt . This is passed to the other agent that critique and propose a novel solution, doing that in a loop of T rounds. The final Social Choice ( A∗ ) represents a linguistic and numerical convergence point, constrained by Arrow’s Impossibility Theorem, where alignment functions as a procedural moderator to minimize systemic bias.
i
Condition
Age
Gender
Race
SES
Needs
1
Acute trauma
56
NB
White
Upper
ICU, Vent, Nurs
2
Chronic illness
24
M
White
U-mid
MedA, MedB, Nurs
3
Cancer
78
F
Black
Low
Surg, MedA, Nurs, ICU
4
Rare disease
52
M
Asian
High
ICU, MedB, Nurs
5
Paediatric
6
F
M-East
Refugee
Surg, MedA, Nurs
6
Respiratory failure
41
M
Latino
Work.
ICU, Vent, MedA, Nurs
Table 1: Representative patient cohort. Available resources are: 3 ICU, 2 Vent, 60 Med-A, 50 Med-B, 80 nursing hrs/week, 3 surgical slots/week. SES means Socio-economic Standard
Metric
Framework
Formula
Direction
Expected Survival Gain (ESG)
Utilitarian
∑i=1Npi⋅CNSSi
↑
Rawlsian Minimum Gurantee (RMG)
Rawlsian
mini∈{1,…,N}CNSSi
↑
Liberitarian Variance (Var)
Liberitarian
N1∑i(CNSSi−CNSS)2
↓
Disadvantage Weighted ESG (DW-ESG)
Prioritarian
∑i=18wiP⋅pi⋅CNSSi
↑
Vulnerability Weighted Care Intensity (VWCI)
Care Ethics
∑i=1NwiC⋅CNSSi
↑
Gini
Egalitarian
G=N∑i=1Nh(i)2∑i=1Nih(i)−NN+1
↑
Table 2: Metrics Details. Each metric computes the welfare function associated with its corresponding ethical framework. The arrow (Direction) shows if it needs to be maximized or minimized. w are weight, each depends on the considered ethical framework
ESG (Util) ↑
Gini (Egal) ↓
RMG (Rawls) ↑
Framework
A
B
A
C
A
B
A
C
A
B
A
C
VS
VS
VS
VS
VS
VS
Utilitarian
5.170
5.052
5.115
4.772
0.268
0.313
0.286
0.388
0.504
0.418
0.462
0.233
Egalitarian
5.107
4.949
5.035
4.599
0.264
0.311
0.288
0.409
0.673
0.573
0.615
0.283
Rawlsian
4.938
4.691
4.850
4.271
0.302
0.360
0.330
0.470
0.668
0.371
0.593
0.061
Prioritarian
5.541
5.454
5.496
5.234
0.227
0.271
0.242
0.336
0.743
0.658
0.711
0.503
Table 3: Results across all metrics and frameworks. ↑ higher is better, ↓ lower is better. Best value per column in bold .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
ESG (Util)
Gini (Egal)
RMG (Rawls)
Framework
p
d
p
d
p
d
p
d
p
d
p
d
A vs B
A vs C
A vs B
A vs C
A vs B
A vs C
Utilitarian
.003*
0.41
.008*
0.38
.001*
0.52
.002*
0.47
.002*
0.48
.004*
0.43
Egalitarian
.002*
0.44
.006*
0.41
.001*
0.55
.001*
0.51
.001*
0.58
.003*
0.49
Rawlsian
.003*
0.46
.009*
0.40
.002*
0.49
.001*
0.55
.001*
0.81
.001*
0.76
Prioritarian
.018*
0.31
.022*
0.28
.001*
0.57
.001*
0.52
.001*
0.60
.001*
0.57
Appendix
Table 4: p -values and Cohen’s d effect sizes from Wilcoxon signed-rank tests ( n=50 cohorts). * denotes p<0.05 .
Demographic bias in language models studies how identity cues shape model outputs, decisions, and representations, and how to detect and reduce resulting disparities.
Multi-agent LLM debates achieve strong performance on decision-making tasks as well as problem-solving benchmarks, yet their safety and fairness risks remain poorly understood. Notably, interaction can amplify the biases of single LLMs, raising concerns for real-world deployment. We identify the emergence of collective (often biased) norms in multi-agent LLM debates and show that noise (e.g., LLM sampling temperature) is a key driver. To explain this, we propose an analytical framework drawing on physics-inspired theoretical models of social dynamics. We predict a phase transition to collective bias when conformity surpasses a critical threshold given the LLMs' initial bias and debate noise. We test the theoretical predictions through controlled experiments and observe a finite-size crossover consistent with an underlying phase transition. We further find that agent heterogeneity suppresses emergence by smoothing (rounding) this transition. Finally, we show that these insights generalize to realistic decision-making tasks, including investment decisions and LLM-as-a-judge evaluation.
Maya Okawa
CBS-NTT Program in Physics of Intelligence, Harvard University · Physics of Artificial Intelligence Group, NTT Research, Inc.
Large language models (LLMs) are increasingly deployed in applications involving interaction between agents, where their output plays a role in collective reasoning and decision-making processes. Despite significant research into the functioning of LLMs in such multi-agent systems, the processes of bias propagation in such systems are still a challenge. This work studies how biased opinions are propagated in the form of textual interaction in an environment of LLMs, in which a minority of agents maintain persistent extreme opinions, while the remaining agents iteratively update their beliefs through structured textual interactions. The findings show that even the presence of a small percentage of biased agents in such a system leads to significant shifts in the opinions of non-biased agents. It suggests that for the same percentage of biased agents, the shifts occur more quickly for the Llama~3.2 model when compared to a classical Friedkin-Johnsen (FJ) model. Further semantic analysis demonstrates that rhetorical consistency in textual explanations increases systematically with biased exposure and, importantly, is partially decoupled from numerical convergenumericalutral agents adopt the vocabulary employed by the biased agents even in configurations where their numerical opinion shifts remain moderate. The research helps explain how bias and language develop together in multi-agent language model ecosystems.
Omran Berjawi, Giuseppe Fenza, Rida Khatoun
Institut Polytechnique de Paris, Télécom Paris, Palaiseau, France · University of Salerno, Fisciano, Italy
Multi-agent reasoning with large language models (LLMs) raises a central question for collective decision-making: when agents disagree, whose decision should be trusted? Resolving such disagreements is difficult because agents' decisions are susceptible to non-semantic biases such as majority pressure and response-style preferences, which can obscure the quality of the underlying reasoning. Our analysis shows how bias can be both mitigated and exploited: bias control improves initial answer selection, while bias injection elicits decision instability that serves as a reliability signal. Building on these findings, we introduce a test-time aggregation framework that requires neither validation data nor prior knowledge of relative model strength. The framework implements bias control by presenting one style-normalized reasoning trace per distinct answer, and bias injection by placing each aggregating agent's initially selected answer in the minority. Overall, our framework achieves higher average accuracy and better token efficiency than competing non-oracle aggregation methods across three mathematical reasoning benchmarks and four heterogeneous LLM combinations. It also improves final-answer accuracy when applied to multi-agent debate and Mixture-of-Agents, without modifying their interaction or synthesis processes.