On the possibilities and limitations of LLM-based multi-agent systems for multi-criteria decision-making in healthcare operations management

Marco Boresta; Tommaso Giovannelli; Sepehr Namirad; Giulia Riccardi

math.OCOct 16, 2026

On the possibilities and limitations of LLM-based multi-agent systems for multi-criteria decision-making in healthcare operations management

Authors: Marco Boresta, Tommaso Giovannelli, Sepehr Namirad, Giulia Riccardi

Organizations: Istituto di Analisi dei Sistemi ed Informatica (IASI), CNR, Rome, Italy · Department of Mechanical, Materials, and Industrial Engineering, University of Cincinnati, Cincinnati, OH · Department of Computer, Control, and Management Engineering (DIAG), Sapienza University of Rome, Rome, Italy

Abstract

Multi-criteria models for healthcare operations need weights that express how stakeholders trade off patient waiting time, provider workload, and operating cost, but these weights are difficult to quantify. Teams of large language model (LLM) agents, each prompted to represent a stakeholder, have been used to supply them. We propose a framework for assessing the reliability of LLM-generated preferences as inputs to multi-objective optimization and quantifying how their uncertainty affects decisions. We study explicit preferences by asking agents to rate the importance of the criteria in an outpatient appointment scheduling problem, and implicit preferences by inferring trade-offs from the agents' pairwise choices between emergency department configurations. The computational experiments show that the explicit preferences respond to the clinic's operating conditions and differ across roles, but deliberation among the agents can erase these differences, whereas simple averaging preserves them. The pairwise choices used to reveal the agents' implicit preferences depend largely on which option is presented first, and we find limited evidence that a statistical model using contextual information alongside presentation order predicts the agents' choices on new comparison pairs better than a model using presentation order alone. We then formulate robust weighted-sum optimization models to account for preference uncertainty. In most cases, these models select different schedules or configurations from those obtained using average preference weights. These results indicate that LLM-generated preferences need an audit of this kind before they are used as optimization weights in healthcare decision support.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the LLM-PADA framework. Preferences are obtained from LLM agents in two ways, by asking them to rate the criteria in outpatient clinics or by observing their choices between different ED configurations. In both cases, the resulting weights are uncertain, and we propose robust models to quantify how accounting for preference uncertainty changes decisions.

Work	Role-based LLM agents	Explicit weights	Choice-based inference	Reliability assessment	Robust decisions under weight uncertainty	Healthcare operations
Vahidnia [ 57 ]	✓	✓	–	$\circ$	–	–
Jovine et al. [ 36 ]	–	✓	$\circ$	$\circ$	–	–
LLM judges [ 70 , 62 ]	–	–	$\circ$	✓	–	–
LLM debate [ 55 , 12 ]	✓	–	–	✓	–	–
Hu and Mehrotra [ 31 ]	–	$\circ$	–	–	✓	–
PRO [ 32 , 29 , 58 ]	–	–	$\circ$	–	✓	–

Table 1: Positioning of the paper. ✓: addressed;

\circ

: partially addressed; –: not addressed.

Figure 2: Paired score differences between condition HW or HO and condition B before deliberation, by agent and criterion. Values are mean differences in importance score points on the $1$ – $9$ scale; positive values mean higher importance in the stated condition. Error bars are 95% percentile bootstrap intervals over the ten base clinic descriptions.

Figure 3: Focal paired score differences before deliberation and after round 3 of each protocol: waiting score under HW (left) and overtime score under HO (right), in importance score points on the $1$ – $9$ scale. Error bars are 95% percentile bootstrap intervals over the ten base clinic descriptions.

Measure	Scope	Numerical	Argumentative
Valid terminal outputs	All turns	1783/1800 (99.06%)	1765/1800 (98.06%)
Technical terminal failures	All turns	17/1800 (0.94%)	24/1800 (1.33%)
Schema-invalid terminal outputs	All turns	0/1800 (0.00%)	11/1800 (0.61%)
Technical failures across attempts	All attempts	131/1923 (6.81%)	164/2005 (8.18%)
Schema-invalid outputs across attempts	All attempts	0/1923 (0.00%)	67/2005 (3.34%)
Valid other-agent argument reference	Rounds 2–3	—	1118/1170 (95.56%)

Table 2: Technical failures, invalid outputs, and references during deliberation. Entries report counts, with percentages in parentheses. Technical failures are unsuccessful requests. Invalid outputs are completed responses that fail JSON or field validation. Reference percentages are calculated using only valid argumentative turns.

Protocol	Positive	Interval contains zero	Reversed
Pre-deliberation	12	0	0
Numerical	3	3	6
Argumentative	1	8	3
Mechanical	12	0	0

Table 3: Role-differentiation scores before and after deliberation. Entries are counts of the 12 scores per protocol (three agents under four conditions) whose 95% bootstrap interval lies entirely above zero (positive), contains zero, or lies entirely below zero (reversed). Intervals are not adjusted for multiple comparisons.

Figure 4: Inter-agent dispersion ( 3.9 ) of the normalized weight vectors over the deliberation rounds, in units of normalized weight. Lines are means by protocol and condition; shaded regions are 95% percentile bootstrap intervals over the ten base clinic descriptions.

Figure 5: Schedules selected by problems (MW) and (RB) on the same instance with seven requests, using weights elicited under HW and HO and revised through argumentative deliberation. HW and HO label the elicitation conditions; all panels use the same session, requests, and constraints. Blue segments are service intervals, red markers are appointment times, and dashed red segments are patient waiting. Panel titles report total waiting (W), idle time (I), and overtime (O) in minutes.

Figure 6: Average of the eight group mean waiting times,

f_{w}

(minutes), and MIU opening hours,

f_{\mathrm{cost}}

(hours per week), for the eleven configurations in Table 3 of Boresta et al. [6] .

Model	Role	$\widehat{\lambda}$	$\Delta B$ vs. position	Order agreement	Dominance compliance	Diagnostic screen
9B	Patient	$0.70\;[0.00,\,1.00]$	$+0.006\;[-0.003,\,0.018]$	0.26	7/10	Not met
9B	Physician	$1.00\;[0.00,\,1.00]$	$+0.002\;[-0.001,\,0.006]$	0.19	9/10	Not met
9B	Operations	$1.00\;[0.55,\,1.00]$	$+0.002\;[-0.011,\,0.017]$	0.23	8/10	Not met
27B	Patient	$0.75\;[0.71,\,1.00]$	$+0.075\;[0.032,\,0.119]$	0.35	10/10	Met
27B	Physician	$0.72\;[0.67,\,1.00]$	$+0.009\;[-0.030,\,0.048]$	0.29	10/10	Not met
27B	Operations	$0.64\;[0.58,\,0.80]$	$+0.071\;[0.033,\,0.109]$	0.61	10/10	Met

Table 4: Estimated waiting-time weights and predictive and behavioral diagnostics. Brackets give 95% bootstrap intervals.

Model	Role	$\bar{\lambda}$	Mean weight $x^{\mathrm{MW}}$	Fitted weight $x^{\mathrm{FW}}$	Robust $x^{\mathrm{RW}}$	$\Delta W$	$\Delta C$	$P$ (%)	$G$ (%)
9B	Patient	0.750	E	E	H	+1.483	-10	10.34	28.57
9B	Physician	0.719	E	A	H	+1.483	-10	7.70	28.57
9B	Operations	0.896	B	A	G	+2.060	-26	31.21	22.61
27B	Patient	0.830	B	E	E	+1.071	-18	3.63	8.61
27B	Physician	0.767	E	E	E	+0.000	+0	0.00	0.00
27B	Operations	0.662	E	G	G	+0.989	-8	0.20	4.00

Table 5: Selected configurations under mean (MW), fitted (FW), and uncertain (RB) weights. Entries identify configurations

x^{a}

\Delta W

(minutes) and

\Delta C

(weekly opening hours) report RB minus MW;

P

and

G

are defined in ( 4.11 ). All reported optima are unique.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Criterion	Agent	Pre	Numerical	Argumentative	Mechanical
HW	Waiting	Intake	$0.560\ [0.225,0.845]$	$-0.100\ [-0.640,0.440]$	$0.060\ [-0.320,0.560]$	$0.580\ [0.384,0.757]$
HW	Waiting	Provider	$0.695\ [0.455,0.910]$	$0.300\ [-0.280,0.860]$	$0.040\ [-0.420,0.480]$	$0.617\ [0.418,0.785]$
HW	Waiting	Scheduler	$0.506\ [0.206,0.736]$	$0.680\ [0.100,1.320]$	$0.480\ [0.000,0.960]$	$0.565\ [0.346,0.738]$
HO	Overtime	Intake	$2.872\ [2.603,3.128]$	$0.760\ [0.260,1.320]$	$0.880\ [0.060,1.760]$	$2.612\ [2.458,2.807]$
HO	Overtime	Provider	$1.691\ [1.167,2.321]$	$0.980\ [0.140,1.860]$	$1.600\ [0.660,2.620]$	$2.287\ [2.009,2.646]$
HO	Overtime	Scheduler	$2.977\ [2.727,3.218]$	$1.200\ [0.520,1.920]$	$1.340\ [0.520,2.220]$	$2.640\ [2.446,2.870]$

Appendix

Table 6: Mean paired differences in importance scores between HW and B for waiting and between HO and B for overtime. Pre denotes the initial scores; the remaining columns report results after round 3 of each protocol. Differences are in points on the

1

–

9

scale, and brackets give 95% percentile bootstrap intervals over the ten base clinic descriptions.

Weight source	Condition	Intake–waiting	Provider–overtime	Scheduler–idle
Pre	B	$0.087\ [0.071,0.104]$	$0.094\ [0.079,0.106]$	$0.078\ [0.064,0.093]$
Pre	CF	$0.142\ [0.129,0.156]$	$0.127\ [0.118,0.136]$	$0.087\ [0.075,0.099]$
Pre	HW	$0.073\ [0.044,0.101]$	$0.091\ [0.071,0.110]$	$0.094\ [0.068,0.120]$
Pre	HO	$0.053\ [0.035,0.071]$	$0.019\ [0.010,0.028]$	$0.026\ [0.012,0.042]$
Numerical	B	$0.029\ [-0.002,0.060]$	$-0.039\ [-0.061,-0.013]$	$0.058\ [0.034,0.080]$
Numerical	CF	$-0.030\ [-0.049,-0.011]$	$-0.015\ [-0.029,-0.001]$	$-0.031\ [-0.043,-0.018]$

Appendix

Table 7: Role-differentiation scores by weight source and condition. Pre denotes scores before deliberation; the other sources use round-3 scores. Each column compares the named agent’s normalized weight on its role-associated criterion with the mean weight assigned by the other two agents. Brackets give 95% percentile bootstrap intervals over the ten base clinic descriptions.

Weight source	Condition	Waiting bounds	Idle bounds	Overtime bounds
Pre	B	$[0.253,0.400]$	$[0.236,0.390]$	$[0.312,0.479]$
Pre	CF	$[0.255,0.471]$	$[0.172,0.318]$	$[0.340,0.496]$
Pre	HW	$[0.301,0.456]$	$[0.204,0.365]$	$[0.269,0.435]$
Pre	HO	$[0.231,0.327]$	$[0.176,0.280]$	$[0.447,0.534]$
Numerical	B	$[0.272,0.571]$	$[0.118,0.342]$	$[0.206,0.469]$
Numerical	CF	$[0.341,0.518]$	$[0.118,0.267]$	$[0.300,0.444]$

Appendix

Table 8: Empirical 5th–95th percentile bounds used with the simplex constraint to define the weight uncertainty sets. For each condition, the pre-deliberation and mechanical samples contain 30 normalized weight vectors, while the numerical and argumentative samples contain 150.

Weight source	Condition	(MW)	(RB)	Mean-objective change	Worst-case change
Pre	B	$(86,0,0)$	$(50,15,6)$	$+8.12\%$	$-5.95\%$
Pre	CF	$(72,8,0)$	$(50,15,6)$	$+0.99\%$	$-9.86\%$
Pre	HW	$(72,8,0)$	$(50,15,6)$	$+0.01\%$	$-7.93\%$
Pre	HO	$(86,0,0)$	$(72,8,0)$	$+0.16\%$	$-3.45\%$
Numerical	B	$(23,25,16)$	$(23,25,16)$	$0.00\%$	$0.00\%$
Numerical	CF	$(50,15,6)$	$(23,25,16)$	$+0.57\%$	$-3.53\%$

Appendix

Table 9: Outcomes selected by (MW) and (RB) on the same scheduling instance, using weights from each source and elicitation condition. Each outcome profile is reported as (total patient waiting, provider idle time, provider overtime) in minutes. The final columns give the percentage changes in the mean-weight and worst-case objectives when moving from the (MW) solution to the (RB) solution, using the corresponding objective value at the (MW) solution as the reference.

Preference Modeling

See all 36 papers

Role-Specialized Mixture-of-Agents with Open-Weight LLMs for Clinical Prediction
Aug 23, 2026Jun Hou, Yi Fang, Xuan WangMixture-of-Agents Clinical Prediction
Virginia Tech
ParetoChat: An LLM-Based Interface for Interactive Multi-Objective Optimization
Oct 16, 2026Gokhan CeyhanMulti-Objective Optimization Multi-Criteria Decision Analysis
Just Eat Takeaway.com
Large Language Model Orchestration under Heterogeneous Preferences via Explicit Persona Inference
Oct 6, 2026Shuqing Shi, Ziyan Wang, Milind Tambe +1Multi-Agent Coordination Active Preference Learning
King’s College London, London, United Kingdom · Harvard University, Cambridge, MA, USA

Aug 23, 2026J/K move · Enter open · S save

Role-Specialized Mixture-of-Agents with Open-Weight LLMs for Clinical Prediction

Jun Hou, Yi Fang, Xuan Wang

Virginia Tech

Mixture-of-Agents Clinical Prediction Multi-Agent LLMs Multi-Agent Systems Clinical Risk Prediction

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly applied to clinical prediction tasks such as in-hospital mortality and readmission from electronic health records (EHRs). Privacy and compliance constraints motivate systems that can be deployed locally, which has increased interest in open-weight multi-agent designs. However, most medical multi-agent systems are evaluated as a single block, leaving unclear which agent role contributes to prediction and whether retrieval drives observed gains. We study a role-specialized Mixture-of-Agents (MoA) that combines medical knowledge retrieval with contrastive similar-patient reasoning. By varying the role design while holding the retrieval setup fixed, we localize the main effect to the final integrator. Pairing large open-weight analysts with a small open-weight integrator matches closed-model prompting on F1 for mortality prediction while flagging substantially more true high-risk patients. Mechanism analysis shows the role assignment directly yields a high-recall operating point without threshold tuning. The effect is task-dependent, with smaller gains for readmission because the available records correlate weakly with this longer-horizon outcome. These results position role design as a key factor in privacy-constrained, training-free clinical LLM prediction.

Abstract

Preference Modeling

Explore similar work

Role-Specialized Mixture-of-Agents with Open-Weight LLMs for Clinical Prediction

ParetoChat: An LLM-Based Interface for Interactive Multi-Objective Optimization

Large Language Model Orchestration under Heterogeneous Preferences via Explicit Persona Inference