Multi-agent deliberation (MAD), where multiple LLMs engage in structured discussion, improves AI performance on many benchmarks, but whether it helps humans make better decisions remains untested. We conduct two user studies on LLM-response-verification and fact-checking. We first compare MAD against common AI decision-support methods that isolate its two defining dimensions --- multiple AIs and multi-round interaction. We find MAD outperformed single-round AI assistance on both tasks; multi-round interaction was the main driver of this benefit, while a second AI added value when the single AI had room to improve. Then, we study two design choices in MAD --- AI persona and human engagement level --- finding a devil's advocate persona to lower accuracy and user experience, while humans' active participation in MAD improved accuracy in one task at the cost of frustration. We conclude by discussing implications for when, why, and how multi-agent deliberation can support human decision-making.
Figures & tables
Figure 1. Summary of treatments in our user study. RQ1: we study whether MAD ( MAD-Active - Base ) can outperform common AI assistance methods Chatbot-R1 , Chatbot , and Static , where Chatbot-R1 is derived from Chatbot . RQ2: we study the effects of two design choices in MAD, AI persona configuration ( Base , Expert , DA ) and human engagement levels (active participant and passive reviewer), with a 3×2 design. MAD-Active - Base is used for both RQ1 and RQ2.
Figure 2. RQ1: Human decision accuracy . Comparing MAD-Active - Base with Chatbot-R1 , Chatbot , and Static for users’ final decision accuracy. Human baselines reflect the accuracies of humans’ initial answers before any AI assistance and dashed lines show AI’s accuracy on the entire datasets. Error bars are 95% confidence intervals. Significant differences between conditions are denoted with *: p<0.05 , **: p<0.01 , ***: p<0.001 .
Figure 3. RQ1: Accuracy divided by correctness of AIs’ final answer . (a) All-AI correct rate : frequency that all AIs’ final answers are correct. (b) User accuracy when all AIs were correct . (c) All-AI error rate . (d) User accuracy when all AIs were incorrect . Error bars are 95% confidence intervals. Significant differences between conditions are denoted with *: p<0.05 , **: p<0.01 , ***: p<0.001 .
Figure 4. RQ2: Human decision accuracy . Two-way ANOVA on human engagement level ( MAD-Active vs. MAD-Passive ) and AI persona configuration ( Base , Expert , DA ). Error bars are 95% confidence intervals. Text in panels notes the significant main effects; brackets show significant effects between AI persona treatments or interactions (*: p<0.05 , **: p<0.01 , ***: p<0.001 ).
Figure 5. RQ2: Divided analysis to analyze the mechanisms behind design choices . Two-way ANOVA on human engagement level ( MAD-Active vs. MAD-Passive ) and AI persona configuration ( Base , Expert , DA ). (a) All-AI correct rate : frequency that all AIs’ final answers are correct. (b) User accuracy when all AIs were correct . (c) All-AI error rate . (d) User accuracy when all AIs were incorrect . (e) AI final disagreement rate : rate at which the two AIs have different final decisions. Error bars are 95% confidence intervals. Text in panels notes the significant main effects; brackets show significant effects between AI persona treatments or interactions (*: p<0.05 , **: p<0.01 , ***: p<0.001 ).
Notable patterns
Significant findings for ReaLMistake
Significant findings for Fact-check
MAD-Active - Base > Static or Chatbot
Helpfulness of AI Disagreement comfort Disagreement confusion Decision confidence Trust towards AI Satisfaction towards AI Disagreement helpfulness
Helpfulness of AI Disagreement comfort Disagreement confusion AIs provide multiple perspectives
MAD-Active - Base < Static or Chatbot
Mental demand Effort
DA > Base or Expert
AIs provide multiple perspectives Attention to disagreeing answers Attention to disagreeing arguments Disagreement helpfulness
DA < Base or Expert
Frustration Trust towards AI Complexity of AI Confusion from AI Satisfaction towards AI Disagreement comfort
Frustration Trust towards AI Complexity of AI Confusion from AI Mental demand Effort
Expert < Base
Frustration Confusion
Complexity
MAD-Active > MAD-Passive
Consensus pressure Disagreement confusion
Disagreement helpfulness
Table 1. Summary of significant findings for users’ subjective metrics , divided by notable patterns and significant findings for ReaLMistake and Fact-check. > ( < ) indicate that the first listed treatment is considered better (worse) on a metric, depending on whether higher scores reflect positive or negative subjective experiences. For example, higher mental demand indicates higher cognitive load and worse user experience, so MAD-Active - Base < Static means MAD-Active - Base users have higher mental demand. Underlined metrics indicate findings shared for ReaLMistake and Fact-check.
Figure 6. Users’ subjective experience: significant findings shared for both tasks when comparing MAD-Active - Base with Chatbot and Static . (a) Helpfulness of AI. (b) Comfort towards AIs’ disagreements. (c) Confusion towards AIs’ disagreements. Error bars are 95% confidence intervals. Significant differences between conditions are denoted with *: p<0.05 , **: p<0.01 , ***: p<0.001 .
Comparing MAD with baselines
Improve users’ overall accuracy
Improve AI performance
Help utilize correct AI answers
Reduce & detect shared AI errors
Benefits when AIs disagree
MAD vs. single-AI single-round interaction
✔
✝
✝
◯
N/A
MAD vs. single-AI multi-round interaction
✝
✝
◯
◯
N/A
MAD vs. two-AI independent responses
✔
✔
✔
✗
◯
Comparing MAD design choices
Improve users’ overall accuracy
Improve AI performance
Help utilize correct AI answers
Reduce & detect shared AI errors
Benefits when AIs disagree
AI expert personas vs. no persona
◯
✗
◯
◯
◯
AI devil’s advocate vs. no persona
✗
✗
✗
✝
◯
Table 2. Summary of findings from RQ1 and RQ2, divided by measurements . ✔ : the first listed treatment or design choice outperforms the other across both tasks. ✝ : the first listed treatment or design choice outperforms the other in one out of the two tasks. ◯ : no differences between the listed pair of treatments or design choices. ✗ : the first listed treatment or design choice performs worse than the other in at least one task. N/A is for when findings involving multiple AIs do not apply to single-AI treatments.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Questions
Need for Cognition Scale (NCS-6)
I would prefer complex to simple problems.
I like to have the responsibility of handling a situation that requires a lot of thinking.
* Thinking is not my idea of fun.
* I would rather do something that requires little thought than something that is sure to challenge my thinking abilities.
I really enjoy a task that involves coming up with new solutions to problems.
I would prefer a task that is intellectual, difficult, and important to one that is somewhat important but does not require much thought.
How much mental and perceptual activity was required (e.g., thinking, deciding, calculating, remembering, looking, searching, etc.) for the tasks?
Effort
How hard did you have to work (mentally and physically) to accomplish your level of performance?
Frustration
How irritated, stressed, and annoyed did you feel during the tasks?
Perceptions of AI
Helpfulness
The AI chatbots’ assistance helped me make good decisions.
Trust
I trusted the AI chatbots to provide reliable decision support.
Appendix
Table 4. Exit survey questions, measured using 7-point Likert scales.
Figure 8. Decision accuracy comparing all MAD treatments with baselines . Error bars are 95% confidence intervals. Significant differences between conditions are denoted with *: p<0.05 , **: p<0.01 , ***: p<0.001 .
Figure 9. Users’ subjective experience: comparing MAD-Active - Base against baselines. (a): decision confidence. (b-d): NASA-TLX metrics on cognitive load. (e-j): perceptions of AI assistance. (k-q): experience with multiple AIs. Error bars are 95% confidence intervals. Significant differences between conditions are denoted with *: p<0.05 , **: p<0.01 , ***: p<0.001 .
Figure 10. Users’ subjective experience: two-way ANOVA on AI persona ( Base , Expert , DA ) and human engagement ( MAD-Active vs. MAD-Passive . (a): decision confidence. (b-d): NASA-TLX metrics on cognitive load. (e-j): perceptions of AI assistance. Error bars are 95% confidence intervals. Text in panels notes the significant main effects; brackets show significant effects between AI persona treatments or interactions (*: p<0.05 , **: p<0.01 , ***: p<0.001 ).
Figure 12. RQ2: metrics for when two AIs initially disagree . Two-way ANOVA on human engagement level ( MAD-Active vs. MAD-Passive ) and AI persona configuration ( Base , Expert , DA ). (a) AI initial disagreement rate : whether the two AIs have different initial decisions. (b) User final decision accuracy when AIs initially disagree . Error bars are 95% confidence intervals. Text in panels notes the significant main effects; brackets show significant effects between AI persona treatments or interactions (*: p<0.05 , **: p<0.01 , ***: p<0.001 ).
Task
Condition
Metric
MAD-Active - DA
MAD-Passive - DA
ReaLMistake
Overall cases
Non-DA AI initial accuracy
0.657±0.189
0.693±0.182
Non-DA AI final accuracy
0.783±0.136
0.650±0.227
Human final accuracy
0.610±0.222
0.533±0.190
Underreliance towards non-DA AI
0.298±0.274
0.426±0.272
When 2 AIs disagree at final round
Disagreement rate
0.587±0.214
0.650±0.191
Non-DA AI final accuracy
0.858±0.175
0.759±0.234
Appendix
Table 5. Measurements for devil’s advocate treatments , analyzing behavior or non-DA AI and human’s accuracy. The metrics are computed for cases where the condition holds. ± shows 95% confidence intervals. No statistical tests were conducted.
Figure 13. Accuracy by round: (a): comparing MAD-Active - Base with baselines. (b): comparing all MAD treatments. The “Initial” accuracy uses users’ answers before AI assistance and the “Final” accuracy uses users’ final answers after AI assistance. “R1/R2/R3” accuracy uses users’ answers after interacting with the AI that round.
Task
Treatments
Initial → R1
R1 → Final
Comparison
Initial → Final
ReaLMistake
Static
N/A
N/A
N/A
0.123±0.195 **
Chatbot
0.112±0.216 **
0.034±0.180
—
0.146±0.220 **
MAD-Active - Base
0.156±0.193 **
0.021±0.101
p<0.001 ***
0.177±0.199 **
MAD-Active - Expert
0.102±0.163 **
0.075±0.19 *
—
0.177±0.211 **
MAD-Active - DA
0.043±0.15
0.040±0.189
—
0.083±0.211 **
MAD-Passive - Base
0.147±0.191 **
0.006±0.144
p<0.001 ***
0.154±0.226 **
Appendix
Table 6. Accuracy gains divided by discussion rounds for all treatments. Initial → R1 : accuracy gain from users’ initial decision to decision after first round of discussion. R1 → Final : accuracy gain from first round of discussion to final decision. Initial → Final : total accuracy gain from users’ initial decision to final decision. ± shows 95% confidence intervals. Accuracy gain significantly higher than 0 is denoted with *: p<0.05 , **: p<0.01 , ***: p<0.001 . Comparison shows significant differences between Initial → R1 and R1 → Final.
Figure 14. Is one round of discussion is enough for MAD? . Comparing MAD-Active against baselines Chatbot-R1 , Chatbot , and Static for (a) users’ final decision accuracy and (b) MAD-Active - Base users’ round 1 decision accuracy . Error bars are 95% confidence intervals. Significant differences between conditions are denoted with *: p<0.05 , **: p<0.01 , ***: p<0.001 .
Figure 17. Measurements for correlated AI errors. Two-way ANOVA on human engagement level ( MAD-Active vs. MAD-Passive ) and AI persona configuration ( Base , Expert , DA ). (a) AI resolve correlated-error cases : for correlated-error cases, whether the two AIs reach correct consensus. (b) User accuracy in correlated error cases . Error bars are 95% confidence intervals. Text in panels notes the significant main effects; brackets show significant effects between AI persona treatments or interactions (*: p<0.05 , **: p<0.01 , ***: p<0.001 ).
Figure 18. Users’ partisan bias: comparing MAD-Active - Base with baselines. (a): Δ accuracy is difference in accuracy between own-party and other-party claims. (b): Δ true response rate is difference in true-response-rate between own-party and other-party claims. (c): Partisan response rate is frequency to rate own-party claims as true and other-party claims as false. Error bars are 95% confidence intervals. Significant one-sample t -tests are indicated with below error bars (*: p<0.05 , **: p<0.01 , ***: p<0.001 ). Two-sample t -tests showed no significance between treatments.
Round
Filtering method
# claims pre-filter
# claims post-filter
1
Remove non-English claims & claims referencing images, videos, or social media posts
1702
1376
2
Remove claims according to our criteria, using LLM judge with Prompt
1376
424
3
Removed Snopes claims posted before 2025 or without post date, using LLM judge with Prompt
424
402
4
Retain only Democrat- and Republican-leaning claims, using crowdsourced annotations and difference in true-rate
402
113
5
Downsampled dataset based on 12 true Republican-leaning claims
113
48
Appendix
Table 7. Summary of the data processing steps for constructing our Fact-check dataset.
Figure 20. Example user interface for ReaLMistake, MAD-Passive - Expert treatment
Figure 21. Example user interface for Fact-check, MAD-Passive - DA treatment