More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs
Organizations: Department of Computer Science and Engineering, University of Nevada, Reno, NV 89557, USA
Abstract
Evaluations of multi-agent language models often compare communicating architectures against a single greedy pass. This practice confounds coordination with sample multiplicity and inference compute. We synthesize nine primary empirical evaluations comprising 516 unique multi-agent configurations organized along a resource vector R with six dimensions and a baseline ladder with four categories. Against baselines with a single pass (Category A), multi-agent accuracy differences are negative overall and near zero when omitting the largest contributing study. When evaluated against baselines matched on sample count (Category B) or resource budget (Category C), multi-agent systems average -0.042 in accuracy, and multilevel random effects modeling on the log odds scale remains negative across sensitivity checks. Multi-agent accuracy equals or falls below the baseline in 74% of these comparable contrasts, and study means are nonpositive in almost all evaluated studies. Across studies reporting both baselines, moving from a greedy pass to a sampled baseline shifts accuracy differences by -0.038 on average. Because communication adds no information when agents share identical evidence, positive returns require partitioned evidence, external verification, or error diversity across models. We provide a reviewer checklist and accounting utilities.
Figures & tables
| Study | Matching condition | Baseline(s) | Category | ||
| Tran and Kiela | Thinking tokens | SAS, equal budget | 96 | C (Budget-matched) | |
| Kim et al. | Iterations | Single, budget-matched | 18 | C (Budget-matched) | |
| Jwalapuram et al. | Cost audited | CoT; CoT-SC (5) | 107 | A (Greedy), B (Sampled) | |
| Choi et al. | Agent count | Single run; majority vote | 14 | / | A (Greedy), B (Sampled) |
| Wunderlich et al. | Comparable (Pareto) | CoT; SC (Pareto) | 2 | D (Pareto) | |
| Ke et al. | Unreported | CoT agent; SC agent | 29 | A (Greedy), B (Sampled) |
| Category | Set | Studies | Contrasts | Contrast mean | Study mean | MAS ahead |
| Cat. A (Single-pass) | with Zhang | 5 | 358 | 0.37 | ||
| Cat. A (Single-pass) | without Zhang | 4 | 178 | 0.48 | ||
| Cat. B (Sample-matched) | with Zhang | 6 | 360 | 0.24 | ||
| Cat. B (Sample-matched) | without Zhang | 5 | 220 | 0.27 | ||
| Cat. C (Budget-matched) | all | 2 | 114 | 0.32 | ||
| Cat. B/C (Combined) | with Zhang | 8 | 474 | 0.26 |
| Model | Studies/contrasts | SE | Wald 95% CI | CR2 95% CI | ||
| A9: Category A, single-pass | 5/358 | 0.087 | [ , ] | [ , ] ( ) | 90 / 9 | |
| : Category B, sampled | 6/360 | 0.056 | [ , ] | [ , ] ( ) | 95 / 4 | |
| B9: Category B/C, matched | 8/474 | 0.052 | [ , ] | [ , ] ( ) | 93 / 6 | |
| B9 Z: Category B/C without Zhang et al. | 7/334 | 0.039 | [ , ] | [ , ] ( ) | 86 / 4 | |
| D: Baseline-strength check (sampling gap) | 5/318 | 0.047 | [ , ] | [ , ] ( ) | 72 / 21 |
| Field | Content |
| Baseline rung | Category A, B, C or D of the single-agent baseline, the rule that assigns it, and the matched resource if the rung is C ( pace classify ). |
| Resource vector | with , , , , and measured for both systems; a dimension the log does not report is stated as unreported and is not estimated. |
| Token accounting | Prompt plus completion tokens of every call, including calls reused from another strategy, and whether cached prompt tokens are counted. |
| Resource ratios | Multi-agent over single-agent ratios of , and , and whether token parity holds within the stated tolerance ( pace audit ). |
| Frontier gap | Accuracy minus the self-consistency frontier at the system’s mean tokens, with the 95% paired bootstrap interval over items, the pool size and the number of items ( pace report ). |
| Evidence and routes | Shared or split evidence, whether an external verifier is available, whether the models are heterogeneous, the numbers of items and seeds, and the release of code and logs ( pace dynamics for the transition rates of debate logs). |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Specification | Studies/contrasts | SE | Wald 95% CI | CR2 95% CI | CR2 | |
| Panel A: Item-correlation sweep for B9 ( , ) | ||||||
| 8/474 | 0.055 | [ , ] | [ , ] | 7.1 | ||
| 8/474 | 0.054 | [ , ] | [ , ] | 7.1 | ||
| 8/474 | 0.052 | [ , ] | [ , ] | 7.2 | ||
| 8/474 | 0.051 | [ , ] | [ , ] | 7.3 | ||
| 8/474 | 0.050 | [ , ] | [ , ] | 7.4 | ||
| Study | Benchmark | Share | ||||
|---|---|---|---|---|---|---|
| Tran & Kiela | FRAMES | 48 | 0.450 | 0.454 | 0.31 | |
| MuSiQue (4-hop) | 48 | 0.341 | 0.323 | 0.35 | ||
| Kim et al. | SWE-bench Verified | 9 | 0.522 | 0.506 | 0.33 | |
| Terminal-Bench | 9 | 0.344 | 0.278 | 0.22 | ||
| Jwalapuram et al. | GPQA-Diamond | 23 | 0.718 | 0.728 | 0.52 | |
| HLE-Maths | 23 | 0.168 | 0.194 | 0.65 |
| Benchmark | Cost ratio vs CoT | Cost ratio vs SC | ||||||
| GPQA-Diamond | 23 | 0.718 | 0.738 | 0.728 | 7.8 | 1.1 | ||
| HLE-Maths | 23 | 0.168 | 0.194 | 0.194 | 8.5 | 1.6 | ||
| SWE-bench Lite | 17 | 0.310 | 0.357 | 0.246 | 7.6 | 0.9 | ||
| BrowseComp-Plus | 23 | 0.737 | 0.759 | 0.682 | 10.9 | 1.9 | ||
| SMFR | 21 | 0.292 | 0.351 | 0.329 | 6.7 | 1.7 | ||
| All | 107 | 0.455 | 0.489 | 0.448 | 8.1 | 1.6 |
| Framework | Share CoT | Share CoT-SC | |||
| DyLAN | 18 | 0.44 | 0.39 | ||
| MAS-Zero | 18 | 0.33 | 0.28 | ||
| AFlow | 18 | 0.33 | 0.06 | ||
| MaAS | 18 | 0.56 | 0.28 | ||
| ADAS | 18 | 0.28 | 0.17 | ||
| MAS-Orchestra | 14 | 0.50 | 0.21 |
| Paper (arXiv id) | Status | Reason |
| Ke et al., MAS-Orchestra (2601.14652) | Included | Table with accuracy for standalone agents and orchestration systems; test sizes stated. |
| Yang et al., Revisiting Multi-Agent Debate as Test-Time Scaling (2505.22960) | Included | Accuracy for self-consistency, refinement, and debate at equal maximum generations; AIME size derived as 60 (30 problems in each of two years); MATH500 stated as 500. |
| Zhang et al., Stop Overvaluing Multi-Agent Debate (2502.08788) | Included | Single-agent, CoT, self-consistency, and five debate frameworks on nine benchmarks; dataset sizes stated in the appendix; evaluated counts and AGIEval count not stated ( placeholder for AGIEval). |
| Smit et al., Should we be going MAD? (2311.17371) | Included | Best and median scores over hyperparameter configurations; dataset sizes stated for six benchmarks, evaluated counts not always confirmed. |
| Chu et al., MASBench (2610.04672) | Excluded | Reports only multi-agent settings with communication costs; no single-agent condition. |
| Parmar et al., Multi-Agent Debate for Data Cleaning (2606.02866) | Excluded | F1-type metrics on error detection, mainly one backbone; not item accuracy. |
| Source | Reported Pattern | Untested Coordination Hypothesis |
| Wunderlich et al. [12] | Debate ( ) and MoA ( ) exceed 10-sample SC ( ) at comparable compute read off Pareto fronts on MMLU-Pro ( ; Llama 3.1 70B). The MoA proposers are copies of one model (single-proposer setting), not different models. | Dense multiple choice knowledge verification and aggregation by a synthesizing layer may yield positive gains over sampling at comparable compute; the source reports no cross-model comparison in this setting. |
| ColMAD [ 37 ] | Collaborative debate outperforms standard competitive debate toward downstream judge accuracy at matched tokens. | Converting inter-agent debate from zero-sum competition into supportive critique aimed at a downstream judge may mitigate oversight errors under token budgets. |
| Zhu et al. [36] | Confidence-weighted aggregation and diverse initializations exceed majority voting on six QA tasks. | Preserving panelist diversity and weighting responses by calibrated confidence may avoid consensus collapse observed in homogeneous debate. |
| Zhang et al. [25] | Homogeneous debate methods show isolated positive cells (the AGIEval mean over four models and five methods is ) despite overall vs. CoT. A separate heterogeneous-model experiment in the source (Heter-MAD, its Table 4) reports gains on AGIEval that our extraction does not include. | Model diversity may supply complementary priors on selected tasks, though net aggregate differences across nine benchmarks remain negative ( vs. SC) for the homogeneous methods we extracted. |
| Expert MAS [ 11 ] | Reaches at N=58848.5%57.0%20.2%63.0%$ automated MAS. | Domain-specific external tooling and verifiers ( ) appear to drive accuracy gains when compared against unequipped single-agent baselines. |
| Arm | Mean | Median | MAS ahead | |
| Debate | 48 | 0.42 | ||
| Sequential | 48 | 0.25 | ||
| Subtask-parallel | 48 | 0.12 | ||
| Parallel roles | 48 | 0.40 | ||
| Ensemble | 48 | 0.19 | ||
| All five arms | 240 |
| Study (condition) | ||||
| Tran (matched thinking tokens) | 70 | 26 | ||
| Jwalapuram (unmatched cost) | 54 | 53 | ||
| Kim (matched iterations) | 9 | 9 | ||
| Choi (matched on agent count; MAD only) | 2 | 12 |
| Sample | Slope below knot | Slope above knot | Difference (above below) | |
| All four studies, study fixed effects | 235 | (0.09) | (0.09) | (0.17) |
| Excluding Tran at 100 thinking tokens | 219 | (0.09) | (0.09) | (0.17) |
| Tran and Kiela only | 96 | (0.05) | (0.07) | (0.11) |
| Jwalapuram et al. only | 107 | (0.09) | (0.12) | (0.20) |
| Kim et al. only | 18 | (0.36) | (0.40) | (0.73) |
| Choi et al. only | 14 | fewer than 3 contrasts on one side of the knot | ||
Multi-Agent LLM Systems
Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.
See all 1,119 papers