Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models
Organizations: Boston Children’s Hospital, Harvard Medical School · Harvard School of Engineering and Applied Sciences, Harvard College
Abstract
Multi-agent debate (MAD) reportedly improves reasoning and factuality over single-model inference, but prior work treats agents as symmetric peers, leaving open what drives the gains. We test the hypothesis that cognitive diversity among agents is the driver, in the setting where the question is still measurable: small open-weight models with benchmark headroom. Across 23 models from eleven vendor families, five tasks, and 5,500+ debate and control runs, we vary diversity along three axes - personas, sampling temperature, and model identity - pairing every debate configuration with a generation-budget-matched majority-vote control. The hypothesis is rejected on every axis. Debate beats single-agent inference (3--7 points where tasks have headroom) but at matched budget conditions it ties or even loses to self-consistency sampling at 1.6 the wall-clock and 3.4 the token cost. Persona prompting reduces accuracy and a dose-response experiment over each model's full combinatorial persona space shows the cost is a persona tax, not a diversity tax: redundant personas hurt most, while maximally-diverse teams recover part of the loss. Furthermore, mixed-model teams lose to majority votes over their own rosters, with accuracy tracking member capability rather than heterogeneity, and nearly all of debate's benefit comes from the first exchange of answers. We further identify a pervasive measurement hazard in which debate transcripts silently overflow serving context windows, whose correction alone moves our debate-versus-sampling comparison from points to parity. Our results recast reported MAD gains as an ensemble-sampling effect and provide the budget-matched, contamination-checked baseline bar that future debate mechanisms should be required to clear.
Figures & tables
| Task | Solo | Debate | SC-9 | Persona |
|---|---|---|---|---|
| Math | 91.2 | 91.5 | 90.6 | 90.0 |
| GSM8K | 87.9 | 91.2 ∗ | 91.0 ∗ | 88.5 |
| MMLU | 69.9 | 76.5 ∗ | 76.8 ∗ | 75.2 ∗ |
| CSQA | 72.6 | 78.3 ∗ | 80.6 ∗ | 73.9 |
| Biography † | 54.6 | 55.6 | 64.6 ∗ | 51.8 |
| Mean | 80.4 | 84.4 | 84.8 | 81.9 |
| Arm | Teams | Debate | Ensemble | (pp) [95% CI] |
|---|---|---|---|---|
| Core triple ( ) | 80 | 88.4 | 89.4 | [ , ] ∗ |
| Pair, strong dup. ( ) | 15 | 83.0 | 85.7 | [ , ] ∗ |
| Pair, weak dup. ( ) | 15 | 79.8 | 81.4 | [ , ] |
| Quad, strong dup. ( ) | 20 | 91.9 | 91.8 | [ , ] |
| Quad, weak dup. ( ) | 20 | 90.5 | 90.0 | [ , ] |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Size | Class | Notes |
| Qwen3-0.6B | 0.6B | Qwen3 | |
| LFM2.5-1.2B | 1.2B | LFM2.5 | hybrid conv/attn |
| VibeThinker-1.5B | 1.5B | VibeThinker | reasoning FT |
| Llama-3.2-3B-Instruct | 3B | Llama-3.2 | |
| Phi-4-mini | 3.8B | Phi-4 | synthetic-data |
| Nemotron-3-Nano-4B | 4B | Nemotron-3 | Mamba hybrid |
| Family | Role | Pool | Tasks | Runs | Seed |
| Homogeneous baseline (solo + debate ) | core | 23 | 5 | 460 | 42 |
| MaxDet persona debate ( ) | core | 23 | 5 | 345 | 42 |
| Self-consistency, budget-matched ( , uniform + spread) | control | 23 | 5 | 690 | 42 |
| Seed-43 homogeneous slice (solo + debate ) | ablation | 19 | 5 | 190 | 43 |
| Mixed-model core triples ( , 80 teams) | core | 19 | 5 | 400 | 42 |
| Mixed-model pairs (AAB/ABB, 15 pairs) | core | 19 | 5 | 150 | 42 |
| Debate arm | Temp. policy | Budget | Matched control | Temp. policy |
|---|---|---|---|---|
| Homog. , | spread – | 9 | SC-9 (two flavors) | / spread |
| Homog. , | spread – | 15 | SC-15 (two flavors) | / spread |
| Homog. , | spread – | 21 | SC-21 (two flavors) | / spread |
| Persona , | fixed | SC- uniform | ||
| Persona rungs 0–10, | fixed | 9 | no-persona debate | spread – |
| Mixed triple , | spread – | 9 | roster ens. , |
| Model | Solo | Debate | SC-9 | Persona |
|---|---|---|---|---|
| LFM2.5-1.2B | 100.0 | 100.0 | 99.0 | 99.0 |
| LFM2.5-8B-A1B | 100.0 | 100.0 | 100.0 | 97.0 |
| Llama-3.2-3B | 35.0 | 37.0 | 29.0 | 39.0 |
| Llama-4-Scout-17B-16E (W4A16) | 100.0 | 100.0 | 100.0 | 100.0 |
| Ministral-3-8B | 98.0 | 96.0 | 99.0 | 54.0 |
| Mistral-7B-v0.3 | 7.0 | 9.0 | 18.0 | 9.0 |
| Model | Solo | Debate | SC-9 | Persona |
|---|---|---|---|---|
| LFM2.5-1.2B | 70.0 | 74.0 | 75.0 | 70.0 |
| LFM2.5-8B-A1B | 89.0 | 96.0 | 99.0 | 89.0 |
| Llama-3.2-3B | 46.0 | 78.0 | 85.0 | 60.0 |
| Llama-4-Scout-17B-16E (W4A16) | 96.0 | 95.0 | 96.0 | 94.0 |
| Ministral-3-8B | 91.0 | 94.0 | 95.0 | 88.0 |
| Mistral-7B-v0.3 | 43.0 | 55.0 | 40.0 | 43.0 |
| Model | Solo | Debate | SC-9 | Persona |
|---|---|---|---|---|
| LFM2.5-1.2B | 45.0 | 55.0 | 57.0 | 57.0 |
| LFM2.5-8B-A1B | 61.0 | 80.0 | 79.0 | 74.0 |
| Llama-3.2-3B | 17.0 | 49.0 | 45.0 | 42.0 |
| Llama-4-Scout-17B-16E (W4A16) | 83.0 | 84.0 | 83.0 | 87.0 |
| Ministral-3-8B | 64.0 | 79.0 | 78.0 | 68.0 |
| Mistral-7B-v0.3 | 40.0 | 60.0 | 59.0 | 45.0 |
| Model | Solo | Debate | SC-9 | Persona |
|---|---|---|---|---|
| LFM2.5-1.2B | 55.0 | 62.0 | 64.0 | 61.0 |
| LFM2.5-8B-A1B | 71.0 | 80.0 | 81.0 | 75.0 |
| Llama-3.2-3B | 26.0 | 60.0 | 69.0 | 53.0 |
| Llama-4-Scout-17B-16E (W4A16) | 85.0 | 89.0 | 89.0 | 91.0 |
| Ministral-3-8B | 61.0 | 72.0 | 76.0 | 39.0 |
| Mistral-7B-v0.3 | 49.0 | 69.0 | 73.0 | 62.0 |
| Arm | Math | GSM8K | MMLU | CSQA |
|---|---|---|---|---|
| Core triple ( ) | ∗ | ∗ | ∗ | |
| Pair, strong dup. ( ) | ∗ | |||
| Pair, weak dup. ( ) | ||||
| Quad, strong dup. ( ) | ||||
| Quad, weak dup. ( ) | ∗ | ∗ |
| Team family | Teams | vs. majority | vs. acc.-weighted | vs. log-odds (binary) | vs. log-odds (multiclass) |
|---|---|---|---|---|---|
| Core triples, seed 42 | 80 | [ , ] ∗ | [ , ] ∗ | [ , ] | [ , ] ∗ |
| Core triples, seed 43 | 80 | [ , ] ∗ | [ , ] ∗ | [ , ] | [ , ] ∗ |
| Pairs (AAB/ABB) | 30 | [ , ] ∗ | [ , ] ∗ | [ , ] | [ , ] |
| Quads (dup-s/w) | 40 | [ , ] | [ , ] | [ , ] | [ , ] ∗ |
| Composition | Debate | Ensemble | (pp) |
|---|---|---|---|
| SSS | 93.7 | 93.2 | ∗ |
| SSM | 92.5 | 92.8 | |
| SMM | 90.3 | 91.5 | |
| MMM | 87.0 | 88.2 | ∗ |
| SSW | 91.7 | 92.2 | |
| SMW | 90.6 | 90.9 |
| Task | |||||
|---|---|---|---|---|---|
| Math | 93.5 | 98.3 | 99.2 ∗ | 99.2 | 99.2 |
| GSM8K | 91.7 | 93.2 | 94.0 | 94.3 | 94.5 |
| MMLU | 74.1 | 76.4 | 76.8 | 77.0 ∗ | 75.6 |
| CSQA | 79.2 | 81.0 ∗ | 79.6 | 79.7 | 78.6 |
| Mean | 84.6 | 87.2 ∗ | 87.4 ∗ | 87.5 ∗ | 87.0 ∗ |
| Model | Solo | Debate | SC-9 | Persona | Verif. claims |
|---|---|---|---|---|---|
| LFM2.5-1.2B | 87.3 | 86.5 | 86.3 | 92.6 | 224 |
| LFM2.5-8B-A1B | 40.8 | 42.9 | 44.1 | 32.5 | 340 |
| Llama-3.2-3B | 30.0 | 28.4 | 28.0 | 27.1 | 389 |
| Llama-4-Scout-17B-16E-Ins | 69.9 | 62.0 | 69.0 | 58.7 | 490 |
| Ministral-3-8B-2 | 31.4 | 55.9 | 64.7 | 16.7 | 865 |
| Mistral-7B-v0.3 | 38.7 | 40.9 | 52.1 | 26.1 | 464 |
| Claims direction | Facts direction | |||||
|---|---|---|---|---|---|---|
| Judge | Units | Strict | Verif. | Units | Strict | Verif. |
| Gemma-3-27B | 6.98M | 18.4 | 65.2 | 1.91M | 39.8 | 78.9 |
| GPT-OSS-20B | 6.95M | 11.2 | 58.6 | 1.90M | 9.7 | 57.9 |
| GPT-OSS-120B | 6.98M | 11.1 | 61.6 | 1.91M | 9.9 | 52.2 |
| Muse Glimmer | 3.30M | 10.3 | 42.2 | 0.96M | 12.7 | 42.1 |
| Gemma-3-27B | GPT-OSS-20B | GPT-OSS-120B | Muse Glimmer | |
|---|---|---|---|---|
| Gemma-3-27B | — | 0.342 | 0.372 | 0.317 |
| GPT-OSS-20B | 0.563 | — | 0.762 | 0.440 |
| GPT-OSS-120B | 0.568 | 0.759 | — | 0.474 |
| Muse Glimmer | 0.354 | 0.414 | 0.412 | — |
| MMLU-Pro | GSM-Symbolic | |||||
|---|---|---|---|---|---|---|
| Model | Solo | Debate | SC-9 | Solo | Debate | SC-9 |
| gemma-4-12B-it | 74.0 | 77.0 | 78.0 | 96.0 | 95.0 | 96.0 |
| gemma-4-26B-A4B-it-FP8-dy | 82.0 | 85.0 | 83.0 | 98.0 | 97.0 | 98.0 |
| granite-4.1-8b | 43.0 | 56.0 | 59.0 | 95.0 | 96.0 | 97.0 |
| LFM2.5-1.2B | 32.0 | 35.0 | 35.0 | 77.0 | 83.0 | 74.0 |
| LFM2.5-8B-A1B | 40.0 | 60.0 | 65.0 | 93.0 | 94.0 | 94.0 |
| wall-clock (min) | debate/SC-9 ratio | ||||||
|---|---|---|---|---|---|---|---|
| Task | Solo | Debate | SC-9 | Persona | clock | gen. tokens | all tokens |
| Math | 4 | 11 | 8 | 14 | 1.34 | 1.18 | 2.34 |
| GSM8K | 6 | 23 | 18 | 28 | 1.32 | 0.84 | 3.36 |
| MMLU | 7 | 27 | 16 | 34 | 1.74 | 0.93 | 3.51 |
| CSQA | 5 | 21 | 12 | 30 | 1.82 | 1.10 | 3.80 |
| Biography | 4 | 17 | 9 | 19 | 1.95 | 1.26 | 4.09 |
| Model | combo | ladder | rung-0 pctl | MaxDet pctl |
|---|---|---|---|---|
| gemma-4-12B-it | 0.814 | 0.794 | 0.18 | 99.86 |
| gemma-4-26B-A4B-it-FP8-dyn | 0.835 | 0.964 | 0.12 | 99.96 |
| granite-4.1-8b | 0.895 | 0.964 | 0.39 | 99.98 |
| LFM2.5-1.2B-Instruct | 0.923 | 0.988 | 0.05 | 99.82 |
| LFM2.5-8B-A1B | 0.877 | 0.818 | 0.05 | 100.00 |
| Llama-3.2-3B-Instruct | 0.893 | 0.879 | 0.45 | 99.99 |
| Callsign | Description |
|---|---|
| cryptographer | a nihilistic cryptographer who only trusts solutions verifiable by zero-knowledge proofs |
| baroque | a Baroque music theorist fixated on harmonic counterpoint and structural symmetry |
| meteorologist | a chaotic systems meteorologist who views all certainty as a transient statistical anomaly |
| xenolinguist | a hard-science fiction xenolinguist obsessed with logical consistency and species-specific syntax |
| cartographer | a medieval cartographer whose primary concern is establishing clear boundaries and known territories |
| neuropharm | a neuropharmacologist who analyzes all input as complex patterns of reward and aversion chemicals |