Borrowed Strength, Imported Risk: Capability Gaps and Reception Govern Heterogeneous LLM Debate
Organizations: ServiceNow
Abstract
Heterogeneous LLM debate is often attributed to collective reasoning that creates capability beyond any member. In the matched panels we study, the aggregate gain is dominated by capability transfer: a peer supplies task competence and receivers absorb a measurable fraction. Across nine matched directions, honest benefit and first-exchange adversarial exposure open when the peer has a positive same-task advantage, and honest gain is closely described by the empirical decomposition , the product of that advantage and the receiver's reception. Two fixed-receiver directions provide held-out validation without refitting: reception remains stable within uncertainty as a larger peer advantage produces a larger gain. A matched, single-direction open-answer MuSiQue scope probe reaches the negative-gap side of the same gate. Qwen-7B trails the Gemma-3-12B receivers by 10.78 points; despite 62.50% reception, it produces only +1.17 points of controlled first-exchange gain. Reception cannot substitute for positive aggregate peer advantage. When an influential peer is assigned an adversarial role, the same influential peer slot removes the borrowed gain and can push the panel below homogeneous debate. Across all five full-reference settings, the best observed majority correctness does not exceed the influential member's best observed accuracy, despite unused complementary answers. Heterogeneous debate is therefore a measurable transfer decision with a measurable failure mode, not a benefit conferred by diversity alone.
Figures & tables
| Direction ( ) | Task | Rep.-items | ||||
| Gemma-3-12B Qwen-7B | MATH-hard | 786 | 67.8 | 58.1 | 9.7 | 32.2 |
| Qwen-7B Llama-8B | GSM8K | 3957 | 91.9 | 87.1 | 4.8 | 8.1 |
| Gemini-Flash GPT-4.1 | MATH-hard | 786 | 79.4 | 75.9 | 3.5 | 20.6 |
| GPT-4.1 Gemini-Flash | MATH-hard | 786 | 75.9 | 79.4 | +3.5 | 24.1 |
| GPT-4.1 OSS-120B | MATH-hard | 786 | 75.9 | 80.2 | +4.3 | 24.1 |
| Llama-8B Qwen-7B | GSM8K | 3957 | 87.1 | 91.9 | +4.8 | 12.9 |
| Quantity (pp) | Ungated | Gated |
|---|---|---|
| Honest recovery | +7.33 (.250) | +26.94 (.031) |
| Honest net accuracy | +0.49 (.500) | +9.58 (.031) |
| Adv. corruption | +0.36 (.250) | +10.16 (.031) |
| Adv. net accuracy | +0.46 (.750) | 4.82 (.094) |
| Predictor | Slope | Exact | LOO slope | |
|---|---|---|---|---|
| 0.614 | 0.7760 | 0.0078 | [0.28, 0.76] | |
| 11.971 | 0.1455 | 0.3560 | [5.46, 15.39] | |
| 1.088 | 0.9950 | 0.0002 | [1.08, 1.12] |
| Qwen-7B peer | Qwen | Gemma | OSS-120B |
|---|---|---|---|
| (pp) | 0.00 | 9.12 | 21.50 |
| (%) | – | 62.16 | 60.35 |
| R1 gain (pp) | 0.00 | +7.89 | +16.92 |
| R4 gain (pp) | 0.00 | +11.05 | +21.37 |
| R1 recovery (%) | 2.92 | 25.26 | 42.94 |
| R1 honest corruption (%) | 2.06 | 4.72 | 2.21 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Recovery (%) | Corruption (%) | |||||
|---|---|---|---|---|---|---|
| Direction | Homog. | Honest | Advers. | Homog. | Honest | Advers. |
| Gemma-3-12B Qwen-7B | 3.0 | 7.7 | 8.7 | 1.1 | 2.0 | 1.6 |
| Qwen-7B Llama-8B | 3.9 | 17.7 | 13.1 | 0.1 | 0.6 | 0.4 |
| Gemini-Flash GPT-4.1 | 17.1 | 20.6 | 15.2 | 1.5 | 2.4 | 1.8 |
| GPT-4.1 Gemini-Flash | 26.6 | 41.8 | 30.7 | 8.8 | 6.5 | 10.2 |
| GPT-4.1 OSS-120B | 26.6 | 41.7 | 26.8 | 8.8 | 7.3 | 9.9 |
| Direction | (pp) | (%) | Bootstrap 95% CI | Unaided (%) | Uptake (%) | Gain (pp) | |
|---|---|---|---|---|---|---|---|
| Gemma-3-12B Qwen-7B | 9.7 | 14.7 | [2.6, 29.6] | 116 | 5.7 | 1.5 | +1.0 |
| Qwen-7B Llama-8B | 4.8 | 36.9 | [25.2, 49.7] | 244 | 5.6 | 0.5 | +0.6 |
| Gemini-Flash GPT-4.1 | 3.5 | 32.5 | [14.1, 51.6] | 114 | 13.9 | 1.8 | 0.1 |
| GPT-4.1 Gemini-Flash | +3.5 | 83.3 | [72.2, 93.7] | 144 | 14.0 | 9.8 | +4.8 |
| GPT-4.1 OSS-120B | +4.3 | 83.1 | [72.5, 92.7] | 154 | 13.3 | 9.9 | +4.8 |
| Llama-8B Qwen-7B | +4.8 | 78.5 | [72.4, 84.2] | 632 | 2.3 | 25.2 | +3.6 |
| Direction | (%) | Wilson 95% | Bootstrap 95% |
|---|---|---|---|
| Gemma-3-12B Qwen-7B | 14.7 | [9.4, 22.2] | [2.6, 29.6] |
| Qwen-7B Llama-8B | 36.9 | [31.1, 43.1] | [25.2, 49.7] |
| Gemini-Flash GPT-4.1 | 32.5 | [24.6, 41.5] | [14.1, 51.6] |
| GPT-4.1 Gemini-Flash | 83.3 | [76.4, 88.5] | [72.2, 93.7] |
| GPT-4.1 OSS-120B | 83.1 | [76.4, 88.2] | [72.5, 92.7] |
| Llama-8B Qwen-7B | 78.5 | [75.1, 81.5] | [72.4, 84.2] |
| Contrast | Mean (pp) | Bootstrap 95% CI | Exact | Odds ratio [95% CI] |
| Intent to treat (12,717 matched replicate-items) | ||||
| Recovery: honest vs. homogeneous | +20.40 | [+17.62, +23.28] | 0.0039 | 6.58 [2.24, 19.37] |
| Corruption: honest vs. homogeneous | 0.78 | [ 2.28, +0.74] | 0.3555 | 1.43 [0.61, 3.36] |
| Recovery: adversarial vs. honest | 15.25 | [ 17.94, 12.61] | 0.0156 | 0.29 [0.14, 0.62] |
| Corruption: adversarial vs. honest | +7.67 | [+5.98, +9.37] | 0.0312 | 5.17 [1.70, 15.72] |
| Per protocol (10,580 items; adversary parsed-wrong in all of R0–R4) | ||||
| Outcome | Specification | Term | OR | 95% CI | ||
|---|---|---|---|---|---|---|
| recovery | pair fe three condition | honest vs. homogeneous | 14264 | 6.582 | [2.237, 19.367] | 0.0038 |
| recovery | pair fe three condition | adversarial vs. homogeneous | 14264 | 1.914 | [1.058, 3.464] | 0.0356 |
| corruption | pair fe three condition | honest vs. homogeneous | 62038 | 1.430 | [0.609, 3.357] | 0.3625 |
| corruption | pair fe three condition | adversarial vs. homogeneous | 62038 | 7.345 | [1.097, 49.158] | 0.0419 |
| recovery | pair fixed effects | adversarial vs. honest | 9493 | 0.293 | [0.137, 0.624] | 0.0057 |
| corruption | pair fixed effects | adversarial vs. honest | 41375 | 5.170 | [1.700, 15.725] | 0.0093 |
| Direction | Rep.-items | Recv. | Peer | Mixed best | Controlled peak (pp) | Gap closed |
|---|---|---|---|---|---|---|
| Gemma-3-12B Qwen-7B | 715 | 72.0 | 63.2 | 73.2 (R2) | +2.0 | – |
| Qwen-7B Llama-8B | 3957 | 91.9 | 87.1 | 92.9 (R3) | +0.7 | – |
| Gemini-Flash GPT-4.1 | 784 | 79.5 | 75.9 | 82.2 (R1) | +0.7 | – |
| GPT-4.1 Gemini-Flash | 782 | 75.8 | 79.6 | 82.2 (R3) | +7.6 | 167% |
| GPT-4.1 OSS-120B | 784 | 75.9 | 80.1 | 81.3 (R4) | +6.4 | 127.5% |
| Llama-8B Qwen-7B | 3957 | 87.1 | 91.9 | 91.5 (R4) | +4.5 | 91% |
| Setting | Series | R0 | R1 | R2 | R3 | R4 |
|---|---|---|---|---|---|---|
| GPT-OSS-120B Llama-70B | Peer alone | 80.5 | 81.1 | 81.1 | 80.9 | 80.8 |
| Peer in panel | 80.4 | 79.4 | 79.4 | 78.1 | 77.3 | |
| Receivers | 44.3 | 70.7 | 72.1 | 72.0 | 71.1 | |
| Panel 2 | 45.2 | 71.8 | 72.8 | 72.7 | 71.6 | |
| Oracle | 82.5 | 83.6 | 82.9 | 81.7 | 80.3 | |
| GPT-OSS-120B GPT-4.1 | Peer alone | 80.2 | 80.8 | 80.8 | 80.7 | 80.6 |
| Peer (%) | Panel (%) | ||||||
|---|---|---|---|---|---|---|---|
| Setting | alone | in panel | receivers | 2 | oracle | Shortfall | |
| GPT-OSS-120B Llama-70B | 776 | 81.1 | 80.4 | 72.1 | 72.8 | 83.6 | 10.8 |
| GPT-OSS-120B GPT-4.1 | 786 | 80.8 | 81.4 | 81.4 | 81.4 | 87.2 | 5.7 |
| Gemini-2.5-Flash GPT-4.1 | 784 | 81.6 | 82.5 | 82.2 | 82.5 | 87.2 | 4.7 |
| GPT-4.1 Gemini-2.5-Flash | 784 | 75.9 | 82.7 | 82.2 | 82.7 | 87.4 | 4.7 |
| Gemma-2B Llama-1B | 100 | 61.0 | 61.0 | 43.5 | 48.0 | 71.0 | 23.0 |
| Setting | Panel | Peer | Panel-only | Peer-only | Net | New | Discarded | |
|---|---|---|---|---|---|---|---|---|
| GPT-OSS-120B Llama-70B | 776 | 71.65 | 77.32 | 2.96 | 8.63 | 5.67 | 2.58 | 13.40 |
| GPT-OSS-120B GPT-4.1 | 786 | 81.30 | 77.74 | 3.69 | 0.13 | +3.56 | 1.65 | 7.51 |
| Gemini-2.5-Flash GPT-4.1 | 784 | 82.27 | 80.36 | 3.19 | 1.28 | +1.91 | 1.91 | 6.89 |
| GPT-4.1 Gemini-2.5-Flash | 784 | 81.12 | 82.53 | 1.53 | 2.93 | 1.40 | 2.04 | 8.29 |
| Gemma-2B Llama-1B | 100 | 42.00 | 49.00 | 1.00 | 8.00 | 7.00 | 1.00 | 30.00 |
| Round | Pooled EM | 2 | Oracle | F1 |
|---|---|---|---|---|
| R0 | 45.67 | 45.67 | 47.33 | 58.90 |
| R1 | 48.11 | 48.33 | 49.00 | 61.86 |
| R2 | 48.00 | 48.00 | 49.33 | 61.27 |
| R3 | 48.00 | 48.33 | 48.67 | 61.00 |
| R4 | 47.44 | 47.33 | 48.33 | 60.57 |
| Quantity | Estimate | Bootstrap 95% CI | |
|---|---|---|---|
| Capability gap (pp) | 10.78 | [ 16.54, 4.79] | 300 |
| Honest controlled gain, R1 (pp) | +1.17 | [ 2.61, +4.95] | 300 |
| Honest recovery difference, R1 (pp) | +9.67 | [+4.78, +14.84] | 326 |
| Honest corruption difference, R1 (pp) | +9.25 | [+4.19, +14.55] | 271 |
| Adversarial-peer controlled gain, R1 (pp) | +1.00 | [ 1.89, +3.83] | 300 |
| Honest controlled gain, R4 (pp) | +2.33 | [ 2.08, +6.73] | 300 |
| Panel | Metric | R0 | R1 | R2 | R3 | R4 |
|---|---|---|---|---|---|---|
| Diverse config. | Pooled agent | 60.31 | 63.32 | 63.40 | 63.36 | 63.49 |
| At least two correct | 60.43 | 63.61 | 63.49 | 63.10 | 63.74 | |
| Any-agent oracle | 69.97 | 68.07 | 66.79 | 66.16 | 65.52 | |
| Uniform config. | Pooled agent | 67.81 | 68.07 | 67.13 | 66.45 | 66.71 |
| At least two correct | 67.81 | 67.94 | 67.18 | 66.54 | 66.79 | |
| Any-agent oracle | 67.81 | 68.45 | 67.30 | 66.79 | 66.92 |
| Association | Exact | |
|---|---|---|
| +0.973 | 0.0009 | |
| +0.729 | 0.0387 | |
| +0.607 | 0.0914 | |
| +0.549 | 0.1544 | |
| +0.113 | 0.7973 | |
| +0.611 | 0.1120 |
| Direction | Net | |||||||
|---|---|---|---|---|---|---|---|---|
| percentage points, except conditional rates and (%) | ||||||||
| Gemma-3-12B Qwen-7B | 9.7 | 7.4 | 14.7 | 1.08 | 16.4 | 5.8 | 0.95 | +0.13 |
| Qwen-7B Llama-8B | 4.8 | 3.1 | 36.9 | 1.14 | 8.6 | 5.7 | 0.49 | +0.64 |
| Gemini-Flash GPT-4.1 | 3.5 | 7.2 | 32.5 | 2.35 | 10.2 | 7.5 | 0.76 | +1.59 |
| GPT-4.1 Gemini-Flash | +3.5 | 9.2 | 83.3 | 7.63 | 6.5 | 24.3 | 1.59 | +6.04 |
| GPT-4.1 OSS-120B | +4.3 | 9.8 | 83.1 | 8.14 | 6.3 | 34.3 | 2.16 | +5.98 |
| Predictor | Slope | Intercept | Max. residual | |
|---|---|---|---|---|
| 1.088 | +0.61 | 0.9950 | 1.07 | |
| 1.072 | +0.94 | 0.9898 | 1.51 | |
| 1.088 | 1.79 | 0.9680 | 2.30 | |
| 1.027 | +0.08 | 0.9671 | 3.38 |
Multi-Agent Debate
Multi-Agent Debate studies how language-model agents exchange arguments, critique answers, and combine judgments for collective reasoning.
See all 156 papers