Multi-agent Scaling Across Disjunctive and Compensatory Tasks

Carolina Fortuna; Blaz Bertalanic

cs.AISep 25, 2026

Multi-agent Scaling Across Disjunctive and Compensatory Tasks

Authors: Carolina Fortuna, Blaz Bertalanic

Abstract

Multi-agent LLM systems are often expected to improve as team size increases, yet the scaling behavior may depend on task structure. Our central contribution is to introduce Steiner's taxonomy of group tasks as a framework for analyzing multi-agent LLM scaling and focusing the analysis on disjunctive and compensatory tasks. We model independently sampled agents as conditionally independent given the item, which yields their large-team limits: plurality voting converges to the model's modal answer, and averaging converges to the model's item-level bias. Across selected representative benchmarks, 13 open-weight models, and teams of up to 30 agents, we find qualitatively different scaling behavior. On disjunctive tasks, the probability that at least one agent is correct grows by 5-20 points with team size, but plurality voting over agents that answer directly realises almost none of this potential, as the model predicts to within 0.5 points on average. Multi-round revision raises accuracy considerably, yet the gain is nearly the same with one peer as with 29. In contrast, scaling provides little benefit on Fermi estimation, despite its natural suitability for aggregation: item-level biases shared across the samples of a model account for about 87% of the squared error, so averaging reduces error by only about 6%. Combining model families helps on Fermi estimation but does not surpass the strongest member on disjunctive tasks. These results show that task structure, together with the mechanism combining member outputs, is a fundamental determinant of team scaling.

Figures & tables

Task	Combination Operator	Output Domain	Top-Conference Benchmarks (2024–2026)	LLM Aggregation Mechanism
Dis- junc- tive	$Y=\max_{i}\{y_{i}\}$ (or plurality)	Discrete ( $\mathcal{Y}$ )	MATH-500, GSM8K, MMLU-Pro ( Wang et al., 2024 ) , GPQA ( Rein et al., 2024 ) , BigCodeBench ( Zhuo et al., 2025 ) , ARC	Plurality voting, verifier selection, multi-agent debate
Com- pen- satory	$Y=\left(\prod_{i=1}^{n}x_{i}\right)^{\frac{1}{n}}$ (or median)	Continuous ( $\mathbb{R}^{+},[0,1]$ )	ForecastBench ( Karger et al., 2025 ) , Guesstimate ( Chuang et al., 2025 ) , RealFP ( Kalyan et al., 2021 ) , SynthFP, SciOly	Geometric/arithmetic mean, median, calibrated Brier aggregation
Addi- tive	$Y=\sum_{i}y_{i}$ or $\bigcup_{i}S_{i}$	Cumulative sets	DAT ( Olson et al., 2021 ) , open-ended ideation, parallel instruction synthesis	Deduplicated set union, parallel greedy generation
Con- junc- tive	$Y=\min_{i}\{y_{i}\}$	Boolean / state chain	SWE-bench ( Jimenez et al., 2024 ) , TheAgentCompany ( Xu et al., 2026 ) , OSWorld ( Xie et al., 2024 ) , BrowseComp ( Kim et al., 2026 )	When modeled as strict sequential pipeline, multi-step dependency verification
Dis- cre- tionary	$Y=f_{\text{group}}(\{y_{i}\})$ Dynamic	Heteroge- neous	AgentBench ( Liu et al., 2024 ) , ChatEval ( Chan et al., 2024 ) , MoA ( Wang et al., 2025 )	Meta-agent router, judge arbitration, dynamic role assignment

Table 1: Categorization of tasks according to the Steiner taxonomy. Disjunctive and compensatory tasks are briefly discussed in this section; with additional details on all tasks in Appendix A .

Figure 1: Scaling with team size (average over 13 models; bands are 95% item-bootstrap intervals). The team answer is the plurality vote on the disjunctive tasks and the geometric mean on a representative compensatory RealFP (right), where accuracy means an estimate within one order of magnitude. Exact values are given in Appendix Table 4 .

Homogeneous Architectures (Multi-Turn Deliberation, Round 3)
Model	Solo	$\mathbf{N=5}$	$\mathbf{N=30}$	$\mathbf{\Delta_{5}}$	$\mathbf{\Delta_{\text{decay}}}$	$\bm{\rho}$	$\mathbf{N_{\text{eff}}}$	$\mathbf{N_{\text{eff}}}$
	Acc	Acc	Acc	Gain			(5)	(30)
smollm3-3b	32.43	45.09	45.61	$+12.66$ [+11.80, +13.55]	$+0.52$ [-0.11, +1.15]	0.67	1.36	1.47
qwen2.5-3b	27.08	32.32	32.58	$+5.23$ [+4.57, +5.88]	$+0.26$ [-0.31, +0.80]	0.82	1.16	1.20
qwen3-4b	42.13	51.21	52.71	$+9.09$ [+8.34, +9.87]	$+1.50$ [+1.05, +1.97]	0.86	1.13	1.16
mistral-7b	26.14	28.96	27.34	$+2.82$ [+2.34, +3.30]	$-1.62$ [-2.09, -1.19]	0.84	1.15	1.18

Table 2: Per-Model Scaling and Correlation Breakdown on Disjunctive Tasks (Round 3; accuracy in %, macro-averaged over the five benchmarks).

\Delta_{5}=\text{Acc}_{5}-\text{Solo}

and

\Delta_{\text{decay}}=\text{Acc}_{30}-\text{Acc}_{5}

, with paired 95% item-bootstrap intervals in brackets;

\rho

is the intra-class correlation of correctness. † Best five-model pool, selected and evaluated on disjoint halves of the items (mean over 10 splits; brackets give the range across splits); for pools,

\rho

is the mean correlation between members.

Model	Solo	$\mathbf{N=5}$	$\mathbf{N=30}$	Drop	Solo	$\mathbf{N=5}$	$\bm{\rho}$	$\mathbf{N_{\text{eff}}}$
	MAE	MAE	MAE	%	Acc	Acc		(5)
smollm3-3b	1.85	1.71	1.67	$+7.3$ [+4.7, +9.8]	31.9	33.1 [29.4, 36.8]	0.81	1.18
qwen2.5-3b	1.94	1.86	1.84	$+4.1$ [+2.0, +6.1]	28.6	26.1 [22.9, 29.5]	0.87	1.12
mistral-7b	2.08	1.91	1.91	$+8.1$ [+6.1, +11.0]	23.4	27.7 [23.6, 31.6]	0.87	1.12
olmo2-7b	1.80	1.70	1.68	$+5.5$ [+3.1, +7.3]	34.1	34.5 [31.3, 38.7]	0.87	1.11
qwen2.5-7b	1.80	1.72	1.71	$+3.9$ [+2.3, +5.6]	34.0	33.9 [30.1, 37.7]	0.90	1.08

Table 3: Per-Model Scaling Breakdown on Compensatory Fermi Estimation (RealFP, 529 items, Round 1, decomposition-prompt runs). MAE is the mean absolute log-error of the geometric mean (clipped at

\pm 3

); “Drop” is its relative reduction from

N=1

N=5

, with a paired 95% item-bootstrap interval;

\rho

is the intra-class correlation of signed log-errors. ‡ The five 7B–8B models of Section 3.3 , one answer each (

N=5

) or six each (

N=30

); “Solo” is their mean, and

\rho

their mean cross-model bias correlation.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Bench.	Team Size $\mathbf{N}$									95% CI
		2	3	5	7	10	15	20	25	30	half-width
Panel A: Round 1 (Independent Ensembles / Statistical Aggregation)
Disj.	arc	82.88	83.42	83.49	83.66	83.62	83.70	83.55	83.65	83.66	$\pm$ 1.3
Disj.	gsm8k	34.06	34.16	34.14	34.34	34.33	34.25	34.36	34.27	34.31	$\pm$ 1.2
Disj.	gsmhard	18.03	18.35	18.67	18.73	18.83	18.96	18.95	19.05	19.07	$\pm$ 1.1
Disj.	math500	30.80	31.13	31.31	31.59	31.74	31.93	32.07	32.05	32.08	$\pm$ 2.1

Appendix

Table 4: Empirical Scaling Across Disjunctive and Compensatory Tasks. Mean accuracy (%) across the 13-model panel, comparing Round 1 (independent answers, plurality vote or geometric mean), the Round-1 oracle success rate pass@

N

(at least one agent correct), and Round 3 (after two revision rounds) across team sizes

N=2

N=30

. The last column gives the largest half-width of the 95% item-bootstrap interval across team sizes in each row.

Figure 2: Observed Round-1 plurality accuracy against the accuracy predicted by resampling from the estimated item-level answer distributions, for 13 models, five tasks, and all team sizes (650 points). The dotted line is the identity. Marker shape and colour denote the task; each point is one model–task–team-size configuration.

Figure 3: Validation of the voting-gain bound (Eq. 3 ). Observed gain of the predicted large-team plurality limit over mean single-agent accuracy,

|\hat{\pi}-\bar{p}|

, against the bound

2\bar{p}(1-\bar{p})(1-\rho)

, for the 13 panel models and five disjunctive tasks (65 points). The dotted line is the identity; every point lies below it.

Task	$T$	Solo	R1, $N{=}30$	Gain	pass@30	$\bm{\rho}$	Distinct	R3, $N{=}30$
GSM-Hard	0.2	11.0	10.0	$-1.0$ [-2.0, -0.2]	13.5	0.92	3.3	27.0
	0.4	11.0	10.8	$-0.2$ [-1.5, +1.0]	18.2	0.85	6.2	28.0
	0.7	11.0	10.8	$-0.2$ [-2.0, +1.5]	19.8	0.79	10.0	27.3
	1.0	10.2	11.0	$+0.8$ [-1.0, +2.8]	22.8	0.75	13.0	29.0
MMLU-Hard	0.2	54.5	53.2	$-1.2$ [-3.0, +0.2]	55.5	0.98	1.0	58.0
	0.4	55.2	53.0	$-2.2$ [-4.2, -0.5]	58.2	0.95	1.1	57.0

Appendix

Table 5: Temperature sweep on disjunctive tasks (qwen2.5-7b, answer-first prompts, one run). Accuracy in %. “Gain” is Round-1 plurality accuracy at

N=30

minus single-agent accuracy, with a paired 95% item-bootstrap interval;

\rho

is the intra-class correlation of correctness; “Distinct” is the mean number of distinct answers among the 30 Round-1 answers to an item.

Bench-	R1	R1	$\mathbf{N^{*}}$	R3	R3	$\mathbf{\Delta_{\text{decay}}}$	$\mathbf{\Delta_{\text{delib}}}$
mark	$N=2$	$N=30$	Peak		$N=30$
arc	82.88	83.66	25	84.55	84.54	$-0.01$ [-0.13, +0.13]	$+1.66$ [+1.38, +1.95]
gsm8k	34.06	34.31	7	61.95	60.88	$-1.07$ [-1.39, -0.73]	$+27.89$ [+27.01, +28.79]
gsmhard	18.03	19.07	20	33.53	33.08	$-0.45$ [-0.69, -0.21]	$+15.50$ [+14.49, +16.52]
math500	30.80	32.08	15	39.90	39.13	$-0.77$ [-1.22, -0.33]	$+9.10$ [+8.08, +10.08]
mmlu_hard	52.18	52.94	5	55.18	54.81	$-0.37$ [-0.73, -0.01]	$+3.00$ [+2.42, +3.58]

Appendix

Table 6: Anatomy of the Disjunctive Deliberation Peak and Decay. Mean accuracy (%) across the 13-model panel: Round 1 at

N=2

and

N=30

, Round 3 optimal team size

N^{*}

, peak accuracy, accuracy at

N=30

, post-peak change

\Delta_{\text{decay}}=\text{Acc}_{N=30}-\text{Acc}_{N^{*}}

, and net deliberation gain at the peak

\Delta_{\text{delib}}=\text{Acc}_{N^{*}}-\text{Acc}_{\text{R1},N=2}

, each with a paired 95% item-bootstrap interval. Because

N^{*}

is selected as the maximum,

\Delta_{\text{decay}}

is biased downward.

Figure 4: Validation of the compensatory decomposition (Eq. 4 ). For each panel model, the mean squared error of the geometric mean of clipped log-errors predicted from item-level bias and within-item variance (line) and observed for single agents and actual Round-1 teams (points), together with the bias floor

\mathbb{E}_{k}[b_{k}^{2}]

(dotted). Decomposition-prompt runs. The

y

-axes do not start at zero, so the vertical scale exaggerates the decrease; the observed reduction in mean squared error from

N=1

N=30

is at most 13% for seven of the ten models and 15%, 17%, and 29% for llama3.1-8b, smollm3-3b, and marin-8b, respectively.

Clipping	MAE	MAE	MAE	Red.	Red.	$\beta$	$r$
	$\mathbf{N=1}$	$\mathbf{N=5}$	$\mathbf{N=30}$	1 $\to$ 5	1 $\to$ 30
$\pm 2$	1.41	1.32	1.30	6.9%	7.9%	0.86	0.61
$\pm 3$	1.84	1.73	1.71	6.1%	7.1%	0.87	0.63
$\pm 5$	2.41	2.28	2.26	5.5%	6.4%	0.88	0.65
none	3.82	3.68	3.66	3.7%	4.2%	0.94	0.84

Appendix

Table 7: Sensitivity of the RealFP Results to Clipping. Mean absolute log-error (MAE) of the geometric mean in Round 1 (single agents at

N=1

), its relative reduction from

N=1

N=5

and to

N=30

, bias share

\beta=\mathbb{E}[b_{k}^{2}]/(\mathbb{E}[b_{k}^{2}]+\mathbb{E}[\sigma_{k}^{2}])

, and mean pairwise cross-model correlation of item-level biases

r

, macro-averaged over the ten models with decomposition-prompt runs. The

\pm 3

row corresponds to the values reported in the main text.

	$T=0.4$			$T=0.8$			$T=1.0$
Model	$N{=}2$	$N{=}5$	$N{=}10$	$N{=}2$	$N{=}5$	$N{=}10$	$N{=}2$	$N{=}5$	$N{=}10$
qwen3-4b	34.3 $\pm$ 3.9	35.5 $\pm$ 3.7	35.3 $\pm$ 3.6	33.5 $\pm$ 3.6	34.7 $\pm$ 3.9	35.8 $\pm$ 3.8	33.4 $\pm$ 3.4	35.2 $\pm$ 3.8	36.2 $\pm$ 3.9
qwen3-8b	38.3 $\pm$ 4.0	38.4 $\pm$ 3.9	38.5 $\pm$ 4.1	39.4 $\pm$ 3.7	38.6 $\pm$ 3.9	38.7 $\pm$ 3.7	39.2 $\pm$ 3.9	38.8 $\pm$ 3.9	38.9 $\pm$ 4.3
qwen3-14b	38.1 $\pm$ 4.0	38.2 $\pm$ 4.2	38.3 $\pm$ 4.4	38.1 $\pm$ 3.7	37.6 $\pm$ 4.1	38.9 $\pm$ 4.2	38.9 $\pm$ 3.9	38.0 $\pm$ 4.0	38.4 $\pm$ 4.1
gpt-oss-20b	50.3 $\pm$ 3.9	55.3 $\pm$ 3.9	55.2 $\pm$ 4.0	49.9 $\pm$ 3.7	55.0 $\pm$ 3.9	55.4 $\pm$ 4.0	51.0 $\pm$ 3.8	56.1 $\pm$ 3.9	54.8 $\pm$ 4.2

Appendix

Table 8: Within-one-order-of-magnitude accuracy (%) on RealFP by temperature and team size (Round 1, median aggregation, 529 items, one to three runs per cell);

\pm

gives the half-width of the 95% item-bootstrap interval.

Model	Obs.	$\mathbf{\mathbb{E}[e_{i}]}$ (Mean $\mathbf{b}$ )	Median	$\bm{\sigma}$	Under	Over
		(OOM)	(OOM)	(Std)	(%)	(%)
smollm3-3b	1,058	$+0.48$ [+0.32, +0.66]	$+0.49$	2.10	41.2	57.9
qwen2.5-3b	1,058	$+0.65$ [+0.47, +0.84]	$+0.92$	2.12	36.1	63.5
mistral-7b	522	$+0.71$ [+0.53, +0.89]	$+1.21$	2.23	35.6	63.6
olmo2-7b	1,158	$+0.19$ [+0.02, +0.37]	$+0.28$	2.11	45.1	54.5
qwen2.5-7b	1,587	$+0.11$ [-0.05, +0.28]	$+0.02$	2.12	48.9	50.1

Appendix

Table 9: Intra-Model Bias and Distribution of Signed Errors. Mean signed error

b=\mathbb{E}[e_{i}]

(clipped at

\pm 3

; 95% item-cluster bootstrap interval), median signed error, standard deviation

\sigma

(clipped), and percentage of estimates under- and over-estimating the ground truth. Evaluated on single-agent estimates of the decomposition-prompt runs, sorted by parameter count.

	smollm3-3b	qwen2.5-3b	mistral-7b	olmo2-7b	qwen2.5-7b	llama3.1-8b	marin-8b	qwen3-8b	phi4-14b	qwen3-14b
smollm3-3b	1.00	0.69	0.55	0.66	0.64	0.64	0.65	0.62	0.58	0.64
qwen2.5-3b	0.69	1.00	0.50	0.68	0.67	0.64	0.60	0.64	0.59	0.64
mistral-7b	0.55	0.50	1.00	0.52	0.53	0.65	0.55	0.46	0.59	0.52
olmo2-7b	0.66	0.68	0.52	1.00	0.66	0.61	0.66	0.66	0.61	0.66
qwen2.5-7b	0.64	0.67	0.53	0.66	1.00	0.69	0.61	0.66	0.71	0.67
llama3.1-8b	0.64	0.64	0.65	0.61	0.69	1.00	0.61	0.60	0.68	0.67

Appendix

Table 10: Cross-Model Correlation of Item-Level Biases (

\mathbf{r}

). Pairwise Pearson correlation of item-level mean signed log-errors (clipped at

\pm 3

) across the ten models with decomposition-prompt runs, on the items with estimates from all models, sorted by parameter count. The mean off-diagonal correlation is 0.63 (95% item-bootstrap interval [0.60, 0.67]).

Intervention	Acc. $N{=}2$	Acc. $N{=}3$	Acc. $N{=}5$	Error $N{=}5$
Baseline (no intervention)	28.9	31.4	33.1 [29.3, 36.9]	3.41 [2.91, 3.96]
Decomposition anchors	28.0	30.4	33.1 [29.3, 37.1]	3.44 [2.98, 3.96]
Unit systems + logcalc	18.0	20.2	25.0 [21.4, 28.9]	4.52 [3.95, 5.15]
Logcalc	18.5	24.4	28.9 [22.2, 35.6] ( $n=180$ )	5.03 [3.81, 6.35]
Grounding + logcalc	21.0	19.7	21.1 [10.5, 31.6] ( $n=57$ )	4.35 [2.96, 6.42]
Distinct scaffolds	30.2	33.6	36.3 [32.1, 40.6]	3.31 [2.83, 3.88]

Appendix

Table 11: Heterogeneous-team interventions on RealFP (Round 1, median aggregation). Accuracy within one order of magnitude (%) and mean absolute

\log_{10}

error (unclipped, and therefore not comparable with the clipped values of Section 3.4 ; see Table 7 ) at

N=5

, with 95% item-bootstrap intervals at

N=5

. Where fewer than 529 items were completed,

n

is given.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,039 papers

May 31, 2026

The Ringelmann Effect in Multi-Agent LLM Systems: A Scaling Law for Effective Team Size

Inference-time multi-agent LLM scaling lacks a shared unit: counting nominal agents conflates cost with independent evidence. We derive a two-parameter scaling law

R(N) = N_\text{eff}/N = 1/(1+c(N-1)N^{-β})

where the regime exponent

β

classifies any configuration into one of three asymptotic regimes -- hard-ceiling at

1/c

(

β= 0

), sublinear at

N^β/c

(

0 < β< 1

), or linear (

β\ge 1

), and a mean-field theorem predicts that peer count

k

and rounds

τ

during agent debate enter the dynamics only through their product

kτ

. The law applies at two levels: answer diversity and correctness redundancy. Across 44 (model

\times

task

\times

condition) cells spanning peer debate, self-correction, random-noise placebo, self-consistency, three open-weight families (Qwen, Llama, Ministral) at scales from 7B to 32B with a frontier API check (Gemini), thinking models, heterogeneous teams, and sparse communication, the functional form fits every condition at

R^2 > 0.99

; only

(c, β)

shifts. On free-form math, dense peer influence collapses the answer-level regime from sublinear into hard-ceiling; correctness-level fits remain hard-ceiling throughout. Three findings have practical implications. (i)~Thirty dense debating agents produce no more answer diversity than one on MMLU-Hard. (ii)~A noise placebo tracks self-correction on free-form math and at

4\times

scale, so within homogeneous teams the gain commonly attributed to ``debate'' comes from re-evaluation, not peer content. (iii)~A single

N \le 5

pilot predicts the

N=30

structural ceiling, and within the configurations tested only architectural diversity (heterogeneous teams) lowers

c

and escapes the hard-ceiling regime, communication-mode interventions do not.

Blaž Bertalanič, Carolina Fortuna

Jozef Stefan Institute

Multi-Agent LLM Systems Multi-Agent Debate Multi-Agent Communication

Ask PDF

Mar 27, 2026

Scaling Teams or Scaling Time? Memory Enabled Lifelong Learning in LLM Multi-Agent Systems

Large language model (LLM) multi-agent systems can scale along two distinct dimensions: by increasing the number of agents and by improving through accumulated experience over time. Although prior work has studied these dimensions separately, their interaction under realistic cost constraints remains unclear. In this paper, we introduce a conceptual scaling view of multi-agent systems that jointly considers team size and lifelong learning ability, and we study how memory design shares this landscape. To this end, we propose LLMA-Mem, a lifelong memory framework for LLM multi-agent systems under flexible memory topologies. We evaluate LLMA-Mem on MultiAgentBench across coding, research, and database environments. Empirically, LLMA-Mem consistently improves long-horizon performance over baselines while reducing cost. Our analysis further reveals a non-monotonic scaling landscape: larger teams do not always produce better long-term performance, and smaller teams can outperform larger ones when memory better supports the reuse of experience. These findings position memory design as a practical path for scaling multi-agent systems more effectively and more efficiently over time.

Shanglin Wu, Yuyang Luo, Yueqing Liang +4

Emory University · Illinois Institute of Technology · University of Notre Dame +1

Multi-Agent LLM Systems Multi-Agent Systems Memory-Augmented Language Model Agents Lifelong Learning Agents

Ask PDF

Oct 20, 2026

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

Practitioner guidance holds that a large language model reviews a document better when the work is divided among agents. We test a narrow form of it in a controlled comparison of single-agent and multi-agent setups on a financial task, reviewing credit assessment memos against requirements. One agent checks six requirements in one call. A team of three on the same model checks the same memo, two requirements each, in the same words, with no roles or later stage. Labels are known for every requirement on 300 synthetic memos, so misses and false alarms are measured separately. Two primary predictions are tested on four models at a Bonferroni-corrected level. Under an instruction to decide each requirement once, the team catches 0.08 to 0.10 more of the defects that carry an explanation on the three models whose single agent misses some, at the corrected level on two. It raises more false alarms on all four, at the corrected level on three. Balanced accuracy, an exploratory measure, changes by 0.02 or less, and the team uses 2.2 to 2.4 times the tokens. Showing each agent the requirements it was not given reduces the added false alarms on the two Claude models, at the corrected level on one. In exploratory comparisons, sampling the single agent three times and flagging on any draw gives higher balanced accuracy than the team on those models, and the team's results depend on the instruction.

Pei Liu

School of Engineering and Applied Science, George Washington University, Washington, DC, USA

Multi-Agent LLM Systems LLM Agent Evaluation LLM Agent Efficiency Financial Language Model Evaluation

Ask PDF

Abstract

Multi-Agent LLM Systems

Explore similar work

The Ringelmann Effect in Multi-Agent LLM Systems: A Scaling Law for Effective Team Size

Scaling Teams or Scaling Time? Memory Enabled Lifelong Learning in LLM Multi-Agent Systems

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task