An Exact Generate - Transform Decomposition of Small-LLM Team Scaling Across Orchestration Architectures

Blaz Bertalanic; Carolina Fortuna

cs.AISep 28, 2026

An Exact Generate - Transform Decomposition of Small-LLM Team Scaling Across Orchestration Architectures

Authors: Blaz Bertalanic, Carolina Fortuna

Organizations: Jožef Stefan Institute, Ljubljana, Slovenia

Abstract

Replacing one LLM agent with a collaborating team can raise accuracy, but whether scaling the team helps, and which architecture to scale, is unclear. Sweeping eight agent orchestration architectures across five instruction-tuned 7-9B models, five short-answer benchmarks, and an executable-code benchmark up to 30 calls, we find that the returns to team scaling are sharply task-dependent: from three to thirty calls accuracy rises by up to 17 points on the two arithmetic word-problem benchmarks (GSM8K, GSMHard) but by at most four on ARC, GPQA, and MMLU, for every architecture, a split the usual task-averaged number conceals. Proposer-Critic captures the arithmetic gains, scaling steepest and, in aggregate, surpassing every other architecture at the largest budget (item-clustered intervals exclude zero), though it ranks among the weakest elsewhere, and no architecture wins across tasks. We explain these trajectories with an exact generate-transform decomposition. Partitioning any workflow into proposal coverage and a downstream transform, any accuracy change splits exactly into an extensive coverage dividend and an intensive transformation change. The decomposition diagnoses each task: arithmetic offers coverage headroom that a critic-guided transform converts, whereas the multiple-choice benchmarks either saturate in coverage or fail to convert it, and on open-ended code generative recovery nearly vanishes so accuracy tracks coverage. At equal call budgets token cost still varies 2.1x. Extra calls therefore create candidate opportunity that only some architectures, on some tasks, convert. Team scaling is a task- and architecture-specific bet, not a uniform lever.

Figures & tables

Figure 1: The eight orchestration architectures, nodes colored by functional stage: generate (the

L

proposals), transform (critics, refiners, synthesizers, duelists), the manager, and Diamond’s non-answering planner. Tournament selects through pairwise duels, Tree merges via fan-in-three synthesis, and Persona-Star shares Star’s graph but differs in prompts. Counts are illustrative.

Figure 2: Where team scaling pays off. (a) The scaling gain

\Delta Y

(the

N=3

N=30

accuracy change, Section 3.3 ) for every architecture on every benchmark (equal-cell over five models), grouped by task family, with Proposer-Critic marked. Returns concentrate on arithmetic and are near-zero on ARC, GPQA, and MMLU for all architectures. (b) Arithmetic accuracy versus team size: Proposer-Critic (red) starts mid-field and separates with scale.

Figure 3: Accuracy gain over a single agent (the mean individual worker), by task category: (a) arithmetic, (b) multiple-choice, (c) code, each cell an equal mean over the category’s model–task cells with per-panel color scales. Gains on arithmetic (25 to 40 points) are two to three times those on multiple-choice (8 to 14) or code, and Proposer-Critic tops arithmetic at $N=30$ while trailing on multiple-choice. (d) QA tokens per problem versus team size: cost climbs at architecture-dependent rates to a 2.1 $\times$ spread by $N=30$ . Diamond is undefined at $N=2$ .

Figure 4: Two-margin decomposition of the

N=3\to 30

accuracy change by task category: (a) arithmetic, (b) multiple-choice, (c) code. Each architecture’s change splits into an extensive coverage dividend

\mathcal{E}

(blue) and an intensive transformation change

\mathcal{I}

(red, extending left when the transform sheds coverage), summing to the net

\Delta Y

(diamond), equal-cell over each category’s model–task cells. Coverage is added in every category, but the transform converts it only on arithmetic, where Proposer-Critic alone posts a positive

\mathcal{I}

and the largest net gain. On multiple-choice every proposal-expanding design sheds most of its coverage and nets almost nothing, and on code the same shedding, with recovery near zero, leaves the net close to the coverage floor.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture	Actual calls	$L$	Other non-final roles	Manager input	Tiers
Star	30	29	none	29 worker reports	2
Persona-Star	30	29	none	29 persona-worker reports	2
Proposer-Critic	30	15	14 critics	14 critics + 1 free proposal	3
Tournament	29	15	13 non-final duelists	2 finalists	5
Tree	30	22	7 fan-in-three synthesizers	7 syntheses + 1 orphan	3
Chain	30	1	28 one-parent refiners	all 29 preceding reports	30

Appendix

Table S1: Exact architecture composition at requested budget

N=30

L

is the first answer-producing tier used for proposal metrics. Tournament rounds down to the largest pairwise bracket within the requested budget.

Architecture	$\mathcal{E}$	$\bar{O}\Delta s$	$(1-\bar{O})\Delta g$	$\mathcal{I}$	$\Delta Y$	$\Delta Y>0$	$\mathcal{E}>0$	$\mathcal{I}<0$
Star	+7.28	-2.65	-2.23	-4.88	+2.40	17/25	25/25	22/25
Persona-Star	+11.32	-5.14	-1.80	-6.94	+4.39	20/25	25/25	23/25
Prop.-Critic	+6.68	-0.67	+1.21	+0.54	+7.22	21/25	25/25	18/25
Tournament	+5.72	-3.12	-1.14	-4.26	+1.46	18/25	25/25	22/25
Tree	+6.55	-2.46	-0.56	-3.02	+3.53	20/25	25/25	21/25
Chain	+0.00	-0.18	+4.17	+3.99	+3.99	24/25	14/25	2/25

Appendix

Table S2: Exact two-margin decomposition of QA scaling from requested

N=3

N=30

, in equal-cell percentage points. The intensive term is split into changes in covered-case success and recovery. Every row satisfies

\Delta Y=\mathcal{E}+\mathcal{I}

before rounding. The last three columns count signs across 25 model–task cells.

Architecture	$\pi_{11}$	$Y^{11}_{3}$	$Y^{11}_{30}$	$\Delta Y_{11}$	$\pi_{01}$	$c_{01}$	$c_{11}$	$\Delta Y$
Star	48	85.4	88.6	+3.19	13	+1.48	+1.00	+2.40
Persona-Star	49	83.7	88.8	+5.10	20	+2.20	+1.55	+4.39
Prop.-Critic	44	79.5	86.2	+6.73	15	+2.50	+1.42	+7.22
Tournament	48	86.0	85.0	-0.94	11	+1.31	-0.22	+1.46
Tree	48	85.8	88.0	+2.23	12	+1.55	+0.88	+3.53
Diamond	45	83.6	83.6	-0.01	26	+4.45	-0.26	+4.39

Appendix

Table S3: Paired coverage-transition decomposition of QA scaling from

N=3

N=30

, equal-cell means over 25 model–task cells. Each trial is paired across the two budgets by (task, run, item) and stratified by its coverage transition

(O_{3},O_{30})

; match rate is 100%.

\pi_{11}

is the share of trials covered at both budgets and

\pi_{01}

the newly-covered share (percent).

Y^{11}_{3}

and

Y^{11}_{30}

are final accuracies on the both-covered stratum and

\Delta Y_{11}

their difference;

c_{01}

and

c_{11}

are the strata contributions to the total

\Delta Y

(points). The both-covered stratum holds item difficulty fixed, so a non-negative

\Delta Y_{11}

argues against a same-item transform decline: accuracy there rises for the four proposal-expanding designs Star, Persona-Star, Proposer-Critic, and Tree, is flat for Diamond, and falls only for Tournament. The negative intensive term of Table S2 is therefore consistent with the weak conversion of harder newly-covered items rather than the transform discarding answers it previously returned.

Architecture	$O$	Loss	$P(Y{=}0\mid O{=}1)$	Recovery	$P(Y{=}1\mid O{=}0)$	Net	$Y$
Star	61.5	10.8	19.6	5.7	12.6	-5.1	56.4
Persona-Star	69.7	16.7	25.5	4.8	11.7	-11.9	57.8
Prop.-Critic	59.1	11.4	20.2	11.3	27.0	0.0	59.1
Tournament	59.2	10.9	21.8	7.1	15.2	-3.9	55.4
Tree	60.6	10.5	19.9	7.2	15.1	-3.3	57.3
Chain	44.6	3.3	10.5	17.0	29.2	+13.8	58.4

Appendix

Table S4: Proposal-to-team transfer at

N=30

on QA (equal mean over 25 model–task cells).

O

and

Y

are proposal coverage and final accuracy. Loss and recovery are unconditional masses needed by the accounting identity; the conditional columns measure downstream behavior given the corresponding opportunity. Net is recovery minus loss.

Architecture	$O$	$s$	$g$	$v$	$O^{\star}$	$Y$	$Y-O$
Star	61.5	82.4	14.9	0.675	45.8	56.4	-5.1
Persona-Star	69.7	76.0	15.9	0.601	39.9	57.8	-11.9
Prop.-Critic	59.1	80.8	27.7	0.531	59.0	59.1	-0.0
Tournament	59.2	81.6	17.3	0.642	48.4	55.4	-3.9
Tree	60.6	82.7	18.2	0.645	51.3	57.3	-3.3
Chain	44.6	92.7	30.7	0.619	80.7	58.4	+13.8

Appendix

Table S5: Response-law boundary at

N=30

. Rates pool the equal-cell joint masses, so

Y=g+(s-g)O

holds exactly for every row. A generative transformer finishes above its proposal oracle exactly when

O<O^{\star}=g/(1-s+g)

. All values are percentages except proposal leverage

v=s-g

Calibration $B$	Comparable cells	Agree	Agreement
3	196	149	76.0%
5	199	177	88.9%
7	199	185	93.0%
10	198	190	96.0%

Appendix

Table S6: Held-out-budget stability of the oracle-crossing label. The label

O<O^{\star}

(final accuracy above proposal coverage) at calibration budget

B

predicts the corresponding label at

N=30

. Cells tied at either endpoint are excluded. Because the boundary exactly encodes the sign of

Y-O

at each budget, this evaluates label persistence rather than an independently fitted classifier.

Arithmetic (GSM8K, GSMHard)
Architecture	Lla.	Min.	Nem.	Q2.5	Q3
Star	45.7	66.0	58.4	36.4	50.4
Persona-Star	45.0	69.0	61.3	37.1	56.7
Prop.-Critic	52.4	64.0	64.2	51.4	65.5
Tournament	28.3	62.0	58.6	43.0	54.6
Tree	41.5	65.8	61.8	46.1	52.4

Appendix

Table S7: Team accuracy (%) at

N=30

, equal-cell mean over each category’s tasks, by model. On arithmetic the architecture spread is large (mean 14 points) and Proposer-Critic leads four of five models, whereas on multiple-choice every design is within a few points (mean spread 4). Single is the mean individual worker, one non-thinking call. Bold marks the best architecture within a model and category.

Architecture	Lla.	Min.	Nem.	Q2.5	Q3
Star	58.9	66.1	84.3	69.3	78.3
Persona-Star	49.8	76.8	84.3	72.8	79.7
Prop.-Critic	54.3	59.8	84.6	70.9	77.6
Tournament	54.3	75.6	84.6	72.8	79.7
Tree	59.1	68.3	84.6	73.4	79.5
Chain	47.0	72.2	81.5	72.8	77.0

Appendix

Table S8: HumanEval+ pass rate (%) at

N=30

by architecture and model, with the mean individual-worker one-call baseline (Single). Bold is the best architecture within a model.

$N$	PC	runner-up	$\Delta$	95% CI
2	44.6	Tree	-0.2	$[-0.7,+0.2]$
3	44.9	Chain	-2.3 †	$[-2.9,-1.7]$
5	49.0	Chain	-0.0	$[-0.6,+0.6]$
7	51.7	Persona-Star	+0.5	$[-0.1,+1.1]$
10	55.1	Persona-Star	+2.8 ∗	$[+2.2,+3.5]$
15	56.1	Persona-Star	+2.7 ∗	$[+2.0,+3.4]$

Appendix

Table S9: Task-type specialization with scale on the two arithmetic word-problem benchmarks (GSM8K, GSMHard), equal-cell over 5 models

\times

2 tasks. PC is Proposer-Critic accuracy (%); runner-up is the best non-PC architecture at that budget (chosen on the full sample);

\Delta

is their difference (points) with an item-clustered 95% bootstrap interval (2,000 replicates resampling questions within benchmark). ∗ / † mark intervals excluding zero above/below. PC is significantly behind the field-best at

N=3

and significantly ahead from

N=10

; at

N=30

it beats every other architecture (all seven intervals exclude zero,

+5.7

+10.2

Subject	Star	Pers.	PC	Tourn.	Tree	Diamond	Chain	C.-Ch.	win
college mathematics	54	52	55	53	53	51	58	56	Chain
college physics	62	65	67	65	63	64	70	68	Chain
econometrics	56	56	55	56	57	56	58	56	Chain
formal logic	58	59	57	57	56	60	59	58	Diam.
professional accounting	57	59	55	57	57	58	59	57	Chain

Appendix

Table S10: MMLU-hard accuracy (%) at

N=30

by subject and architecture (pooled over 5 models). The five hard subjects behave alike (accuracy 54–62%) and Chain leads in four of five, so the no-universal-winner picture is not an artifact of pooling subjects.

Figure S1: Matched Star–Persona-Star prompt intervention at

N=30

. (a) The intervention’s two-margin decomposition: an extensive coverage dividend of

+5.08

points is mostly offset by a

-3.72

intensive transformation change, leaving

+1.36

final points. (b) Final accuracy split by which system has a correct proposal available (neither, Persona-Star only, Star only, or both), with equal-cell shares annotated. The margin Persona newly covers converts weakly, and where both cover, Persona-Star is slightly more accurate, so the aggregate intensive term is not a same-item causal effect.

Model	Star $A_{\rm pair}$	Persona $A_{\rm pair}$	$\Delta O$	$\Delta\ell$	$\Delta Y$
Llama-3.1	0.656	0.551	+8.4	+6.1	+0.2
Ministral-3	0.676	0.561	+7.7	+6.2	+0.9
Nemotron	0.713	0.641	+4.3	+4.2	+1.1
Qwen2.5	0.823	0.685	+12.9	+9.2	+0.9
Qwen3	0.854	0.734	+7.4	+3.6	+3.7

Appendix

Table S11: Matched Star–Persona-Star intervention at

N=30

, averaged over five QA tasks. The two agreement columns report exact-answer pairwise agreement. The remaining columns are Persona-minus-Star percentage-point changes.

Model	Task	$\Delta A_{\rm pair}$	$\Delta$ accuracy (pp)
Llama-3.1	arc	-0.091 [-0.100,-0.082]	+0.8 [-0.3,+1.9]
Llama-3.1	gpqa	-0.123 [-0.146,-0.102]	-0.5 [-3.7,+3.2]
Llama-3.1	gsm8k	-0.098 [-0.107,-0.089]	-0.6 [-2.6,+1.2]
Llama-3.1	gsmhard	-0.074 [-0.084,-0.062]	-0.8 [-2.4,+0.8]
Llama-3.1	mmlu_hard	-0.140 [-0.151,-0.128]	+2.1 [-0.0,+4.1]
Ministral-3	arc	-0.085 [-0.092,-0.078]	-0.4 [-1.3,+0.5]

Appendix

Table S12: Paired Persona-Star minus Star effects at

N=30

. Brackets are item-clustered 95% bootstrap intervals.

Quantity	Mean $\Delta$ (pp)	Aggregate 95% CI	$\Delta>0$	Cell CI $>0$	Cell CI $<0$
Proposal oracle	+8.13	[+7.50, +8.73]	23/25	23	1
Transfer loss	+5.85	[+5.20, +6.49]	20/25	19	2
Recovery	-0.91	[-1.13, -0.70]	4/25	2	13
Net transfer	-6.77	[-7.45, -6.07]	5/25	3	20
Final accuracy	+1.36	[+0.94, +1.82]	18/25	7	0

Appendix

Table S13: Paired Persona-Star minus Star effects at

N=30

. Sign counts summarize 25 item-clustered model–task intervals; the aggregate interval uses 5,000 benchmark-stratified item-bootstrap replicates, carries each sampled item across five models, and weights cells equally. The means close

\Delta Y=\Delta O+\Delta r-\Delta\ell

Coverage stratum	Share (%)	Star $Y$ (%)	Persona $Y$ (%)	Cond. $\Delta Y$ (pp)	Contribution (pp)
Neither covered ( $0\!\to\!0$ )	27.94	9.49	10.12	+0.63	+0.08
Persona only ( $0\!\to\!1$ )	10.53	20.40	31.11	+10.71	+0.94
Star only ( $1\!\to\!0$ )	2.40	37.99	23.41	-14.58	-0.23
Both covered ( $1\!\to\!1$ )	59.14	81.80	82.99	+1.19	+0.58
Total	100.00	—	—	—	+1.36

Appendix

Table S14: Paired coverage transitions for Persona-Star versus Star at

N=30

. Each row is a realized

(O_{\rm Star},O_{\rm Persona})

stratum. Conditional

\Delta Y

compares final accuracy on the same paired trials. Contribution is stratum share times conditional

\Delta Y

, computed within each of the 25 model–task cells and then averaged, so it need not equal the product of the displayed aggregate columns. It sums to the overall +1.36-point effect.

Architecture	$P(Y=0\mid O=1)$	$A_{\rm pair}$ : lost	preserved	$\Delta A_{\rm pair}$
Star	19.6	0.590	0.793	+0.202 (25/25)
Persona-Star	25.5	0.504	0.690	+0.186 (25/25)

Appendix

Table S15: Proposal conflict and downstream loss at

N=30

, restricted to trials with an available correct proposal (

O=1

). Means weight the 25 model–task cells equally. The agreement gap is preserved minus lost and is positive in every cell.

Final-manager instruction	$\Delta O$	$\Delta\ell$	$\Delta r$	$\Delta Y$
Tally–decide	-0.02 [-0.15, +0.11]	+0.49 [+0.12, +0.87]	-2.67 [-2.85, -2.49]	-3.18 [-3.59, -2.78]
Critique–synthesize	+0.08 [-0.10, +0.26]	+0.27 [-0.12, +0.67]	-1.27 [-1.43, -1.11]	-1.46 [-1.87, -1.06]

Appendix

Table S16: Diamond manager-prompt stress test at

N=30

. Each entry is the prompt variant minus the baseline manager in percentage points, with a 95% benchmark-stratified item-bootstrap interval. The DAG, upstream instructions, model, sampling settings, and seed recipe are unchanged; separately launched decoding is not exact frozen-evidence replay.

Architecture	Coverage	Vote	Final agent	Final $-$ vote
Star	61.5	45.9	56.4	+10.5
Persona-Star	69.7	45.6	57.8	+12.2
Prop.-Critic	59.1	45.8	59.1	+13.3
Tournament	59.2	45.6	55.4	+9.7
Tree	60.6	45.8	57.3	+11.5
Chain	44.6	44.6	58.4	+13.8

Appendix

Table S17: Active final-agent synthesis versus deterministic plurality over the same proposal tier at

N=30

(equal mean over QA cells). Plurality is an offline control and invokes no additional LLM.

Architecture	Acc.	$\Delta$ vs. $N=1$	$\Delta>0$
Star	42.3	+3.7	11/15
Persona-Star	43.6	+5.0	11/15
Prop.-Critic	45.2	+6.6	11/15
Tournament	41.6	+3.0	9/15
Tree	42.8	+4.3	10/15
Chain	44.8	+6.2	12/15

Appendix

Table S18: Fixed

N=30

sparse architectures versus the direct one-call baseline on 15 hard-QA model–task cells. Accuracy and change are equal-cell means in percentage points. The final row is a descriptive post-hoc upper envelope, not a deployable policy.

Model	Task	Items	Direct	Long-10 $\times$	$\Delta$ (pp)	Long tokens
Llama-3.1	gpqa	198	26.8	24.7	-2.0 [-8.1,+4.0]	2.77k
Llama-3.1	gsmhard	1017	5.3	5.4	+0.1 [-1.1,+1.3]	5.15k
Llama-3.1	mmlu_hard	724	46.1	45.4	-0.7 [-3.5,+2.1]	1.31k
Ministral-3	gsmhard	200	27.0	58.0	+31.0 [+23.5,+38.0]	1.97k
Ministral-3	mmlu_hard	724	55.4	66.3	+10.9 [+7.2,+15.1]	1.92k
Nemotron	gpqa	198	33.3	53.0	+19.7 [+11.6,+27.8]	5.29k

Appendix

Table S19: One-agent 10

\times

-cap long-reasoning control on shared hard-QA items. The maximum generation length rises from 1,024 to 10,240 tokens and the prompt requests extended verification; hence this is a combined prompt-and-cap control, not a pure token intervention.

\Delta

is Long minus Direct with a paired item-bootstrap 95% CI. No Ministral–GPQA long-cap run is available.

Architecture	$\mathcal{E}$	$\mathcal{I}$	$\Delta Y$	$O$	$g$	$Y-O$
Star	$+6.34$	$-4.38$	$+1.95$	84.5	1.0	$-13.09$
Persona-Star	$+8.49$	$-7.59$	$+0.89$	90.3	0.0	$-17.64$
Prop.-Critic	$+7.70$	$-7.46$	$+0.24$	82.9	7.2	$-13.50$
Tournament	$+5.22$	$-1.36$	$+3.86$	83.0	9.0	$-9.63$
Tree	$+5.81$	$-1.87$	$+3.94$	83.8	4.4	$-10.81$
Chain	$-0.33$	$+0.62$	$+0.28$	71.9	20.5	$-1.83$

Appendix

Table S20: Exact two-margin decomposition on the open-ended HumanEval+ task, equal-cell means over the five models.

\mathcal{E}

\mathcal{I}

, and

\Delta Y

are the requested

N=3\to 30

change in percentage points (

\Delta Y=\mathcal{E}+\mathcal{I}

exactly).

O

, generative recovery

g=P(Y{=}1\mid O{=}0)

, and

Y-O

are at

N=30

. Coverage rises as on QA, but

g

nearly vanishes for the wide star-family and every architecture finishes below its proposal oracle (

Y<O

Model	Task	Single	Self R3	Debate R2	Debate R3	Debate $-$ self (pp)	Cost / self	Cost / best sparse
Llama-3.1	gpqa	26.8	31.8	28.3	29.0	-2.8 [-8.1,+2.0]	3.7 $\times$	7.4 $\times$
Llama-3.1	gsmhard	5.3	31.1	22.9	27.6	-3.4 [-5.5,-1.5]	5.2 $\times$	19.8 $\times$
Llama-3.1	mmlu_hard	46.1	48.6	46.4	47.4	-1.2 [-4.1,+1.5]	5.1 $\times$	12.3 $\times$
Ministral-3	gpqa	38.9	41.4	43.9	45.5	+4.0 [-1.5,+9.1]	2.5 $\times$	8.3 $\times$
Ministral-3	gsmhard	26.2	59.9	62.4	59.3	-0.6 [-2.8,+1.5]	2.7 $\times$	7.3 $\times$
Ministral-3	mmlu_hard	55.4	67.1	66.0	64.5	-2.6 [-5.0,+0.0]	2.9 $\times$	8.6 $\times$

Appendix

Table S21: Direct one-call accuracy and

N=30

three-round controls. In Self, each agent revises only its own prior answer; Debate exposes every agent to the other 29 reports. Self R3 and both Debate columns use their common item support; the 95% item bootstrap pairs the two round-3 outcomes. Both cost ratios accumulate rounds 1–3. All 15 hard-QA cells are available.

Explore similar work

Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams
May 28, 2026Yinsicheng Jiang, Liang Cheng, Yeqi Huang +6Agentic Benchmarks Large Language Model Agents
University of Edinburgh · 2Microsoft Research
How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval
May 29, 2026Nazmus AshrafiCode Generation Multi-Agent Large Language Model Systems
Independent Researcher
When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty
Aug 1, 2026Nicolas Leins, Nico Pelleriti, Jana Gonnermann-Müller +1Large Language Model Workflows Inference Cost
Zuse Institute Berlin & TU Berlin, Berlin, Germany · Zuse Institute Berlin & Weizenbaum Institute Berlin, Berlin, Germany

May 28, 2026, cs.MAJ/K move · Enter open · S save

Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams

Yinsicheng Jiang, Liang Cheng, Yeqi Huang, Yufan Zhao+5

University of Edinburgh · 2Microsoft Research

Agentic Benchmarks Large Language Model Agents Role Card

Open paper AskPDF arXiv

LLM agents are increasingly deployed as multi-role teams, where tasks are divided across specialized roles such as planner, executor, and verifier. In these systems, cost and accuracy are no longer properties of a single model: they depend on which model fills each role and where it is hosted, including API, self-hosted, and hybrid deployment. Existing agentic benchmarks typically evaluate fixed models or fixed agent configurations, and therefore offer limited guidance for cost-accuracy-optimal deployment. We introduce AgentCARD, a role-aware benchmark suite for evaluating LLM agent teams across role assignment and deployment mode. AgentCARD combines a role-decomposed evaluation harness, a unified API/self-hosted cost model, Pareto-frontier analysis, and a Shapley-based diagnostic for identifying role bottlenecks. Our evaluation shows that heterogeneous teams consistently occupy the cost-accuracy frontier. They improve accuracy by up to

44\%

over cost-equivalent homogeneous teams, or match the strongest homogeneous team at up to

12\times

lower per-task cost through hybrid deployment. We further find that the best role assignment is domain-dependent: some domains are planner-bottlenecked, while others are executor-bottlenecked. Finally, AgentCARD extends beyond planner--executor teams to workflows with additional roles such as verification, and supports continual evaluation as new domains and team structures emerge. Our code is released at: https://github.com/Auto-CAP/AgentCAP

Abstract

Explore similar work

Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams

How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval

When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty