An Exact Generate - Transform Decomposition of Small-LLM Team Scaling Across Orchestration Architectures

Blaz Bertalanic; Carolina Fortuna

cs.AISep 28, 2026

An Exact Generate - Transform Decomposition of Small-LLM Team Scaling Across Orchestration Architectures

Authors: Blaz Bertalanic, Carolina Fortuna

Organizations: Jožef Stefan Institute, Ljubljana, Slovenia

Abstract

Replacing one LLM agent with a collaborating team can raise accuracy, but whether scaling the team helps, and which architecture to scale, is unclear. Sweeping eight agent orchestration architectures across five instruction-tuned 7-9B models, five short-answer benchmarks, and an executable-code benchmark up to 30 calls, we find that the returns to team scaling are sharply task-dependent: from three to thirty calls accuracy rises by up to 17 points on the two arithmetic word-problem benchmarks (GSM8K, GSMHard) but by at most four on ARC, GPQA, and MMLU, for every architecture, a split the usual task-averaged number conceals. Proposer-Critic captures the arithmetic gains, scaling steepest and, in aggregate, surpassing every other architecture at the largest budget (item-clustered intervals exclude zero), though it ranks among the weakest elsewhere, and no architecture wins across tasks. We explain these trajectories with an exact generate-transform decomposition. Partitioning any workflow into proposal coverage and a downstream transform, any accuracy change splits exactly into an extensive coverage dividend and an intensive transformation change. The decomposition diagnoses each task: arithmetic offers coverage headroom that a critic-guided transform converts, whereas the multiple-choice benchmarks either saturate in coverage or fail to convert it, and on open-ended code generative recovery nearly vanishes so accuracy tracks coverage. At equal call budgets token cost still varies 2.1x. Extra calls therefore create candidate opportunity that only some architectures, on some tasks, convert. Team scaling is a task- and architecture-specific bet, not a uniform lever.

Figures & tables

Figure 1: The eight orchestration architectures, nodes colored by functional stage: generate (the

L

proposals), transform (critics, refiners, synthesizers, duelists), the manager, and Diamond’s non-answering planner. Tournament selects through pairwise duels, Tree merges via fan-in-three synthesis, and Persona-Star shares Star’s graph but differs in prompts. Counts are illustrative.

Figure 2: Where team scaling pays off. (a) The scaling gain

\Delta Y

(the

N=3

N=30

accuracy change, Section 3.3 ) for every architecture on every benchmark (equal-cell over five models), grouped by task family, with Proposer-Critic marked. Returns concentrate on arithmetic and are near-zero on ARC, GPQA, and MMLU for all architectures. (b) Arithmetic accuracy versus team size: Proposer-Critic (red) starts mid-field and separates with scale.

Figure 3: Accuracy gain over a single agent (the mean individual worker), by task category: (a) arithmetic, (b) multiple-choice, (c) code, each cell an equal mean over the category’s model–task cells with per-panel color scales. Gains on arithmetic (25 to 40 points) are two to three times those on multiple-choice (8 to 14) or code, and Proposer-Critic tops arithmetic at $N=30$ while trailing on multiple-choice. (d) QA tokens per problem versus team size: cost climbs at architecture-dependent rates to a 2.1 $\times$ spread by $N=30$ . Diamond is undefined at $N=2$ .

Figure 4: Two-margin decomposition of the

N=3\to 30

accuracy change by task category: (a) arithmetic, (b) multiple-choice, (c) code. Each architecture’s change splits into an extensive coverage dividend

\mathcal{E}

(blue) and an intensive transformation change

\mathcal{I}

(red, extending left when the transform sheds coverage), summing to the net

\Delta Y

(diamond), equal-cell over each category’s model–task cells. Coverage is added in every category, but the transform converts it only on arithmetic, where Proposer-Critic alone posts a positive

\mathcal{I}

and the largest net gain. On multiple-choice every proposal-expanding design sheds most of its coverage and nets almost nothing, and on code the same shedding, with recovery near zero, leaves the net close to the coverage floor.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture	Actual calls	$L$	Other non-final roles	Manager input	Tiers
Star	30	29	none	29 worker reports	2
Persona-Star	30	29	none	29 persona-worker reports	2
Proposer-Critic	30	15	14 critics	14 critics + 1 free proposal	3
Tournament	29	15	13 non-final duelists	2 finalists	5
Tree	30	22	7 fan-in-three synthesizers	7 syntheses + 1 orphan	3
Chain	30	1	28 one-parent refiners	all 29 preceding reports	30

Appendix

Table S1: Exact architecture composition at requested budget

N=30

L

is the first answer-producing tier used for proposal metrics. Tournament rounds down to the largest pairwise bracket within the requested budget.

Architecture	$\mathcal{E}$	$\bar{O}\Delta s$	$(1-\bar{O})\Delta g$	$\mathcal{I}$	$\Delta Y$	$\Delta Y>0$	$\mathcal{E}>0$	$\mathcal{I}<0$
Star	+7.28	-2.65	-2.23	-4.88	+2.40	17/25	25/25	22/25
Persona-Star	+11.32	-5.14	-1.80	-6.94	+4.39	20/25	25/25	23/25
Prop.-Critic	+6.68	-0.67	+1.21	+0.54	+7.22	21/25	25/25	18/25
Tournament	+5.72	-3.12	-1.14	-4.26	+1.46	18/25	25/25	22/25
Tree	+6.55	-2.46	-0.56	-3.02	+3.53	20/25	25/25	21/25
Chain	+0.00	-0.18	+4.17	+3.99	+3.99	24/25	14/25	2/25

Appendix

Table S2: Exact two-margin decomposition of QA scaling from requested

N=3

N=30

, in equal-cell percentage points. The intensive term is split into changes in covered-case success and recovery. Every row satisfies

\Delta Y=\mathcal{E}+\mathcal{I}

before rounding. The last three columns count signs across 25 model–task cells.

Architecture	$\pi_{11}$	$Y^{11}_{3}$	$Y^{11}_{30}$	$\Delta Y_{11}$	$\pi_{01}$	$c_{01}$	$c_{11}$	$\Delta Y$
Star	48	85.4	88.6	+3.19	13	+1.48	+1.00	+2.40
Persona-Star	49	83.7	88.8	+5.10	20	+2.20	+1.55	+4.39
Prop.-Critic	44	79.5	86.2	+6.73	15	+2.50	+1.42	+7.22
Tournament	48	86.0	85.0	-0.94	11	+1.31	-0.22	+1.46
Tree	48	85.8	88.0	+2.23	12	+1.55	+0.88	+3.53
Diamond	45	83.6	83.6	-0.01	26	+4.45	-0.26	+4.39

Appendix

Table S3: Paired coverage-transition decomposition of QA scaling from

N=3

N=30

, equal-cell means over 25 model–task cells. Each trial is paired across the two budgets by (task, run, item) and stratified by its coverage transition

(O_{3},O_{30})

; match rate is 100%.

\pi_{11}

is the share of trials covered at both budgets and

\pi_{01}

the newly-covered share (percent).

Y^{11}_{3}

and

Y^{11}_{30}

are final accuracies on the both-covered stratum and

\Delta Y_{11}

their difference;

c_{01}

and

c_{11}

are the strata contributions to the total

\Delta Y

(points). The both-covered stratum holds item difficulty fixed, so a non-negative

\Delta Y_{11}

argues against a same-item transform decline: accuracy there rises for the four proposal-expanding designs Star, Persona-Star, Proposer-Critic, and Tree, is flat for Diamond, and falls only for Tournament. The negative intensive term of Table S2 is therefore consistent with the weak conversion of harder newly-covered items rather than the transform discarding answers it previously returned.

Architecture	$O$	Loss	$P(Y{=}0\mid O{=}1)$	Recovery	$P(Y{=}1\mid O{=}0)$	Net	$Y$
Star	61.5	10.8	19.6	5.7	12.6	-5.1	56.4
Persona-Star	69.7	16.7	25.5	4.8	11.7	-11.9	57.8
Prop.-Critic	59.1	11.4	20.2	11.3	27.0	0.0	59.1
Tournament	59.2	10.9	21.8	7.1	15.2	-3.9	55.4
Tree	60.6	10.5	19.9	7.2	15.1	-3.3	57.3
Chain	44.6	3.3	10.5	17.0	29.2	+13.8	58.4

Appendix

Table S4: Proposal-to-team transfer at

N=30

on QA (equal mean over 25 model–task cells).

O

and

Y

are proposal coverage and final accuracy. Loss and recovery are unconditional masses needed by the accounting identity; the conditional columns measure downstream behavior given the corresponding opportunity. Net is recovery minus loss.

Architecture	$O$	$s$	$g$	$v$	$O^{\star}$	$Y$	$Y-O$
Star	61.5	82.4	14.9	0.675	45.8	56.4	-5.1
Persona-Star	69.7	76.0	15.9	0.601	39.9	57.8	-11.9
Prop.-Critic	59.1	80.8	27.7	0.531	59.0	59.1	-0.0
Tournament	59.2	81.6	17.3	0.642	48.4	55.4	-3.9
Tree	60.6	82.7	18.2	0.645	51.3	57.3	-3.3
Chain	44.6	92.7	30.7	0.619	80.7	58.4	+13.8

Appendix

Table S5: Response-law boundary at

N=30

. Rates pool the equal-cell joint masses, so

Y=g+(s-g)O

holds exactly for every row. A generative transformer finishes above its proposal oracle exactly when

O<O^{\star}=g/(1-s+g)

. All values are percentages except proposal leverage

v=s-g

Calibration $B$	Comparable cells	Agree	Agreement
3	196	149	76.0%
5	199	177	88.9%
7	199	185	93.0%
10	198	190	96.0%

Appendix

Table S6: Held-out-budget stability of the oracle-crossing label. The label

O<O^{\star}

(final accuracy above proposal coverage) at calibration budget

B

predicts the corresponding label at

N=30

. Cells tied at either endpoint are excluded. Because the boundary exactly encodes the sign of

Y-O

at each budget, this evaluates label persistence rather than an independently fitted classifier.

Arithmetic (GSM8K, GSMHard)
Architecture	Lla.	Min.	Nem.	Q2.5	Q3
Star	45.7	66.0	58.4	36.4	50.4
Persona-Star	45.0	69.0	61.3	37.1	56.7
Prop.-Critic	52.4	64.0	64.2	51.4	65.5
Tournament	28.3	62.0	58.6	43.0	54.6
Tree	41.5	65.8	61.8	46.1	52.4

Appendix

Table S7: Team accuracy (%) at

N=30

, equal-cell mean over each category’s tasks, by model. On arithmetic the architecture spread is large (mean 14 points) and Proposer-Critic leads four of five models, whereas on multiple-choice every design is within a few points (mean spread 4). Single is the mean individual worker, one non-thinking call. Bold marks the best architecture within a model and category.

Architecture	Lla.	Min.	Nem.	Q2.5	Q3
Star	58.9	66.1	84.3	69.3	78.3
Persona-Star	49.8	76.8	84.3	72.8	79.7
Prop.-Critic	54.3	59.8	84.6	70.9	77.6
Tournament	54.3	75.6	84.6	72.8	79.7
Tree	59.1	68.3	84.6	73.4	79.5
Chain	47.0	72.2	81.5	72.8	77.0

Appendix

Table S8: HumanEval+ pass rate (%) at

N=30

by architecture and model, with the mean individual-worker one-call baseline (Single). Bold is the best architecture within a model.

$N$	PC	runner-up	$\Delta$	95% CI
2	44.6	Tree	-0.2	$[-0.7,+0.2]$
3	44.9	Chain	-2.3 †	$[-2.9,-1.7]$
5	49.0	Chain	-0.0	$[-0.6,+0.6]$
7	51.7	Persona-Star	+0.5	$[-0.1,+1.1]$
10	55.1	Persona-Star	+2.8 ∗	$[+2.2,+3.5]$
15	56.1	Persona-Star	+2.7 ∗	$[+2.0,+3.4]$

Appendix

Table S9: Task-type specialization with scale on the two arithmetic word-problem benchmarks (GSM8K, GSMHard), equal-cell over 5 models

\times

2 tasks. PC is Proposer-Critic accuracy (%); runner-up is the best non-PC architecture at that budget (chosen on the full sample);

\Delta

is their difference (points) with an item-clustered 95% bootstrap interval (2,000 replicates resampling questions within benchmark). ∗ / † mark intervals excluding zero above/below. PC is significantly behind the field-best at

N=3

and significantly ahead from

N=10

; at

N=30

it beats every other architecture (all seven intervals exclude zero,

+5.7

+10.2

Subject	Star	Pers.	PC	Tourn.	Tree	Diamond	Chain	C.-Ch.	win
college mathematics	54	52	55	53	53	51	58	56	Chain
college physics	62	65	67	65	63	64	70	68	Chain
econometrics	56	56	55	56	57	56	58	56	Chain
formal logic	58	59	57	57	56	60	59	58	Diam.
professional accounting	57	59	55	57	57	58	59	57	Chain

Appendix

Table S10: MMLU-hard accuracy (%) at

N=30

by subject and architecture (pooled over 5 models). The five hard subjects behave alike (accuracy 54–62%) and Chain leads in four of five, so the no-universal-winner picture is not an artifact of pooling subjects.

Figure S1: Matched Star–Persona-Star prompt intervention at

N=30

. (a) The intervention’s two-margin decomposition: an extensive coverage dividend of

+5.08

points is mostly offset by a

-3.72

intensive transformation change, leaving

+1.36

final points. (b) Final accuracy split by which system has a correct proposal available (neither, Persona-Star only, Star only, or both), with equal-cell shares annotated. The margin Persona newly covers converts weakly, and where both cover, Persona-Star is slightly more accurate, so the aggregate intensive term is not a same-item causal effect.

Model	Star $A_{\rm pair}$	Persona $A_{\rm pair}$	$\Delta O$	$\Delta\ell$	$\Delta Y$
Llama-3.1	0.656	0.551	+8.4	+6.1	+0.2
Ministral-3	0.676	0.561	+7.7	+6.2	+0.9
Nemotron	0.713	0.641	+4.3	+4.2	+1.1
Qwen2.5	0.823	0.685	+12.9	+9.2	+0.9
Qwen3	0.854	0.734	+7.4	+3.6	+3.7

Appendix

Table S11: Matched Star–Persona-Star intervention at

N=30

, averaged over five QA tasks. The two agreement columns report exact-answer pairwise agreement. The remaining columns are Persona-minus-Star percentage-point changes.

Model	Task	$\Delta A_{\rm pair}$	$\Delta$ accuracy (pp)
Llama-3.1	arc	-0.091 [-0.100,-0.082]	+0.8 [-0.3,+1.9]
Llama-3.1	gpqa	-0.123 [-0.146,-0.102]	-0.5 [-3.7,+3.2]
Llama-3.1	gsm8k	-0.098 [-0.107,-0.089]	-0.6 [-2.6,+1.2]
Llama-3.1	gsmhard	-0.074 [-0.084,-0.062]	-0.8 [-2.4,+0.8]
Llama-3.1	mmlu_hard	-0.140 [-0.151,-0.128]	+2.1 [-0.0,+4.1]
Ministral-3	arc	-0.085 [-0.092,-0.078]	-0.4 [-1.3,+0.5]

Appendix

Table S12: Paired Persona-Star minus Star effects at

N=30

. Brackets are item-clustered 95% bootstrap intervals.

Quantity	Mean $\Delta$ (pp)	Aggregate 95% CI	$\Delta>0$	Cell CI $>0$	Cell CI $<0$
Proposal oracle	+8.13	[+7.50, +8.73]	23/25	23	1
Transfer loss	+5.85	[+5.20, +6.49]	20/25	19	2
Recovery	-0.91	[-1.13, -0.70]	4/25	2	13
Net transfer	-6.77	[-7.45, -6.07]	5/25	3	20
Final accuracy	+1.36	[+0.94, +1.82]	18/25	7	0

Appendix

Table S13: Paired Persona-Star minus Star effects at

N=30

. Sign counts summarize 25 item-clustered model–task intervals; the aggregate interval uses 5,000 benchmark-stratified item-bootstrap replicates, carries each sampled item across five models, and weights cells equally. The means close

\Delta Y=\Delta O+\Delta r-\Delta\ell

Coverage stratum	Share (%)	Star $Y$ (%)	Persona $Y$ (%)	Cond. $\Delta Y$ (pp)	Contribution (pp)
Neither covered ( $0\!\to\!0$ )	27.94	9.49	10.12	+0.63	+0.08
Persona only ( $0\!\to\!1$ )	10.53	20.40	31.11	+10.71	+0.94
Star only ( $1\!\to\!0$ )	2.40	37.99	23.41	-14.58	-0.23
Both covered ( $1\!\to\!1$ )	59.14	81.80	82.99	+1.19	+0.58
Total	100.00	—	—	—	+1.36

Appendix

Table S14: Paired coverage transitions for Persona-Star versus Star at

N=30

. Each row is a realized

(O_{\rm Star},O_{\rm Persona})

stratum. Conditional

\Delta Y

compares final accuracy on the same paired trials. Contribution is stratum share times conditional

\Delta Y

, computed within each of the 25 model–task cells and then averaged, so it need not equal the product of the displayed aggregate columns. It sums to the overall +1.36-point effect.

Architecture	$P(Y=0\mid O=1)$	$A_{\rm pair}$ : lost	preserved	$\Delta A_{\rm pair}$
Star	19.6	0.590	0.793	+0.202 (25/25)
Persona-Star	25.5	0.504	0.690	+0.186 (25/25)

Appendix

Table S15: Proposal conflict and downstream loss at

N=30

, restricted to trials with an available correct proposal (

O=1

). Means weight the 25 model–task cells equally. The agreement gap is preserved minus lost and is positive in every cell.

Final-manager instruction	$\Delta O$	$\Delta\ell$	$\Delta r$	$\Delta Y$
Tally–decide	-0.02 [-0.15, +0.11]	+0.49 [+0.12, +0.87]	-2.67 [-2.85, -2.49]	-3.18 [-3.59, -2.78]
Critique–synthesize	+0.08 [-0.10, +0.26]	+0.27 [-0.12, +0.67]	-1.27 [-1.43, -1.11]	-1.46 [-1.87, -1.06]

Appendix

Table S16: Diamond manager-prompt stress test at

N=30

. Each entry is the prompt variant minus the baseline manager in percentage points, with a 95% benchmark-stratified item-bootstrap interval. The DAG, upstream instructions, model, sampling settings, and seed recipe are unchanged; separately launched decoding is not exact frozen-evidence replay.

Architecture	Coverage	Vote	Final agent	Final $-$ vote
Star	61.5	45.9	56.4	+10.5
Persona-Star	69.7	45.6	57.8	+12.2
Prop.-Critic	59.1	45.8	59.1	+13.3
Tournament	59.2	45.6	55.4	+9.7
Tree	60.6	45.8	57.3	+11.5
Chain	44.6	44.6	58.4	+13.8

Appendix

Table S17: Active final-agent synthesis versus deterministic plurality over the same proposal tier at

N=30

(equal mean over QA cells). Plurality is an offline control and invokes no additional LLM.

Architecture	Acc.	$\Delta$ vs. $N=1$	$\Delta>0$
Star	42.3	+3.7	11/15
Persona-Star	43.6	+5.0	11/15
Prop.-Critic	45.2	+6.6	11/15
Tournament	41.6	+3.0	9/15
Tree	42.8	+4.3	10/15
Chain	44.8	+6.2	12/15

Appendix

Table S18: Fixed

N=30

sparse architectures versus the direct one-call baseline on 15 hard-QA model–task cells. Accuracy and change are equal-cell means in percentage points. The final row is a descriptive post-hoc upper envelope, not a deployable policy.

Model	Task	Items	Direct	Long-10 $\times$	$\Delta$ (pp)	Long tokens
Llama-3.1	gpqa	198	26.8	24.7	-2.0 [-8.1,+4.0]	2.77k
Llama-3.1	gsmhard	1017	5.3	5.4	+0.1 [-1.1,+1.3]	5.15k
Llama-3.1	mmlu_hard	724	46.1	45.4	-0.7 [-3.5,+2.1]	1.31k
Ministral-3	gsmhard	200	27.0	58.0	+31.0 [+23.5,+38.0]	1.97k
Ministral-3	mmlu_hard	724	55.4	66.3	+10.9 [+7.2,+15.1]	1.92k
Nemotron	gpqa	198	33.3	53.0	+19.7 [+11.6,+27.8]	5.29k

Appendix

Table S19: One-agent 10

\times

-cap long-reasoning control on shared hard-QA items. The maximum generation length rises from 1,024 to 10,240 tokens and the prompt requests extended verification; hence this is a combined prompt-and-cap control, not a pure token intervention.

\Delta

is Long minus Direct with a paired item-bootstrap 95% CI. No Ministral–GPQA long-cap run is available.

Architecture	$\mathcal{E}$	$\mathcal{I}$	$\Delta Y$	$O$	$g$	$Y-O$
Star	$+6.34$	$-4.38$	$+1.95$	84.5	1.0	$-13.09$
Persona-Star	$+8.49$	$-7.59$	$+0.89$	90.3	0.0	$-17.64$
Prop.-Critic	$+7.70$	$-7.46$	$+0.24$	82.9	7.2	$-13.50$
Tournament	$+5.22$	$-1.36$	$+3.86$	83.0	9.0	$-9.63$
Tree	$+5.81$	$-1.87$	$+3.94$	83.8	4.4	$-10.81$
Chain	$-0.33$	$+0.62$	$+0.28$	71.9	20.5	$-1.83$

Appendix

Table S20: Exact two-margin decomposition on the open-ended HumanEval+ task, equal-cell means over the five models.

\mathcal{E}

\mathcal{I}

, and

\Delta Y

are the requested

N=3\to 30

change in percentage points (

\Delta Y=\mathcal{E}+\mathcal{I}

exactly).

O

, generative recovery

g=P(Y{=}1\mid O{=}0)

, and

Y-O

are at

N=30

. Coverage rises as on QA, but

g

nearly vanishes for the wide star-family and every architecture finishes below its proposal oracle (

Y<O

Model	Task	Single	Self R3	Debate R2	Debate R3	Debate $-$ self (pp)	Cost / self	Cost / best sparse
Llama-3.1	gpqa	26.8	31.8	28.3	29.0	-2.8 [-8.1,+2.0]	3.7 $\times$	7.4 $\times$
Llama-3.1	gsmhard	5.3	31.1	22.9	27.6	-3.4 [-5.5,-1.5]	5.2 $\times$	19.8 $\times$
Llama-3.1	mmlu_hard	46.1	48.6	46.4	47.4	-1.2 [-4.1,+1.5]	5.1 $\times$	12.3 $\times$
Ministral-3	gpqa	38.9	41.4	43.9	45.5	+4.0 [-1.5,+9.1]	2.5 $\times$	8.3 $\times$
Ministral-3	gsmhard	26.2	59.9	62.4	59.3	-0.6 [-2.8,+1.5]	2.7 $\times$	7.3 $\times$
Ministral-3	mmlu_hard	55.4	67.1	66.0	64.5	-2.6 [-5.0,+0.0]	2.9 $\times$	8.6 $\times$

Appendix

Table S21: Direct one-call accuracy and

N=30

three-round controls. In Self, each agent revises only its own prior answer; Debate exposes every agent to the other 29 reports. Self R3 and both Debate columns use their common item support; the 95% item bootstrap pairs the two round-3 outcomes. Both cost ratios accumulate rounds 1–3. All 15 hard-QA cells are available.

Explore similar work

May 28, 2026cs.MA

Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams

LLM agents are increasingly deployed as multi-role teams, where tasks are divided across specialized roles such as planner, executor, and verifier. In these systems, cost and accuracy are no longer properties of a single model: they depend on which model fills each role and where it is hosted, including API, self-hosted, and hybrid deployment. Existing agentic benchmarks typically evaluate fixed models or fixed agent configurations, and therefore offer limited guidance for cost-accuracy-optimal deployment. We introduce AgentCARD, a role-aware benchmark suite for evaluating LLM agent teams across role assignment and deployment mode. AgentCARD combines a role-decomposed evaluation harness, a unified API/self-hosted cost model, Pareto-frontier analysis, and a Shapley-based diagnostic for identifying role bottlenecks. Our evaluation shows that heterogeneous teams consistently occupy the cost-accuracy frontier. They improve accuracy by up to

44\%

over cost-equivalent homogeneous teams, or match the strongest homogeneous team at up to

12\times

lower per-task cost through hybrid deployment. We further find that the best role assignment is domain-dependent: some domains are planner-bottlenecked, while others are executor-bottlenecked. Finally, AgentCARD extends beyond planner--executor teams to workflows with additional roles such as verification, and supports continual evaluation as new domains and team structures emerge. Our code is released at: https://github.com/Auto-CAP/AgentCAP

Yinsicheng Jiang, Liang Cheng, Yeqi Huang +6

University of Edinburgh · 2Microsoft Research

Agentic Benchmarks Large Language Model Agents Role Card

Ask PDF

May 29, 2026cs.SE

How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval

Large-language-model code generation has shifted from single-shot prompting to multi-agent orchestrations - analyst, coder, tester, and debugger pipelines - and is evaluated almost exclusively on functional correctness. Whether these architectures also affect the structural complexity of the code they produce, and which orchestration layers carry the cost, remains largely unexamined: prior work has documented prompt-level effects on code complexity, but the architecture-level question is open. We compare six widely-used multi-agent configurations (Basic, AC, ACT, Debugger, AC+Debugger, ACT+Debugger) under two models from the GPT-4o family across all 164 HumanEval tasks - 1,968 paired observations - using the five RADON complexity metrics (SLOC, cyclomatic complexity, and Halstead Volume, Difficulty, and Effort). We apply a paired non-parametric statistical pipeline (Friedman omnibus, Wilcoxon signed-rank post-hoc with Holm correction, Kendall's

W

and matched-pairs rank-biserial effect sizes) in both all-completions and passing-only conditions. The six architectures collapse into two indistinguishable complexity clusters separated by a 50-130% gap, the same partition in both models and under both conditions; among the architectural layers, the analyst-coder split inflates complexity, the runtime debugger does not - and on the analyst-coder background actively deflates it - and the tester re-inflates it. The heavy cluster's additional complexity buys no pass@1 advantage: the leanest architectures match or beat the heaviest on accuracy. Architectural elaboration in LLM code generation should therefore be justified by measured benefit on the dimensions that matter, not assumed.

Nazmus Ashrafi

Independent Researcher

Code Generation Multi-Agent Large Language Model Systems Multi-Agent Orchestration

Ask PDF

Aug 1, 2026cs.AI

When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty

LLM orchestration is often assumed to improve reasoning by allocating additional inference-time computation, yet its gains may not justify its cost. Existing comparisons also frequently overlook differences in optimization effort, making it difficult to isolate the value of orchestration itself. We conduct a controlled evaluation of Self-Refine, Best-of-

N

, and Debate against task-only and chain-of-thought (CoT) single-call baselines across five LLM backbones and three domains: competitive programming, chess puzzles, and mathematics. For comparability, we optimize each method with GEPA under the same optimization budget and evaluate all methods on the same difficulty-stratified benchmark items. Orchestration yields moderate but benchmark-dependent gains: averaged across backbones within each benchmark, the largest improvement is 4.6 percentage points over optimized CoT inference and 4.5 points over task-only inference, while requiring approximately 2 to 4 times the mean total tokens of task-only inference. Human-derived difficulty is associated with lower absolute accuracy in all three benchmarks, but within-benchmark analyses do not indicate that orchestration effects increase with task difficulty. By contrast, exploratory mixed-effects analyses reveal strong interactions between orchestration method and backbone model across all three benchmarks, showing that orchestration effectiveness depends substantially on the underlying model. Our results suggest that orchestration decisions should be model-specific and account for whether moderate accuracy gains justify the additional inference cost. More broadly, evaluations of LLM orchestrations should control optimization effort and report model-specific accuracy--cost trade-offs rather than treating additional inference-time structure as uniformly beneficial.

Nicolas Leins, Nico Pelleriti, Jana Gonnermann-Müller +1

Zuse Institute Berlin & TU Berlin, Berlin, Germany · Zuse Institute Berlin & Weizenbaum Institute Berlin, Berlin, Germany

Large Language Model Workflows Inference Cost Orchestration Large Language Model Backbones

Ask PDF