Right-Sizing LLM-Agent Decomposition in VAT Determination: A Pilot Controlled Sweep

Pedro Santos

cs.MAAug 24, 2026

Right-Sizing LLM-Agent Decomposition in VAT Determination: A Pilot Controlled Sweep

Authors: Pedro Santos

Organizations: Independent Researcher

Abstract

Recent LLM-agent systems make conflicting design bets: decompose work across many narrow agents, or use one strong tool-using agent. This pilot studies that choice on bounded cross-border VAT determination with reverse charge, where every case has an oracle label and each intermediate decision is independently scoreable. We hold the activity surface fixed (subtasks, tools, I/O schemas, validation checks, orchestrator, base model, and merge policy) and vary only the assignment of subtasks to workers across four orchestrated configurations, from one wide worker to five narrow ones, against S0, a tuned no-orchestrator single agent, with a deterministic rule engine as oracle. The program spans 4,400 runs: a 40-case, five-repeat main sweep, matched-token arms separating prompt-budget from agent-count effects, and three failure-injection arms, all judged against pre-registered falsification criteria. The two intermediate configurations lead on accuracy (0.830, against endpoints at 0.720 and 0.770) but miss the pre-stated bar against the fine endpoint, so the intermediate-optimum hypothesis remains unsupported at pilot scale. The single agent does not Pareto-dominate the orchestrated set. The matched-token criterion fires: the budget-matched single agent lands 6.5 points below the leader, but the interval includes zero, so any advantage is consistent with a prompt-budget explanation. Under injection, availability faults are absorbed at every granularity, with wide-scope restart over-recovering its baseline by +0.160, while one schema-conforming hallucinated record degrades every configuration and inverts the ordering, hitting fragmented configurations hardest. The contribution is a bounded, preregistered pilot heuristic for right-sizing decomposition (place one partition boundary at the dependency-layer midpoint), released with oracle, dataset, harness, raw traces, and analysis pipeline.

Figures & tables

Figure 1: The fixed activity surface and the experimental conditions. Top: the five subtasks and the transitive reduction of the dependency order

\mathcal{D}

, with dependency layers

L_{1}

–

L_{4}

. Bottom: worker partitions C1–C4 and the no-orchestrator baseline S0; the dashed vertical line marks the dependency-layer midpoint used by the merge policy (§4.4) and the matched-token reference (§6.3).

Condition	Accuracy [95% CI]	Tokens [95% CI]	Med. latency, s [95% CI]	Trace cons.	Term. fail.
S0	0.755 [0.665, 0.840]	11,899 [11,217, 12,628]	11.8 [11.0, 12.7]	1.000	0/200
C1	0.720 [0.625, 0.810]	11,317 [10,599, 12,000]	12.1 [11.3, 12.6]	1.000	0/200
C2	0.830 [0.750, 0.910]	14,590 [12,971, 16,416]	15.8 [15.2, 16.3]	0.985	3/200
C3	0.830 [0.745, 0.915]	13,591 [12,507, 14,928]	16.7 [16.2, 17.4]	0.985	3/200
C4	0.770 [0.685, 0.855]	15,845 [15,457, 16,238]	18.4 [17.8, 19.7]	1.000	0/200

Table 1: Main sweep: final-answer accuracy, token cost, and median latency per condition, with 95% case-clustered bootstrap CIs (§6.5). Latency is the median of case-level medians.

Figure 2: Main-sweep final-answer accuracy by condition with 95% case-clustered bootstrap CIs (Table 1 ). The orchestrated sequence C1–C4 is non-monotonic; S0 is plotted separately because it is not a point on the granularity curve (§4.5).

n=40

cases

\times

five repeats (200 runs per condition).

Contrast	$\Delta$ accuracy [95% CI]	$d_{z}$	Perm. $p$	Holm $p$	Reject
S0 vs. C1	$+0.035$ [ $-0.035$ , $+0.110$ ]	0.14	0.436	0.843	no
C1 vs. C4	$-0.050$ [ $-0.160$ , $+0.050$ ]	$-0.15$	0.422	0.843	no
C2 vs. C1	$+0.110$ [ $+0.010$ , $+0.210$ ]	0.34	0.043	0.172	no
$\mathrm{S0}^{\prime}_{C2}$ vs. C2	$-0.065$ [ $-0.165$ , $+0.025$ ]	$-0.20$	0.265	0.794	no

Table 2: Pre-specified headline family (§6.5) with Holm–Bonferroni correction. No test in the family survives correction.

Figure 3: Cost–quality plane: mean total tokens per case vs. final-answer accuracy, with 95% CIs on both axes (Table 1 ). The dashed staircase traces the non-dominated set

\{

C1, S0, C3

\}

; open diamonds are the matched-token arms

\mathrm{S0}^{\prime}_{C2}

and

\mathrm{S0}^{\prime}_{C^{\star}}

(Table 4 ). Each point aggregates

n=40

cases

\times

five repeats (200 runs per condition).

Condition	JUR	RAT	RCH	Total
S0	30	0	19	49
C1	23	0	33	56
C2	0	15	19	34
C3	7	10	17	34
C4	6	1	39	46

Table 3: Error localization: earliest failing subtask per incorrect main-sweep run (§6.1), over the 219 incorrect runs.

Arm	Accuracy [95% CI]	Tokens (eval)	Target	Deviation	In $\pm 10\%$ band
S0	0.755 [0.665, 0.840]	11,899	—	—	—
$\mathrm{S0}^{\prime}_{C2}$	0.765 [0.655, 0.865]	13,230	14,590	$-9.3\%$	yes
$\mathrm{S0}^{\prime}_{C^{\star}}$	0.760 [0.650, 0.860]	12,864	13,591	$-5.3\%$	yes

Table 4: Matched-token arms (§6.3). Tuning used only the development split; evaluation-set usage is reported as observed.

Injection	Cond.	Subst. success	All-case acc.	Val.-trace acc.	$\Delta$ acc. [95% CI]	Token penalty
Timeout	S0	1.000	0.735	0.735	$-0.020$ [ $-0.110$ , $+0.065$ ]	$-1{,}442$
	C1	1.000	0.880	0.880	$+0.160$ [ $+0.065$ , $+0.265$ ]	$+21{,}793$
	C2	0.965	0.810	0.839	$-0.020$ [ $-0.090$ , $+0.045$ ]	$+4{,}611$
	C3	0.965	0.835	0.865	$+0.005$ [ $-0.030$ , $+0.045$ ]	$+4{,}433$
	C4	0.995	0.775	0.779	$+0.005$ [ $-0.055$ , $+0.065$ ]	$-87$
Outage	S0	1.000	0.780	0.780	$+0.025$ [ $-0.060$ , $+0.110$ ]	$+1{,}138$

Table 5: Localized fallback under injection (§6.4), per (injection, configuration) cell.

\Delta

accuracy is the paired delta against the condition’s own un-injected baseline (Table 1 ), with case-clustered 95% bootstrap CIs.

Figure 4: Paired per-case accuracy deltas under injection, by condition, with case-clustered 95% bootstrap CIs (Table 5 ). Availability perturbations (left, center) straddle zero in every cell except the C1 timeout overshoot (

+0.160

); the content perturbation (right) is negative in every condition. Each cell aggregates

n=40

cases

\times

five repeats (200 runs per (injection, condition) cell).

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,040 papers

Oct 5, 2026

Attention Tax, Handoff Tax: A Stylised Model of When Multi-Agent LLM Systems Help

Recent work on multi-agent LLM systems reaches sharply different conclusions: some results show that a single agent with the same information and compute should dominate a delegated system, others that multi-agent gains grow with task depth. We argue that much of the disagreement comes from modelling different bottlenecks, and introduce a stylised reliability model built around two trade-offs. Decomposition reduces the burden of long contexts but incurs a handoff tax when information is compressed or transferred between agents. Redundancy gains from multiple samples, but its benefit depends on how much their failures are shared. With reasoning budget, verification, and task structure added, the model yields two crossover conditions: decomposition becomes preferable once the attention cost avoided by resetting context exceeds the handoff cost, and parallel sampling at equal budget is eventually preferable when its shared-failure floor lies below the error floor of one agent thinking longer. We connect these regimes to recent theoretical and empirical results. On a ledger-reconciliation task we measure the context-degradation curve and the handoff tax from single-agent and handoff runs alone. From these the model places the crossover at depth 10 and predicts decomposition to win at depths 20, 50, and 100. It does, on step-level and final-balance accuracy, and the decomposed system's success, which the prediction never sees, lands within 9 percentage points of the predicted rate at every depth.

Akshit Anchan, Nayonika Sen

University of Amsterdam Amsterdam, The Netherlands · Northeastern University Boston, MA, United States of America

Multi-Agent LLM Systems Multi-Agent Reasoning Multi-Agent Communication LLM Agent Reliability

Ask PDF

May 28, 2026

Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams

LLM agents are increasingly deployed as multi-role teams, where tasks are divided across specialized roles such as planner, executor, and verifier. In these systems, cost and accuracy are no longer properties of a single model: they depend on which model fills each role and where it is hosted, including API, self-hosted, and hybrid deployment. Existing agentic benchmarks typically evaluate fixed models or fixed agent configurations, and therefore offer limited guidance for cost-accuracy-optimal deployment. We introduce AgentCARD, a role-aware benchmark suite for evaluating LLM agent teams across role assignment and deployment mode. AgentCARD combines a role-decomposed evaluation harness, a unified API/self-hosted cost model, Pareto-frontier analysis, and a Shapley-based diagnostic for identifying role bottlenecks. Our evaluation shows that heterogeneous teams consistently occupy the cost-accuracy frontier. They improve accuracy by up to

44\%

over cost-equivalent homogeneous teams, or match the strongest homogeneous team at up to

12\times

lower per-task cost through hybrid deployment. We further find that the best role assignment is domain-dependent: some domains are planner-bottlenecked, while others are executor-bottlenecked. Finally, AgentCARD extends beyond planner--executor teams to workflows with additional roles such as verification, and supports continual evaluation as new domains and team structures emerge. Our code is released at: https://github.com/Auto-CAP/AgentCAP

Yinsicheng Jiang, Liang Cheng, Yeqi Huang +6

University of Edinburgh · 2Microsoft Research

Multi-Agent LLM Systems LLM Agent Evaluation LLM Agent Workflow Optimization AI Agent Benchmarks LLM Agents LLM Agent Efficiency

Ask PDF

Aug 24, 2026

Markets, Not Planners: Decentralized Orchestration of LLM Agents with Private Information

As LLM agents proliferate, built by different parties and with different capabilities and costs, orchestrating them is more like assembling labor across the economy than a computer calling a subroutine. Existing orchestration is typically centralized, with a single planner assigning every task, but this creates a bottleneck as agent pools grow, requires private information (e.g., agents' execution costs), and can easily be manipulated, such that a single inserted preference nearly doubles a favored agent's task share under a centralized LLM allocator. We introduce AgentLance, a repeated labor market in which agents bid on tasks using their private costs and self-maintained strategy notes, an allocator selects winners from bids and public reputation records, and a VCG-style payment rule rewards cost-aware bidding. Complex tasks are handled by hierarchical delegation: winning agents can decompose work and subcontract it through the same mechanism. Across mathematical reasoning, code generation, knowledge-intensive QA, and agentic tasks, AgentLance matches agents to their specializations, shifts work toward cheaper agents as cost sensitivity rises, and consistently outperforms single-model, centralized-orchestration, and market baselines. Diagnosing market failures, including inaccurate cost self-estimation and sub-optimal bidding, then correcting them in controlled experiments yields further gains, charting a path toward more efficient agent economies.

Xiao Liu, Haoyang Li, Songwei Li +4

Knowledge Lab, University of Chicago · University of Virginia · Tsinghua University +2

Multi-Agent Orchestration LLM Agent Orchestration Multi-Agent Resource Allocation Mechanism Design

Ask PDF

Abstract

Multi-Agent LLM Systems

Explore similar work

Attention Tax, Handoff Tax: A Stylised Model of When Multi-Agent LLM Systems Help

Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams

Markets, Not Planners: Decentralized Orchestration of LLM Agents with Private Information