Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies

Siddhant Kulkarni; Yukta Kulkarni

cs.AIMar 24, 2026

Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies

Authors: Siddhant Kulkarni, Yukta Kulkarni

Organizations: Department of Computer Science and Engineering, New York University, New York, NY 10012

Abstract

The adoption of large language models (LLMs) for structured information extraction from financial documents has accelerated rapidly, yet production deployments face fundamental architectural decisions with limited empirical guidance. We present a systematic benchmark comparing four multi-agent orchestration architectures: sequential pipeline, parallel fan-out with merge, hierarchical supervisor-worker and reflexive self-correcting loop. These are evaluated across five frontier and open-weight LLMs on a corpus of 10,000 SEC filings (10-K, 10-Q and 8-K forms). Our evaluation spans 25 extraction field types covering governance structures, executive compensation and financial metrics, measured along five axes: field-level F1, document-level accuracy, end-to-end latency, cost per document and token efficiency. We find that reflexive architectures achieve the highest field-level F1 (0.943) but at 2.3x the cost of sequential baselines, while hierarchical architectures occupy the most favorable position on the cost-accuracy Pareto frontier (F1 0.921 at 1.4x cost). We further present ablation studies on semantic caching, model routing and adaptive retry strategies, demonstrating that hybrid configurations can recover 89% of the reflexive architecture's accuracy gains at only 1.15x baseline cost. Our scaling analysis from 1K to 100K documents per day reveals non-obvious throughput-accuracy degradation curves that inform capacity planning. These findings provide actionable guidance for practitioners deploying multi-agent LLM systems in regulated financial environments.

Figures & tables

Panel of Figure 1 — Fig. 1: Architecture A: Sequential Pipeline. Documents flow through a fixed chain of agents with cumulative context passing.

Panel of Figure 2 — Fig. 2: Architecture B: Parallel Fan-Out with Merge. Independent extraction branches execute concurrently before reconciliation.

Panel of Figure 3 — Fig. 3: Architecture C: Hierarchical Supervisor-Worker. The supervisor dynamically allocates tasks and selectively re-extracts low-confidence fields.

Panel of Figure 4 — Fig. 4: Architecture D: Reflexive Self-Correcting Loop. Verification failures trigger iterative critique and correction cycles (up to 3 iterations).

Filing Type	Count	Avg. Pages	Avg. Tokens
10-K	4,000	142	187,340
10-Q	4,000	68	82,150
8-K	2,000	12	14,820

TABLE I: Dataset composition by filing type.

Model	Provider	Context	Cost ($/1M tok.)
GPT-4o (2024-11-20)	OpenAI	128K	2.50 / 10.00
Claude 3.5 Sonnet	Anthropic	200K	3.00 / 15.00
Gemini 1.5 Pro	Google	1M	1.25 / 5.00
Llama 3 70B	Meta	128K	0.60 / 0.80
Mixtral 8x22B	Mistral	64K	0.50 / 0.70

TABLE II: Models evaluated in the benchmark.

Architecture	Metric	GPT-4o	Claude 3.5S	Gemini 1.5P	Llama3 70B	Mixtral 8x22B
Sequential (A)	F1	0.897	0.903	0.881	0.834	0.812
	Doc Acc	0.631	0.648	0.597	0.487	0.453
	Lat (s)	34.2	38.7	29.1	22.4	19.8
	Cost ($)	0.142	0.187	0.098	0.038	0.031
Parallel (B)	F1	0.908	0.914	0.893	0.851	0.829
	Doc Acc	0.659	0.672	0.623	0.521	0.488

TABLE III: Primary benchmark results. Best value per metric column shown in bold .

Domain	GPT-4o	Claude 3.5S	Gemini 1.5P
Financial Metrics	0.952	0.958	0.941
Governance	0.914	0.921	0.896
Exec. Compensation	0.887	0.898	0.871
Domain	Llama3 70B	Mixtral 8x22B
Financial Metrics	0.912	0.893
Governance	0.852	0.824

TABLE IV: Per-domain F1 for the hierarchical architecture.

Panel of Figure 9 — Fig. 5: Per-domain F1 across all five models under the hierarchical architecture. Financial metrics consistently achieve the highest scores due to standardized GAAP formats; executive compensation is the most challenging domain across all models.

Cache Config	F1	Cost/doc	Lat (s)	Hit Rate
No cache	0.929	$0.261	46.2	—
Section-level	0.927	$0.198	34.8	24.1%
Field-level	0.924	$0.171	29.3	38.7%
Hybrid (adaptive)	0.926	$0.182	31.4	31.4%

TABLE V: Impact of semantic caching on the hierarchical architecture.

Routing Strategy	F1	Cost/doc	Lat (s)
All Claude 3.5 Sonnet	0.929	$0.261	46.2
All GPT-4o	0.921	$0.198	41.8
All Mixtral 8x22B	0.843	$0.044	23.1
2-tier (Claude + Mixtral)	0.912	$0.127	31.6
3-tier (Claude + GPT + Mixtral)	0.918	$0.143	33.2

TABLE VI: Impact of model routing strategies.

Retry Strategy	F1	Cost/doc	Lat (s)	Retry Rate
No retry	0.908	$0.214	38.1	0%
Fixed retry (1x)	0.929	$0.261	46.2	14.2%
Confidence-gated	0.926	$0.243	43.7	10.8%
Escalation (model+)	0.931	$0.258	45.1	12.1%
Adaptive threshold	0.929	$0.247	44.0	11.6%

TABLE VII: Retry strategy comparison.

Configuration	F1	Cost/doc	Lat (s)
Sequential baseline	0.903	$0.187	38.7
Hierarchical baseline	0.929	$0.261	46.2
Reflexive baseline	0.943	$0.430	74.1
Hierarchical-Optimized	0.924	$0.148	30.2

TABLE VIII: Combined optimization: Hierarchical-Optimized configuration.

Panel of Figure 14 — Fig. 6: Ablation configurations plotted on the cost-F1 plane relative to the three baselines (gray squares). Routing variants achieve the largest cost reductions; the combined Hierarchical-Optimized configuration (green star) sits below baseline cost while recovering most of the accuracy gain.

Docs/Day	Seq. F1	Par. F1	Hier. F1	Refl. F1
1,000	0.903	0.914	0.929	0.943
5,000	0.903	0.914	0.928	0.941
10,000	0.901	0.913	0.926	0.937
25,000	0.899	0.911	0.919	0.924
50,000	0.894	0.906	0.907	0.898
100,000	0.886	0.898	0.891	0.871

TABLE IX: Scaling characteristics (Claude 3.5 Sonnet, fixed 8-node cluster). F1 scores at varying daily throughput.

Panel of Figure 16 — Fig. 7: F1 degradation as daily document volume scales from 1K to 100K. Reflexive accuracy collapses sharply beyond 25K docs/day due to timeout truncation of correction loops, while sequential degrades most gradually.

Architecture	$\boldsymbol{p_{50}}$	$\boldsymbol{p_{95}}$	$\boldsymbol{p_{99}}$
Sequential	38.7	62.4	89.1
Parallel	21.3	41.7	68.3
Hierarchical	46.2	78.3	124.6
Reflexive	74.1	148.7	247.3

TABLE X: Latency percentiles in seconds (Claude 3.5 Sonnet, 10K docs).

Panel of Figure 18 — Fig. 8: Latency percentile distribution ( $p_{50}$ , $p_{95}$ , $p_{99}$ ) per architecture (Claude 3.5 Sonnet, 10K documents). The reflexive architecture shows the widest spread, with a $p_{99}/p_{50}$ ratio of $3.34\times$ , making it unsuitable for strict SLA environments.

Panel of Figure 19 — Fig. 9: Radar chart comparing all four architectures across five normalized performance dimensions (higher = better on all axes; Speed and Cost Efficiency are inverted from their raw measures). Hierarchical dominates on token efficiency and balances accuracy and cost well. Reflexive leads on accuracy but scores zero on speed and cost efficiency. Parallel uniquely leads on speed. Sequential is the most cost-efficient and scale-resilient.

Panel of Figure 20 — Fig. 10: Cost-accuracy Pareto frontier across all architecture-model configurations. Each marker represents one architecture-model pair. The dashed line traces the Pareto frontier. The Hierarchical-Optimized point (green star) offers the best cost-efficiency knee, achieving F1 $=0.924$ at only $0.148/doc.

Failure Mode	Seq.	Par.	Hier.	Refl.
Cross-table reference failure	28.4	12.1	14.3	8.7
Temporal confusion (FY vs. QTR)	18.2	17.8	12.1	6.4
Unit/scale error (M vs. K vs. raw)	14.7	14.2	10.8	4.2
Proxy stmt. vs. 10-K confusion	9.3	6.8	7.2	5.1
Restated figure extraction	8.1	9.4	8.7	7.8
Compensation vesting misparse	7.6	8.3	9.1	8.4

TABLE XI: Failure mode prevalence (% of errors attributable to each mode).

Panel of Figure 22 — Fig. 11: Failure mode heatmap showing the percentage of errors attributable to each failure category per architecture. Darker red indicates higher prevalence. The reflexive architecture’s dominant failure (ambiguous disclosure resolution, 39.3%) and the sequential architecture’s cross-table reference failures (28.4%) are clearly visible.

Architecture	Input	Output	Total	Eff.
Sequential	142,340	3,820	146,160	2.61%
Parallel	168,720	4,210	172,930	2.43%
Hierarchical	197,480	5,640	203,120	2.78%
Reflexive	312,670	8,940	321,610	2.78%

TABLE XII: Token consumption breakdown (Claude 3.5 Sonnet, mean per document).

Panel of Figure 24 — Fig. 12: Stacked token consumption per architecture (input + output, mean per document, Claude 3.5 Sonnet). Output tokens are a small fraction in all cases. The reflexive architecture consumes $2.2\times$ more input tokens than sequential, explaining the cost differential despite identical token efficiency ratios.

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,039 papers

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task
Oct 20, 2026Pei LiuMulti-Agent LLM Systems LLM Agent Evaluation
School of Engineering and Applied Science, George Washington University, Washington, DC, USA
Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness
Mar 29, 2026Phat Nguyen, Thang PhamMulti-Agent LLMs LLM Agent Evaluation
Georgia Institute of Technology, Atlanta GA 30332, USA · Adobe Inc., San Jose CA 95110, USA
YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition
Jun 4, 2026PSBC LLM Team, Huawei LLM Team, Ruihan Long +56Financial Language Model Evaluation LLM Compression

Oct 20, 2026J/K move · Enter open · S save

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

Pei Liu

School of Engineering and Applied Science, George Washington University, Washington, DC, USA

Multi-Agent LLM Systems LLM Agent Evaluation LLM Agent Efficiency Financial Language Model Evaluation

Open paper AskPDF arXiv

Practitioner guidance holds that a large language model reviews a document better when the work is divided among agents. We test a narrow form of it in a controlled comparison of single-agent and multi-agent setups on a financial task, reviewing credit assessment memos against requirements. One agent checks six requirements in one call. A team of three on the same model checks the same memo, two requirements each, in the same words, with no roles or later stage. Labels are known for every requirement on 300 synthetic memos, so misses and false alarms are measured separately. Two primary predictions are tested on four models at a Bonferroni-corrected level. Under an instruction to decide each requirement once, the team catches 0.08 to 0.10 more of the defects that carry an explanation on the three models whose single agent misses some, at the corrected level on two. It raises more false alarms on all four, at the corrected level on three. Balanced accuracy, an exploratory measure, changes by 0.02 or less, and the team uses 2.2 to 2.4 times the tokens. Showing each agent the requirements it was not given reduces the added false alarms on the two Claude models, at the corrected level on one. In exploratory comparisons, sampling the single agent three times and flagging on any draw gives higher balanced accuracy than the team on those models, and the team's results depend on the instruction.

Abstract

Multi-Agent LLM Systems

Explore similar work

The More the Better? A Comparison of Single- and Multi-Agent LLM Setups on a Financial Review Task

Toward Reliable Evaluation of LLM-Based Financial Multi-Agent Systems: Taxonomy, Coordination Primacy, and Cost Awareness

YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition