Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies
Authors: Siddhant Kulkarni, Yukta Kulkarni
Organizations: Department of Computer Science and Engineering, New York University, New York, NY 10012
The adoption of large language models (LLMs) for structured information extraction from financial documents has accelerated rapidly, yet production deployments face fundamental architectural decisions with limited empirical guidance. We present a systematic benchmark comparing four multi-agent orchestration architectures: sequential pipeline, parallel fan-out with merge, hierarchical supervisor-worker and reflexive self-correcting loop. These are evaluated across five frontier and open-weight LLMs on a corpus of 10,000 SEC filings (10-K, 10-Q and 8-K forms). Our evaluation spans 25 extraction field types covering governance structures, executive compensation and financial metrics, measured along five axes: field-level F1, document-level accuracy, end-to-end latency, cost per document and token efficiency. We find that reflexive architectures achieve the highest field-level F1 (0.943) but at 2.3x the cost of sequential baselines, while hierarchical architectures occupy the most favorable position on the cost-accuracy Pareto frontier (F1 0.921 at 1.4x cost). We further present ablation studies on semantic caching, model routing and adaptive retry strategies, demonstrating that hybrid configurations can recover 89% of the reflexive architecture's accuracy gains at only 1.15x baseline cost. Our scaling analysis from 1K to 100K documents per day reveals non-obvious throughput-accuracy degradation curves that inform capacity planning. These findings provide actionable guidance for practitioners deploying multi-agent LLM systems in regulated financial environments.
Figures & tables
Fig. 1: Architecture A: Sequential Pipeline. Documents flow through a fixed chain of agents with cumulative context passing.
Fig. 2: Architecture B: Parallel Fan-Out with Merge. Independent extraction branches execute concurrently before reconciliation.
Fig. 3: Architecture C: Hierarchical Supervisor-Worker. The supervisor dynamically allocates tasks and selectively re-extracts low-confidence fields.
Fig. 4: Architecture D: Reflexive Self-Correcting Loop. Verification failures trigger iterative critique and correction cycles (up to 3 iterations).
Filing Type
Count
Avg. Pages
Avg. Tokens
10-K
4,000
142
187,340
10-Q
4,000
68
82,150
8-K
2,000
12
14,820
TABLE I: Dataset composition by filing type.
Model
Provider
Context
Cost ($/1M tok.)
GPT-4o (2024-11-20)
OpenAI
128K
2.50 / 10.00
Claude 3.5 Sonnet
Anthropic
200K
3.00 / 15.00
Gemini 1.5 Pro
Google
1M
1.25 / 5.00
Llama 3 70B
Meta
128K
0.60 / 0.80
Mixtral 8x22B
Mistral
64K
0.50 / 0.70
TABLE II: Models evaluated in the benchmark.
Architecture
Metric
GPT-4o
Claude 3.5S
Gemini 1.5P
Llama3 70B
Mixtral 8x22B
Sequential (A)
F1
0.897
0.903
0.881
0.834
0.812
Doc Acc
0.631
0.648
0.597
0.487
0.453
Lat (s)
34.2
38.7
29.1
22.4
19.8
Cost ($)
0.142
0.187
0.098
0.038
0.031
Parallel (B)
F1
0.908
0.914
0.893
0.851
0.829
Doc Acc
0.659
0.672
0.623
0.521
0.488
TABLE III: Primary benchmark results. Best value per metric column shown in bold .
Domain
GPT-4o
Claude 3.5S
Gemini 1.5P
Financial Metrics
0.952
0.958
0.941
Governance
0.914
0.921
0.896
Exec. Compensation
0.887
0.898
0.871
Domain
Llama3 70B
Mixtral 8x22B
Financial Metrics
0.912
0.893
Governance
0.852
0.824
TABLE IV: Per-domain F1 for the hierarchical architecture.
Fig. 5: Per-domain F1 across all five models under the hierarchical architecture. Financial metrics consistently achieve the highest scores due to standardized GAAP formats; executive compensation is the most challenging domain across all models.
Cache Config
F1
Cost/doc
Lat (s)
Hit Rate
No cache
0.929
$0.261
46.2
—
Section-level
0.927
$0.198
34.8
24.1%
Field-level
0.924
$0.171
29.3
38.7%
Hybrid (adaptive)
0.926
$0.182
31.4
31.4%
TABLE V: Impact of semantic caching on the hierarchical architecture.
Fig. 6: Ablation configurations plotted on the cost-F1 plane relative to the three baselines (gray squares). Routing variants achieve the largest cost reductions; the combined Hierarchical-Optimized configuration (green star) sits below baseline cost while recovering most of the accuracy gain.
Docs/Day
Seq. F1
Par. F1
Hier. F1
Refl. F1
1,000
0.903
0.914
0.929
0.943
5,000
0.903
0.914
0.928
0.941
10,000
0.901
0.913
0.926
0.937
25,000
0.899
0.911
0.919
0.924
50,000
0.894
0.906
0.907
0.898
100,000
0.886
0.898
0.891
0.871
TABLE IX: Scaling characteristics (Claude 3.5 Sonnet, fixed 8-node cluster). F1 scores at varying daily throughput.
Fig. 7: F1 degradation as daily document volume scales from 1K to 100K. Reflexive accuracy collapses sharply beyond 25K docs/day due to timeout truncation of correction loops, while sequential degrades most gradually.
Fig. 8: Latency percentile distribution ( p50 , p95 , p99 ) per architecture (Claude 3.5 Sonnet, 10K documents). The reflexive architecture shows the widest spread, with a p99/p50 ratio of 3.34× , making it unsuitable for strict SLA environments.
Fig. 9: Radar chart comparing all four architectures across five normalized performance dimensions (higher = better on all axes; Speed and Cost Efficiency are inverted from their raw measures). Hierarchical dominates on token efficiency and balances accuracy and cost well. Reflexive leads on accuracy but scores zero on speed and cost efficiency. Parallel uniquely leads on speed. Sequential is the most cost-efficient and scale-resilient.
Fig. 10: Cost-accuracy Pareto frontier across all architecture-model configurations. Each marker represents one architecture-model pair. The dashed line traces the Pareto frontier. The Hierarchical-Optimized point (green star) offers the best cost-efficiency knee, achieving F1 =0.924 at only $0.148/doc.
Failure Mode
Seq.
Par.
Hier.
Refl.
Cross-table reference failure
28.4
12.1
14.3
8.7
Temporal confusion (FY vs. QTR)
18.2
17.8
12.1
6.4
Unit/scale error (M vs. K vs. raw)
14.7
14.2
10.8
4.2
Proxy stmt. vs. 10-K confusion
9.3
6.8
7.2
5.1
Restated figure extraction
8.1
9.4
8.7
7.8
Compensation vesting misparse
7.6
8.3
9.1
8.4
TABLE XI: Failure mode prevalence (% of errors attributable to each mode).
Fig. 11: Failure mode heatmap showing the percentage of errors attributable to each failure category per architecture. Darker red indicates higher prevalence. The reflexive architecture’s dominant failure (ambiguous disclosure resolution, 39.3%) and the sequential architecture’s cross-table reference failures (28.4%) are clearly visible.
Architecture
Input
Output
Total
Eff.
Sequential
142,340
3,820
146,160
2.61%
Parallel
168,720
4,210
172,930
2.43%
Hierarchical
197,480
5,640
203,120
2.78%
Reflexive
312,670
8,940
321,610
2.78%
TABLE XII: Token consumption breakdown (Claude 3.5 Sonnet, mean per document).
Fig. 12: Stacked token consumption per architecture (input + output, mean per document, Claude 3.5 Sonnet). Output tokens are a small fraction in all cases. The reflexive architecture consumes 2.2× more input tokens than sequential, explaining the cost differential despite identical token efficiency ratios.
Practitioner guidance holds that a large language model reviews a document better when the work is divided among agents. We test a narrow form of it in a controlled comparison of single-agent and multi-agent setups on a financial task, reviewing credit assessment memos against requirements. One agent checks six requirements in one call. A team of three on the same model checks the same memo, two requirements each, in the same words, with no roles or later stage. Labels are known for every requirement on 300 synthetic memos, so misses and false alarms are measured separately. Two primary predictions are tested on four models at a Bonferroni-corrected level. Under an instruction to decide each requirement once, the team catches 0.08 to 0.10 more of the defects that carry an explanation on the three models whose single agent misses some, at the corrected level on two. It raises more false alarms on all four, at the corrected level on three. Balanced accuracy, an exploratory measure, changes by 0.02 or less, and the team uses 2.2 to 2.4 times the tokens. Showing each agent the requirements it was not given reduces the added false alarms on the two Claude models, at the corrected level on one. In exploratory comparisons, sampling the single agent three times and flagging on any draw gives higher balanced accuracy than the team on those models, and the team's results depend on the instruction.
Pei Liu
School of Engineering and Applied Science, George Washington University, Washington, DC, USA
Multi-agent systems based on large language models (LLMs) for financial trading have grown rapidly since 2023, yet the field lacks a shared framework for understanding what drives performance or for evaluating claims credibly. This survey makes three contributions. First, we introduce a four-dimensional taxonomy, covering architecture pattern, coordination mechanism, memory architecture, and tool integration; applied to 12 multi-agent systems and two single-agent baselines. Second, we formulate the Coordination Primacy Hypothesis (CPH): inter-agent coordination protocol design is a primary driver of trading decision quality, often exerting greater influence than model scaling. CPH is presented as a falsifiable research hypothesis supported by tiered structural evidence rather than as an empirically validated conclusion; its definitive validation requires evaluation infrastructure that does not yet exist in the field. Third, we document five pervasive evaluation failures (look-ahead bias, survivorship bias, backtesting overfitting, transaction cost neglect, and regime-shift blindness) and show that these can reverse the sign of reported returns. Building on the CPH and the evaluation critique, we introduce the Coordination Breakeven Spread (CBS), a metric for determining whether multi-agent coordination adds genuine value net of transaction costs, and propose minimum evaluation standards as prerequisites for validating the CPH.
Phat Nguyen, Thang Pham
Georgia Institute of Technology, Atlanta GA 30332, USA · Adobe Inc., San Jose CA 95110, USA
Large language models (LLMs) drive significant financial innovations, yet their high-concurrency deployment is severely bottlenecked by KV cache memory overhead, which inflates infrastructure costs and throttles scalability. To address this, we propose YouZhi-LLM, a highly efficient financial LLM empowered by a comprehensive structural transition and training pipeline natively built on the Huawei Ascend ecosystem. At its algorithmic core, YouZhi-LLM features a layer-adaptive GQA-to-MLA transition framework that dynamically assigns per-layer FreqFold sizes, maximizing KV-cache compression while minimizing perplexity degradation. To recover representation capacity and inject domain expertise, the Ascend-based training pipeline seamlessly integrates generalized knowledge distillation with financial-specific supervised fine-tuning. Evaluations demonstrate the superiority of this systematic approach, with the adaptive transition reducing perplexity degradation by up to 35% over uniform baselines. Crucially, when evaluated on Ascend NPUs via vLLM-Ascend, the massive KV-cache reduction translates directly into deployment efficiency. Compared to their respective base models, YouZhi-7B yields a 12.3% improvement in average financial benchmark score alongside a 2.69× increase in maximum concurrency; similarly, YouZhi-14B achieves a 7.0% accuracy gain and a 2.43× concurrency boost, establishing a new paradigm for cost-effective, high-throughput financial inference.