LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis

Yen-Hsun Huang; Yu-Shiou Lin

cs.IRMar 29, 2026

LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis

Authors: Yen-Hsun Huang, Yu-Shiou Lin

Organizations: Department of Education, Taipei Veterans General Hospital, Taipei, Taiwan · Department of Psychiatry, Taipei Veterans General Hospital, Taipei, Taiwan

Abstract

Systematic reviews and meta-analyses (SR/MA) remain the gold standard for evidence synthesis, yet completing one typically requires 67 weeks and substantial expert effort. Recent large language model (LLM) systems have demonstrated strong performance on individual SR phases - screening (otto-SR: 96.7% sensitivity), extraction (Gartlehner et al.: 91.0% accuracy), and search (TrialMind: 0.83 recall) - but no study has reported what it actually costs to run an end-to-end pipeline, how cost distributes across phases, or how architectural choices affect the cost-quality trade-off. We present LUMEN, an open-source multi-agent pipeline that automates six SR/MA phases using 11 specialized LLM agents with deliberate model routing. We evaluate LUMEN on seven datasets: five self-conducted domain reviews (psychiatry, psychology, surgery, vaccinology, cardiology) and two SYNERGY screening benchmarks. Across 13 ground-truth-comparable outcomes, LUMEN achieves 100% directional agreement with published meta-analyses, with effect sizes within 1% for homogeneous study designs. The primary contribution is the first empirical cost and operational characterization of such a pipeline: a complete review costs 19 to 29 USD (median 22.65 USD), with title-abstract screening and data extraction together dominating expenditure. A three-arm extraction ablation reveals a phase-dependent architecture reversal: multi-agent design hurts screening but is essential for extraction, producing 5.7x more poolable analyses than single-model alternatives while eliminating clinically dangerous direction errors. A two-dataset screening benchmark demonstrates that model ranking is domain-dependent and not transferable across review topics. All code and cost logs are publicly available.

Figures & tables

Figure 1: LUMEN pipeline architecture. Six phases automate the SR/MA workflow using 11 specialized LLM agents with deliberate model routing. Phase 3.1 (expanded) shows the dual-screener mechanism with five-point relevance scoring and arbitration. Median per-phase costs are annotated.

Agent	Phase	Model	Role
Strategist	1	Claude Sonnet 4.6	PICO $\to$ search strategy
Rescue Screener	2.5	Gemini Flash Lite	Quarantine rescue
Screener 1	3.1	Gemini 3.1 Pro	5-point T/A screening
Screener 2	3.1	GPT-4.1 Mini	5-point T/A screening
Arbiter	3.1	Claude Sonnet 4.6	Conflict resolution
Full-text Screener	3.3	Claude Sonnet 4.6	PDF-level PICO verification

Table 1: Agent inventory.

ID	Domain	Yield	Screened	Included	GT Reference
D1	Antidepressants in elderly	5,168	3,756	16	Lenouvel 2024
D2	CBT-I for insomnia + depression	5,535	3,172	10	Furukawa 2024
D3	Lap. vs open cholecystectomy	6,233	4,334	46	Roy 2024
D4	Pneumococcal vaccines	2,010	1,389	49	Farrar 2023
D5	SGLT2i in heart failure	8,062	4,839	11	Vaduganathan 2022
S1	SYNERGY: Bos_2018	—	4,757	10	Expert labels

Table 2: Dataset characteristics.

Phase	D1	D2	D3	D4	D5	Range
P3.1 T/A Screening	20.37	16.80	5.69	7.30	25.00	$5.69–$ 25.00
P4 Extraction	3.72	3.69	9.50	10.67	2.92	$2.92–$ 10.67
P3.3 Full-text	0.66	1.19	2.41	2.70	0.04	$0.04–$ 2.70
P6 Manuscript	0.79	0.41	0.72	0.74	0.62	$0.41–$ 0.79
Other †	0.56	0.42	1.19	1.24	1.08	$0.42–$ 1.24
Total	26.10	22.51	19.51	22.65	29.04	$19.51–$ 29.04

Table 3: Cost breakdown by phase (USD).

Figure 2: Cost breakdown Per-phase cost breakdown across five domain reviews. Title–abstract screening (blue) dominates in high-yield domains (D1, D2, D5), while extraction (orange) dominates in high-inclusion domains (D3, D4). Dashed line indicates median total cost ($22.65).

Dataset	Arm	Sensitivity	Specificity	Missed	Review %	Cost
Bos_2018 (10 GT)	Gemini	0.800 (8/10)	0.649	2	24	$2.36
	GPT	0.900 (9/10)	0.700	1	19	$0.33
	Claude	0.900 (9/10)	0.691	1	17	$34.71
	Dual	0.900 (9/10)	0.618	1	28	$2.88
van_de_Schoot (38 GT)	Gemini	0.947 (36/38)	0.773	2	22	$17.53
	GPT	0.842 (32/38)	0.824	6	17	$2.40

Table 4: Screening benchmark at the sensitive operating point (score

\geq

3).

Figure 3: Screening scatter Screening sensitivity versus human review burden at the sensitive operating point (score $\geq$ 3) across four LLM arms on two SYNERGY datasets. Filled markers: Bos_2018; open markers: van_de_Schoot_2018; arrows trace cross-dataset movement. Upper-right = ideal (high sensitivity, low review). Claude (orange) is the only arm moving toward this ideal on both datasets. GPT (green) reverses from 0.90 to 0.84, missing 6 of 38 studies. Dual (purple) maintains top-tier sensitivity; its higher review% reflects a structured disagreement queue.

Domain	Arm A	Arm C	Arm D	GT (A)	Cost ratio
D1 Antidepressants	6	0	4	1/1 $\checkmark$	2.8 $\times$
D2 CBT-I	4	0	0	1/1 $\checkmark$	3.9 $\times$
D3 Cholecystectomy	11	2	8 ∗	4/4 $\checkmark$	2.8 $\times$
D4 Vaccines	6	1	4	3/3 $\checkmark$	2.6 $\times$
D5 SGLT2i	7	3	6	4/4 $\checkmark$	3.0 $\times$
Total	34	6	22	13/13	2.6–3.9 $\times$

Table 5: Extraction ablation: poolable analyses and GT agreement.

Figure 4: Extraction ablation Number of poolable meta-analyses produced by each extraction arm. The LUMEN 3-pass pipeline (Arm A, blue) consistently produces the most analyses with 100% GT directional agreement. Arm C (Sonnet, orange) fails in D1 and D2 due to verbatim naming. Asterisk on D3 Arm D indicates a clinically dangerous direction error.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Forest plot Forest plot comparison of LUMEN-derived pooled estimates (blue) versus published ground truth (orange) for select outcomes in Domains 3 and 5. All estimates agree directionally, with Domain 5 hazard ratios within 1% of published values.

Phase	API Calls	Input (M tok.)	Output (K tok.)	Wall-Clock (min)	Fail
P1 Strategy	2	0.008	5.0	0.4	0%
P2.5 Pre-screen	56 (14–80)	0.095	2.5	2.0	0%
P3.1 T/A Screen	2,352 (1,748–7,520)	5.16	288	242	0%
P3.3 FT Screen	41 (1–86)	0.457	9.7	18	0%
P4 Extraction	48 (3–147)	0.434	338	54	0%
P4.5 Planning	2 (2–3)	0.011	7.8	$<$ 2	0%

Appendix

Table S1: Per-phase operational metrics aggregated across five domain reviews. Median values reported with range in parentheses. Zero retries and zero failures across all five runs (

\sim

27,430 total API calls).

Domain	Screened	Included	Human Rev.	Cohen’s $\kappa$	PABAK	Arbiter
D1	3,756	22	49	0.74	0.99	8
D2	3,172	20	55	0.76	0.99	5
D3	4,334	185	296	0.73	0.95	70
D4	1,389	45	51	0.62	0.94	26
D5	4,839	—	309	0.21	0.93	54

Appendix

Table S2: Phase 3.1 dual-screener agreement metrics. PABAK = prevalence-adjusted bias-adjusted kappa. D5

\kappa=0.21

reflects a kappa paradox (

\sim

97% exclude); PABAK

=0.93

corrects for prevalence.

Analysis	Arm A	Arm C	Arm D
D1: Antidepressants in Elderly
Depression (all AD)	$k$ =8, SMD=0.12, $I^{2}$ =59%	—	—
Depression (SSRIs)	$k$ =6, SMD= $-$ 0.02, $I^{2}$ =51%	—	—
Response/remission	$k$ =3, OR=2.12, $I^{2}$ =58%	—	—
Cognition	$k$ =3, SMD=0.36, $I^{2}$ =91%	—	—
Adverse events	$k$ =7, OR=1.23, $I^{2}$ =59%	—	—

Appendix

Table S4: Extraction ablation per-analysis results. Arm A = LUMEN 3-pass; Arm C = single-pass Sonnet; Arm D = single-pass Gemini. “—” = no poolable analysis. ∗ Direction error (OR=0.88 vs. correct 2.15).

Analysis	Measure	$k$	ES	95% CI	$p$	$I^{2}$
D1: Antidepressants in Elderly
All AD – Depression	SMD	8	0.119	[ $-$ 0.42, 0.66]	0.58	58.5%
SSRIs – Depression	SMD	6	$-$ 0.019	[ $-$ 0.73, 0.70]	0.95	50.8%
Sertraline – Response	OR	3	2.12	[0.20, 22.20]	0.22	57.9%
Sertraline – Cognition	SMD	3	0.364	[ $-$ 3.41, 4.14]	0.60	90.8%
All AD – Adverse events	OR	7	1.23	[0.67, 2.24]	0.42	59.0%

Appendix

Table S5: Complete Phase 5 meta-analysis results (REML + Knapp–Hartung via R metafor ). Arm A only.

Dataset	Arm	Model	Sens $\geq$ 4	Sens $\geq$ 3	Spec $\geq$ 4	Spec $\geq$ 3	Rev.%	Cost
Bos_2018	A	Gemini	0.500	0.800	0.892	0.649	24	$2.36
(10 GT)	B	GPT	0.600	0.900	0.893	0.700	19	$0.33
	C	Claude	0.800	0.900	0.866	0.691	17	$34.71
	D	Dual	0.500	0.900	0.901	0.618	28	$2.88
vdS_2018	A	Gemini	0.579	0.947	0.994	0.773	22	$17.53
(38 GT)	B	GPT	0.526	0.842	0.993	0.824	17	$2.40

Appendix

Table S6: Screening validation on two SYNERGY datasets at both thresholds (conservative:

\geq

4; sensitive:

\geq

3). Bold = best per dataset. One study per dataset was missed by all four arms at both thresholds.

Systematic Review Automation

See all 18 papers

meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis
Apr 5, 2026Hsieh-Ting Lin, Jiunn-Tyng YehSystematic Review Automation AI-Assisted Scientific Research
Koo Foundation Sun Yat-Sen Cancer Center, Department of Oncology, Taipei, Taiwan · Duke Institute for Health Innovation, Durham, NC, USA
ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews
Oct 15, 2026Zhaoqian Xue, Shaoze Zhou, Xuexin Li +10Systematic Review Automation AI-Assisted Scientific Research
University of Arizona · Florida International University · Vanderbilt University +7
Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
Jun 15, 2026Anzhe Xie, Weihang Su, Yujia Zhou +2Systematic Review Automation AI-Assisted Scientific Research
Tsinghua University Beijing, China

Apr 5, 2026J/K move · Enter open · S save

meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis

Hsieh-Ting Lin, Jiunn-Tyng Yeh

Koo Foundation Sun Yat-Sen Cancer Center, Department of Oncology, Taipei, Taiwan · Duke Institute for Health Innovation, Durham, NC, USA

Systematic Review Automation AI-Assisted Scientific Research Scientific Workflow Automation LLM Agent Orchestration Human-in-the-Loop AI

Open paper AskPDF arXiv

Objective: To describe the architecture and design rationale of meta-pipe, an open-source large language model (LLM)-agent pipeline that integrates the complete systematic review and meta-analysis (SR/MA) workflow -- from literature search through statistical analysis, manuscript generation, and quality assurance -- with mandatory human oversight at critical decision points. Study Design and Setting: We developed a 10-stage modular pipeline integrating Claude (Anthropic; Opus 4 for reasoning, Haiku 3.5 for classification) for LLM-assisted screening and extraction, Python (~3,600 lines of code) for automation, R (meta, metafor, gemtc, netmeta) for statistical analysis, and Quarto for manuscript rendering. Five mandatory human decision points enforce oversight. We systematically compared meta-pipe's capabilities with five existing SR automation tools based on published documentation as of March 2026. Results: meta-pipe offers four capabilities not available in any single existing tool: automated manuscript generation from analysis outputs, semi-automated GRADE assessment, overclaim detection (12 predefined patterns), and dual-paradigm network meta-analysis (Bayesian and frequentist). Estimated API cost is $15-30 per typical 5-10 study review. No validation data are reported; this is a system description, not a validation study. Conclusion: End-to-end AI-assisted evidence synthesis is architecturally feasible as an open-source tool with mandatory human oversight. Formal validation reproducing published Cochrane reviews is underway and essential before routine use.

Abstract

Systematic Review Automation

Explore similar work

meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis

ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews

Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio