meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis
Organizations: Koo Foundation Sun Yat-Sen Cancer Center, Department of Oncology, Taipei, Taiwan · Duke Institute for Health Innovation, Durham, NC, USA
Abstract
Objective: To describe the architecture and design rationale of meta-pipe, an open-source large language model (LLM)-agent pipeline that integrates the complete systematic review and meta-analysis (SR/MA) workflow -- from literature search through statistical analysis, manuscript generation, and quality assurance -- with mandatory human oversight at critical decision points. Study Design and Setting: We developed a 10-stage modular pipeline integrating Claude (Anthropic; Opus 4 for reasoning, Haiku 3.5 for classification) for LLM-assisted screening and extraction, Python (~3,600 lines of code) for automation, R (meta, metafor, gemtc, netmeta) for statistical analysis, and Quarto for manuscript rendering. Five mandatory human decision points enforce oversight. We systematically compared meta-pipe's capabilities with five existing SR automation tools based on published documentation as of March 2026. Results: meta-pipe offers four capabilities not available in any single existing tool: automated manuscript generation from analysis outputs, semi-automated GRADE assessment, overclaim detection (12 predefined patterns), and dual-paradigm network meta-analysis (Bayesian and frequentist). Estimated API cost is $15-30 per typical 5-10 study review. No validation data are reported; this is a system description, not a validation study. Conclusion: End-to-end AI-assisted evidence synthesis is architecturally feasible as an open-source tool with mandatory human oversight. Formal validation reproducing published Cochrane reviews is underway and essential before routine use.
Figures & tables
| Stage | Module | Key Tasks | Outputs | Quality Gate | Human Decision |
|---|---|---|---|---|---|
| 00 | Topic intake | Brainstorming, feasibility assessment | PICO definition, eligibility criteria | Completeness check | Yes: define scope |
| 01-02 | Search & bibliography | Database search, deduplication | dedupe.bib, search strategy | Duplicate detection | No |
| 03 | Screening | LLM-assisted screening with test-retest check | decisions.csv, kappa statistic | Kappa 0.60 | Yes: resolve disagreements |
| 03b | Analysis gate | Pairwise vs NMA determination | analysis-type-decision.md | Network geometry check | Yes: confirm analysis type |
| 04 | Full-text management | PDF retrieval, Unpaywall integration | manifest.csv, full-text PDFs | Retrieval rate | No |
| 05 | Data extraction | LLM-assisted extraction, validation | extraction.csv, data dictionary | Range/consistency checks | Yes: review flagged items |
| Dimension | Covidence | TrialMind | otto-SR | Nested Knowledge | DistillerSR | meta-pipe |
| Intended scope | SR mgmt | Search Extract | Screen Extract+RoB | Full SR/MA | SR mgmt | Full SR/MA |
| Search strategy | - | AI | - | AI | - | AI |
| T/A screening | AI | AI | AI | AI | AI | AI |
| Full-text screening | Manual | AI | AI | Manual | Manual | AI |
| Data extraction | AI a | AI | AI | AI | AI | AI |
| Risk of bias | Template | - | AI | Template | Template b | AI |
| Stage Boundary | Format | Key Fields / Schema | Validation |
|---|---|---|---|
| 00 01 | JSON | pico.json : population, intervention, comparator, outcomes, study_design, date_range, databases | Documentation only |
| 01 02 | BibTeX | Standard BibTeX fields (author, title, year, journal, abstract, doi) | Documentation only |
| 02 03 | BibTeX | dedupe.bib : deduplicated bibliography with unique IDs | Automated (duplicate check) |
| 03 03b | CSV | decisions.csv : study_id, title, decision (include/exclude/uncertain), rationale, confidence, pass1, pass2, agreement | Automated (kappa computation) |
| 03b 04 | Markdown | analysis-type-decision.md : pairwise or NMA, rationale, human approval | Documentation only |
| 04 05 | CSV | manifest.csv : study_id, pdf_path, retrieval_method (unpaywall/manual), retrieval_status | Automated (file existence check) |
| # | Pattern | Description | Example Flagged Text |
|---|---|---|---|
| 1 | Causal language for non-randomized data | Causal verbs (caused, led to, resulted in) used for observational findings | “Smoking caused increased mortality” (in cohort study) |
| 2 | Overgeneralization beyond study population | Claims extending beyond the actual study population | “All patients with cancer should…” (when study was TNBC only) |
| 3 | Recommendations without evidence level | Treatment recommendations without stating evidence quality | “Clinicians should prescribe…” (without GRADE context) |
| 4 | Unsupported superlatives | “First,” “only,” “best,” “most effective” without citation | “The most effective treatment…” |
| 5 | Certainty language for uncertain findings | “Clearly,” “definitely,” “proves” for findings with wide CIs | “This clearly demonstrates …” |
| 6 | Extrapolation to unexamined outcomes | Claims about outcomes not measured in included studies | Safety conclusions drawn from efficacy-only data |