stat.MEApr 3, 2026

Eligibility-Aware Evidence Synthesis: An Agentic Framework for Clinical Trial Meta-Analysis

Authors: Yao Zhao, Zhiyue Zhang, Yanxun Xu

Organizations: Department of Applied Mathematics and Statistics, Johns Hopkins University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Clinical evidence synthesis requires identifying relevant trials from large registries and aggregating results that account for population differences. While recent LLM-based approaches have automated components of systematic review, they do not support end-to-end evidence synthesis. Moreover, conventional meta-analysis weights studies by statistical precision without considering clinical compatibility reflected in eligibility criteria. We propose EligMeta, an agentic framework that integrates automated trial discovery with eligibility-aware meta-analysis, translating natural-language queries into reproducible trial selection and incorporating eligibility alignment into study weighting to produce cohort-specific pooled estimates. EligMeta employs a hybrid architecture separating LLM-based reasoning from deterministic execution: LLMs generate interpretable rules from natural-language queries and perform schema-constrained parsing of trial metadata, while all logical operations, weight computations, and statistical pooling are executed deterministically to ensure reproducibility. The framework structures eligibility criteria and computes similarity-based study weights reflecting population alignment between target and comparator trials. In a gastric cancer landscape analysis, EligMeta reduced 4,044 candidate trials to 39 clinically relevant studies through rule-based filtering, recovering all 13 guideline-cited trials. In an olaparib adverse events meta-analysis across four trials, eligibility-aware weighting shifted the pooled risk ratio from 2.18 (95% CI: 1.71-2.79) under conventional Mantel-Haenszel estimation to 1.97 (95% CI: 1.76-2.20), demonstrating quantifiable impact of incorporating eligibility alignment. EligMeta bridges automated trial discovery with eligibility-aware meta-analysis, providing a scalable and reproducible framework for evidence synthesis in precision medicine.

Figures & tables

Figure 1: Overview of the EligMeta framework for clinical trial evidence synthesis. A free-text clinical query is translated into structured, auditable rule sets defining study selection criteria and a target eligibility profile. These rules are applied through deterministic evaluation to retrieve and filter relevant trials from public registries, with all intermediate artifacts available for expert review. The resulting studies support downstream landscape analysis and eligibility-aware meta-analysis, in which each study’s contribution to the pooled estimate is weighted by its population similarity to the target trial.
Figure 2: Trial selection and structuring workflow. Left: A free-text clinical query is translated into a reviewed set of structured rules defining study selection criteria and target conditions. Each rule is transformed to a function plan via LLM reasoning. Right: The function plans are applied to trial records using format-constrained parsing and deterministic evaluation, producing a filtered set of trials and structured summaries. LLM usage is restricted to rule specification and parsing, while all selection operations are deterministic and auditable.
Figure 3: Eligibility-aware meta-analysis workflow. Left: Free-text eligibility criteria, together with a target disease specification, are translated into explicit, human-readable penalty rules that capture clinically meaningful differences between trial populations and the target scenario. These rules are evaluated on a structured representation of eligibility criteria to produce trial-level penalty scores through deterministic operations. Right: Penalty scores are transformed into eligibility weights, which are then incorporated into weighted meta-analysis to produce population-aligned effect estimates. LLM usage is restricted to penalty rule specification and schema-constrained parsing, while penalty evaluation, score transformation, and statistical estimation are deterministic and auditable.
Figure 4: Stepwise filtering results for the gastric cancer use case. The diagram follows the six rules described in the text, showing the number of clinical trials remaining and excluded at each filtering step.
 OlaparibPlacebo     
StudyEventsTotalEventsTotalRRCI low{}_{\text{low}}CI high{}_{\text{high}}wi,MHdispw_{i,MH}^{disp}wi,EWdispw_{i,EW}^{disp}
Golan 201918919601.320.632.7413.634.6
Moore 2018104260191302.741.764.2631.720.5
Ledermann 201443136181282.251.373.6923.224.4
Pujade–Lauraine 20177319519991.951.253.0431.520.4
Total (original result from Ricci et al. [26] )2.181.712.79  
Table 2: Olaparib maintenance treatment vs placebo, outcome: risk ratio (RR) of vomiting (all-grade). wi,MHdispw_{i,MH}^{disp} : conventional precision-based display weights from Ricci et al. [ 26 ] . wi,EWdispw_{i,EW}^{disp} : eligibility-weighted display weights using Golan et al. [ 27 ] as target population. CI: 95% confidence interval.
Figure 5: Comparison of pooled risk ratio estimates for all-grade vomiting under classical precision-based weighting (left) and eligibility-aware weighting (right), for olaparib maintenance versus placebo.
More in this topic

Evidence Selection

Evidence selection chooses the documents, passages, claims, or observations that best support an answer or decision.

See all 60 papers

Explore similar work

CardsList
Jun 15, 2026

Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio

Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442 expert-curated meta-analyses from Nature Portfolio journals. Each entry pairs a research question with PI/ECO criteria, a retrieval corpus of 140k PubMed articles, verified positive studies, hard negatives that are topically similar but PI/ECO-ineligible, and complete search strategies and date bounds. Benchmarking twelve pipeline configurations (nine RAG variants and a protocol-driven agent) reveals a critical screening bottleneck: despite a retrieval ceiling of 90.9% recall at K=200, no system recovers more than 52.7% of ground-truth included literature. Current LLMs fail to reliably separate eligible studies from PI/ECO-failing distractors in pools of comparable topical relevance. Stage-attributed metrics capture where systems succeed and fail; a single end-to-end score does not.

Anzhe Xie, Weihang Su, Yujia Zhou +2

Tsinghua University Beijing, China

Systematic Review AutomationAI-Assisted Scientific ResearchLLM Agent EvaluationAI Agent BenchmarksBiomedical IREvidence Retrieval+1
AskPDF
May 10, 2026

MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies

Large language models (LLMs) have saturated standard medical benchmarks that test factual recall, yet their ability to perform higher-order reasoning, such as synthesizing evidence from multiple sources, remains critically under-explored. To address this gap, we introduce MedMeta, the first benchmark designed to evaluate an LLM's ability to generate conclusions from medical meta-analyses using only the abstracts of cited studies. MedMeta comprises 81 meta-analyses from PubMed (2018--2025) and evaluates models using two distinct workflows: a Retrieval-Augmented Generation (Golden-RAG) setting with ground-truth abstracts, and a Parametric-only approach relying on internal knowledge. Our evaluation framework is validated by a well-structured analysis showing our LLM-as-a-judge protocol strongly aligns with human expert ratings, as evidenced by high Pearson's r correlation (0.81) and Bland-Altman analysis revealing negligible systematic bias, establishing it as a reliable proxy for scalable evaluation. Our findings underscore the critical importance of information grounding: the Golden-RAG workflow consistently and significantly outperforms the Parametric-only approach across models. In contrast, the benefits of domain-specific fine-tuning are marginal and largely neutralized when external material is provided. Furthermore, stress tests show that all models, regardless of architecture, fail to identify and reject negated evidence, highlighting a critical vulnerability in current RAG systems. Notably, even under ideal RAG conditions, current LLMs achieve only slightly above-average performance (~2.7/5.0). MedMeta provides a challenging new benchmark for evidence synthesis and demonstrates that for clinical applications, developing robust RAG systems is a more promising direction than model specialization alone.

Huy Hoang Ha, Benoit Favre, Francois Portet

HealthcareRAG EvaluationClinical Language Model Evaluation
AskPDF
Apr 5, 2026

meta-pipe: An LLM-agent pipeline for end-to-end automated systematic review and meta-analysis

Objective: To describe the architecture and design rationale of meta-pipe, an open-source large language model (LLM)-agent pipeline that integrates the complete systematic review and meta-analysis (SR/MA) workflow -- from literature search through statistical analysis, manuscript generation, and quality assurance -- with mandatory human oversight at critical decision points. Study Design and Setting: We developed a 10-stage modular pipeline integrating Claude (Anthropic; Opus 4 for reasoning, Haiku 3.5 for classification) for LLM-assisted screening and extraction, Python (~3,600 lines of code) for automation, R (meta, metafor, gemtc, netmeta) for statistical analysis, and Quarto for manuscript rendering. Five mandatory human decision points enforce oversight. We systematically compared meta-pipe's capabilities with five existing SR automation tools based on published documentation as of March 2026. Results: meta-pipe offers four capabilities not available in any single existing tool: automated manuscript generation from analysis outputs, semi-automated GRADE assessment, overclaim detection (12 predefined patterns), and dual-paradigm network meta-analysis (Bayesian and frequentist). Estimated API cost is $15-30 per typical 5-10 study review. No validation data are reported; this is a system description, not a validation study. Conclusion: End-to-end AI-assisted evidence synthesis is architecturally feasible as an open-source tool with mandatory human oversight. Formal validation reproducing published Cochrane reviews is underway and essential before routine use.

Hsieh-Ting Lin, Jiunn-Tyng Yeh

Koo Foundation Sun Yat-Sen Cancer Center, Department of Oncology, Taipei, Taiwan · Duke Institute for Health Innovation, Durham, NC, USA

Systematic Review AutomationAI-Assisted Scientific ResearchScientific Workflow AutomationLLM Agent OrchestrationHuman-in-the-Loop AI
AskPDF