Organizations: Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, IL, USA · Keiji AI Inc, Seattle, WA, USA · Institute of Scientific and Industrial Research, The University of Osaka, Suita, Japan · University of Pennsylvania, Philadelphia, PA, USA · AbbVie Inc, North Chicago, IL, USA · Carle Illinois College of Medicine, University of Illinois Urbana-Champaign, Urbana, IL, USA
Nearly 90% of drugs entering clinical development ultimately fail, despite billions of dollars in investment. Pharmaceutical companies therefore rely on clinical development planning (CDP) and probability of technical and regulatory success assessment to anticipate development risks, yet these decisions remain labor-intensive and subjective, requiring experts across clinical science, statistics, regulatory affairs, and competitive intelligence to jointly acquire, synthesize, and reason over heterogeneous evidence. Here, we introduce TrialAtlas, a memory-augmented multi-agent research organization for CDP that mirrors this collaborative process by coordinating specialized agents for literature synthesis, competitive trial intelligence, regulatory precedent analysis, and integrated reasoning over trial design and development risk. TrialAtlas further learns from historical clinical trials and regulatory outcomes, including prior New Drug Applications (NDAs), to ground its decisions in accumulated development experience. To evaluate these capabilities in an authentic regulatory setting, we introduce TrialAtlasBench, constructed from 291 FDA Complete Response Letters and spanning three practical tasks: detecting trial design deficiencies, recommending actionable design improvements, and predicting technical and regulatory success. TrialAtlas achieves an F1 score of 50.0% for deficiency detection, outperforming the strongest baseline by 6.1 points, and reaches 85.3% balanced accuracy and 84.7% F1 for prediction of technical and regulatory success, improving over the best baselines by 6.7 points in balanced accuracy and 12.0 points in Cohen's kappa. In expert evaluation, 86.4% of TrialAtlas-generated concerns were judged valid, compared with 83.1% for OpenAI DeepResearch and 59.3% for Gemini DeepResearch.
Figures & tables
Figure 1
Figure 2: TrialAtlas improves detection of FDA-aligned trial design deficiencies. a, Task formulation for trial design deficiency detection. Given structured clinical trial design information, including interventions, eligibility criteria, study arms, outcome measures, phase and others, systems are required to identify all FDA-aligned design deficiencies that may affect regulatory approval. b, Quantitative performance comparison on the deficiency detection task, evaluated using precision, recall, and F1 score, for TrialAtlas and the four baselines (a standalone LLM, a ReAct agent, Biomni, and TrialGenie). c, Example intermediate process in TrialAtlas , illustrating PubMed literature search and evidence collection based on parsed trial design information. d, Qualitative comparison of deficiency detection with and without the memory base. Without memory, TrialAtlas may incorrectly flag design elements that conform to established regulatory norms. With memory, prior experience is incorporated, preventing spurious deficiency flags in trials that follow established standards.
Figure 3: TrialAtlas supports actionable clinical trial design optimization. a, Task formulation for trial design optimization. Given structured clinical trial design information, systems are required to generate FDA-aligned, protocol-level optimization suggestions that improve design quality while preserving feasibility and regulatory acceptability. b, Quantitative performance comparison on the design optimization task, evaluated using precision, recall, and F1 score, for TrialAtlas and the four baselines (a standalone LLM, a ReAct agent, Biomni, and TrialGenie). c, Example outputs of TrialAtlas , including planning, department-level analyses (Competing Trial Analysis, Literature Evidence Synthesis, and Drug Benchmarking), and the resulting prioritized optimization recommendations.
Figure 4: TrialAtlas predicts technical and regulatory success from trial design. a, Task formulation for the prediction of technical and regulatory success, where a trial’s approval outcome is inferred from structured clinical trial design information. b, Performance comparison across systems on the prediction of technical and regulatory success, evaluated using balanced accuracy, F1 score, and Cohen’s kappa. c, Example analysis results produced by TrialAtlas to contextualize approval predictions, including enrollment distributions stratified by trial status and distributions of treatment categories from relevant PubMed literature.
Figure 5: Human expert evaluation of AI-generated trial design analyses. a, Design of the blinded expert study. For each held-out trial, three systems produced trial design deficiency reports: TrialAtlas and two frontier general-purpose deep-research agents, OpenAI DeepResearch and Gemini DeepResearch. Six experts from the pharmaceutical industry then rated every reported deficiency as a valid concern or not, blind to which system had produced it. b, Structure of the evaluation form. For each trial, an expert was shown the structured trial design (intervention, condition, eligibility, study arms, outcome measures, and phase), the corresponding FDA Complete Response Letter as a reference for the regulatory concerns, and the system’s full report. The expert then rated each flagged deficiency as a valid concern or not a valid concern. c, Proportion of each system’s flagged deficiencies that experts judged to be valid concerns. TrialAtlas achieves the highest proportion (86.4%) among the three systems, ahead of OpenAI DeepResearch (83.1%) and Gemini DeepResearch (59.3%).
Department of Computer Science, University of Pisa, Pisa, Italy · Department of Computer Engineering, Modeling, Electronics and Systems, University of Calabria, Rende, Italy. · Institute of Information Science and Technologies “Alessandro Faedo”, National Research Council, Pisa, Italy. +2