Organizations: Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, IL, USA · Keiji AI Inc, Seattle, WA, USA · Institute of Scientific and Industrial Research, The University of Osaka, Suita, Japan · University of Pennsylvania, Philadelphia, PA, USA · AbbVie Inc, North Chicago, IL, USA · Carle Illinois College of Medicine, University of Illinois Urbana-Champaign, Urbana, IL, USA
Nearly 90% of drugs entering clinical development ultimately fail, despite billions of dollars in investment. Pharmaceutical companies therefore rely on clinical development planning (CDP) and probability of technical and regulatory success assessment to anticipate development risks, yet these decisions remain labor-intensive and subjective, requiring experts across clinical science, statistics, regulatory affairs, and competitive intelligence to jointly acquire, synthesize, and reason over heterogeneous evidence. Here, we introduce TrialAtlas, a memory-augmented multi-agent research organization for CDP that mirrors this collaborative process by coordinating specialized agents for literature synthesis, competitive trial intelligence, regulatory precedent analysis, and integrated reasoning over trial design and development risk. TrialAtlas further learns from historical clinical trials and regulatory outcomes, including prior New Drug Applications (NDAs), to ground its decisions in accumulated development experience. To evaluate these capabilities in an authentic regulatory setting, we introduce TrialAtlasBench, constructed from 291 FDA Complete Response Letters and spanning three practical tasks: detecting trial design deficiencies, recommending actionable design improvements, and predicting technical and regulatory success. TrialAtlas achieves an F1 score of 50.0% for deficiency detection, outperforming the strongest baseline by 6.1 points, and reaches 85.3% balanced accuracy and 84.7% F1 for prediction of technical and regulatory success, improving over the best baselines by 6.7 points in balanced accuracy and 12.0 points in Cohen's kappa. In expert evaluation, 86.4% of TrialAtlas-generated concerns were judged valid, compared with 83.1% for OpenAI DeepResearch and 59.3% for Gemini DeepResearch.
Figures & tables
Figure 1
Figure 2: TrialAtlas improves detection of FDA-aligned trial design deficiencies. a, Task formulation for trial design deficiency detection. Given structured clinical trial design information, including interventions, eligibility criteria, study arms, outcome measures, phase and others, systems are required to identify all FDA-aligned design deficiencies that may affect regulatory approval. b, Quantitative performance comparison on the deficiency detection task, evaluated using precision, recall, and F1 score, for TrialAtlas and the four baselines (a standalone LLM, a ReAct agent, Biomni, and TrialGenie). c, Example intermediate process in TrialAtlas , illustrating PubMed literature search and evidence collection based on parsed trial design information. d, Qualitative comparison of deficiency detection with and without the memory base. Without memory, TrialAtlas may incorrectly flag design elements that conform to established regulatory norms. With memory, prior experience is incorporated, preventing spurious deficiency flags in trials that follow established standards.
Figure 3: TrialAtlas supports actionable clinical trial design optimization. a, Task formulation for trial design optimization. Given structured clinical trial design information, systems are required to generate FDA-aligned, protocol-level optimization suggestions that improve design quality while preserving feasibility and regulatory acceptability. b, Quantitative performance comparison on the design optimization task, evaluated using precision, recall, and F1 score, for TrialAtlas and the four baselines (a standalone LLM, a ReAct agent, Biomni, and TrialGenie). c, Example outputs of TrialAtlas , including planning, department-level analyses (Competing Trial Analysis, Literature Evidence Synthesis, and Drug Benchmarking), and the resulting prioritized optimization recommendations.
Figure 4: TrialAtlas predicts technical and regulatory success from trial design. a, Task formulation for the prediction of technical and regulatory success, where a trial’s approval outcome is inferred from structured clinical trial design information. b, Performance comparison across systems on the prediction of technical and regulatory success, evaluated using balanced accuracy, F1 score, and Cohen’s kappa. c, Example analysis results produced by TrialAtlas to contextualize approval predictions, including enrollment distributions stratified by trial status and distributions of treatment categories from relevant PubMed literature.
Figure 5: Human expert evaluation of AI-generated trial design analyses. a, Design of the blinded expert study. For each held-out trial, three systems produced trial design deficiency reports: TrialAtlas and two frontier general-purpose deep-research agents, OpenAI DeepResearch and Gemini DeepResearch. Six experts from the pharmaceutical industry then rated every reported deficiency as a valid concern or not, blind to which system had produced it. b, Structure of the evaluation form. For each trial, an expert was shown the structured trial design (intervention, condition, eligibility, study arms, outcome measures, and phase), the corresponding FDA Complete Response Letter as a reference for the regulatory concerns, and the system’s full report. The expert then rated each flagged deficiency as a valid concern or not a valid concern. c, Proportion of each system’s flagged deficiencies that experts judged to be valid concerns. TrialAtlas achieves the highest proportion (86.4%) among the three systems, ahead of OpenAI DeepResearch (83.1%) and Gemini DeepResearch (59.3%).
Clinical development is sequential decision-making under uncertainty, where a sponsor must plan a portfolio of experiments from heterogeneous evidence. We study this setting by framing oncology clinical development as an offline decision-making problem in which an agent predicts the next six-month trial portfolio of an oncology drug program from information available at the decision date. To support this, we construct a temporal dataset that combines 31.7k heterogeneous public data records, including trial registries, regulatory reviews, sponsor filings, utilization data, and epidemiology, into 881 offline decision episodes across 45 historical programs. We compare four offline objectives: behavioral cloning, reward-weighted behavioral cloning, learned-reward training, and value-based implicit Q-learning against four frontier LLM agents that share a common date-gated retrieval scaffold across held-out drug, sponsor, drug-class, and temporal splits. Models trained offline outperform the non-fine-tuned baselines, particularly in the post-August 2025 contamination-clean holdout. Reward-weighted behavioral cloning performs the best, obtaining 46.2% indication F1 and 14.2% strict F1 against 25.0% and 2.1%, respectively, for the best-performing tool agent on each metric. These results suggest that structured offline learning can teach agents to plan clinical experiments.
William Bolton, Philip Torr
Department of Engineering Science, University of Oxford, Oxford, United Kingdom.
Querying clinical trial registries remains a manual and error-prone process, requiring researchers to navigate large volumes of semi-structured data without support for natural language interaction or cross-source synthesis. To address this, we introduce ClinAgent, a conversational system based on agentic Retrieval-Augmented Generation (RAG) that enables clinicians and researchers to query clinical trial information in plain language and receive grounded, up-to-date responses across multi-turn interactions. The system centers on a Large Language Model (LLM) agent following the ReAct paradigm, which iteratively reasons over queries, selects among a set of integrated tools, and refines its actions based on intermediate outputs. These tools include a ClinicalTrials.gov search interface, a PubMed module, and a Python-based analyzer operating on a locally cached structured dataset of clinical trials. We evaluate the system using a three-phase framework assessing operational effectiveness, planning quality, tool-use efficiency, and expert qualitative judgments, comparing three LLM backends: Gemini 3.0 Flash and two variants of DeepSeek V3.2 (thinking and non-thinking). Results reveal complementary strengths, with DeepSeek (thinking mode) excelling in planning quality, while Gemini achieves the highest overall performance and strongest expert ratings. Overall, our findings highlight the potential of agentic AI systems to improve the accessibility and synthesis of clinical trial information, supporting more efficient and user-centered biomedical research workflows.
Department of Computer Science, University of Pisa, Pisa, Italy · Department of Computer Engineering, Modeling, Electronics and Systems, University of Calabria, Rende, Italy. · Institute of Information Science and Technologies “Alessandro Faedo”, National Research Council, Pisa, Italy. +2
Artificial intelligence (AI) agents promise to accelerate drug discovery by compressing interpretation and decision-making loops, but practical deployment requires trusted evaluation on realistic program decisions. We introduce TherapeuticsBench Preclinical Pharmacology (TxBench-PP), a verifiable benchmark for small-molecule preclinical pharmacology and the first focused slice of a broader TherapeuticsBench effort across drug-discovery stages and therapeutic modalities. TxBench-PP tests whether agents can recover accurate conclusions from real-world assay data rather than memorized facts from literature. The benchmark contains 100 evaluations indexed by program stage, assay type, and task structure, spanning mechanism-of-action (MoA) and pharmacodynamic (PD) reasoning, compound-target engagement, causal target validation, developability and safety, and translational efficacy. Agents receive realistic workflow snapshots, inspect files in a coding environment, and return structured answers graded deterministically. Across 16 model-harness configurations, comprising 11 models and 4,800 trajectories, no system reliably recovered preclinical pharmacology decisions. The strongest configuration, Claude Opus 4.8 / Pi, passed 59.3% of endpoint attempts (178/300; 95% CI, 51.1-67.6), followed by GPT-5.5 / Pi at 55.3% (166/300; 47.0-63.6).