cs.AINov 17, 2026

VITAL: A Framework for Validating Agentic AI in Pharmaceutical Drug Development and Clinical Trial Operations

Authors: Sandeep Burugupalli, Pedro Matos-Filipe, Andrea Hita Ardiaca, Adam Taylor, Alex Grigorenko, Natalia Jakubiak, Xin Yu, Raghav Mani, 

Organizations: AstraZeneca Boston. 35 Gatehouse Drive, Waltham, MA 02451, United States · AstraZeneca Barcelona. Avinguda de Roma, 81, 08029 Barcelona, Spain · AstraZeneca Cambridge. Francis Crick Avenue Cambridge Biomedical Campus Cambridge, CB2 0AA, United Kingdom · NVIDIA. 2788 San Tomas Expy, Santa Clara, CA 95051, United States · 

Abstract

Current assurance approaches do not establish whether evidence is sufficient to validate, monitor and govern agentic artificial intelligence (AI) within a pharmaceutical context of use. Plausible outputs may conceal failures in retrieval, reasoning, orchestration or human-control points, where reference standards are unavailable. We developed and operationalised VITAL (Validation by Intended use, Traceability, Autonomy and Lifecycle evidence), a validation framework for agents in clinical development, through landscape analysis, regulatory and standards mapping, expert interviews and multi-stakeholder refinement. VITAL integrates context of use, controls, behavioural validation and lifecycle evidence within a risk-adjusted framework. We applied VITAL across epidemiology and treatment-landscape analysis, site selection, enrolment rescue and trial-design trade-off support to translate autonomy, reversibility, transparency and patient impact into proportionate validation activities, evidence requirements and auditable reconstruction of system behaviour. VITAL may inform evidence requirements and lifecycle accountability for higher-risk pharmaceutical uses; evaluation should assess reproducibility across assessors, organisations and system configurations.

Figures & tables

Explore similar work

Jul 31, 2026

Beyond Component Testing: Validating Agentic AI Systems

Agentic AI systems act through multi-step trajectories that combine planning, tool use, memory, interaction, and adaptation. This behavior stretches validation practice beyond component testing and one-shot input-output evaluation, because acceptable system behavior now depends on how decisions unfold over time and under changing environmental conditions. This systematic mapping study synthesizes 262 papers spanning agent evaluation, software assurance, cyber-physical systems, runtime monitoring, and regulatory guidance in order to characterize the validation problem for agentic systems. The review is organized around a five-dimension taxonomy covering behavioral, safety, temporal, regulatory, and multi-agent concerns, and uses that taxonomy to map current approaches and identify dense and sparse pairings of approaches and dimensions. The resulting map shows that the literature concentrates on behavioral evaluation (105 of 262 papers) and is thinnest on temporal validity (14 papers); regulatory work rests mainly on assurance cases and regulatory analysis, largely from IEEE-indexed venues, and multi-agent work mainly on benchmarks. Three cross-domain case studies (medical care, industrial operations, smart-mobility systems) provide operational illustrations of how the five taxonomy dimensions recur in safety-critical settings, motivated by the failure patterns documented in the reviewed literature. The paper concludes with a lifecycle-oriented research agenda centered on bounded-autonomy specifications, adversarial trajectory generation, runtime monitoring, and audit-ready evidence structures. The central claim is that trustworthy deployment of agentic AI depends on validating trajectories in context rather than assessing isolated components alone.
Mar 4, 2026

A Multi-Agent Framework for Interpreting Multivariate Physiological Time Series

Continuous physiological monitoring is central to emergency care, yet deploying trustworthy AI is challenging. While LLMs can translate complex physiological signals into clinical narratives, it is unclear how agentic systems perform relative to zero-shot inference. To address these questions, we present Vivaldi, a role-structured multi-agent system that explains multivariate physiological time series. Due to regulatory constraints that preclude live deployment, we instantiate Vivaldi in a controlled, clinical pilot to a small, highly qualified cohort of emergency medicine experts, whose evaluations reveal a context-dependent picture that contrasts with prevailing assumptions that agentic reasoning uniformly improves performance. Our experiments show that agentic pipelines substantially benefit non-thinking and medically fine-tuned models, improving expert-rated explanation justification and relevance by +6.9 and +9.7 points, respectively. Contrarily, for thinking models, agentic orchestration often degrades explanation quality, including a 14-point drop in relevance, while improving diagnostic precision (ESI F1 +3.6). We also find that explicit tool-based computation is decisive for codifiable clinical metrics, whereas subjective targets, such as pain scores and length of stay, show limited or inconsistent changes. Expert evaluation further indicates that gains in clinical utility depend on visualization conventions, with medically specialized models achieving the most favorable trade-offs between utility and clarity. Together, these findings show that the value of agentic AI lies in the selective externalization of computation and structure rather than in maximal reasoning complexity, and highlight concrete design trade-offs and learned lessons, broadly applicable to explainable AI in safety-critical healthcare settings.
Aug 1, 2026

Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims. No formal framework has characterized how validity degrades across these stages. We present a knowledge-based three-layer compounding validity model, Vtotal≤V1×V2×V3V_{\rm total} \leq V_1 \times V_2 \times V_3, that captures multiplicative degradation across task generation (V1V_1), human-simulator calibration (V2V_2), and automated judgment (V3V_3). Applying it to five widely-used agentic benchmark configurations, we find V1V_1 and V2V_2 independently measurable and both degraded, but V3V_3 not currently quantifiable for any: three of the four benchmarks score with a single deterministic rule, so no second rater exists and reliability is undefined, and the fourth's partial LLM grader has no published grader-vs-human measurement. A structured survey of 55 published agentic evaluation papers finds that 26 of them report no chance-corrected inter-rater statistic of any kind, a determination four independent raters recover at Krippendorff's α=0.834α= 0.834. On the broader judgment of whether a reported metric fits its rating design, approximately 80% of this purposive sample use structurally mismatched, incomplete, or absent inter-rater reliability (IRR) metrics (four-rater range 60.7%-84.4%). Both are consistent with systematic V3V_3 collapse. We further document V1V_1 failures (task validity flaws in 7 of 10 benchmarks) and V2V_2 miscalibration (up to 9 percentage points inter-simulator variance, with systematic demographic disparities for non-Standard American English speakers). We derive eight prescriptions grounded in psychometric science and domain-stratified reliability thresholds (intraclass correlation ≥0.70\geq 0.70; α≥0.67/0.70/0.80α\geq 0.67/0.70/0.80 by consequence level) that practitioners can apply immediately.