Medical AI Agents

Latest papers

38 papers7 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
May 4, 2026Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10AI Agent Benchmarks Electronic Health Records
Stanford University
Healthcare AI GYM for Medical Agents
May 1, 2026Minbyul JeongMedical LLMs RL for Tool Use
Upstage AI
RadAgent: A tool-using AI agent for stepwise interpretation of chest computed tomography
Apr 16, 2026Mélanie Roschewitz, Kenneth Styppa, Yitian Tao +10Tool-Using Vision-Language Agents Healthcare
Department of Biosystems Science and Engineering, ETH Zurich, Basel, Switzerland · ETH AI Center, Zurich, Switzerland · Department of Computer Science, ETH Zurich, Zurich, Switzerland +5
Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve
Apr 15, 2026Weixiang Shen, Bailiang Jian, Jun Li +6Medical AI Agents LLM Agent Self-Improvement
Technical University of Munich (TUM) · TUM University Hospital · LMU Munich +3
BAAI Cardiac Agent: An intelligent multimodal agent for automated reasoning and diagnosis of cardiovascular diseases from cardiac magnetic resonance imaging
Apr 5, 2026Taiping Qu, Hongkai Zhang, Lantian Zhang +18Medical AI Agents Medical Diagnosis
Beijing Academy of Artificial Intelligence, No. 150 Chengfu Road, Haidian District, Beijing 100084, China · Department of Radiology, Beijing Anzhen Hospital, Beijing Institute of Heart, Lung & Vascular Diseases, Capital Medical University, 2 Anzhen Road, Beijing 100029, China · Department of MR, the First Affiliated Hospital, Henan Medical University, 88 Jiankang Road, Weihui 453100, China +2
Doctorina MedBench: A Dialogue-Based Benchmark and Evaluation Framework for Agent-Based Medical AI
Mar 26, 2026Anna Kozlova, Stanislau Salavei, Pavel Satalkin +3Medical AI Agents AI Agent Benchmarks
A.I. Doctor Medical Assist LTD, 13 Myrtiotissis Street, Aqua Mansions, Office 1, Germasogeia, 4041 Limassol, Cyprus
Meissa: Multi-modal Medical Agentic Intelligence
Mar 9, 2026Yixiong Chen, Xinyi Bai, Yue Pan +2Medical AI Agents Medical LLMs
Johns Hopkins University, Baltimore, MD, USA · Cornell University, Ithaca, NY, USA
Which Tool Response Should I Trust? Tool-Expertise-Aware Chest X-ray Agent with Multimodal Agentic Learning
Feb 25, 2026Zheang Huai, Honglong Yang, Xiaomeng LiMedical AI Agents Tool-Using Vision-Language Agents
The Hong Kong University of Science and Technology, Kowloon, Hong Kong

May 4, 2026J/K move · Enter open · S save

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments

Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler, Kavita Renduchintala+9

Stanford University

AI Agent Benchmarks Electronic Health Records Long-Horizon Agent Evaluation Healthcare LLM Agents Clinical Language Model Evaluation+1

Open paper AskPDF arXiv

We introduce PhysicianBench, a benchmark for evaluating LLM agents on physician tasks grounded in real clinical setting within electronic health record (EHR) environments. Existing medical agent benchmarks primarily focus on static knowledge recall, single-step atomic actions, or action intent without verifiable execution against the environment. As a result, they fail to capture the long-horizon, composite workflows that characterize real clinical systems. PhysicianBench comprises 100 long-horizon tasks adapted from real consultation cases between primary care and subspecialty physicians, with each task independently reviewed by a separate panel of physicians. Tasks are instantiated in an EHR environment with real patient records and accessed through the same standard APIs used by commercial EHR vendors. Tasks span 21 specialties (e.g., cardiology, endocrinology, oncology, psychiatry) and diverse workflow types (e.g., diagnosis interpretation, medication prescribing, treatment planning), requiring an average of 27 tool calls per task. Solving each task requires retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. Each task is decomposed into structured checkpoints (670 in total across the benchmark) capturing distinct stages of completion graded by task-specific scripts with execution-grounded verification. Across 13 proprietary and open-source LLM agents, the best-performing model achieves only 46% success rate (pass@1), while open-source models reach at most 19%, revealing a substantial gap between current agent capabilities and the demands of real-world clinical workflows. PhysicianBench provides a realistic and execution-grounded benchmark for measuring progress toward autonomous clinical agents.