Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve

Weixiang Shen; Bailiang Jian; Jun Li; Che Liu; Johannes Moll; Xiaobin Hu; Daniel Rueckert; Hongwei Bran Li

cs.AIApr 15, 2026

Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve

Authors: Weixiang Shen, Bailiang Jian, Jun Li, Che Liu, Johannes Moll, Xiaobin Hu, Daniel Rueckert, Hongwei Bran Li,

Organizations: Technical University of Munich (TUM) · TUM University Hospital · LMU Munich · National University of Singapore ·

Abstract

Tool-augmented large language model (LLM) agents can orchestrate specialist classifiers, segmentation models, and visual question-answering modules to interpret chest X-rays. However, these agents still solve each case in isolation: they fail to accumulate experience across cases, correct recurrent reasoning mistakes, or adapt their tool-use behavior without expensive reinforcement learning. While a radiologist naturally improves with every case, current agents remain static. In this work, we propose Evo-MedAgent, a self-evolving memory module that equips a medical agent with the capacity for inter-case learning at test time. Our memory comprises three complementary stores: (1)~Retrospective Clinical Episodes that retrieve problem-solving experiences from similar past cases, (2)~an Adaptive Procedural Heuristics bank curating priority-tagged diagnostic rules that evolves via reflection, much like a physician refining their internal criteria, and (3)~a Tool Reliability Controller that tracks per-tool trustworthiness. On ChestAgentBench, Evo-MedAgent raises multiple-choice question (MCQ) accuracy from 0.68 to 0.79 on GPT-5-mini, and from 0.76 to 0.87 on Gemini-3 Flash. With a strong base model, evolving memory improves performance more effectively than orchestrating external tools on qualitative diagnostic tasks. Because Evo-MedAgent requires no training, its per-case overhead is bounded by one additional retrieval pass and a single reflection call, making it deployable on top of any frozen model.

Figures & tables

Figure 1: Overview of Evo-MedAgent. A tool-augmented LLM agent processes CXR questions sequentially while reading from and writing to a self-evolving memory via test-time learning. Before solving, the agent retrieves relevant episodic cases, procedural rules, and tool governance guidance. After receiving ground-truth feedback, it reflects and refines all three memory stores.

Configuration	GPT-5-mini		Gemini-3 Flash		GPT-5.2
Configuration	MCQ	Open	MCQ	Open	MCQ
Baseline (Tool-Enabled)	0.68	–	0.73	–	–
Baseline (Tool-Free)	0.68	0.29	0.76	0.21	0.69
+ $\mathcal{S}$ (Procedural)	0.71	–	0.81	–	–
+ $\mathcal{E}$ (Episodic)	0.78	–	0.84	–	–
Full Memory ( $\mathcal{S}$ + $\mathcal{E}$ )	0.79 (0.76–0.82)	0.40	0.87	0.38	0.81

Table 1: Ablation results on ChestAgentBench (tool-free).

\mathcal{S}

= Procedural memory,

\mathcal{E}

= Episodic memory. Full-memory GPT-5-mini MCQ is reported as mean (range) across three case-order permutations (

\mu{=}0.79

\sigma{=}0.025

). Dashes indicate incomplete runs. Best scores are marked in bold.

Panel of Figure 2 — Figure 2: Cumulative MCQ accuracy over sequential benchmark streams. Left : GPT-5-mini under tool-free baselines and full memory. Right : Gemini-3-Flash under tool-enabled baseline and full memory.

Figure 3: Evo-MedAgent showcase across two chest X-ray cases. Earlier Failure : the agent incorrectly attributes a left pleural effusion to pneumonia; ground truth is haemothorax from a rib spur. Memory Evolution : episodic reflection identifies the missed bone review and distils two new procedural rules. Memory-guided Success : on a later case, the without-memory agent misdiagnoses right lower lobe opacification as atelectasis, while the memory-augmented agent applies the retrieved rules (meniscus sign, absence of air bronchograms) to correctly identify a malignant pleural-based mass.

LLM Agent Self-Improvement

Large language model agent self-improvement studies how agents use experience and feedback to improve their behavior, tools, and workflows.

See all 257 papers

EMR: Self-Evolving Medical Multi-Agent System via Experience Mining and Reuse
Sep 14, 2026Dongsheng Shi, Yue Li, Xin Yi +1Continual Learning for LLM Agents Multi-Agent LLM Systems
East China Normal University · City University of Hong Kong
Evolving Medical Imaging Agents via Experience-driven Self-skill Discovery
Mar 6, 2026Lin Fan, Pengyu Dai, Zhipeng Deng +4Self-Evolving Agents LLM Agent Skill Learning
Southwest Jiaotong University, China · The University of Tokyo, Japan · RIKEN, Japan +2
MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agents
Aug 22, 2026Md Asaduzzaman Jabin, Khoa Le, Lin Zhao +1LLM Agent Memory Memory-Augmented Language Model Agents
University of Georgia, Athens, GA, USA · New Jersey Institute of Technology, Newark, NJ, USA

Sep 14, 2026J/K move · Enter open · S save

EMR: Self-Evolving Medical Multi-Agent System via Experience Mining and Reuse

Dongsheng Shi, Yue Li, Xin Yi, Linlin Wang

East China Normal University · City University of Hong Kong

Continual Learning for LLM Agents Multi-Agent LLM Systems LLM Agent Self-Improvement Clinical Reasoning Medical LLMs Memory-Augmented Language Model Agents

Open paper AskPDF arXiv

Large language model (LLM) driven multi-agent systems have shown promise in complex clinical reasoning, yet existing approaches rely on static strategies and lack persistent clinical memory, preventing self-evolving from prior diagnostic successes and failures. We present EMR, a self-evolving medical multi-agent system via Experience Mining and Reuse. EMR introduces a hierarchical clinical experience library that organizes accumulated knowledge into three levels: clinical principles, diagnostic patterns, and representative cases. During inference, EMR emulates multidisciplinary consultation: a planner agent coordinates domain-specific department agents for specialized reasoning, while a summary agent synthesizes their analyses into a final decision. Critically, EMR automatically extracts correct diagnostic insights and failure-related warnings from multi-agent reasoning trajectories, incrementally updating the experience library to guide future cases. Experiments on medical reasoning benchmarks demonstrate that EMR consistently outperforms state-of-the-art medical multi-agent baselines. Further analysis reveals that the hierarchical experience enables cross-specialty generalization and transfer across diverse LLM backbones, offering a scalable and in

Abstract

LLM Agent Self-Improvement

Explore similar work

EMR: Self-Evolving Medical Multi-Agent System via Experience Mining and Reuse

Evolving Medical Imaging Agents via Experience-driven Self-skill Discovery

MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agents