Evolving Medical Imaging Agents via Experience-driven Self-skill Discovery

Lin Fan; Pengyu Dai; Zhipeng Deng; Haolin Wang; Xun Gong; Yefeng Zheng; Yafei Ou

cs.AIMar 6, 2026

Evolving Medical Imaging Agents via Experience-driven Self-skill Discovery

Authors: Lin Fan, Pengyu Dai, Zhipeng Deng, Haolin Wang, Xun Gong, Yefeng Zheng, Yafei Ou

Organizations: Southwest Jiaotong University, China · The University of Tokyo, Japan · RIKEN, Japan · Westlake University, China ·

Abstract

Clinical image interpretation is inherently multi-step and tool-centric: clinicians iteratively combine visual evidence with patient context, quantify findings, and refine their decisions through a sequence of specialized procedures. While LLM-based agents promise to orchestrate such heterogeneous medical tools, existing systems treat tool sets and invocation strategies as static after deployment. This design is brittle under real-world domain shifts, across tasks, and evolving diagnostic requirements, where predefined tool chains frequently degrade and demand costly manual re-design. We propose MACRO, a self-evolving, experience-augmented medical agent that shifts from static tool composition to experience-driven tool discovery. From verified execution trajectories, the agent autonomously identifies recurring effective multi-step tool sequences, synthesizes them into reusable composite tools, and registers these as new high-level primitives that continuously expand its behavioral repertoire. A lightweight image-feature memory grounds tool selection in a visual-clinical context, while a GRPO-like training loop reinforces reliable invocation of discovered composites, enabling closed-loop self-improvement with minimal supervision. Extensive experiments across diverse medical imaging datasets and tasks demonstrate that autonomous composite tool discovery consistently improves multi-step orchestration accuracy and cross-domain generalization over strong baselines and recent state-of-the-art agentic methods, bridging the gap between brittle static tool use and adaptive, context-aware clinical AI assistance. Code will be available upon acceptance.

Figures & tables

Figure 1: Existing medical agents vs. our MACRO. Existing medical agents rely on static, predefined tool sets and invocation strategies that struggle to adapt to variations across different imaging domains or intra-class sample differences, often leading to failures. In contrast, MACRO enhances adaptability by discovering and integrating composite tools, validated tool sequences capable of executing multi-step operations. These sequences are distilled from repeatedly successful trajectories in real-world workflows, enabling more robust performance in the face of clinical variability.

Figure 2: Pipeline of proposed MACRO. For each input image and multi-step trajectory, MACRO first retrieves relevant experiences from a memory store using image feature similarity. At every step, the model generates a response; if the generated tool sequence contains any registered composite tool $\mathbf{c}\in\mathcal{C}$ , a positive reward is assigned. Tool calls are executed, and their results are appended to the evidence, informing subsequent steps. At the end of the trajectory, if the final answer matches the ground truth, the entire tool sequence is stored back into $\mathcal{M}$ , and its contiguous subsequences increment the frequency counts in the composite registry $\mathcal{C}$ , enabling online discovery of new composite tools.

Type	Method	REFUGE2 [ 12 ]		MITEA [ 51 ]
Type	Method	BACC $\uparrow$	F1 $\uparrow$	BACC $\uparrow$	F1 $\uparrow$
General	GPT-4o [ 1 ]	56.4	21.1	56.8	28.1
	Janus-Pro-7B [ 7 ]	53.4	13.3	52.3	10.7
	LLaVA-Med [ 21 ]	50.0	0.0	50.0	0.0
	BioMedClip [ 50 ]	58.1	21.3	47.0	37.8
	Qwen2.5-7B-VL [ 41 ]	54.3	16.3	50.0	0.0

Table 1: Comparison with general VLMs and medical agentic systems on REFUGE2 [ 12 ] , MITEA [ 51 ] . The best results in each column are highlighted in bold , and the second-best values are underlined .

Method	BACC $\uparrow$	F1 $\uparrow$	ACC $\uparrow$	SEN $\uparrow$	SPC $\uparrow$	PRE $\uparrow$
MobileViT [ 28 ]	52.64	11.85	81.42	21.06	84.23	9.31
ResNet [ 15 ]	51.75	10.89	78.27	23.10	80.40	7.79
MobileNet [ 16 ]	47.84	10.79	74.08	17.02	78.66	9.07
LeViT [ 14 ]	49.29	6.73	84.17	8.49	90.09	8.99
EfficientFormer [ 22 ]	50.63	12.40	72.04	27.90	73.37	8.82
MedMamba [ 47 ]	50.83	6.91	86.56	8.94	92.73	11.56

Table 2: BE classification results obtained on the Test set of RAM-W600 [ 45 ] . The best results in each column are highlighted in bold , and the second-best values are underlined .

Components			REFUGE2 [ 12 ]		MITEA [ 51 ]
Memory	Composite	GRPO	BACC	F1	BACC	F1
✗	✗	✗	82.8	69.7	69.2	64.8
✓	✗	✗	83.9	75.8	72.4	65.3
✓	✓	✗	90.6	80.3	75.1	71.8
✓	✓	✓	92.7	80.3	77.2	74.9

Table 3: Ablation on key components, including memory store, composite tool discovery, and GRPO. The best results in each column are highlighted in bold , and the second-best values are underlined .

Figure 3: Closed-loop learning in MACRO yields two measurable benefits: (a) tool complexity decreases as higher-level tools progressively replace multi-step patterns; (b) As MACRO acquires stronger tools and better demonstrations, performance with abstract tools improves by 8.8% over the basic tool lib.

Figure 4: A case study on glaucoma diagnosis, illustrating the detailed workflow within the MACRO framework. The green text represents the call action operations, while the surrounding reasoning reflects the model’s explanation and thought process behind each action.

Self-Evolving Agents

Self-evolving agents improve their skills, memory, tools, policies, or environments by learning from interaction and evaluation feedback.

See all 401 papers

Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve
Apr 15, 2026Weixiang Shen, Bailiang Jian, Jun Li +6LLM Agent Self-Improvement LLM Agent Memory
Technical University of Munich (TUM) · TUM University Hospital · LMU Munich +3
The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy
Jul 13, 2026Chunzheng Zhu, Lei Tian, Bohan Tan +15Healthcare LLM Agent Self-Improvement
1Hunan University · 2ByteDance · 3Duke University +6
MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolution
Sep 21, 2026Junde Wu, Jiayuan Zhu, Minghao Hu +2Self-Improving Agents Self-Evolving Agents
University of Oxford · Stanford University

Apr 15, 2026J/K move · Enter open · S save

Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve

Weixiang Shen, Bailiang Jian, Jun Li, Che Liu+5

Technical University of Munich (TUM) · TUM University Hospital · LMU Munich+3

LLM Agent Self-Improvement LLM Agent Memory Continual Learning for LLM Agents LLM Agent Orchestration Medical Image Analysis

Open paper AskPDF arXiv

Tool-augmented large language model (LLM) agents can orchestrate specialist classifiers, segmentation models, and visual question-answering modules to interpret chest X-rays. However, these agents still solve each case in isolation: they fail to accumulate experience across cases, correct recurrent reasoning mistakes, or adapt their tool-use behavior without expensive reinforcement learning. While a radiologist naturally improves with every case, current agents remain static. In this work, we propose Evo-MedAgent, a self-evolving memory module that equips a medical agent with the capacity for inter-case learning at test time. Our memory comprises three complementary stores: (1)~Retrospective Clinical Episodes that retrieve problem-solving experiences from similar past cases, (2)~an Adaptive Procedural Heuristics bank curating priority-tagged diagnostic rules that evolves via reflection, much like a physician refining their internal criteria, and (3)~a Tool Reliability Controller that tracks per-tool trustworthiness. On ChestAgentBench, Evo-MedAgent raises multiple-choice question (MCQ) accuracy from 0.68 to 0.79 on GPT-5-mini, and from 0.76 to 0.87 on Gemini-3 Flash. With a strong base model, evolving memory improves performance more effectively than orchestrating external tools on qualitative diagnostic tasks. Because Evo-MedAgent requires no training, its per-case overhead is bounded by one additional retrieval pass and a single reflection call, making it deployable on top of any frozen model.

Abstract

Self-Evolving Agents

Explore similar work

Evo-MedAgent: Beyond One-Shot Diagnosis with Agents That Remember, Reflect, and Improve

The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomy

MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolution