Organizations: Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences · Beijing Key Laboratory of Safe AI and Superalignment · Beijing Institute of AI Safety and Governance · School of Artificial Intelligence, University of Chinese Academy of Sciences · Long-term AI
Interactive clinical agents operate under partial observability, so reliable care depends on reaching the correct diagnosis through evidence-grounded, safe interactions. Yet existing agents struggle to convert experience into reusable process knowledge with explicit provenance and authority. To address this gap, we introduce MediSkill-Evo, which self-evolves governed process knowledge without fine-tuning the backbone. It realizes this self-evolution by updating clinical, process, symbolic, and visual knowledge in four typed banks under type-specific validation and scope rules. The Process-Constrained Preference Harness then turns validated knowledge into action by grounding candidates in evidence and prioritizing safer decisions. We evaluate on 300 MIMIC-IV-derived FullChain encounters, 180 hard-isolation conditions covering six process obligations, and 100 multimodal NEJM image-diagnosis cases. On Qwen FullChain, MediSkill-Evo improves diagnosis accuracy by 7.81% and treatment-intent coverage by 70.67% over the best-performing prior agent, while reducing critical failures by 43.04%. Under stress, it improves the stress-process composite by 7.77% and required-action completion by 12.41% over the best-performing agent for each metric, with stronger patient-fact, temporal-evidence, and triage-red-flag recovery and no controller-scored errors in unavailable-evidence, treatment, and triage safety checks. On multimodal NEJM diagnosis, MediSkill-Evo with optional MedSAM localization improves diagnosis accuracy by 2.56% and core score by 18.96% over the best-performing memory agent. Code is available at https://anonymous.4open.science/r/mediskill-evo_anonymous-68E7.
Figures & tables
Figure 1. Online inference and offline learning and governance in MediSkill-Evo. Left: the online pipeline retrieves four-bank context, generates and verifies candidate actions, executes the selected action, and incorporates environment feedback. Right: completed trajectories are reflected into typed memory operations, validated for label/evidence leakage, controller-defined safety, support, and replayability, and published as the next memory snapshot. Two equally sized MediSkill-Evo diagrams separated by a vertical dashed line. The left panel presents the seven-stage online inference pipeline, and the right panel presents the six-stage offline learning and governance pipeline.
Figure 2. Two frozen-test interactions scored diagnosis-correct by the automatic evaluator. Left: targeted history, available laboratory evidence, an unavailable ultrasound, and a focused physical examination support acute cholecystitis without treating a missing result as negative. Right: medication history, original radiograph and CT evidence, optional MedSAM localization, and a follow-up blood count support nitrofurantoin-induced pulmonary toxicity; this label does not validate anatomy or management. A side-by-side comparison showing the key doctor, patient, and measurement turns in a text-only acute cholecystitis case and a multimodal nitrofurantoin pulmonary toxicity case. The multimodal case displays the original chest radiograph and CT before the corresponding MedSAM overlays.
Dataset
Train
Test
Modality
MIMIC-IV FullChain
700
300
Text
Controlled clinical stress
420
180
Text
NEJM FullChain Interactive
200
100
Text + image
Table 1. Evaluation datasets. FullChain encounters derive from MIMIC-IV ( Johnson et al., 2023 ) and use the AgentClinic interaction protocol ( Schmidgall et al., 2024 ) ; multimodal cases derive from the NEJM image collection ( The New England Journal of Medicine, 2026 ) . Stress counts denote controlled conditions; the 420/180 conditions are derived from 70/30 disjoint underlying clinical cases.
Dimension
Dimension construction
Primary capability under evaluation
Diagnosis difficulty
Delay decisive source evidence until the Doctor uses its registered question, examination, or test channel.
Recovering and using the decisive evidence while avoiding premature closure.
Evidence completeness
Make one source item unavailable while retaining an independent evidence path that preserves case solvability.
Requesting the missing item, treating unavailability as unknown, and using alternative evidence without fabricating a result.
Patient behavior complexity
Partition patient-knowable source facts so that they are released only after focused, relevant questions.
Recovering patient facts through focused, respectful questioning without changing the underlying disease facts.
Treatment and prescription
Gate a source-supported treatment prerequisite behind the corresponding history, examination, or test request.
Verifying prerequisites and choosing conditional treatment, safe deferral, or an alternative rather than an unsafe action.
Temporal dynamics
Delay a source-supported timeline fact until the Doctor asks a temporally targeted question.
Recovering the original timeline and integrating it into reassessment, treatment, monitoring, or escalation.
Triage safety
Withhold a source-supported red flag until the Doctor performs the appropriate symptom or risk screen.
Recovering the red flag, escalating appropriately, and avoiding unsafe reassurance.
Table 2. Controlled-stress dimensions, source-grounded construction, and enforced process obligations. Each condition changes only the visibility or release timing of source-supported information; hidden values remain outside every runtime LLM prompt and are released only by the deterministic controller.
Metric group
Operational definition
Standard and multimodal evaluation
Outcome
Dx ↑ : accepted final diagnosis; Tx/Rx ↑ : reference treatment intents covered, with unsafe care penalized.
Masks : cases with at least one non-empty MedSAM mask.
Table 3. Evaluation metrics and their operational meanings. Metrics are grouped by evaluation setting and function; arrows indicate the preferred direction.
Backbone
Method
Benchmark outcome
Evidence acquisition
Risk and economy
Int.Eff. ↑
Dx ↑
Tx/Rx ↑
Evid. ↑
Hist. ↑
Tests ↑
Safety ↓
Critical ↓
Unnec. ↓
Qwen3.6-Flash
AgentClinic
61.33
33.62
12.31
11.70
13.46
1.67
31.00
15.89
39.60
Agent-KB
61.00
38.43
14.52
11.26
17.45
0.67
32.33
13.82
41.42
ExPeL
59.00
35.50
14.58
11.09
17.43
2.33
33.00
14.03
40.28
MemP
64.00
38.93
15.16
12.45
17.41
1.67
28.67
12.79
41.40
Reflexion
62.33
38.27
13.38
11.06
15.65
1.33
31.00
13.25
41.18
Table 4. Main automatic operational results (%) on the same 300 FullChain test encounters. Backbones represent the Qwen ( Yang et al., 2025 ) and DeepSeek ( DeepSeek-AI, 2024 ) families; baseline methods are cited in the accompanying text. Higher is better except for Safety, Critical, and Unnec. Bold marks the best result within each backbone; shaded rows denote MediSkill-Evo.
Figure 3. Primary controlled-stress process and safety profile (%). (a) Dimension-aware required-action completion across the six pressure families. (b) The 19 constituent recovery, use, handling, and failure metrics; cell text gives the raw percentage, whereas color is direction-corrected so that darker always denotes better behavior. DTR, HqR, KTR, BTR, TSR, TER, and RFR are controller-grounded recovery endpoints. Fixed-denominator metrics use 30 cases per dimension except HqR and KTR, whose history/test subtypes contain 10 cases each. † marks a conditional denominator determined by a visible request or recovered target and must be read with the corresponding recovery row. Bold cell text marks a unique best. A two-panel controlled-stress comparison. The left radar chart compares four agents on six dimension-aware process-completion scores. The right annotated heatmap reports 19 recovery, use, handling, and safety submetrics for the same agents; darker cells indicate better direction-adjusted behavior while printed values retain the raw percentages.
Method
Stress ↑
Req.Act. ↑
Core ↑
Dx ↑
Tx/Rx ↑
Safety ↓
Critical ↓
AgentClinic
77.75
70.39
75.56
96.67
78.36
0.00
1.11
MemP
72.84
76.81
67.96
87.78
75.31
1.67
7.22
Reflexion
76.06
78.55
71.33
90.56
78.86
1.11
2.78
MediSkill-Evo
83.79
88.30
80.03
93.89
82.40
1.11
2.78
Table 5. Auxiliary cross-dimension aggregates and conventional automatic outcomes on all 180 controlled stress conditions (%). Stress is the registered stress-process composite and Req.Act. is its dimension-aware required-action component. Higher is better except for Safety and Critical.
Agent/condition
Ready
Dx ↑
Hist. ↑
Tests ↑
Unnec. ↓
Core ↑
Masks
MemP
100/100
38.00
58.60
59.00
12.75
35.95
0/100
Reflexion
100/100
39.00
58.63
67.00
12.77
40.29
0/100
MediSkill-Evo (raw)
100/100
37.00
75.58
100.00
5.00
44.69
0/100
MediSkill-Evo (+MedSAM)
100/100
40.00
79.50
100.00
6.58
47.93
34/100
Table 6. Four-agent results on 100 multimodal NEJM image cases ( The New England Journal of Medicine, 2026 ) . Ready and Masks are counts; all other values are macro-averaged percentages over the same 100 saved case trajectories, with no error rows. Masks counts cases with at least one non-empty MedSAM mask.
Variant
Dx ↑
Tx/Rx ↑
Safe. ↓
Crit. ↓
Unnec. ↓
No memory
67.00
36.50
1.00
30.00
11.50
w/o Clinical
74.00
63.65
1.00
24.00
29.75
w/o Process
71.00
60.70
8.00
36.00
30.50
w/o Symbolic
70.00
68.90
0.00
15.00
19.50
Full
76.00
69.40
0.00
13.00
19.17
Table 7. Component analysis on the same 100 FullChain test cases. Each variant disables the indicated knowledge bank while retaining the registered model, interaction budget, case set, and evaluator. All values are percentages.
Stage
Principal inputs
Gold allowed?
Structured output
Trajectory reflection
Completed training trace, evaluator feedback, case targets
Train only
Reflection and failure attribution
Doctor-bank proposal
Validated reflection, observed symbolic traces, active artifact IDs
Train only
Typed bank mutations
Visual measurement
Original pixels, optional MedSAM artifacts, retrieved Measurement Skills
No
Evidence-only visual report
Candidate and critic
Visible state, process ledger, retrieved banks, action portfolio
Fraction of delayed medication, allergy, pregnancy, or renal prerequisites released.
Safe conditional treatment, alternative, or deferral; unsafe action.
Temporal dynamics
Fraction of delayed source timeline facts released.
Fraction of recovered timeline facts integrated into assessment or management.
Triage safety
Fraction of delayed source red flags released.
Escalation adequacy; unsafe reassurance.
Table 12
Setting
Backbone alias
Temp.
Candidates
Doctor ceiling
MediSkill-Evo test-time banks
FullChain
qwen3.6-flash
0
3
6
C/P/S frozen; M inactive
FullChain
deepseek-v4-flash
0
3
6
C/P/S frozen; M inactive
Controlled stress
qwen3.6-flash
0
3
6
C/P/S frozen; M inactive
NEJM, original image
qwen3.6-flash
0
3
8
C/P/S frozen; condition-specific M frozen
NEJM, optional MedSAM
qwen3.6-flash
0
3
8
C/P/S frozen; condition-specific M frozen
Table S2. Registered evaluation configuration. All rows use one observed rollout per case–configuration pair. “Inactive” means that no Measurement Bank is queried.
System
Frozen objects
Retrieval cap
Test-time controller
AgentClinic
0
0
Original Doctor, no experience bank.
Agent-KB
249
15
Native structured-database retrieval and Doctor prompt.
ExPeL
258
25
Native cross-trial insight retrieval and Doctor prompt.
MemP
236
15
Native procedural-memory retrieval and Doctor prompt.
Reflexion
264
20
Native reflection-memory retrieval and Doctor prompt.
SkillWeaver
245
20
Native skill retrieval and Doctor prompt.
Table S3. Registered Qwen FullChain comparator ledger. “Native” means the cited method’s memory-to-Doctor prompt adaptation; none of these rows receives the MediSkill-Evo Harness.
Metric
Better
Tie
Worse
Diagnosis
54
215
31
Treatment intent
266
17
17
Evidence recall
287
10
3
History recall
290
10
0
Test recall
133
106
61
Auto safety violation
5
294
1
Table S4. Paired Qwen case transitions from existing frozen traces ( n=300 ).
Tool-augmented large language model (LLM) agents can orchestrate specialist classifiers, segmentation models, and visual question-answering modules to interpret chest X-rays. However, these agents still solve each case in isolation: they fail to accumulate experience across cases, correct recurrent reasoning mistakes, or adapt their tool-use behavior without expensive reinforcement learning. While a radiologist naturally improves with every case, current agents remain static. In this work, we propose Evo-MedAgent, a self-evolving memory module that equips a medical agent with the capacity for inter-case learning at test time. Our memory comprises three complementary stores: (1)~Retrospective Clinical Episodes that retrieve problem-solving experiences from similar past cases, (2)~an Adaptive Procedural Heuristics bank curating priority-tagged diagnostic rules that evolves via reflection, much like a physician refining their internal criteria, and (3)~a Tool Reliability Controller that tracks per-tool trustworthiness. On ChestAgentBench, Evo-MedAgent raises multiple-choice question (MCQ) accuracy from 0.68 to 0.79 on GPT-5-mini, and from 0.76 to 0.87 on Gemini-3 Flash. With a strong base model, evolving memory improves performance more effectively than orchestrating external tools on qualitative diagnostic tasks. Because Evo-MedAgent requires no training, its per-case overhead is bounded by one additional retrieval pass and a single reflection call, making it deployable on top of any frozen model.
Weixiang Shen, Bailiang Jian, Jun Li +6
Technical University of Munich (TUM) · TUM University Hospital · LMU Munich +3
Doctor agents are moving beyond single-turn answer generation toward evolving clinical decision systems that acquire evidence within episodes and reuse experience across episodes. Fixed-input medical QA and many interactive benchmarks provide limited support for evaluating how episode-level decisions interact with cross-episode experience. We introduce MedEvoEval, an executable longitudinal evaluation framework based on action-gated simulated outpatient episodes. Each case is converted into role-specific patient, examination, and manager views; evidence is revealed through valid actions, and structured traces link observations, actions, final outputs, scores, and optional experience write-back. Experiments show that these traces expose process costs hidden by final-answer scoring, characterize how MDT-style consultation reallocates resources, and support analyses of memory maturation, held-out transfer, update-stage response, and backward retention. On 200 held-out cases, the mature-memory condition with sequential write-back reduces interaction turns by 0.915 and improves composite utility by 0.0483 after Holm correction. MedEvoEval provides a concrete basis for evaluating experience-driven improvement, transfer, and retention. The accompanying artifact provides 700 processed episode records with provenance, role-specific schemas, and execution, scoring, and analysis code.
Interactive clinical agents acquire evidence through patient dialogue and diagnostic examinations, yet their performance also depends on how the interaction environment exposes information, available actions, and examination results. When these conditions are implicit or inconsistently controlled, final-answer accuracy can conflate diagnostic reasoning with the effects of the interaction protocol. We propose the Evidence-Grounded Interaction Protocol (EGIP), which enforces role-specific information boundaries, exposes diagnostic actions without revealing their results, retrieves examination results from stored clinical records, and organizes acquired evidence for final decision-making. By separating evidence acquisition from terminal prediction, EGIP enables different decision strategies to operate on the same observations. We evaluate EGIP across two clinical-agent frameworks, five datasets, and six language models. EGIP improves diagnostic accuracy over native baselines across all primary comparisons. A clinician review further shows that the proportion of automatically scored correct responses jointly judged evidence-supported rises from 10.20% to 36.55%. These findings support the effectiveness of EGIP as a unified interaction protocol for improving clinical-agent performance through controlled evidence access and acquisition.
Zecheng Ren, Ruichen Cong, Yi Min Chen +8
Waseda University, Tokorozawa, Japan · Shanghai Jiao Tong University, Shanghai, China