Multi-modal large language models (MM-LLMs) have shown strong performance in medical image understanding and clinical reasoning. Recent medical agent systems extend them with tool use and multi-agent collaboration, enabling complex decision-making. However, these systems rely almost entirely on frontier models (e.g., GPT), whose API-based deployment incurs high cost, high latency, and privacy risks that conflict with on-premise clinical requirements. We present Meissa, a lightweight 4B-parameter medical MM-LLM that brings agentic capability offline. Instead of imitating static answers, Meissa learns both when to engage external interaction (strategy selection) and how to execute multi-step interaction (strategy execution) by distilling structured trajectories from frontier models. Specifically, we propose: (1) Unified trajectory modeling: trajectories (reasoning and action traces) are represented within a single state-action-observation formalism, allowing one model to generalize across heterogeneous medical environments. (2) Three-tier stratified supervision: the model's own errors trigger progressive escalation from direct reasoning to tool-augmented and multi-agent interaction, explicitly learning difficulty-aware strategy selection. (3) Prospective-retrospective supervision: pairing exploratory forward traces with hindsight-rationalized execution traces enables stable learning of effective interaction policies. Trained on 40K curated trajectories, Meissa matches or exceeds proprietary frontier agents in 10 of 16 evaluation settings across 13 medical benchmarks spanning radiology, pathology, and clinical reasoning. Using over 25x fewer parameters than typical frontier models like Gemini-3, Meissa operates fully offline with 22x lower end-to-end latency compared to API-based deployment. Data, models, and environments are released at https://github.com/Schuture/Meissa.
Figures & tables
Figure 1: Overview of Meissa : Trajectory-based agentic behavior distillation. Left: Stratified trajectory supervision uses the model’s own errors to progressively escalate interaction depth, teaching strategy selection. Center: Four agent environments serve as diverse trajectory sources. Right: Prospective-retrospective supervision teaches both exploration and optimal execution policies.
Figure 2: Four agent environments as trajectory sources. Each environment produces trajectories with distinct state–action–observation patterns: (a) tool calling trajectories with vision tool chains, (b) interleaved image-text trajectories with visual feedback loops, (c) multi-agent trajectories with expert debate and synthesis, (d) clinical simulation trajectories with multi-turn information gathering.
Benchmark
Domain
Modality
Environment
ID/OOD
MIMIC-CXR-VQA [ 1 ]
Radiology
CXR + text
Tool Calling + Col.
ID
ChestAgentBench [ 13 ]
Radiology
Multi-CXR + text
Tool Calling
OOD †
PathVQA [ 16 ]
Pathology
Path img + text
Think img + Col.
ID
SLAKE [ 30 ]
Mixed
Med img + text
Think img
ID
VQA-RAD [ 27 ]
Radiology
Rad img + text
Think img
ID
OmniMed [ 19 ]
Mixed
Med img + text
Think img
OOD †
Table 1: Evaluation benchmarks (13 total across 4 agent environments). † = strict OOD: excluded from training and verified by n -gram de-contamination ( n=8 ).
Framework I: Continuous Tool Calling
ChestAgentBench †
MIMIC-CXR-VQA
Direct Inference
GPT-4o
56.4
40.0
Gemini-3-flash
76.2
43.6
Qwen3-VL-4B (SFT)
53.3
49.7
Agent-based (MedRAX)
Table 2: Comparison under four agent evaluation frameworks. Bold : best per column; underline : second-best per column. Parenthetical superscripts on Meissa rows denote absolute improvement ( Δ ) over the untrained Qwen3-VL-4B baseline within the same agent framework. † = strict OOD benchmark. All Qwen3-VL and Gemini-3-flash results are from our own evaluation, all other results are from the original papers.
Strategy
Avg Accuracy
Avg Actions
Avg Tokens
Latency
Gemini-3-flash
72.7
4.03
≥860‡
87.2s ( 72× )
Always direct
54.2
1.0
479
1.23s ( 1× )
Always agentic
45.9
3.52
2082
10.64s ( 9× )
Oracle (upper bound)
63.2
1.54
933
3.41s ( 3× )
Learned routing (Ours)
62.8
1.71
959
4.12s ( 3× )
Table 3: Counterfactual strategy comparison on ChestAgentBench. Learned routing achieves near-oracle accuracy at ∼3× lower cost than always-agentic inference, while avoiding the accuracy degradation caused by over-invoking tools on easy queries. ‡ = Gemini-3-flash API does not report token usage; value is a lower-bound estimate.
Figure 3: Strategy selection analysis. ( Left ) Tier 1 easy queries are answered directly in 96% of cases, while Tier 3 hard queries trigger agentic interaction 97% of the time, confirming difficulty-aware routing. ( Center ) Meissa accuracy peaks near 2,000 tokens / query and then drops at 4,000 tokens (capacity limit of the 4B model), whereas frontier models scale monotonically. This motivates depth allocation for lightweight models. ( Right ) Depth-constrained accuracy ( Tmax∈{0,1,2,3,∞} ): accuracy improves consistently with interaction depth but exhibits diminishing returns beyond Tmax=3 .
Configuration
ID Avg
OOD Avg
Overall
(a) Stratified trajectory supervision
Agentic only (Tier 3)
61.2
46.6
56.6
Direct + Agentic (Tier 1+3)
59.8
46.9
55.8
Enhanced + Agentic (Tier 2+3)
63.7
51.0
59.7
Stratified (Tier 1+2+3)
66.3
50.3
61.3
(b) Execution policy learning
Table 4: Ablation studies. (a) Stratified trajectory supervision shows each tier contributes. (b) Prospective and retrospective supervision are complementary.
Perturbation
PathVQA
SLAKE
VQA-RAD
Δ Avg
Meissa (unperturbed)
75.2
80.3
67.1
—
Paraphrased tool outputs
76.7
76.6
66.9
−0.8
Visual feedback → blank
72.0
67.2
62.9
−6.8
Visual feedback → random
56.6
66.5
57.8
−13.9
Tools disabled ( T=0 )
72.1
75.7
66.3
−2.8
Altered prompt format
73.7
76.4
65.5
−2.3
Table 5: Policy robustness under perturbation for interleaved thinking with image framework. Δ = accuracy change vs. unperturbed Meissa on agentic-only queries ( T>0 ). The model relies causally on observation content and visual feedback.
Figure 4: Case study. Each panel shows a query with Meissa ’s reasoning trace. (a) T=0 : bilateral nodular infiltrates with cavitation are directly recognizable; no tool is invoked. (b) T=1 : the model calls chest_xray_expert to confirm mediastinal air before diagnosing pneumomediastinum. (c) T=3 : the report generator misses the opacity, but the expert and phrase-grounding tools correctly identify a dense mass ( red bounding box); the model reconciles the conflicting outputs. (d) T=3 , conflict resolution: llava_med_qa hallucinates a pulse oximeter and ventilator ( red highlight), while two other tools confirm a clean CXR; the model identifies the hallucinated output. (e) T=4 , progressive diagnosis: BiomedParse failed to capture the target regions, so Meissa actively zoom in the image to confirm the findings.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Tier 1
Tier 2
Tier 3
Total
Tier 3 Discard
Continuous Tool Calling
MIMIC-CXR-VQA
1.2k
0.2k
3.5k
4.9k
14.3k
Interleaved Thinking with Image
PathVQA
0.5k
1.1k
6.0k
6.6k
19.1k
SLAKE
0.3k
0.3k
1.1k
1.7k
2.7k
VQA-RAD
0.4k
0.1k
0.7k
1.2k
1.1k
Appendix
Table 6: Training data statistics by tier and environment. Each row shows the number of trajectories contributed by each source. “Tier 3 Discard” counts samples that failed the data filtering.
Benchmark
Domain
Test Size
Format
Metric
ID/OOD
Framework I: Continuous Tool Calling
MIMIC-CXR-VQA
Radiology
1,000
Open-ended
Soft-match acc.
ID
ChestAgentBench †
Radiology
2,500
Open-ended (7 cat.)
Per-category acc.
OOD
Framework II: Interleaved Thinking with Image
PathVQA
Pathology
3,362
Yes/No + Open
Accuracy
ID
SLAKE
Mixed
1,061
Yes/No + Open
Accuracy
ID
Appendix
Table 7: Evaluation benchmark details by agent environment. Test set sizes, question formats, and evaluation metrics. Benchmarks shared across environments (e.g., MedQA) are listed under each with environment-specific test sizes. † = strict OOD.
Tool
Input
Output
ChestXRayClassifier
CXR image
14 pathology probabilities
ChestXRaySegmentation
CXR image, anatomy
Seg. mask + region stats
CheXagent VQA
CXR image, question
Free-text answer
LLaVA-Med VQA
Medical image, question
Free-text answer
XRayPhraseGrounding
CXR image, text phrase
Bounding box coordinates
ChestXRayReportGen
CXR image
Structured radiology report
Appendix
Table 8: MedRAX tool specifications. Each tool is defined as a JSON schema in the system prompt.
Dimension
Prospective (Forward)
Retrospective (Backward)
Generation time
During real-time inference
After correct answer is known
Answer knowledge
Agent does not know y
Agent is given y as context
Uncertainty level
High (planning, hypotheses, contingencies)
Low (hindsight rationalization)
Error content
May contain wrong assumptions, backtracking
Clean, logically consistent
Reasoning style
Process-driven, exploratory
Result-driven, decisive
Action sequence
Original (may include suboptimal steps)
Same actions, re-narrated reasoning
Appendix
Table 9: Prospective vs. retrospective trajectory comparison.
MIMIC
ChestAg
PathVQA
SLAKE
VQA-RAD
MedQA
PubMedQA
NEJM
MIMIC-IV
Meissa
± 0.8
± 1.2
± 0.6
± 0.9
± 1.5
± 1.1
± 1.3
± 2.4
± 1.6
Appendix
Table 10: Meissa results with standard deviations. Mean ±σ over 3 seeds.
Category
Qwen3-VL-4B
Answer-only SFT
GPT-4o
MedRAX+GPT-4o
Meissa
Detection
46.9
53.5
58.7
64.1
62.7
Classification
45.2
51.8
54.6
62.9
62.1
Localization
48.8
55.6
59.0
63.6
63.0
Comparison
50.0
55.4
55.5
61.8
61.5
Relationship
51.5
56.5
59.0
63.1
62.8
Diagnosis
43.0
50.3
52.6
62.5
62.0
Appendix
Table 11: ChestAgentBench per-category accuracy. Meissa ’s improvement over the base model is most pronounced on characterization and diagnosis categories where tool-generated structured outputs provide explicit clinical grounding.
Difficulty
Direct ( T=0 )
Agentic ( T>0 )
Avg Depth
Easy
96%
4%
0.1
Medium
62%
38%
0.8
Hard
3%
97%
2.4
Overall
72%
28%
0.6
Appendix
Table 12: Routing distribution by difficulty. Percentage of queries routed to each strategy. Easy = student-solvable; Medium = teacher-solvable without tools; Hard = requires agent environment.
Strategy
Accuracy
Avg Actions
Avg Tokens
Relative Cost
Base model (Qwen3-VL-4B)
46.6
1.0
452
1×
Answer-only SFT
53.3
1.0
485
1×
Shuffled trajectories
55.1
1.6
812
1.7 ×
Random-sample agentic
58.5
1.8
965
2.0 ×
Always direct ( T=0 )
54.2
1.0
479
1×
Always agentic
45.9
3.5
2,082
4.3 ×
Appendix
Table 13: Full efficiency comparison. Accuracy vs. cost for all strategy variants. Shuffled trajectories destroy causal action order while preserving data content.
Teacher Model
ID Avg
OOD Avg
Overall
Gemini-3-flash
66.3
50.3
61.3
Qwen3-VL-32B-Instruct
63.5
47.2
58.0
Difference
− 2.8
− 3.1
− 3.3
Appendix
Table 14: Teacher sensitivity analysis. Replacing Gemini-3-flash with Qwen3-32B yields comparable results, confirming that the supervision structure matters more than the specific teacher.
Tool Calling
Thinking with Image
MDAgents
Perturbation
MIMIC
ChestAg
PathVQA
SLAKE
MedQA
PubMedQA
Unperturbed
72.0
66.5
75.2
80.3
57.2
77.9
Paraphrased outputs
71.2
65.8
76.7
76.6
56.5
76.8
Bbox noise ( ± 10px)
69.5
63.1
73.8
77.4
—
—
Visual → blank
64.3
58.2
72.0
67.2
—
—
Visual → random
58.6
51.4
56.6
66.5
—
—
Appendix
Table 15: Per-perturbation robustness breakdown. Accuracy on agentic-only queries ( T>0 ) under each perturbation, with Δ vs. unperturbed.
Tmax
Overall Acc
Easy Acc
Hard Acc
Avg Depth
Avg Tokens
0 (direct only)
55.6
61.2
38.5
0.0
479
1
58.3
60.8
48.7
0.4
618
2
60.0
60.5
54.6
0.9
762
3
60.4
60.3
57.1
1.2
864
∞ (unconstrained)
61.3
59.6
60.4
1.7
959
Appendix
Table 16: Accuracy under maximum depth constraints. Tmax = maximum allowed interaction depth at inference. Tmax=0 forces direct reasoning for all queries.
Hyperparameter
Value
Base model
Qwen3-VL-4B-Instruct
Fine-tuning method
LoRA
LoRA rank
32
LoRA alpha
64
LoRA target modules
All linear layers
Learning rate
5×10−5
Appendix
Table 17: Training hyperparameters.
Figure 5: Per-query latency distributions on ChestAgentBench. (a) Meissa completes the majority of queries in under 3 seconds; the long tail corresponds to queries invoking multiple tools. (b) Gemini-3-flash + MedRAX averages 87.2s per query due to multiple API calls and remote tool execution, resulting in ∼22× higher latency than Meissa . Note the different x -axis scales.
Figure 6: Chest X-ray for Patterns 1 and 2 (case 18321). An 18-year-old with bronchiolitis obliterans. Both PA and lateral views show clear lungs with no bronchial wall thickening or pleural thickening – a normal-appearing chest X-ray despite clinical disease.
Figure 7: Chest X-ray for Pattern 3 (case 6577). A 70-year-old woman with a mechanical mitral valve. PA view shows homogeneous opacification occupying the lower two-thirds of the left hemithorax. Lateral view reveals a well-circumscribed mass with a visible prosthetic valve. The correct diagnosis is intrapulmonary teratoma.
Figure 8: Chest X-ray for Pattern 4 (case 12819). Trauma case with rib fracture and pulmonary laceration. Arrows on the PA view indicate left lower lobe opacity (arrowhead), subcutaneous emphysema (small arrows), and a pulmonary laceration (large arrow). Lateral views show the fracture site at the 8th intercostal space (arrows). Despite two tool failures, Meissa correctly localizes the fracture via visual reasoning.
Figure 9: Pathology image for Pattern 5. Gross specimen showing bilateral Krukenberg tumors: lobulated, multinodular masses on both ovaries. The question “Is this true bilateral Krukenberg?” requires examining both sides of the image. The base model repeatedly zoomed into the left side without ever examining the right.
Figure 10: Microscopy image for Pattern 6. The image shows fungal hyphae and spores under microscopy – no gastrointestinal tissue is present. BioMedParseTextSeg returned a large segmented area (121,312 pixels) when queried for “gastrointestinal tissue,” leading the model to a false positive conclusion.
Figure 11: Microscopy image for Pattern 7. Histological cross-sections of pinworms ( Enterobius vermicularis ). After 4 rounds of progressive zoom caused a crash, the model recovered by switching to BioMedParseTextSeg , correctly confirming pinworm presence.
Clinical decision-making is inherently experience-driven: physicians progressively refine their reasoning by synthesizing patient history, multimodal observations, and prior diagnostic experiences across interactions. In contrast, current multimodal large language model (MLLM)-based medical AI agents largely operate as stateless inference systems, generating decisions independently for each interaction without retaining or internalizing experiential knowledge. This discrepancy limits their ability to progressively improve reasoning reliability through usage and adapt to longitudinal patient contexts in real-world clinical workflows. In this study, we propose Medical Structured Multimodal Memory (MSM-Mem), an agentic memory framework that enables medical AI agents to evolve through accumulated clinical experiences. MSM-Mem organizes heterogeneous clinical experiences into semantic, episodic, and visual memory and incrementally updates them during inference, allowing the agent to retrieve prior experiences to inform current reasoning and progressively refine decision-making over time. Evaluations on MoE-LLaVA backbones demonstrate consistent performance improve- ments with further gains observed through continued usage. In general, MSM-Mem offers a viable pathway toward medical AI agents capable of evolving their reasoning competence in a manner analogous to the way clinicians learn from practice over time.
Md Asaduzzaman Jabin, Khoa Le, Lin Zhao +1
University of Georgia, Athens, GA, USA · New Jersey Institute of Technology, Newark, NJ, USA
While interpretable prototype networks offer compelling case-based reasoning for clinical diagnostics, their raw continuous outputs lack the semantic structure required for medical documentation. Bridging this gap via standard Retrieval-Augmented Generation (RAG) routinely triggers ``retrieval sycophancy,'' where Large Language Models (LLMs) hallucinate post-hoc rationalizations to align with visual predictions. We introduce ProtoMedAgent, a framework that formalizes multimodal clinical reporting as an iterative, zero-gradient test-time optimization problem over a strict neuro-symbolic bottleneck. Operating on a frozen prototype backbone, we distill latent visual and tabular features into a discrete semantic memory. Online generation is strictly constrained by exact set-theoretic differentials and a reflective Scribe-Critic loop, mathematically precluding unsupported narrative claims. To safely bound data disclosure, we introduce a semantic privacy gate governed by k-anonymity and ℓ-diversity. Evaluated on a 4,160-patient clinical cohort, ProtoMedAgent achieves 91.2% Comparison Set Faithfulness where it fundamentally outperforms standard RAG (46.2%). ProtoMedAgent additionally leverages a binding ℓ-diversity phase transition to systematically reduce artifact-level membership inference risks by an absolute 9.8%.
We introduce MediX-R1, an open-ended Reinforcement Learning (RL) framework for medical multimodal large language models (MLLMs) that enables clinically grounded, free-form answers beyond multiple-choice formats. MediX-R1 fine-tunes a baseline vision-language backbone with Group Based RL and a composite reward tailored for medical reasoning: an LLM-based accuracy reward that judges semantic correctness with a strict YES/NO decision, a medical embedding-based semantic reward to capture paraphrases and terminology variants, and lightweight format and modality rewards that enforce interpretable reasoning and modality recognition. This multi-signal design provides stable, informative feedback for open-ended outputs where traditional verifiable or MCQ-only rewards fall short. To measure progress, we propose a unified evaluation framework for both text-only and image+text tasks that uses a Reference-based LLM-as-judge in place of brittle string-overlap metrics, capturing semantic correctness, reasoning, and contextual alignment. Despite using only ∼51K instruction examples, MediX-R1 achieves excellent results across standard medical LLM (text-only) and VLM (image + text) benchmarks, outperforming strong open-source baselines and delivering particularly large gains on open-ended clinical tasks. Our results demonstrate that open-ended RL with comprehensive reward signals and LLM-based evaluation is a practical path toward reliable medical reasoning in multimodal models. Our trained models, curated datasets and source code are available at https://medix.cvmbzuai.com
Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Omair Mohamed +5
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI) · Jubilee Mission Medical College and Research Institute · JJM Medical College