cs.AIFeb 21, 2026

LAMMI-Pathology: A Tool-Centric Bottom-Up LVLM-Agent Framework for Molecularly Informed Medical Intelligence in Pathology

Authors: Haoyang Su, Shaoting Zhang, Xiaosong Wang

Organizations: Fudan University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The emergence of tool-calling-based agent systems introduces a more evidence-driven paradigm for pathology image analysis in contrast to the coarse-grained text-image diagnostic approaches. With the recent large-scale experimental adoption of spatial transcriptomics technologies, molecularly validated pathological diagnosis is becoming increasingly open and accessible. In this work, we propose LAMMI-Pathology (LVLM-Agent System for Molecularly Informed Medical Intelligence in Pathology), a scalable agent framework for domain-specific agent tool-calling. LAMMI-Pathology adopts a tool-centric, bottom-up architecture in which customized domain-adaptive tools serve as the foundation. These tools are clustered by domain style to form component agents, which are then coordinated through a top-level planner hierarchically, avoiding excessively long context lengths that could induce task drift. Based on that, we introduce a novel trajectory construction mechanism based on Atomic Execution Nodes (AENs), which serve as reliable and composable units for building semi-simulated reasoning trajectories that capture credible agent-tool interactions. Building on this foundation, we develop a trajectory-aware fine-tuning strategy that aligns the planner's decision-making process with these multi-step reasoning trajectories, thereby enhancing inference robustness in pathology understanding and its adaptive use of the customized toolset.

Figures & tables

Figure 1: Overview of the LAMMI-Pathology framework. The top row illustrates the hierarchical reasoning architecture where component agents orchestrate style-specific tools and aggregate contextual evidence for the LAMMI planner. The middle row depicts the data and tool construction pipeline, where spatial transcriptomics literature paired with histopathology images are processed through LVLM retriever to extract QA pairs, and tools are bottom-up clustered according to their sequential co-occurrence frequency during AEN construction, with adaptive cluster numbers. The bottom row presents the AEN-driven semi-simulated trajectory generation and trajectory-aware fine-tuning methodology, with both the planner and component agents operating on shared fine-tuned model parameters.
Figure 2: Trajectory-aware Adapter architecture. The adapter is injected after the FFN in each Transformer decoder layer. Segment masks are dynamically generated from input sequences to identify Thought , Action , and Action Input segments. Three learnable per-channel scaling vectors are applied, producing a modulation term that is multiplicatively applied to the FFN output.
FrameworkAdaptationPlannerACS ↑\uparrowHR ↓\downarrowTRR ↓\downarrow
OpenAI-Agents-SDKPEGPT-50.7390.1640.033
ReACTPEInternVL3.5 8B0.7980.3010.000
MiniCPM-V 4.50.7150.3530.018
Qwen3 VL 8B Instruct0.7050.2550.036
  InternVL3.5 8B0.4620.6620.049
MAT-AgentLoRA+PEMiniCPM-V 4.50.3730.6600.238
Table 1: Performance comparison on PathSpatial-DocQA dataset. We report ACS, HR, and TRR across different frameworks and base models.
FrameworkAdaptationPlannerTRR ↓\downarrowTCF1 ↑\uparrowTSS ↑\uparrowACS ↑\uparrowHR ↓\downarrow
OpenAI-Agents-SDKPEGPT-50.0250.3700.8610.4920.147
ReACTPEInternVL3.5 8B0.0080.0310.3810.5250.446
MiniCPM-V 4.50.0480.3620.8790.6150.579
Qwen3 VL 8B Instruct0.0920.3770.7920.5870.506
  InternVL3.5 8B0.0540.1540.6550.5340.733
MAT-AgentLoRA+PEMiniCPM-V 4.50.1710.1530.7430.4750.757
Table 2: Performance comparison on ST-Traj dataset. We report TRR, TCF1, TSS, ACS, and HR across different frameworks and base models.
FrameworkPlannerACS ↑\uparrowF1 ↑\uparrowHR ↓\downarrowTRR ↓\downarrow
AtlasEduContentPathCLSPubmed
OpenAI-Agents-SDKGPT-50.7090.7550.7730.4800.7190.2880.000
ReACTInternVL3.5 8B0.5000.3410.3690.1980.4430.6910.000
MiniCPM-V 4.50.5540.4380.4940.2320.5120.5380.092
Qwen3 VL 8B Instruct0.5720.4330.5410.2880.5180.4830.001
 InternVL3.5 8B0.5060.3490.4370.2570.4400.7260.041
Table 3: Performance comparison on PathMMU dataset. We report ACS and F1 scores across four sub-tasks (Atlas, EduContent, PathCLS, PubMed) and trajectory quality metrics (HR, TRR) across different frameworks and base models.
AdaptionPlannerNITR=0.2NITR=0.5NITR=1.0
TRR ↓\downarrowTSS ↑\uparrowTRR ↓\downarrowTSS ↑\uparrowTRR ↓\downarrowTSS ↑\uparrow
 InternVL3.5 8B0.0000.5040.0000.5000.0000.500
PEMiniCPM-V 4.50.0270.8140.0500.7800.0850.801
 Qwen3 VL 8B Instruct0.0220.7200.0490.8390.0220.820
LoRA+PEInternVL3.5 8B0.2300.4350.3120.4680.4110.441
MiniCPM-V 4.50.2820.5570.3270.7160.3860.784
Table 4: Ablation study on adaptation methods under varying NITR on PathSpatial-DocQA dataset. We compare PE, TA+PE, LoRA+PE, and Full+PE across different base models and report TRR and TSS for each NITR value.
Adaptionlτ=3l_{\tau}=3lτ=4l_{\tau}=4lτ=5l_{\tau}=5lτ=6l_{\tau}=6lτ=7l_{\tau}=7lτ=8l_{\tau}=8
w/o TA0.6020.6400.6600.6750.6830.691
w/ TA0.7150.7690.7960.8170.8300.862
Table 5: TSS as a function of maximum iteration limit lτl_{\tau} on ST-Traj dataset.
Figure 3: Memory consumption comparison between LAMMI framework and direct multi-agent approaches. The plot shows GPU memory usage differences between LAMMI and standard MAS.
Figure 4: Trajectory visualization of an open-ended research-oriented query showing a semantically coherent solution path that incorporates hallucination occurrence and evidential reasoning through planner-guided exploratory tool invocation by ImageAgent and GeneAgent.
Tool NameSourceDescription
ImageAgent
BLIPToolModel Training: 3.68M HEST ROI, 15k Visium ROI, 10K ST ROI with corresponding high-expression genes (based on RNA-seq data)Provides genetic answers for pathology images with questions related to gene expressions. Input: text instruction/question and image name. Output: basic genetic information including high-expression genes, species, and tissue.
CLIPToolModel Training: OpenCLIP ViT-H-14 fine-tuned on pathogenomics dataset (training data consistent with BLIPTool)OpenCLIP image-text matching tool. Strictly accepts image file paths as input. Do not provide image descriptions or text; only valid paths allowed.
QwenVLCaptionToolModel Training: Image-text caption pairs including PathCap 0.17M, Quilt-Instruct 0.096M, and PathGen 0.14M samples; Base model: Qwen-VL-2.5 vision-language modelProvides answers for pathology images with associated questions. Input: text instruction/question and image path. Returns model’s textual output.
OncoTreeToolAPI: MSKCC OncoTree (oncotree.mskcc.org)Query OncoTree knowledge graph. Input: tumor, disease, or tissue keyword. Returns upstream/downstream nodes and tissue mapping.
GeneAgent
Table 6: Tool Specifications and Sources
Figure 5: Planner Agent prompt template.
Figure 6: Component Agent base prompt template.
Figure 7: GeneAgent prompt template.
Figure 8: ImageAgent prompt template.
Figure 9: Parsing LLM prompt template for schema compliance.
Figure 10: Typical PathSpatial-DocQA dataset example for case STimage-1K4M_35290801_page_2_2.
Figure 11: Typical PathSpatial-DocQA dataset example for case STimage-1K4M_36283023_page_17_2.
Figure 12: Typical PathSpatial-DocQA dataset example for case STimage-1K4M_38113104_page_9_0.
Figure 13: Typical PathSpatial-DocQA dataset example for case STimage-1K4M_38282415_page_3_0.
Figure 14: Typical PathSpatial-DocQA dataset example for case STimage-1K4M_37468493_page_10_1.
Figure 15: Typical PathSpatial-DocQA dataset example for case STimage-1K4M_38113104_page_4_1.
Figure 16: Typical PathSpatial-DocQA dataset example for case STimage-1K4M_38067175_page_2_2.
Figure 17: Typical PathSpatial-DocQA dataset example for case HEST_page_25_0.
Figure 18: Typical PathSpatial-DocQA dataset example for case STimage-1K4M_36741401_page_2_2.
Figure 19: Typical PathSpatial-DocQA dataset example for case STimage-1K4M_37468583_page_7_2.
Figure 20: Typical ST-Traj dataset example for case 8780_9284_10156.
Figure 21: Typical ST-Traj dataset example for case 1384_8588_9733_10239.
Figure 22: Typical ST-Traj dataset example for case MEND85_b_CATGGAAATGGGACCA-1.
DatasetLVLMPlan.Orch.ACSTRRTCF1TSSHRAvg. F1
PathMMUInternVL ✓0.4410.000//0.7000.325
MiniCPM ✓0.4230.342//0.3000.263
Qwen-VL ✓0.5540.038//0.3750.263
InternVL✓ 0.493///0.5190.329
MiniCPM✓ 0.546///0.5920.307
Qwen-VL✓ 0.500///0.6450.331
Table 7: LAMMI ablations on planning (Plan.) or domain knowledge (Orch.). Missing values indicate inapplicability.
Figure 23: Case study 1: Question and reference answer for sample STimage-1K4M_34650042_page_13_2.
Figure 24: Case study 2: Question and reference answer for sample 3649_8072_10393.
Figure 25: Case study 3: Question and reference answer for sample 1511_6709_6887_8022.
Figure 26: Case study 4: Question and reference answer for sample 1317_5021_7882_9140_9288_9676_9691_10334.
Figure 27: Trajectory execution for LAMMI (MiniCPM-V 4.5) framework on case STimage-1K4M_34650042_page_13_2.
Figure 28: Trajectory execution for MAT-Agent (MiniCPM-V 4.5) framework on case STimage-1K4M_34650042_page_13_2.
Figure 29: Trajectory execution for ReACT (MiniCPM-V 4.5) framework on case STimage-1K4M_34650042_page_13_2.
Figure 30: Trajectory execution for MLLM-Tools (ToolACE 8B) framework on case STimage-1K4M_34650042_page_13_2.
Figure 31: Trajectory execution for OpenAI-Agent-SDK (GPT-5) framework on case STimage-1K4M_34650042_page_13_2.
Figure 32: Trajectory execution for LAMMI (Qwen3 VL 8B Instruct) framework on case 3649_8072_10393.
Figure 33: Trajectory execution for MAT-Agent (Qwen3 VL 8B Instruct) framework on case 3649_8072_10393.
Figure 34: Trajectory execution for ReACT (Qwen3 VL 8B Instruct) framework on case 3649_8072_10393.
Figure 35: Trajectory execution for MLLM-Tools (ToolACE 8B) framework on case 3649_8072_10393.
Figure 36: Trajectory execution for OpenAI-Agent-SDK (GPT-5) framework on case 3649_8072_10393.
Figure 37: Trajectory execution for LAMMI (MiniCPM-V 4.5) framework on case 1511_6709_6887_8022.
Figure 38: Trajectory execution for MAT-Agent (MiniCPM-V 4.5) framework on case 1511_6709_6887_8022.
Figure 39: Trajectory execution for ReACT (MiniCPM-V 4.5) framework on case 1511_6709_6887_8022.
Figure 40: Trajectory execution for MLLM-Tools (Vicuna) framework on case 1511_6709_6887_8022.
Figure 41: Trajectory execution for OpenAI-Agent-SDK (GPT-5) framework on case 1511_6709_6887_8022.
Figure 42: Trajectory execution for LAMMI (Qwen-VL 8B Instruct) framework on case 1317_5021_7882_9140_9288_9676_9691_10334.
Figure 43: Trajectory execution for MAT-Agent (Qwen-VL 8B Instruct) framework on case 1317_5021_7882_9140_9288_9676_9691_10334.
Figure 44: Trajectory execution for ReACT (Qwen-VL 8B Instruct) framework on case 1317_5021_7882_9140_9288_9676_9691_10334.
Figure 45: Trajectory execution for MLLM-Tools (Qwen 8B) framework on case 1317_5021_7882_9140_9288_9676_9691_10334.
Figure 46: Trajectory execution for OpenAI-Agent-SDK (GPT-5) framework on case 1317_5021_7882_9140_9288_9676_9691_10334.
More in this topic

Tool-Using Vision-Language Agents

Tool-using vision-language agents combine visual and language models with external tools to gather evidence, act, and solve multi-step tasks.

See all 139 papers

Explore similar work

CardsList

  1. PathoSage: Towards Multi-Source Evidence Adjudication in Pathology via Experience-Aware Agentic Workflow

    May 18, 2026Chengyang Zhang, Wenchuan Zhang, Bo Li +5Computational PathologyEvidence-Grounded Reasoning

    College of Computer Science, Sichuan University · Department of Pathology and Institute of Clinical Pathology, West China Hospital, Sichuan University · Department of Computer and Information Science, University of Macau +1

  2. Spatial Message Passing in Language Space for Pathology Image Interpretation

    Aug 14, 2026Jing-Cheng Yang, Hao-Jung Wang, Jinhao Du +4Whole-Slide Image AnalysisComputational Pathology

    National Taiwan University, Taipei, Taiwan · University of Oxford, Oxford, United Kingdom · ZYTCA Limited, Oxford, United Kingdom

  3. PlanPath: Exploration-Guided Orchestration for Patient-Level Pathology Reasoning

    Oct 13, 2026Jiusong Ge, Mingzhao He, Qidong Liu +8Pathology VQAMedical VQA

    Xi’an Jiaotong University · University of Cambridge · A*STAR +1