PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs

Jinyue Li; Yuci Liang; Qiankun Li; Xinheng Lyu; Jiayu Qian; Huabao Chen; Kun Wang; Zhigang Zeng

cs.AIMar 10, 2026

PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs

Authors: Jinyue Li, Yuci Liang, Qiankun Li, Xinheng Lyu, Jiayu Qian, Huabao Chen, Kun Wang, Zhigang Zeng,

Organizations: University of Science and Technology of China · Shenzhen University · Nanyang Technological University · IGS, Imperial College London ·

Abstract

Computational pathology demands both visual pattern recognition and dynamic integration of structured domain knowledge, including taxonomy, grading criteria, and clinical evidence. In practice, diagnostic reasoning requires linking morphological evidence with formal diagnostic and grading criteria. Although multimodal large language models (MLLMs) demonstrate strong vision language reasoning capabilities, they lack explicit mechanisms for structured knowledge integration and interpretable memory control. As a result, existing models struggle to consistently incorporate pathology-specific diagnostic standards during reasoning. Inspired by the hierarchical memory process of human pathologists, we propose PathMem, a memory-centric multimodal framework for pathology MLLMs. PathMem organizes structured pathology knowledge as a long-term memory (LTM) and introduces a Memory Transformer that models the dynamic transition from LTM to working memory (WM) through multimodal memory activation and context-aware knowledge grounding, enabling context-aware memory refinement for downstream reasoning. PathMem achieves SOTA performance across benchmarks, improving WSI-Bench report generation (12.8% WSI-Precision, 10.1% WSI-Relevance) and open-ended diagnosis by 9.7% and 8.9% over prior WSI-based models.

Figures & tables

Figure 1: Overview of PathMem . The working memory selectively activates relevant long-term memory traces and transforms them into an updated working memory state for more reliable reasoning.

Figure 2: LTM construction pipeline for pathology knowledge graphs via iterative literature retrieval and LLM-based information extraction.

Figure 3: Framework of PathMem . A memory-augmented MLLMs for computational pathology that aligns visual, textual, and knowledge graph representations, and adaptively activates LTM for knowledge-grounded reasoning about pathology.

		Morphological Analysis			Diagnosis			Treatment Planning			Average
Model	Input	Open		Close	Open		Close	Open		Close
		WSI-P	WSI-R	Acc	WSI-P	WSI-R	Acc	WSI-P	WSI-R	Acc
Quilt-LLaVA Seyfioglu et al. (2024)	Thumbnail	0.448	0.447	0.947	0.586	0.604	0.849	0.788	0.816	1.000	0.721
GPT-4o Islam and Moushi (2024)	Thumbnail	0.220	0.204	0.471	0.472	0.457	0.530	0.496	0.841	0.875	0.507
WSI-VQA Chen et al. (2024c)	WSI	0.395	0.462	0.758	0.436	0.525	0.577	0.791	0.595	0.771	0.590
WSI-LLaVA Liang et al. (2025)	WSI	0.488	0.610	0.951	0.610	0.612	0.863	0.810	0.845	1.000	0.754

Table 1: Quantitative evaluation of WSI and general-purpose MLLMs on our WSI-Bench across three pathological capabilities. WSI-P: WSI-Precision, WSI-R: WSI-Relevance, Acc: accuracy, open: open-ended question, and close: close-ended question.

Models	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	METEOR	WSI-Precision	WSI-Relevance
GPT-4o Islam and Moushi (2024)	0.202	0.069	0.030	0.016	0.132	0.167	0.067	0.138
Quilt-LLaVA Seyfioglu et al. (2024)	0.474	0.351	0.282	0.236	0.475	0.460	0.324	0.333
MI-Gen Chen et al. (2024b)	0.403	0.306	0.248	0.209	0.446	0.407	0.310	0.377
Hist-Gen Guo et al. (2024)	0.406	0.307	0.248	0.208	0.448	0.416	0.300	0.397
WSI-LLaVA Liang et al. (2025)	0.480	0.358	0.287	0.240	0.490	0.465	0.380	0.429
PathMem (Ours)	0.548 $\pm$ 0.151	0.420 $\pm$ 0.173	0.347 $\pm$ 0.188	0.302 $\pm$ 0.196	0.536 $\pm$ 0.155	0.531 $\pm$ 0.158	0.508 $\pm$ 0.147	0.530 $\pm$ 0.144

Table 2: Quantitative evaluation of WSI and general MLLMs in report generation on WSI-Bench.

Model	Input	WSI-VQA	SlideBench-VQA (BCNB)				CPTAC
Model	Input	WSI-VQA	Tumor	Grade	Subtype	Avg	CPTAC
Quilt-LLaVA Seyfioglu et al. (2024)	Thumb	0.132	0.113	0.194	0.181	0.163	0.603
WSI-VQA Chen et al. (2024c)	WSI	0.469	0.039	0.300	0.000	0.113	0.000
SlideChat Chen et al. (2025c)	WSI	-	0.902	0.231	0.175	0.436	-
WSI-LLaVA Liang et al. (2025)	WSI	0.546	0.905	0.463	0.292	0.553	0.721
PathMem (Ours)	WSI	0.572	0.918	0.482	0.313	0.571	0.754

Table 3: Zero-shot external validation on WSI-VQA, SlideBench-VQA, and CPTAC-NSCLC.

Figure 4: Qualitative comparison of generated reports by our method and three baseline approaches on the report generation task. (Red highlights denote incorrect content, while green highlights denote correct content, and orange highlights denote missing content from T-answer.)

Version	Baseline	LTM		PathMem
Version	-	D	S	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	METEOR	WSI-P	WSI-R
a	✓			0.481 $\pm$ 0.181	0.357 $\pm$ 0.196	0.287 $\pm$ 0.205	0.241 $\pm$ 0.212	0.477 $\pm$ 0.177	0.464 $\pm$ 0.178	0.404 $\pm$ 0.162	0.445 $\pm$ 0.163
b	✓	✓		0.528 $\pm$ 0.153	0.405 $\pm$ 0.176	0.332 $\pm$ 0.195	0.284 $\pm$ 0.198	0.518 $\pm$ 0.168	0.508 $\pm$ 0.164	0.492 $\pm$ 0.155	0.516 $\pm$ 0.147
c	✓		✓	0.521 $\pm$ 0.161	0.396 $\pm$ 0.183	0.324 $\pm$ 0.205	0.278 $\pm$ 0.205	0.510 $\pm$ 0.170	0.511 $\pm$ 0.163	0.489 $\pm$ 0.158	0.510 $\pm$ 0.148
d	✓	✓	✓	0.548 $\pm$ 0.151	0.420 $\pm$ 0.173	0.347 $\pm$ 0.188	0.302 $\pm$ 0.196	0.536 $\pm$ 0.155	0.531 $\pm$ 0.158	0.508 $\pm$ 0.147	0.530 $\pm$ 0.144

Table 4: Ablation study on static and dynamic KG retrieval. The full model with both static and dynamic LTM achieves the best performance across all metrics. D denotes Dynamic-activation, and S denotes Static-activation LTM .

Version	Token Cap		PathMem
Version	D	S	BLEU-1	BLEU-2	BLEU-3	BLEU-4	ROUGE-L	METEOR	WSI-P	WSI-R
a	1	1	0.517 $\pm$ 0.161	0.393 $\pm$ 0.184	0.320 $\pm$ 0.198	0.274 $\pm$ 0.210	0.508 $\pm$ 0.172	0.500 $\pm$ 0.169	0.477 $\pm$ 0.163	0.496 $\pm$ 0.151
b	2	2	0.525 $\pm$ 0.160	0.402 $\pm$ 0.181	0.330 $\pm$ 0.204	0.284 $\pm$ 0.205	0.517 $\pm$ 0.168	0.509 $\pm$ 0.164	0.484 $\pm$ 0.159	0.497 $\pm$ 0.148
c	3	3	0.531 $\pm$ 0.153	0.410 $\pm$ 0.175	0.339 $\pm$ 0.193	0.293 $\pm$ 0.196	0.525 $\pm$ 0.162	0.514 $\pm$ 0.161	0.493 $\pm$ 0.155	0.513 $\pm$ 0.147
d	4	4	0.537 $\pm$ 0.154	0.414 $\pm$ 0.173	0.340 $\pm$ 0.194	0.292 $\pm$ 0.197	0.528 $\pm$ 0.158	0.517 $\pm$ 0.161	0.502 $\pm$ 0.149	0.526 $\pm$ 0.146
g	5	5	0.542 $\pm$ 0.151	0.418 $\pm$ 0.174	0.346 $\pm$ 0.188	0.298 $\pm$ 0.197	0.532 $\pm$ 0.156	0.523 $\pm$ 0.158	0.506 $\pm$ 0.147	0.528 $\pm$ 0.143

Table 5: Sensitivity analysis on the maximum number of activated KG tokens (Top-K). Performance comparison under different limits for dynamic and static LTM retrieval. D denotes Dynamic-activation, and S denotes Static-activation LTM .

Metric	Value
Inference Time	$\sim$ 1.45 items (WSI) / s
GPU Utilization	1 $\times$ GPU
GPU Memory Usage	16.3 GB
Image Feature Size	$1728\times 4096$ tokens
KG Construction Cost	Offline embedding + indexing
KG Update Cost	Re-embedding + re-indexing

Table 6: Efficiency and practicality analysis of PathMem .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Qualitative comparison of generated reports by our method and three baseline approaches on the report generation task. (Red highlights denote incorrect content, while green highlights denote correct content, and orange highlights denote missing content from T-answer.)

Figure 6: Comparison of WSI-based and NLU-based evaluations. Green indicates agreement with the reference, red denotes deviations, orange marks missing ground-truth content, underlined text reflects template-style language, and blue highlights knowledge graph–grounded concepts.

Figure 7: Comparison of WSI-based and NLU-based evaluations. Green indicates agreement with the reference, red denotes deviations, orange marks missing ground-truth content, underlined text reflects template-style language, and blue highlights knowledge graph–grounded concepts.

Figure 8: Comparison of WSI-based and NLU-based evaluations. Green indicates agreement with the reference, red denotes deviations, orange marks missing ground-truth content, underlined text reflects template-style language, and blue highlights knowledge graph–grounded concepts.

Figure 9: Comparison of WSI-based and NLU-based evaluations. Green indicates agreement with the reference, red denotes deviations, orange marks missing ground-truth content, underlined text reflects template-style language, and blue highlights knowledge graph–grounded concepts.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 157 papers

PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology
Oct 24, 2026Nadun Rajapaksha, Sandun Herath, Devnith Wijesinghe +3Medical Report Generation Whole-Slide Image Analysis
University of Moratuwa, Sri Lanka · Idiap Research Institute, Switzerland · EPFL, Switzerland
MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding
Mar 24, 2026Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi +6Whole-Slide Image Analysis Medical VLMs
Department of Computer Science, Khalifa University of Science and Technology, UAE. · Information Technology University, Pakistan. · KAU, KSA. +1
PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?
Jul 30, 2026Zongyi Chen, Yu Liang, Jie Lin +1Computational Pathology Medical VQA
National Institute for Data Science in Health and Medicine, Xiamen University · Institute of Artificial Intelligence Xiamen University, Xiamen, China · Department of Computer Science School of Informatics Xiamen University, Xiamen, China

Oct 24, 2026J/K move · Enter open · S save

PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology

Nadun Rajapaksha, Sandun Herath, Devnith Wijesinghe, Isiri Amani Withanawasam+2

University of Moratuwa, Sri Lanka · Idiap Research Institute, Switzerland · EPFL, Switzerland

Medical Report Generation Whole-Slide Image Analysis Explainable Medical Image Analysis MLLM Reasoning

Open paper AskPDF arXiv

Whole Slide Image (WSI) analysis serves as the fundamental method for the clinical diagnosis of cancer in digital histopathology. Recent advances in Multimodal Large Language Models (MLLMs) have enabled state-of-the-art (SOTA) automated histopathology report generation. However, MLLMs reason purely through distributions encoded in their neural parameters, limiting the amount of knowledge and reasoning logic they can contain given the model scale. Their black-box reasoning poses another challenge to the human interpretation of intermediate reasoning. We address these issues by utilizing knowledge graphs to inject external structured knowledge for MLLMs to reason over, while making this reasoning component visible. Knowledge graphs are capable of partially externalizing the containment of medical knowledge, an area that remains underexplored in pathology report generation. We introduce PathKGen, a knowledge graph-assisted MLLM for histopathology report generation. PathKGen has two main branches: (1) a visual path that learns visual token inputs to the MLLM using pathology foundation models, and (2) a knowledge path that learns knowledge token inputs by extracting key morphological features of WSIs through querying a pathology-specific knowledge graph. A comprehensive evaluation on WSI-Bench dataset shows that PathKGen surpasses previous SOTA methods while revealing diagnostic reasoning as visually intuitive graphs. Due to its modular design, the proposed knowledge path is complementary to vision-aligned MLLMs and can be extended to other applications by curating domain-specific knowledge graphs to generate knowledge tokens.

Abstract

Memory-Augmented VLMs

Explore similar work

PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?