PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology

Nadun Rajapaksha; Sandun Herath; Devnith Wijesinghe; Isiri Amani Withanawasam; Hasindri Watawana; Ranga Rodrigo

cs.CVOct 24, 2026

PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology

Authors: Nadun Rajapaksha, Sandun Herath, Devnith Wijesinghe, Isiri Amani Withanawasam, Hasindri Watawana, Ranga Rodrigo

Organizations: University of Moratuwa, Sri Lanka · Idiap Research Institute, Switzerland · EPFL, Switzerland

Abstract

Whole Slide Image (WSI) analysis serves as the fundamental method for the clinical diagnosis of cancer in digital histopathology. Recent advances in Multimodal Large Language Models (MLLMs) have enabled state-of-the-art (SOTA) automated histopathology report generation. However, MLLMs reason purely through distributions encoded in their neural parameters, limiting the amount of knowledge and reasoning logic they can contain given the model scale. Their black-box reasoning poses another challenge to the human interpretation of intermediate reasoning. We address these issues by utilizing knowledge graphs to inject external structured knowledge for MLLMs to reason over, while making this reasoning component visible. Knowledge graphs are capable of partially externalizing the containment of medical knowledge, an area that remains underexplored in pathology report generation. We introduce PathKGen, a knowledge graph-assisted MLLM for histopathology report generation. PathKGen has two main branches: (1) a visual path that learns visual token inputs to the MLLM using pathology foundation models, and (2) a knowledge path that learns knowledge token inputs by extracting key morphological features of WSIs through querying a pathology-specific knowledge graph. A comprehensive evaluation on WSI-Bench dataset shows that PathKGen surpasses previous SOTA methods while revealing diagnostic reasoning as visually intuitive graphs. Due to its modular design, the proposed knowledge path is complementary to vision-aligned MLLMs and can be extended to other applications by curating domain-specific knowledge graphs to generate knowledge tokens.

Figures & tables

Figure 1: PathKGen is a vision-language assistant for generating histopathology reports while exposing the intermediate decision-making in a graphically intuitive, human readable format, thus improving explainability and encouraging clinical adoption.

Figure 2: Overall pipeline architecture showing the connection of visual path (in blue) and knowledge path (in yellow). Slide features are extracted using CONCH and TITAN (jointly serving as the vision encoder) , and projected into the same semantic space of the MLLM. An auxiliary classifier predicts the histology in the WSI, which is subsequently used to query the knowledge graph and retrieve knowledge about morphological features. Visual tokens ( $\mathbf{v}$ ), knowledge tokens ( $\mathbf{k}$ ) and instruction prompt ( $\mathbf{p}$ ) are given to the MLLM as input. See Sec. 3 for detailed methodology.

Figure 3: (a) Detailed working example of the how the subgraph is extracted, starting from the WSI. The auxiliary classifier predicts the histology class using slide embeddings and anatomical site, which acts as the anchor node for querying the pathology knowledge graph. This returns a fixed deterministic subgraph containing the canonical morphology profile of the predicted histology, which is then organized into structured text and tokenized. See Sec. 3.2 for explanation of detailed working. (b) Comparison of the extracted subgraph with the base knowledge graph. (c) Distribution of the 6 categories of relations in our curated pathology knowledge graph.

Figure 4: Knowledge graph curation pipeline. Morphological features are extracted from ground-truth histopathology reports for each histology class and mapped to UMLS concepts using a biomedical concept alignment model. Then semantic relations between histology classes and morphological concepts are assigned based on UMLS semantic types. The resulting knowledge graph is subsequently reviewed and verified by expert pathologists. See Sec. 3.4 for more detailed description.

Input	Acc.	Prec.	Rec.	F1
Primary site only	0.8416	0.6764	0.7393	0.6941
Slide emb. only	0.9356	0.8685	0.8873	0.8754
Slide emb. + site	0.9752	0.9656	0.9848	0.9711

Table 1: Ablation of auxiliary histology classification. Combining slide embeddings with primary site improves performance over either input alone. Abbreviations: Emb. (Embedding), Acc. (Accuracy), Prec. (Precision), Rec. (Recall), F1 (Macro-F1).

Figure 5: Qualitative visualization of sample model predictions, highlighting the extracted feature nodes (in comparison to the entire knowledge graph) which are explicitly reflected in the generated report. Predicted carcinoma is highlighted in blue. Features present in the WSI are highlighted in green, while features with no evidence are highlighted in red. See Sec. 4.3 for the interpretation. Node labels are omitted for clarity and could be toggled on using graph visualization software. The color coding is a post-hoc formatting processed using rule-based text matching. Additional examples with node labels are provided in Appendix E .

Models	B-1	B-2	B-3	B-4	R-L	M	W-P
GPT-4o [ 12 ]	0.202	0.069	0.030	0.016	0.132	0.167	0.067
Quilt-LLaVA [ 29 ]	0.334	0.170	0.093	0.057	0.246	0.291	0.324
MI-Gen [ 3 ]	0.403	0.306	0.248	0.209	0.446	0.407	0.310
Hist-Gen [ 9 ]	0.406	0.307	0.248	0.208	0.448	0.416	0.300
WSI-Agents [ 26 ]	0.444	0.308	0.231	0.183	0.448	0.472	0.440
WSI-LLaVA [ 17 ]	0.480	0.358	0.287	0.240	0.490	0.465	0.380

Table 2: Quantitative comparison of existing models on the report generation task of the WSI-Bench [ 17 ] dataset. PathKGen outperforms the existing WSI MLLMs in multiple metrics. The results are as reported in WSI-Agents [ 26 ] , and WSI-LLaVa [ 17 ] . See Sec. 4.1 for detailed analysis. Abbreviations: B-1/B-2/B-3/B-4 (BLEU-1/2/3/4), R-L (ROUGE-L), M (METEOR), W-P (WSI-Precision).

Knowledge-path input	B-1	B-2	B-3	B-4	R-L	M	W-P
None	0.4922	0.3745	0.3063	0.2548	0.5185	0.4730	0.4661
Predicted histology (H)	0.5020	0.3805	0.3088	0.2558	0.5174	0.4789	0.4570
Primary site (PS) only	0.4959	0.3773	0.3072	0.2549	0.5142	0.4750	0.4480
PathKGen (H + PS + MF)	0.5122	0.3885	0.3154	0.2611	0.5194	0.4849	0.4802

Table 3: Ablation study of the knowledge pathway. The vision pathway is retained in all conditions, while the information supplied through the knowledge pathway is varied. All models are evaluated at the 7B parameter scale. See Sec. 4.4 for further details. Abbreviations: H (predicted histology), PS (primary site), MF (morphological features), B-1/B-2/B-3/B-4 (BLEU-1/2/3/4), R-L (ROUGE-L), M (METEOR), W-P (WSI-Precision).

Method	B-1	B-2	B-3	B-4	R-L	M	W-P
V path only (7B)	0.4922	0.3745	0.3063	0.2548	0.5185	0.4730	0.4661
(V + K) path (1.5B)	0.4792	0.3607	0.2930	0.2426	0.5040	0.4562	0.4290
(V + K) path (3B)	0.4950	0.3723	0.3011	0.2484	0.5082	0.4719	0.4428
(V + K) path (7B) (PathKGen)	0.5122	0.3885	0.3154	0.2611	0.5194	0.4849	0.4802

Table 4: Parameter-efficiency study of the knowledge pathway. Integrating external pathology knowledge enables lower-parameter MLLMs to achieve competitive performance, allowing smaller models to approach the performance of substantially larger models and SOTA systems. Values enclosed in parentheses denote the model parameter size in billions. See Sec. 4.4 for further evaluation details. Abbreviations: B-1/B-2/B-3/B-4 (BLEU-1/2/3/4), R-L (ROUGE-L), M (METEOR), W-P (WSI-Precision), V (Vision), K (Knowledge).

Intervention	W-P
Predicted histology anchor (default)	0.4802
Oracle histology anchor	0.4830
Random-wrong histology anchor	0.2540

Table 5: Counterfactual knowledge-graph interventions at inference time. We perturb the knowledge supplied to a fixed trained PathKGen model to evaluate its sensitivity to the retrieved knowledge. The model’s improved report-generation performance with correct knowledge and its degradation under corrupted knowledge provide evidence that it meaningfully utilizes the retrieved knowledge. Abbreviation: W-P (WSI-Precision).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Class	Precision	Recall	F1-score	Support
Adrenocortical Carcinoma	1.00	1.00	1.00	5
Bladder Urothelial Carcinoma	1.00	0.89	0.94	9
Brain Lower Grade Glioma	1.00	1.00	1.00	15
Breast Invasive Carcinoma	1.00	0.96	0.98	23
Cervical Squamous Cell Carcinoma	1.00	1.00	1.00	2
Cholangiocarcinoma	1.00	1.00	1.00	2

Appendix

Table 6: Per-class histology classification metrics for the auxiliary classifier evaluated on the test set.

Figure 6: Confusion matrix for the auxiliary classifier (Sec. 3.2), evaluated on the test set

Metric	95% CI
B-1	(0.5094, 0.5138)
B-2	(0.3852, 0.3911)
B-3	(0.3122, 0.3185)
B-4	(0.2580, 0.2645)
M	(0.4826, 0.4896)
R-L	(0.5179, 0.5203)

Appendix

Table 7: Confidence intervals for PathKGen’s report-generation metrics, computed over repeated evaluation runs. This indicates the metrics are stable and differences across knowledge path conditions are unlikely to be due to run to run variance alone. Abbreviations: B-1/B-2/B-3/B-4 (BLEU-1/2/3/4), R-L (ROUGE-L), M (METEOR), W-P (WSI-Precision).

Panel of Figure 7 — Figure 7: Generation prompt template with knowledge guidance. During training, the pathologist-written report is given as the answer in the assistant turn; at inference, autoregressive next token generation begins from the assistant turn.

Figure 8: Additional qualitative examples containing triplets of Whole Slide Image (WSI), activated graph nodes, and generated report. Green: evidence present in the WSI, red: evidence absent in the WSI, blue: predicted histology.

Model	Model Responses
WSI
Ground Truth	Microscopic observation of the pathology slide reveals a presence of tumor cells forming irregular glandular structures that moderately deviate from the normal tissue architecture, consistent with moderate differentiation. The nuclei exhibit slight pleomorphism, with moderate variability in size and shape, and there is an intermediate rate of mitosis observed. Vascular invasion is not specified in the observations available, and there is no mention of necrosis or calcification. Based on these observations, the diagnosis is adenocarcinoma, moderately differentiated.
WSI-LLaVA	Microscopic observation of the pathology slide reveals a moderately differentiated adenocarcinoma characterized by irregular clusters of epithelial cells infiltrating the lung tissue. The tumor cells exhibit moderate glandular formation, deviating from normal alveolar architecture. The nuclei show moderate variability in size and shape, indicative of nuclear pleomorphism. There is no evidence of vascular invasion or necrosis within the observed slide. Based on these observations, the diagnosis is adenosquamous carcinoma, moderately-differentiated.
PathKGen	Microscopic observation of the pathology slide reveals a moderately differentiated adenocarcinoma characterized by irregular clusters of epithelial cells infiltrating the stroma, forming glandular structures that moderately deviate from normal tissue architecture. The nuclei exhibit moderate variability in size and shape, indicative of nuclear pleomorphism, and there is a moderate rate of cell division observed. There is no evidence of vascular invasion, necrosis, or calcification within the examined tissue. Based on these observations, the diagnosis is adenoc arcinoma, moderately differentiated.

Appendix

Table 8: Qualitative comparison of pathology reports generated by WSI-LLaVA and PathKGen models against the ground truth for a sample whole slide image.

Model	Model Responses
WSI
Ground Truth	Microscopic observation of the pathology slide reveals well-differentiated endometrioid carcinoma cells invading minimally into the myometrium. The architecture is predominantly glandular, exhibiting preserved tubule formation indicative of low-grade differentiation, corresponding to a FIGO Grade 1 classification. There is an absence of lymphovascular invasion, which is a favorable prognostic indicator. The tumor primarily shows superficial invasion with no significant features of atypical necrosis or calcification. Based on these observations, the diagnosis is well-differentiated endometrioid adenocarcinoma, FIGO Grade I.
WSI-LLaVA	Microscopic observation of the pathology slide reveals well-differentiated glandular structures with minimal deviation from normal endometrial gland architecture, consistent with a FIGO grade 1 classification. The tumor cells exhibit nuclear features ranging from intermediate to high-grade, indicating a nuclear grade of 2-3. There is no evidence of lymphovascular space invasion. The surrounding non-neoplastic endometrium appears atrophic. Based on these observations, the diagnosis is endometrioid adenocarcinoma, FIGo grade 2.carinoma.
PathKGen	Microscopic observation of the pathology slide reveals endometrioid adenocarcinoma characterized by well-differentiated glandular structures, consistent with a FIGO grade 1 classification. The tumor cells exhibit minimal nuclear atypia and a low rate of mitotic activity. There is no evidence of lymphovascular space invasion, and the tumor is confined to the endometrium without invasion into the myometrium. Based on these observations, the diagnosis is endometrial carcinoma, FIGO Grade 1.

Appendix

Table 9: Qualitative comparison of pathology reports generated by WSI-LLaVA and PathKGen models against the ground truth for a sample whole slide image.

Model	Model Responses
WSI
Ground Truth	Microscopic observation of the pathology slide reveals that the serous adenocarcinoma is characterized by high-grade papillary structures with areas of solid and glandular growth patterns. Tumor cells exhibit significant atypia with prominent nucleoli and high mitotic activity. Numerous psammoma bodies , which are concentrically calcified structures, are present throughout the specimen. There is widespread tumor necrosis and lymphovascular invasion, indicating aggressive behavior of the carcinoma. Reactive stromal changes, including fibrosis and secondary inflammation , are noted, suggesting a chronic inflammatory response. Additionally, foci of calcification are seen, which may correspond to previous necrotic or injured tissues. Based on these observations, the diagnosis is high-grade papillary serous carcinoma.
WSI-LLaVA	Microscopic observation of the pathology slide reveals a high-grade neoplasm characterized by the presence of malignant epithelial cells forming complex papillary structures with fibrovascular cores. The tumor cells exhibit marked nuclear atypia, including pleomorphic nuclei with prominent nucleoli and irregular nuclear contours. There is a high mitotic activity indicative of rapid cell division. Areas of necrosis are present, and there is evidence of vascular invasion within the lymphovascular spaces. The ovarian stroma shows infiltration by the tumor, and the tumoral cells extend to the serosal surfaces. Based on these observations, the diagnosis is high- grade mullerian-derived adenocarcinoma, papillay type.
PathKGen	Microscopic observation of the pathology slide reveals a high-grade serous carcinoma characterized by papillary and solid growth patterns. The tumor cells exhibit marked nuclear atypia with prominent nucleoli and frequent mitotic figures, indicating a high proliferative activity. There is evidence of lymphovascular invasion, as tumor cells are observed within vascular spaces. Necrosis is present within the tumor tissue, contributing to its heterogeneous appearance. Based on these observations, the diagnosis is serous adenocarcinoma, high grade.

Appendix

Table 10: Qualitative comparison of pathology reports generated by WSI-LLaVA and PathKGen models against the ground truth for a sample whole slide image.

Medical Report Generation

Medical report generation turns medical images, clinical records, or conversations into structured, clinically useful reports.

See all 82 papers

Mar 10, 2026

PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs

Computational pathology demands both visual pattern recognition and dynamic integration of structured domain knowledge, including taxonomy, grading criteria, and clinical evidence. In practice, diagnostic reasoning requires linking morphological evidence with formal diagnostic and grading criteria. Although multimodal large language models (MLLMs) demonstrate strong vision language reasoning capabilities, they lack explicit mechanisms for structured knowledge integration and interpretable memory control. As a result, existing models struggle to consistently incorporate pathology-specific diagnostic standards during reasoning. Inspired by the hierarchical memory process of human pathologists, we propose PathMem, a memory-centric multimodal framework for pathology MLLMs. PathMem organizes structured pathology knowledge as a long-term memory (LTM) and introduces a Memory Transformer that models the dynamic transition from LTM to working memory (WM) through multimodal memory activation and context-aware knowledge grounding, enabling context-aware memory refinement for downstream reasoning. PathMem achieves SOTA performance across benchmarks, improving WSI-Bench report generation (12.8% WSI-Precision, 10.1% WSI-Relevance) and open-ended diagnosis by 9.7% and 8.9% over prior WSI-based models.

Jinyue Li, Yuci Liang, Qiankun Li +7

University of Science and Technology of China · Shenzhen University · Nanyang Technological University +2

Memory-Augmented VLMs Medical VLMs Computational Pathology Multimodal Memory Whole-Slide Image Analysis

Ask PDF

Mar 24, 2026

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce MLLM-HWSI, a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight Cell-Cell Attention Fusion (CCAF) transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: GitHub.

Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi +6

Department of Computer Science, Khalifa University of Science and Technology, UAE. · Information Technology University, Pakistan. · KAU, KSA. +1

Whole-Slide Image Analysis Medical VLMs

Ask PDF

Sep 1, 2026

Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation

The rapid advancement of vision-language models (VLMs) has accelerated progress in computational pathology; however, whole-slide image (WSI)-based pathology report generation remains limited by the scarcity of large-scale WSI--report datasets and the complexity of mapping spatially distributed visual patterns to structured clinical text. To address this, we introduce a clinically curated Pan-Asia WSI--report dataset of approximately 10,500 pairs from five institutions and establish the REG 2025 benchmark through a MICCAI challenge for systematic evaluation of multimodal models. We analyze submitted methods spanning pretrained VLMs, multiple-instance learning frameworks, hierarchical expert models, retrieval-augmented generation, and cross-modal Transformers. Rather than indicating that VLM use alone was sufficient for superior performance, the results suggest that top-performing methods benefited from structured report representations, hierarchical diagnostic decomposition, and effective multimodal grounding. We identify key limitations, including instability in quantitative attribute estimation (e.g., numeric hallucination) and a tendency toward diagnostic overspecification, with some errors resembling known diagnostic pitfalls in routine pathology. These findings establish REG 2025 as a benchmark for evaluating WSI-based structured report generation and vision-language understanding in computational pathology, providing insights for the design of clinically grounded multimodal pathology models.

Yumi Lee, Harim Oh, Hyoryung Kim +52

Computational Pathology Medical VLMs Medical Image Benchmarks Whole-Slide Image Analysis

Ask PDF

Abstract

Medical Report Generation

Explore similar work

PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

Benchmarking Vision-Language Models for Automated Pathology Diagnosis and Report Generation