cs.CVOct 24, 2026
SavePathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology
Organizations: University of Moratuwa, Sri Lanka · Idiap Research Institute, Switzerland · EPFL, Switzerland
Abstract
Whole Slide Image (WSI) analysis serves as the fundamental method for the clinical diagnosis of cancer in digital histopathology. Recent advances in Multimodal Large Language Models (MLLMs) have enabled state-of-the-art (SOTA) automated histopathology report generation. However, MLLMs reason purely through distributions encoded in their neural parameters, limiting the amount of knowledge and reasoning logic they can contain given the model scale. Their black-box reasoning poses another challenge to the human interpretation of intermediate reasoning. We address these issues by utilizing knowledge graphs to inject external structured knowledge for MLLMs to reason over, while making this reasoning component visible. Knowledge graphs are capable of partially externalizing the containment of medical knowledge, an area that remains underexplored in pathology report generation. We introduce PathKGen, a knowledge graph-assisted MLLM for histopathology report generation. PathKGen has two main branches: (1) a visual path that learns visual token inputs to the MLLM using pathology foundation models, and (2) a knowledge path that learns knowledge token inputs by extracting key morphological features of WSIs through querying a pathology-specific knowledge graph. A comprehensive evaluation on WSI-Bench dataset shows that PathKGen surpasses previous SOTA methods while revealing diagnostic reasoning as visually intuitive graphs. Due to its modular design, the proposed knowledge path is complementary to vision-aligned MLLMs and can be extended to other applications by curating domain-specific knowledge graphs to generate knowledge tokens.
Figures & tables
Figure 1: PathKGen is a vision-language assistant for generating histopathology reports while exposing the intermediate decision-making in a graphically intuitive, human readable format, thus improving explainability and encouraging clinical adoption.
Figure 2: Overall pipeline architecture showing the connection of visual path (in blue) and knowledge path (in yellow). Slide features are extracted using CONCH and TITAN (jointly serving as the vision encoder) , and projected into the same semantic space of the MLLM. An auxiliary classifier predicts the histology in the WSI, which is subsequently used to query the knowledge graph and retrieve knowledge about morphological features. Visual tokens ( ), knowledge tokens ( ) and instruction prompt ( ) are given to the MLLM as input. See Sec. 3 for detailed methodology.
Figure 3: (a) Detailed working example of the how the subgraph is extracted, starting from the WSI. The auxiliary classifier predicts the histology class using slide embeddings and anatomical site, which acts as the anchor node for querying the pathology knowledge graph. This returns a fixed deterministic subgraph containing the canonical morphology profile of the predicted histology, which is then organized into structured text and tokenized. See Sec. 3.2 for explanation of detailed working. (b) Comparison of the extracted subgraph with the base knowledge graph. (c) Distribution of the 6 categories of relations in our curated pathology knowledge graph.
Figure 4: Knowledge graph curation pipeline. Morphological features are extracted from ground-truth histopathology reports for each histology class and mapped to UMLS concepts using a biomedical concept alignment model. Then semantic relations between histology classes and morphological concepts are assigned based on UMLS semantic types. The resulting knowledge graph is subsequently reviewed and verified by expert pathologists. See Sec. 3.4 for more detailed description.
| Input | Acc. | Prec. | Rec. | F1 |
|---|---|---|---|---|
| Primary site only | 0.8416 | 0.6764 | 0.7393 | 0.6941 |
| Slide emb. only | 0.9356 | 0.8685 | 0.8873 | 0.8754 |
| Slide emb. + site | 0.9752 | 0.9656 | 0.9848 | 0.9711 |
Table 1: Ablation of auxiliary histology classification. Combining slide embeddings with primary site improves performance over either input alone. Abbreviations: Emb. (Embedding), Acc. (Accuracy), Prec. (Precision), Rec. (Recall), F1 (Macro-F1).
Figure 5: Qualitative visualization of sample model predictions, highlighting the extracted feature nodes (in comparison to the entire knowledge graph) which are explicitly reflected in the generated report. Predicted carcinoma is highlighted in blue. Features present in the WSI are highlighted in green, while features with no evidence are highlighted in red. See Sec. 4.3 for the interpretation. Node labels are omitted for clarity and could be toggled on using graph visualization software. The color coding is a post-hoc formatting processed using rule-based text matching. Additional examples with node labels are provided in Appendix E .
| Models | B-1 | B-2 | B-3 | B-4 | R-L | M | W-P |
|---|---|---|---|---|---|---|---|
| GPT-4o [ 12 ] | 0.202 | 0.069 | 0.030 | 0.016 | 0.132 | 0.167 | 0.067 |
| Quilt-LLaVA [ 29 ] | 0.334 | 0.170 | 0.093 | 0.057 | 0.246 | 0.291 | 0.324 |
| MI-Gen [ 3 ] | 0.403 | 0.306 | 0.248 | 0.209 | 0.446 | 0.407 | 0.310 |
| Hist-Gen [ 9 ] | 0.406 | 0.307 | 0.248 | 0.208 | 0.448 | 0.416 | 0.300 |
| WSI-Agents [ 26 ] | 0.444 | 0.308 | 0.231 | 0.183 | 0.448 | 0.472 | 0.440 |
| WSI-LLaVA [ 17 ] | 0.480 | 0.358 | 0.287 | 0.240 | 0.490 | 0.465 | 0.380 |
Table 2: Quantitative comparison of existing models on the report generation task of the WSI-Bench [ 17 ] dataset. PathKGen outperforms the existing WSI MLLMs in multiple metrics. The results are as reported in WSI-Agents [ 26 ] , and WSI-LLaVa [ 17 ] . See Sec. 4.1 for detailed analysis. Abbreviations: B-1/B-2/B-3/B-4 (BLEU-1/2/3/4), R-L (ROUGE-L), M (METEOR), W-P (WSI-Precision).
| Knowledge-path input | B-1 | B-2 | B-3 | B-4 | R-L | M | W-P |
|---|---|---|---|---|---|---|---|
| None | 0.4922 | 0.3745 | 0.3063 | 0.2548 | 0.5185 | 0.4730 | 0.4661 |
| Predicted histology (H) | 0.5020 | 0.3805 | 0.3088 | 0.2558 | 0.5174 | 0.4789 | 0.4570 |
| Primary site (PS) only | 0.4959 | 0.3773 | 0.3072 | 0.2549 | 0.5142 | 0.4750 | 0.4480 |
| PathKGen (H + PS + MF) | 0.5122 | 0.3885 | 0.3154 | 0.2611 | 0.5194 | 0.4849 | 0.4802 |
Table 3: Ablation study of the knowledge pathway. The vision pathway is retained in all conditions, while the information supplied through the knowledge pathway is varied. All models are evaluated at the 7B parameter scale. See Sec. 4.4 for further details. Abbreviations: H (predicted histology), PS (primary site), MF (morphological features), B-1/B-2/B-3/B-4 (BLEU-1/2/3/4), R-L (ROUGE-L), M (METEOR), W-P (WSI-Precision).
| Method | B-1 | B-2 | B-3 | B-4 | R-L | M | W-P |
|---|---|---|---|---|---|---|---|
| V path only (7B) | 0.4922 | 0.3745 | 0.3063 | 0.2548 | 0.5185 | 0.4730 | 0.4661 |
| (V + K) path (1.5B) | 0.4792 | 0.3607 | 0.2930 | 0.2426 | 0.5040 | 0.4562 | 0.4290 |
| (V + K) path (3B) | 0.4950 | 0.3723 | 0.3011 | 0.2484 | 0.5082 | 0.4719 | 0.4428 |
| (V + K) path (7B) (PathKGen) | 0.5122 | 0.3885 | 0.3154 | 0.2611 | 0.5194 | 0.4849 | 0.4802 |
Table 4: Parameter-efficiency study of the knowledge pathway. Integrating external pathology knowledge enables lower-parameter MLLMs to achieve competitive performance, allowing smaller models to approach the performance of substantially larger models and SOTA systems. Values enclosed in parentheses denote the model parameter size in billions. See Sec. 4.4 for further evaluation details. Abbreviations: B-1/B-2/B-3/B-4 (BLEU-1/2/3/4), R-L (ROUGE-L), M (METEOR), W-P (WSI-Precision), V (Vision), K (Knowledge).
| Intervention | W-P |
|---|---|
| Predicted histology anchor (default) | 0.4802 |
| Oracle histology anchor | 0.4830 |
| Random-wrong histology anchor | 0.2540 |
Table 5: Counterfactual knowledge-graph interventions at inference time. We perturb the knowledge supplied to a fixed trained PathKGen model to evaluate its sensitivity to the retrieved knowledge. The model’s improved report-generation performance with correct knowledge and its degradation under corrupted knowledge provide evidence that it meaningfully utilizes the retrieved knowledge. Abbreviation: W-P (WSI-Precision).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Class | Precision | Recall | F1-score | Support |
| Adrenocortical Carcinoma | 1.00 | 1.00 | 1.00 | 5 |
| Bladder Urothelial Carcinoma | 1.00 | 0.89 | 0.94 | 9 |
| Brain Lower Grade Glioma | 1.00 | 1.00 | 1.00 | 15 |
| Breast Invasive Carcinoma | 1.00 | 0.96 | 0.98 | 23 |
| Cervical Squamous Cell Carcinoma | 1.00 | 1.00 | 1.00 | 2 |
| Cholangiocarcinoma | 1.00 | 1.00 | 1.00 | 2 |
Table 6: Per-class histology classification metrics for the auxiliary classifier evaluated on the test set.
Figure 6: Confusion matrix for the auxiliary classifier (Sec. 3.2), evaluated on the test set
| Metric | 95% CI |
|---|---|
| B-1 | (0.5094, 0.5138) |
| B-2 | (0.3852, 0.3911) |
| B-3 | (0.3122, 0.3185) |
| B-4 | (0.2580, 0.2645) |
| M | (0.4826, 0.4896) |
| R-L | (0.5179, 0.5203) |
Table 7: Confidence intervals for PathKGen’s report-generation metrics, computed over repeated evaluation runs. This indicates the metrics are stable and differences across knowledge path conditions are unlikely to be due to run to run variance alone. Abbreviations: B-1/B-2/B-3/B-4 (BLEU-1/2/3/4), R-L (ROUGE-L), M (METEOR), W-P (WSI-Precision).
Figure 7: Generation prompt template with knowledge guidance. During training, the pathologist-written report is given as the answer in the assistant turn; at inference, autoregressive next token generation begins from the assistant turn.
Figure 8: Additional qualitative examples containing triplets of Whole Slide Image (WSI), activated graph nodes, and generated report. Green: evidence present in the WSI, red: evidence absent in the WSI, blue: predicted histology.
| Model | Model Responses |
|---|---|
| WSI | |
| Ground Truth | Microscopic observation of the pathology slide reveals a presence of tumor cells forming irregular glandular structures that moderately deviate from the normal tissue architecture, consistent with moderate differentiation. The nuclei exhibit slight pleomorphism, with moderate variability in size and shape, and there is an intermediate rate of mitosis observed. Vascular invasion is not specified in the observations available, and there is no mention of necrosis or calcification. Based on these observations, the diagnosis is adenocarcinoma, moderately differentiated. |
| WSI-LLaVA | Microscopic observation of the pathology slide reveals a moderately differentiated adenocarcinoma characterized by irregular clusters of epithelial cells infiltrating the lung tissue. The tumor cells exhibit moderate glandular formation, deviating from normal alveolar architecture. The nuclei show moderate variability in size and shape, indicative of nuclear pleomorphism. There is no evidence of vascular invasion or necrosis within the observed slide. Based on these observations, the diagnosis is adenosquamous carcinoma, moderately-differentiated. |
| PathKGen | Microscopic observation of the pathology slide reveals a moderately differentiated adenocarcinoma characterized by irregular clusters of epithelial cells infiltrating the stroma, forming glandular structures that moderately deviate from normal tissue architecture. The nuclei exhibit moderate variability in size and shape, indicative of nuclear pleomorphism, and there is a moderate rate of cell division observed. There is no evidence of vascular invasion, necrosis, or calcification within the examined tissue. Based on these observations, the diagnosis is adenoc arcinoma, moderately differentiated. |
Table 8: Qualitative comparison of pathology reports generated by WSI-LLaVA and PathKGen models against the ground truth for a sample whole slide image.
| Model | Model Responses | ||
|---|---|---|---|
| WSI | |||
| Ground Truth | Microscopic observation of the pathology slide reveals well-differentiated endometrioid carcinoma cells invading minimally into the myometrium. The architecture is predominantly glandular, exhibiting preserved tubule formation indicative of low-grade differentiation, corresponding to a FIGO Grade 1 classification. There is an absence of lymphovascular invasion, which is a favorable prognostic indicator. The tumor primarily shows superficial invasion with no significant features of atypical necrosis or calcification. Based on these observations, the diagnosis is well-differentiated endometrioid adenocarcinoma, FIGO Grade I. | ||
| WSI-LLaVA | Microscopic observation of the pathology slide reveals well-differentiated glandular structures with minimal deviation from normal endometrial gland architecture, consistent with a FIGO grade 1 classification. The tumor cells exhibit nuclear features ranging from intermediate to high-grade, indicating a nuclear grade of 2-3. There is no evidence of lymphovascular space invasion. The surrounding non-neoplastic endometrium appears atrophic. Based on these observations, the diagnosis is endometrioid adenocarcinoma, FIGo grade 2.carinoma. | ||
| PathKGen | Microscopic observation of the pathology slide reveals endometrioid adenocarcinoma characterized by well-differentiated glandular structures, consistent with a FIGO grade 1 classification. The tumor cells exhibit minimal nuclear atypia and a low rate of mitotic activity. There is no evidence of lymphovascular space invasion, and the tumor is confined to the endometrium without invasion into the myometrium. Based on these observations, the diagnosis is endometrial carcinoma, FIGO Grade 1. |
Table 9: Qualitative comparison of pathology reports generated by WSI-LLaVA and PathKGen models against the ground truth for a sample whole slide image.
| Model | Model Responses |
|---|---|
| WSI | |
| Ground Truth | Microscopic observation of the pathology slide reveals that the serous adenocarcinoma is characterized by high-grade papillary structures with areas of solid and glandular growth patterns. Tumor cells exhibit significant atypia with prominent nucleoli and high mitotic activity. Numerous psammoma bodies , which are concentrically calcified structures, are present throughout the specimen. There is widespread tumor necrosis and lymphovascular invasion, indicating aggressive behavior of the carcinoma. Reactive stromal changes, including fibrosis and secondary inflammation , are noted, suggesting a chronic inflammatory response. Additionally, foci of calcification are seen, which may correspond to previous necrotic or injured tissues. Based on these observations, the diagnosis is high-grade papillary serous carcinoma. |
| WSI-LLaVA | Microscopic observation of the pathology slide reveals a high-grade neoplasm characterized by the presence of malignant epithelial cells forming complex papillary structures with fibrovascular cores. The tumor cells exhibit marked nuclear atypia, including pleomorphic nuclei with prominent nucleoli and irregular nuclear contours. There is a high mitotic activity indicative of rapid cell division. Areas of necrosis are present, and there is evidence of vascular invasion within the lymphovascular spaces. The ovarian stroma shows infiltration by the tumor, and the tumoral cells extend to the serosal surfaces. Based on these observations, the diagnosis is high- grade mullerian-derived adenocarcinoma, papillay type. |
| PathKGen | Microscopic observation of the pathology slide reveals a high-grade serous carcinoma characterized by papillary and solid growth patterns. The tumor cells exhibit marked nuclear atypia with prominent nucleoli and frequent mitotic figures, indicating a high proliferative activity. There is evidence of lymphovascular invasion, as tumor cells are observed within vascular spaces. Necrosis is present within the tumor tissue, contributing to its heterogeneous appearance. Based on these observations, the diagnosis is serous adenocarcinoma, high grade. |
Table 10: Qualitative comparison of pathology reports generated by WSI-LLaVA and PathKGen models against the ground truth for a sample whole slide image.
Figure 9: Curated Knowledge Graph
More in this topic
Medical Report Generation
Medical report generation turns medical images, clinical records, or conversations into structured, clinically useful reports.
See all 82 papers