Decomposing Whole Slide Image Report Generation with Graph-Constrained Multiple Instance Learning Workflows

Antony Gitau; Martyna Borak; Bjørn-Jostein Singstad; Martin Paulson; Karl Thomas Hjelmervik; Ola Marius Lysaker; Veralia Gabriela Sanchez

cs.CVAug 15, 2026

Decomposing Whole Slide Image Report Generation with Graph-Constrained Multiple Instance Learning Workflows

Authors: Antony Gitau, Martyna Borak, Bjørn-Jostein Singstad, Martin Paulson, Karl Thomas Hjelmervik, Ola Marius Lysaker, Veralia Gabriela Sanchez

Organizations: Faculty of Technology, Natural Sciences and Maritime Sciences, University of South-Eastern Norway, Borre, Norway · Center for Cancer and Blood Diseases, Vestfold Hospital Trust, Tønsberg, Norway · Faculty of Biomedical Engineering, Silesian University of Technology, Zabrze, Poland · Department of Radiology, Vestfold Hospital Trust, Tønsberg, Norway

Abstract

Whole-slide image (WSI) report generation requires recognizing spatially distributed pathological features and organizing them into a coherent diagnostic narrative. Although direct vision-to-text models can yield fluent reports, they obscure the contributions and failure modes of visual recognition, structured reasoning, and language generation. We propose a decomposed framework in which frozen Virchow2 tile embeddings are aggregated by multiple-instance learning (MIL) classification heads that answer organ-specific diagnostic questions. An organ-conditioned graph constrains the assembly of these answers into a structured reasoning chain, which a language model realizes as a pathology report. On the REG2026 held-out set of 2,028 slides, the proposed workflow achieved a chain-Jaccard score of 0.702. Performance fell to 0.420 without graph-based chain construction, 0.398 when the organ-specific graphs were replaced by a single organ-agnostic graph, and 0.371 when the language model constructed the chain freely from MIL predictions. Using the same report generator, graph-structured chains improved the report score from 0.330 to 0.495. On 350 external TCGA WSIs spanning the seven REG organs without fine-tuning, the expected organ graph was selected in 64.0% of cases and ranked among the top three in 86.6%. Providing the correct organ graph increased agreement with coarse TCGA primary-diagnosis labels from 61.8% to 92.6%, identifying organ routing as a main bottleneck under domain shift. Overall, organ-conditioned, graph-constrained chain assembly improves structured reasoning and report generation while enabling stage-specific error localization.

Figures & tables

Figure 1: Overview of the proposed framework. The WSI is densely divided into non-overlapping $224\times 224$ pixel candidate tiles. Tiles are retained when at least 50% of their pixels have a mean RGB intensity below 220, and up to 128 retained tissue tiles are uniformly and randomly sampled during training and inference. A frozen Virchow2 encoder transforms these tiles into an embedding bag used by an organ MIL classifier, which selects the organ-specific reasoning graph, and organ–question-specific MIL classifiers, which predict answers to successive graph questions. Graph-guided reasoning organizes these predictions into a structured question–answer chain, which Qwen2.5-1.5B-Instruct with LoRA converts into the final pathology report.

Organ	$n$	Full	Canon	Shuf	Majority	Heads	No organ	Free-form
Bladder	153	0.678	0.616	0.318	0.512	0.547	0.457	0.490
Breast	442	0.633	0.510	0.236	0.484	0.349	0.403	0.313
Cervix	162	0.737	0.733	0.390	0.640	0.562	0.359	0.448
Colon	399	0.690	0.552	0.337	0.426	0.425	0.378	0.370
Lung	188	0.773	0.734	0.374	0.632	0.510	0.391	0.410
Prostate	333	0.738	0.479	0.363	0.401	0.391	0.476	0.391

Table 1: Per-organ reasoning performance on the held-out set; all values report chain-Jaccard. Canon uses modal-answer transitions, whereas Shuf permutes answer-to-transition mappings within branching nodes.

Question category	Nodes	Occurrences	Accuracy	Error share
Grading / quantification	22	3,689	0.686	38.0%
Diagnosis / subtype	49	7,437	0.849	36.9%
Presence / extent	68	8,274	0.942	15.9%
Workflow / schema	31	7,252	0.963	8.8%
Rare findings	5	83	0.867	0.4%
Total	175	26,735	–	100%

Table 2: Post-hoc category-level analysis of question-node errors on the held-out set. Error share denotes the proportion of all incorrect question-level predictions attributable to each category.

Organ	$n$	Route	Expected organ in top 3	Weak concordance	Oracle-organ concordance
Bladder	50	11/50 (0.220)	29/50 (0.580)	8/42 (0.190)	30/42 (0.714)
Breast	50	42/50 (0.840)	49/50 (0.980)	37/49 (0.755)	44/49 (0.898)
Cervix	50	17/50 (0.340)	33/50 (0.660)	15/49 (0.306)	47/49 (0.959)
Colon	50	38/50 (0.760)	50/50 (1.000)	35/50 (0.700)	45/50 (0.900)
Lung	50	47/50 (0.940)	50/50 (1.000)	47/50 (0.940)	50/50 (1.000)
Prostate	50	46/50 (0.920)	48/50 (0.960)	45/50 (0.900)	49/50 (0.980)

Table 3: External evaluation on 350 TCGA diagnostic WSIs, with 50 slides per organ. Route denotes top-1 organ-routing accuracy and Top-3 indicates whether the expected organ was among the three highest-ranked predictions. Weak concordance measures agreement with coarse GDC primary-diagnosis labels; Oracle reports concordance when the expected organ graph is provided.

Medical Report Generation

Medical report generation turns medical images, clinical records, or conversations into structured, clinically useful reports.

See all 82 papers

TaxoMIL: Taxonomy-Constrained Learning for Hierarchical Whole Slide Image Analysis
Jun 30, 2026Chaeyeon Lee, Khang Nguyen Quoc, Jinsol Song +3Computational Pathology Hierarchical Classification
School of Electrical Engineering, Korea University, Seoul, Republic of Korea · Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul, Republic of Korea
Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation
May 1, 2026Suryakant Singh, Saarthak Kapse, Joel Saltz +1Computational Pathology Medical Report Generation
Stony Brook University
Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generation
May 29, 2026Zhiyuan Yang, Jiahao Cheng, Vincent Quoc-Huy Trinh +1Computational Pathology Medical Report Generation
Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, Canada · Axe Cancer, Centre de recherche du CHUM, Université de Montréal, Montreal, Canada · Institut de recherche en immunologie et cancérologie (IRIC), Université de Montréal

Jun 30, 2026J/K move · Enter open · S save

TaxoMIL: Taxonomy-Constrained Learning for Hierarchical Whole Slide Image Analysis

Chaeyeon Lee, Khang Nguyen Quoc, Jinsol Song, Yosep Chong+2

School of Electrical Engineering, Korea University, Seoul, Republic of Korea · Department of Hospital Pathology, The Catholic University of Korea College of Medicine, Seoul, Republic of Korea

Computational Pathology Hierarchical Classification Histopathology Image Classification Multiple Instance Learning Whole-Slide Image Analysis

Open paper AskPDF arXiv

Whole slide image (WSI) analysis is central to computational pathology, with multiple instance learning (MIL) emerging as the standard pipeline for slide-level diagnosis. However, conventional approaches formulate WSI diagnosis as a flat classification task over discrete labels, contradicting the inherently hierarchical, coarse-to-fine nature of clinical reasoning. Although recent hierarchical classifiers and vision-language models (VLMs) have sought to address this structural gap, they either fail to capture semantic continuity between related diagnoses or suffer from unconstrained text generation that produces taxonomic hallucinations and parent-child label violations. To address these limitations, we propose TaxoMIL, a taxonomy-constrained framework that reformulates WSI diagnosis as a multi-granularity text generation task. TaxoMIL utilizes a dual-head Transformer decoder to generate coarse- and fine-level diagnostic text, and introduces taxonomy-guided objectives that explicitly structure the label embedding space and strictly ground slide-level visual representations within the clinical taxonomy. Extensive experiments across three diverse WSI datasets demonstrate that TaxoMIL consistently outperforms state-of-the-art MIL classifiers and VLM-based generative methods, yielding accurate and hierarchy-aware diagnostic predictions. The code is released at https://github.com/QuIIL/TaxoMIL

Abstract

Medical Report Generation

Explore similar work

TaxoMIL: Taxonomy-Constrained Learning for Hierarchical Whole Slide Image Analysis

Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation

Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generation