An end-to-end-trained vision-language model for native-language prostate pathology report generation
Authors: Christian Grashei, Fabian Gülhan, Maximilian Legnar, Fabian Stögbauer, Cleo-Aron Weis, Carolin Mogler, Peter Schüffler
Organizations: Institute of Pathology, Technical University of Munich, Munich, Germany · Munich Data Science Institute, Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Section Computational Pathology Heidelberg, Institute of Pathology Heidelberg, University Hospital Heidelberg, Heidelberg, Germany · Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University, Heidelberg, Germany
Prostate cancer is among the most frequently diagnosed malignancies worldwide, and structured reporting of each biopsy core burdens pathologists. Existing tools frame this as classification, leaving pathologists to assemble coherent reports, while many slide-level vision-language models rely on English-centric encoders that transfer poorly to other clinical languages. We present a slide-level framework generating prostate biopsy reports that is language-independent by construction: tokenizer and model are trained from scratch, demonstrated here in German. To address paired-data scarcity, an automated pipeline uses a locally deployed large language model to split composite reports into core-specific image-text pairs, yielding 17,344 pairs from 2,402 historical cases without manual annotation. Evaluated for clinical attributes rather than linguistic similarity, the model achieves 96.2% F1 for malignancy detection and 65.2% for Gleason grading, competitive with an FDA-cleared classifier. Grading is further validated on three external cohorts with latent-space augmentation. Institutions can thus train native-language reporting models on their own archives.
Figures & tables
Figure 1: Generation of specimen-level image-text pairs from pathology reports using LLMs. For a given clinical case, the complete pathology report and its associated specimen labels are retrieved from the laboratory information system (LIS). A locally executed LLM is prompted in a few-shot setting to extract information for a specific specimen from a targeted section of the report (1.). This process yields WSI-text pairs, matching the extracted report text to its corresponding WSI. Text applicable to multiple specimens is assigned to all relevant WSIs. Finally, these extracted texts are queried for the desired attributes to establish specimen-level ground truth for evaluation (2.).
Figure 2: Overview of the preprocessing pipeline and model architecture. The WSI is segmented to isolate the tissue area and tessellated into patches at a resolution of 256×256 pixels. A pathology foundation model then encodes all patches, yielding one embedding per patch. All patch embeddings for a WSI, along with the corresponding report text for the WSI, serve as input for the report generation model during training. The report model architecture comprises a projection MLP, a tokenizer, a slide encoder, a language encoder, and a multimodal decoder. The overall training objective combines a contrastive loss, computed between the outputs of the slide and language encoders, and a captioning loss applied to the report text predicted by the multimodal decoder.
Figure 3: Evaluation of predicted infiltration in the generated reports. Depending on the measurement protocol, the infiltration per prostate biopsy is stated as either a percentage or a physical length. Overall, the generated reports provide an accurate estimation of tumor infiltration. However, prediction deviation noticeably increases for infiltration value ranges that are underrepresented in data.
Figure 4: Train and test datasets and evaluation results on the internal Munich test cohort. a Datasets with sample size and distribution of the Gleason scores obtained during LLM extraction. b Classification metrics computed from the extracted outputs of the report generation model versus the two other models. Compared to the ABMIL model, the report generation model achieves a higher F1-score while simultaneously providing a draft report as output. c Confusion matrices for predicted malignancy and grading on the Munich test dataset (rows) for our trained report generation model, an ABMIL classification model and an FDA-cleared model for prostate biopsies. Each column has the results for one model type. Report Generation Model and ABMIL model have been trained on the same data splits. The exact training dataset of the FDA-cleared model is not known.
Figure 5: Visualization of tumor regions and the attention scores of the slide encoder on H&E-stained WSIs. Each row shows a randomly sampled malignant H&E-stained prostate biopsy WSI from the Munich test cohort. Left: tumor regions annotated in red by a board-certified pathologist. Right: attention scores of the slide encoder of the report generation model are highlighted according to the color scale. Regions receiving high attention largely correspond with the annotated tumor areas, indicating that the slide encoder focuses on diagnostically relevant tissue and making the generated reports more interpretable.
Figure 6: Examples of generated reports compared to original report texts on the Munich test cohort. Texts are originally in German and have been translated to English for readability. The texts produced by our model can serve directly as initial drafts for clinical pathology reports. Furthermore, the model’s outputs include details that enhance explainability, such as the specific proportions of Gleason grades. This additional context is particularly valuable for borderline cases. Difficult cases, where immunohistochemistry is needed for final diagnosis, may also be recognized.
Figure 7: UMAP visualization of slide encoder embeddings. A UMAP projection of the slide embeddings from the Munich test cohort reveals two primary clusters, corresponding to benign and malignant biopsies. Additionally, the malignant cluster demonstrates a continuous transition from lower to higher Gleason grades along the y-axis.
Figure 8: Evaluation of the report generation model on external test cohorts. a Grade distribution across the three external test cohorts. b Confusion matrices for the malignancy status extracted from the generated reports. c Confusion matrices for the predicted Gleason grades per biopsy. d Our augmentation strategy consistently improves the model’s ability to distinguish fine-grained differences between Gleason scores, as demonstrated by an increase in the macro F1-score.
Malignancy
Grading
Cohort
Accuracy
Precision
Recall
F1-score
Accuracy
Precision
Recall
F1-score
Heidelberg
93.0
93.7
90.1
91.9
67.9
47.1
46.6
41.3
Karolinska
93.7
97.2
93.0
95.0
65.1
56.4
51.1
50.1
Radboud
92.6
95.5
95.4
95.4
47.7
52.4
47.2
46.2
Table 1: Accuracy, Precision, Recall and F1-score for malignancy and grading of the report generation model on the external cohorts using the described augmentation strategy.
Figure S1: Overview of color variations and augmentation strategy. a H&E-stained WSIs exhibit substantial visual variability depending on the scanner and staining protocol. Sample images from each cohort used in this study highlight this variance. b For every tissue patch extracted for training, two additional color variants are generated: a lighter version with reduced contrast and saturation, and a darker version with increased contrast and saturation. This approach aims to simulate realistic staining variations encountered in clinical practice. c To further diversify the training data, a latent space augmentation strategy is applied. We train an auxiliary augmentation model on a subset of the training images to predict the embedding of an augmented patch, given the original non-augmented patch embedding and the target augmentation parameters.
The allocation of visual attention by pathologists during cancer diagnosis is a highly selective process that critically shapes the information extracted from whole-slide images (WSIs). Human attention helps medical imaging tasks such as classification and segmentation, and becomes a strong semantic cue for identifying diagnostically informative regions for report generation. In this paper, we introduce human attention into the training of pathologist report generation models. To this end, we collected a multimodal human-attention dataset of 121 prostate WSIs annotated with pathologists' multi-scale viewport trajectories synchronized with the pathologists' verbal descriptions and cursor movements for five clinically relevant components (e.g., Gleason patterns). Using this dataset, we finetune two report generation models with an attention-alignment loss that regularizes the model attention over image patches to match the distribution of pathologist attention. We evaluate our approach on prostate cancer report generation and visual question answering using two models with different internal attention mechanisms (i.e., how image tokens are integrated into the language decoder). Experiments show average gains of 10.9% on NLP-based metrics and 19.3% in accuracy across five clinically relevant report components. Further, model attention maps extracted at inference time, with minimal computational overhead, align more closely with pathologist attention, providing stronger visual support for the generated reports by highlighting the regions that most influence the output.
Department of Computer Science, Stony Brook University, USA · Department of Biomedical Informatics, Stony Brook University, USA · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, IHU PRISM, Cancer Data Science Unit, France +4
Vision--language models (VLMs) promise to make computational pathology queryable and able to draft reports, but how much language adds beyond the vision backbone is unclear, because almost every system is evaluated by its developers on test sets of their own choosing. We grade published evidence by how it was produced and test four claims with controlled probes that reuse released checkpoints and public data, varying one input at a time. Report text adds prognostic signal: adding a frozen embedding of the stage-redacted report to a slide embedding raises the pooled C-index by 0.019 for progression-free interval and by 0.016 for overall survival across 1{,}633 TCGA patients, and restoring the stage statement adds nothing. Prompts add no measurable accuracy at inference: with features and pooling fixed, zero-shot classification is statistically indistinguishable from a supervised linear probe for lung and renal cancer subtyping. Slide-level question answering is barely grounded: SlideChat answers 74.82% of SlideBench-VQA(TCGA) items correctly with the slide and 74.41% without it, whereas its language backbone alone reaches 43.52%. Paraphrasing leaves accuracy unchanged but flips 8--23% of answers. Self-reported gains from agentic navigation and few-shot language priors cannot yet be attributed to language. Language thus adds real but narrow value, and we propose minimum controls that would expose these failure modes in future evaluations.
Wenhao Zhang, Zhongliang Zhou, Yi Ping +2
School of Data Science, University of Virginia, Charlottesville, VA 22903, USA · Merck & Co., Inc., Rahway, NJ 07065, USA
The rapid advancement of vision-language models (VLMs) has accelerated progress in computational pathology; however, whole-slide image (WSI)-based pathology report generation remains limited by the scarcity of large-scale WSI--report datasets and the complexity of mapping spatially distributed visual patterns to structured clinical text. To address this, we introduce a clinically curated Pan-Asia WSI--report dataset of approximately 10,500 pairs from five institutions and establish the REG 2025 benchmark through a MICCAI challenge for systematic evaluation of multimodal models. We analyze submitted methods spanning pretrained VLMs, multiple-instance learning frameworks, hierarchical expert models, retrieval-augmented generation, and cross-modal Transformers. Rather than indicating that VLM use alone was sufficient for superior performance, the results suggest that top-performing methods benefited from structured report representations, hierarchical diagnostic decomposition, and effective multimodal grounding. We identify key limitations, including instability in quantitative attribute estimation (e.g., numeric hallucination) and a tendency toward diagnostic overspecification, with some errors resembling known diagnostic pitfalls in routine pathology. These findings establish REG 2025 as a benchmark for evaluating WSI-based structured report generation and vision-language understanding in computational pathology, providing insights for the design of clinically grounded multimodal pathology models.