cs.CVAug 24, 2026
SaveAn end-to-end-trained vision-language model for native-language prostate pathology report generation
Organizations: Institute of Pathology, Technical University of Munich, Munich, Germany · Munich Data Science Institute, Munich, Germany · Munich Center for Machine Learning, Munich, Germany · Section Computational Pathology Heidelberg, Institute of Pathology Heidelberg, University Hospital Heidelberg, Heidelberg, Germany ·
Abstract
Prostate cancer is among the most frequently diagnosed malignancies worldwide, and structured reporting of each biopsy core burdens pathologists. Existing tools frame this as classification, leaving pathologists to assemble coherent reports, while many slide-level vision-language models rely on English-centric encoders that transfer poorly to other clinical languages. We present a slide-level framework generating prostate biopsy reports that is language-independent by construction: tokenizer and model are trained from scratch, demonstrated here in German. To address paired-data scarcity, an automated pipeline uses a locally deployed large language model to split composite reports into core-specific image-text pairs, yielding 17,344 pairs from 2,402 historical cases without manual annotation. Evaluated for clinical attributes rather than linguistic similarity, the model achieves 96.2% F1 for malignancy detection and 65.2% for Gleason grading, competitive with an FDA-cleared classifier. Grading is further validated on three external cohorts with latent-space augmentation. Institutions can thus train native-language reporting models on their own archives.
Figures & tables
Figure 1: Generation of specimen-level image-text pairs from pathology reports using LLMs. For a given clinical case, the complete pathology report and its associated specimen labels are retrieved from the laboratory information system (LIS). A locally executed LLM is prompted in a few-shot setting to extract information for a specific specimen from a targeted section of the report (1.). This process yields WSI-text pairs, matching the extracted report text to its corresponding WSI. Text applicable to multiple specimens is assigned to all relevant WSIs. Finally, these extracted texts are queried for the desired attributes to establish specimen-level ground truth for evaluation (2.).
Figure 2: Overview of the preprocessing pipeline and model architecture. The WSI is segmented to isolate the tissue area and tessellated into patches at a resolution of pixels. A pathology foundation model then encodes all patches, yielding one embedding per patch. All patch embeddings for a WSI, along with the corresponding report text for the WSI, serve as input for the report generation model during training. The report model architecture comprises a projection MLP, a tokenizer, a slide encoder, a language encoder, and a multimodal decoder. The overall training objective combines a contrastive loss, computed between the outputs of the slide and language encoders, and a captioning loss applied to the report text predicted by the multimodal decoder.
Figure 3: Evaluation of predicted infiltration in the generated reports. Depending on the measurement protocol, the infiltration per prostate biopsy is stated as either a percentage or a physical length. Overall, the generated reports provide an accurate estimation of tumor infiltration. However, prediction deviation noticeably increases for infiltration value ranges that are underrepresented in data.
Figure 4: Train and test datasets and evaluation results on the internal Munich test cohort. a Datasets with sample size and distribution of the Gleason scores obtained during LLM extraction. b Classification metrics computed from the extracted outputs of the report generation model versus the two other models. Compared to the ABMIL model, the report generation model achieves a higher F1-score while simultaneously providing a draft report as output. c Confusion matrices for predicted malignancy and grading on the Munich test dataset (rows) for our trained report generation model, an ABMIL classification model and an FDA-cleared model for prostate biopsies. Each column has the results for one model type. Report Generation Model and ABMIL model have been trained on the same data splits. The exact training dataset of the FDA-cleared model is not known.
Figure 5: Visualization of tumor regions and the attention scores of the slide encoder on H&E-stained WSIs. Each row shows a randomly sampled malignant H&E-stained prostate biopsy WSI from the Munich test cohort. Left: tumor regions annotated in red by a board-certified pathologist. Right: attention scores of the slide encoder of the report generation model are highlighted according to the color scale. Regions receiving high attention largely correspond with the annotated tumor areas, indicating that the slide encoder focuses on diagnostically relevant tissue and making the generated reports more interpretable.
Figure 6: Examples of generated reports compared to original report texts on the Munich test cohort. Texts are originally in German and have been translated to English for readability. The texts produced by our model can serve directly as initial drafts for clinical pathology reports. Furthermore, the model’s outputs include details that enhance explainability, such as the specific proportions of Gleason grades. This additional context is particularly valuable for borderline cases. Difficult cases, where immunohistochemistry is needed for final diagnosis, may also be recognized.
Figure 7: UMAP visualization of slide encoder embeddings. A UMAP projection of the slide embeddings from the Munich test cohort reveals two primary clusters, corresponding to benign and malignant biopsies. Additionally, the malignant cluster demonstrates a continuous transition from lower to higher Gleason grades along the y-axis.
Figure 8: Evaluation of the report generation model on external test cohorts. a Grade distribution across the three external test cohorts. b Confusion matrices for the malignancy status extracted from the generated reports. c Confusion matrices for the predicted Gleason grades per biopsy. d Our augmentation strategy consistently improves the model’s ability to distinguish fine-grained differences between Gleason scores, as demonstrated by an increase in the macro F1-score.
| Malignancy | Grading | |||||||
|---|---|---|---|---|---|---|---|---|
| Cohort | Accuracy | Precision | Recall | F1-score | Accuracy | Precision | Recall | F1-score |
| Heidelberg | 93.0 | 93.7 | 90.1 | 91.9 | 67.9 | 47.1 | 46.6 | 41.3 |
| Karolinska | 93.7 | 97.2 | 93.0 | 95.0 | 65.1 | 56.4 | 51.1 | 50.1 |
| Radboud | 92.6 | 95.5 | 95.4 | 95.4 | 47.7 | 52.4 | 47.2 | 46.2 |
Table 1: Accuracy, Precision, Recall and F1-score for malignancy and grading of the report generation model on the external cohorts using the described augmentation strategy.
Figure S1: Overview of color variations and augmentation strategy. a H&E-stained WSIs exhibit substantial visual variability depending on the scanner and staining protocol. Sample images from each cohort used in this study highlight this variance. b For every tissue patch extracted for training, two additional color variants are generated: a lighter version with reduced contrast and saturation, and a darker version with increased contrast and saturation. This approach aims to simulate realistic staining variations encountered in clinical practice. c To further diversify the training data, a latent space augmentation strategy is applied. We train an auxiliary augmentation model on a subset of the training images to predict the embedding of an augmented patch, given the original non-augmented patch embedding and the target augmentation parameters.