Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology

Daniel Shao; Dongmin Bang; Luca L. Weishaupt; Sophia J. Wagner; Nic G. Reitsam; Ming Y. Lu; Long P. Le; Richard J. Chen

cs.CVOct 12, 2026

Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology

Authors: Daniel Shao, Dongmin Bang, Luca L. Weishaupt, Sophia J. Wagner, Nic G. Reitsam, Ming Y. Lu, Long P. Le, Richard J. Chen,

Organizations: MIT · MGB AI Institute · Harvard University · University of Augsburg

Abstract

Contrastive vision-language (VL) pretraining in computational pathology has generally relied on curated image-caption datasets sourced from social media, educational videos, and research publications. These corpora are sparse relative to the diversity of tissue morphology and difficult to scale further, as pathologists do not routinely produce descriptive captions in clinical practice. We propose to instead leverage slide-level pathology reports which can be generated during routine diagnostic workflows in a scalable manner. A key challenge in this weakly-supervised setting is that pathology reports describe macroscopic, diagnostic findings while patch-level visual features describe microscopic findings. We introduce Locked Image Multiple Instance PreTraining (LIMIT), which addresses this gap by freezing a pretrained vision encoder and fine-tuning only the text encoder: report embeddings serve as cross-attention queries over bags of patch features, producing report-conditioned slide representations optimized via InfoNCE. Using LIMIT, we establish CONCH-Z, which aligns the text encoder of CONCH v1.5 to 335k clinical reports. Evaluated across 22 tasks spanning patch-level classification, tumor detection, and slide-level subtyping, CONCH-Z establishes state-of-the-art zero-shot performance over VL encoders trained on curated caption datasets, while simultaneously closing the gap with slide-level foundation models trained with substantially more parameters and compute. The code and model is available at https://github.com/mahmoodlab/LIMIT.

Figures & tables

Figure 1: Overview of LIMIT. A frozen vision encoder extracts patch embeddings from the WSI. The text encoder embeds each candidate report as a cross-attention query over the patch bag, producing a report-conditioned slide representation. Only the text encoder is updated via a difficulty-weighted InfoNCE objective.

Dataset	Organ	PLIP [ 12 ]	QuiltNet [ 13 ]	Musk [ 38 ]	Mi-Zero [ 22 ]	CONCH [ 21 ]	Conch-Z
C16 (2)	Lymph Node (2)	88.4 $\pm$ 0.1	91.7 $\pm$ 0.1	82.7 $\pm$ 0.2	59.9 $\pm$ 0.2	90.3 $\pm$ 0.2	96.7 $\pm$ 0.1
SPIDER-C	Skin (9)	38.0 $\pm$ 0.3	30.8 $\pm$ 0.3	62.1 $\pm$ 0.3	39.2 $\pm$ 0.3	50.8 $\pm$ 0.3	60.8 $\pm$ 0.3
	Colorectal (8)	43.5 $\pm$ 0.3	37.7 $\pm$ 0.4	43.5 $\pm$ 0.3	47.7 $\pm$ 0.4	44.0 $\pm$ 0.4	54.8 $\pm$ 0.4
	Breast (7)	32.8 $\pm$ 0.3	28.2 $\pm$ 0.5	57.8 $\pm$ 0.4	50.6 $\pm$ 0.4	57.9 $\pm$ 0.4	70.2 $\pm$ 0.3
	Thorax (7)	25.0 $\pm$ 0.4	23.6 $\pm$ 0.4	41.7 $\pm$ 0.4	39.2 $\pm$ 0.5	46.2 $\pm$ 0.4	49.4 $\pm$ 0.4
	Overall	34.8 $\pm$ 0.2	30.1 $\pm$ 0.2	51.3 $\pm$ 0.2	44.2 $\pm$ 0.2	49.7 $\pm$ 0.2	58.8 $\pm$ 0.2

Table 1: Patch-level zero-shot. Performance on patch-level tasks for C16 (AUC), SPIDER (balanced accuracy), and CATCH (balanced accuracy). Number of classes shown in parentheses. Best performance in bold ; second-best underlined . Standard deviation according to 1,000 bootstrap trials.

Figure 2: Zero-shot breast cancer metastasis detection on the CAMELYON16 dataset . a) 4-way patch classification performed according to maximum cosine similarity. Metastasis annotations were provided by Camelyon+, while the remaining annotations were generated by an in-house pathologist. b) Model score for the ground truth class. Red indicates the ground truth class received high predicted scores, while blue indicates low scores for the predicted class. Scores are normalized per-patch, and heatmap range is determined per-model. c) Area Under the Precision-Recall Curve for the key tissue categories for CONCHv1.5 (blue) and Conch-Z (red).

Dataset	Organ	Patch foundation model						Slide foundation model
Dataset	Organ	PLIP [ 12 ]	QuiltNet [ 13 ]	Musk [ 38 ]	Mi-Zero [ 22 ]	CONCH [ 21 ]	Conch-Z	TITAN [ 7 ]	PRISM [ 27 ]
HistAI	Breast	41.6 $\pm$ 3.6	32.3 $\pm$ 1.3	49.5 $\pm$ 3.6	44.7 $\pm$ 2.8	34.0 $\pm$ 1.4	47.6 $\pm$ 3.6	46.1 $\pm$ 3.5	49.7 $\pm$ 2.7
	Colorectum	28.4 $\pm$ 2.2	33.8 $\pm$ 3.6	36.2 $\pm$ 3.6	31.4 $\pm$ 2.9	32.9 $\pm$ 3.1	51.4 $\pm$ 4.2	33.2 $\pm$ 2.9	44.3 $\pm$ 3.8
	Skin	47.6 $\pm$ 3.1	33.3 $\pm$ 0.0	48.1 $\pm$ 3.3	46.0 $\pm$ 3.1	53.1 $\pm$ 3.7	65.0 $\pm$ 3.1	58.9 $\pm$ 3.0	56.7 $\pm$ 3.7
	Overall	39.2 $\pm$ 1.7	33.1 $\pm$ 1.3	44.6 $\pm$ 2.1	40.7 $\pm$ 1.7	40.0 $\pm$ 1.8	54.6 $\pm$ 2.2	46.1 $\pm$ 1.7	50.2 $\pm$ 2.0
TCGA	Breast	48.8 $\pm$ 0.6	7.2 $\pm$ 0.9	62.8 $\pm$ 1.6	49.6 $\pm$ 1.8	50.0 $\pm$ 0.6	89.2 $\pm$ 1.2	77.7 $\pm$ 1.7	81.9 $\pm$ 1.3

Table 2: Slide-level zero-shot. HistAI and TCGA-OT per-organ balanced accuracy. Best bold ; second-best underlined . Standard deviation reported according to 1,000 bootstrap trials.

Type	Model	Breast	Lung	Brain	Kidney	Thyroid	Bowel	Uterus	Other	Overall
Supervised	ABMIL [ 14 ]	91.0 $\pm$ 3.6	88.5 $\pm$ 2.6	26.5 $\pm$ 4.5	89.4 $\pm$ 3.2	57.3 $\pm$ 3.0	63.1 $\pm$ 10.1	81.8 $\pm$ 5.1	89.8 $\pm$ 1.0	73.4 $\pm$ 0.9
	TransMIL [ 30 ]	63.6 $\pm$ 5.5	90.8 $\pm$ 2.4	31.8 $\pm$ 4.8	87.7 $\pm$ 3.5	68.1 $\pm$ 3.4	58.6 $\pm$ 10.6	78.6 $\pm$ 5.5	88.8 $\pm$ 1.2	71.0 $\pm$ 1.1
	DSMIL [ 15 ]	80.9 $\pm$ 5.3	87.9 $\pm$ 3.0	32.8 $\pm$ 5.5	88.7 $\pm$ 3.5	48.8 $\pm$ 1.3	59.6 $\pm$ 10.7	84.6 $\pm$ 4.3	85.5 $\pm$ 2.7	71.1 $\pm$ 2.0
	CLAM [ 23 ]	88.9 $\pm$ 4.0	89.7 $\pm$ 2.6	36.8 $\pm$ 5.8	82.0 $\pm$ 4.0	50.2 $\pm$ 1.5	52.2 $\pm$ 10.4	84.1 $\pm$ 4.2	88.2 $\pm$ 1.3	71.5 $\pm$ 1.2
	WiKG [ 16 ]	88.5 $\pm$ 4.6	88.0 $\pm$ 2.9	36.6 $\pm$ 5.9	88.2 $\pm$ 3.7	47.5 $\pm$ 0.4	56.9 $\pm$ 10.4	73.1 $\pm$ 5.7	88.0 $\pm$ 1.2	70.9 $\pm$ 1.1
	$\text{CONCH}_{\text{lin}}$ [ 21 ]	83.9 $\pm$ 5.2	87.1 $\pm$ 3.1	30.1 $\pm$ 7.5	89.7 $\pm$ 3.3	80.4 $\pm$ 3.2	62.9 $\pm$ 9.2	79.2 $\pm$ 5.5	88.9 $\pm$ 1.1	75.3 $\pm$ 1.1

Table 3: TCGA-OT supervised baselines Supervised baselines and linear probes were trained on 8,226 samples and evaluated on 1,348 samples. Balanced accuracy is reported for all cases (

\pm

standard deviation). Overall represents the average balanced accuracy across all organs.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Role	Prompt content
System	You are a pathology text normalizer. Extract the diagnostic information from the text. Remove extraneous details not visible from histology, such as laterality, staining, immunohistochemistry, marker names, and positivity/negativity of genetic testing. Return the cleaned diagnosis text.
User	Clean this pathology report. Report: {report} Example: Input: Metastatic poorly-differentiated adenocarcinoma with giant cell features identified in the liver, positive for CK7, GATA3, and focal p40, with PD-L1 expression in tumor cells. Output: Metastatic poorly-differentiated adenocarcinoma with giant cell features

Appendix

Table A1: Prompt used to normalize pathology reports in the internal training dataset. The placeholder {report} denotes the raw pathology report text.

Label	Count
Dermis	263,813
Epidermis	23,579
Histiocytoma	50,770
Inflamm/Necrosis	32,723
Mast Cell Tumor	166,951
Melanoma	118,835

Appendix

Table A2: Patch Counts by Label for CATCH dataset. A patch label was given a label if >99% of the area within that patch consisted of that label. Patches without >99% area containing a single label were not included for analysis (conflicting labels).

Model	Architecture	Vision corpus	Reports/captions	Source institution(s)	Objective	Granularity
PLIP [ 12 ]	ViT-B/32 + Transformer	208K pairs	(paired)	Twitter (#path), LAION-5B	CLIP InfoNCE	Patch
QuiltNet [ 13 ]	ViT-B/32 + Transformer	1M pairs	(paired)	YouTube, Twitter, web, ARCH	CLIP InfoNCE	Patch
MI-Zero [ 22 ]	CTransPath + BioClinicalBERT	33K pairs	(paired)	ARCH + internal	InfoNCE + MIL pool	Patch / WSI
MUSK [ 38 ]	BEiT-3 (multiway transformer)	50M patches ( $\sim$ 33K WSIs) + 1M pairs	1B tokens (unpaired)	Stanford internal; Quilt-1M	Masked modeling + CLIP	Patch
CONCHv1.5 [ 21 , 7 ]	ViT (CoCa, UNI-init)	1.17M pairs (vision encoder only released)	(paired)	PMC OA, EDU	Contrastive + captioning	Patch
TITAN [ 7 ]	ViT slide encoder (iBOT + CoCa)	335K WSIs + 423K ROI-captions	183K reports	BWH, MGH; GTEx	SSL + contrastive + captioning	WSI

Appendix

Table A3: Summary of baseline vision-language pathology models. “Vision corpus” reports the scale of unique image data (image-text pairs, patches, or WSIs) used for vision-language alignment. “Reports/captions” reports the scale of paired textual supervision when distinct from the vision corpus. “Source institutions” lists the primary clinical or public sources of training data; public corpora are denoted in italics. Citations refer to the model release papers.

Figure A1: Zero-shot breast cancer metastasis in lymph nodes for various vision-language models . a) Patch-level predictions according to the class embedding with the highest cosine similarity to each patch embedding. Annotated regions according to an in-house pathologist are overlayed. b) Direct similarity scores between each class embedding and the patch embedding. More red indicates higher similarity. Scores are normalized within each model. Among the baseline patch models, QuiltNet exhibits the highest degree of overlap with the annotated regions.

Figure A2: Alignment of text embeddings with slide representations on TCGA-OT . T-SNE on top 10 components of canonical correlation analysis of cancer subtypes and slide-level features. Slide-level features were obtained by weighted average with query text embedding "cancer". Textual embeddings of Conch-Z exhibit higher overlap with image embeddings than CONCHv1.5 text embeddings. For instance, Adjusted rand index (ARI) indicates substantially better alignment for Conch-Z (ARI=71.5) compared to CONCHv1.5 (ARI=61.0).

Figure A3: Training data distribution by organ.

Category	Content
Synonyms	Tumor; Cancer; Malignancy; Carcinoma; Neoplasm
Templates	{term}
	an image of {term}
	{term} cells

Appendix

Table A4: Terms and Prompt Templates for SPIDER-C. Synonyms are used as interchangeable terms, and templates define text prompt structures where {term} is replaced by each synonym.

Computational Pathology

Computational pathology uses algorithms to analyze digitized tissue images for diagnosis, prognosis, biomarker discovery, and clinical decision support.

See all 277 papers

Lumen: Parameter-Efficient Alignment of Pretrained Vision and Language Encoders for Zero-Shot Computational Pathology
Sep 15, 2026Kiarash Tajbakhsh, Abdelrahman Faqieh, Michael Jopiti +9Medical VLMs Vision-Language Alignment
Institute of Tissue Medicine and Pathology, University of Bern, Switzerland.
Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generation
May 29, 2026Zhiyuan Yang, Jiahao Cheng, Vincent Quoc-Huy Trinh +1Computational Pathology Medical Report Generation
Department of Computer Science and Software Engineering (CSSE), Concordia University, Montreal, Canada · Axe Cancer, Centre de recherche du CHUM, Université de Montréal, Montreal, Canada · Institut de recherche en immunologie et cancérologie (IRIC), Université de Montréal
PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology
Dec 19, 2025Fengchun Liu, Songhan Jiang, Linghan Cai +2Computational Pathology Vision-Language Grounding
Harbin Institute of Technology, Shenzhen, School of Computer Science and Technology · National University of Singapore, Department of Electronic and Computer Engineering

Sep 15, 2026J/K move · Enter open · S save

Lumen: Parameter-Efficient Alignment of Pretrained Vision and Language Encoders for Zero-Shot Computational Pathology

Kiarash Tajbakhsh, Abdelrahman Faqieh, Michael Jopiti, Javier Garcia-Baroja+8

Institute of Tissue Medicine and Pathology, University of Bern, Switzerland.

Medical VLMs Vision-Language Alignment Histopathology Image Classification Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Pathology vision-language models are commonly built by pretraining or fine-tuning large encoders on paired image-caption data. We asked whether a pathology vision-language model can instead be assembled by parameter-efficient alignment of frozen unimodal foundation models, leaving their pretrained representations untouched. Here we present Lumen, which aligns frozen Virchow2 and BioMedBERT backbones using rank-4 adapters and projection heads, training only 0.40% of the total parameters on the public QUILT-1M corpus. Across nine public zero-shot patch benchmarks, Lumen achieved the highest mean chance-corrected balanced accuracy, 0.546 versus 0.461 for the strongest baseline (paired difference 0.086, 95% CI 0.042-0.136). On lymph-node metastasis detection, Lumen reached an AUROC of 0.964 (95% CI 0.956-0.971) on 4,214 held-out internal slides and 0.955 (95% CI 0.942-0.966) on 2,368 slides across nine external cohorts and six organs. At the internally calibrated threshold, it outperformed all vision-language baselines, with a balanced accuracy of 0.909 (95% CI 0.896-0.923) internally and 0.915 (95% CI 0.902-0.929) externally. Lumen performed competitively across the evaluations, with the exception of cross-modal retrieval, where it ranked third behind CONCH and PathGen-L/14. Fully fine-tuning both encoders gave Lumen no consistent benefit over low-rank adaptation, although it improved retrieval. Aligning frozen unimodal foundation models therefore yields strong and transferable performance at patch and slide level while training only a small fraction of the parameters.

Abstract

Computational Pathology

Explore similar work

Lumen: Parameter-Efficient Alignment of Pretrained Vision and Language Encoders for Zero-Shot Computational Pathology

Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generation

PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology