cs.CVNov 15, 2025

From Classification to Cross-Modal Understanding: Leveraging Vision-Language Models for Fine-Grained Renal Pathology

Authors: Zhenhao Guo, Rachit Saluja, Tianyu Shi, Tianyuan Yao, Quan Liu, Junchao Zhu, Haibo Wang, Daniel Reisenbüchler, +8 more

Organizations: New York University, New York, NY 10012, USA · Cornell Tech, New York, NY 10044, USA · Vanderbilt University, Nashville, TN, 37235, USA · Carnegie Mellon University, Pittsburgh, PA 15213, USA · University of Regensburg, Regensburg, Bavaria 93053, DE · Weill Cornell Medicine, New York, NY 10065, USA · Northwell Health, New Hyde Park, NY 11040, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Fine-grained glomerular subtyping is essential for kidney biopsy interpretation, yet it presents a harder setting than the broad organ- or disease-level recognition tasks commonly used in vision-language model (VLM) evaluation. In this task, all categories arise within the same renal microanatomic unit, and clinically meaningful subtypes often differ by subtle, overlapping, and continuum-like morphological cues. Clinically valuable labels are also scarce, making fully supervised image-only paradigms difficult to translate to practice. Although VLMs are promising for data-efficient pathology learning, it remains unclear how their cross-modal representations should be adapted and evaluated for such within-structure discrimination under realistic few-shot constraints. To our knowledge, this study establishes the first fine-grained renal pathology benchmark for few-shot VLM adaptation and systematically examines how supervision level, model design, and adaptation strategy shape both classification performance and multimodal representation structure. Beyond accuracy, AUC, and F1, we assess image-text alignment, positive-negative similarity separation, embedding geometry, and calibration (ECE), asking when performance gains reflect more reliable cross-modal understanding. Across renal pathology cohorts, additional supervision consistently improves discrimination, but reliable adaptation is not explained by alignment alone; stronger performance is associated with larger class separation, more coherent embedding organization, and better awareness of confidence-related failure modes under domain variation. Within this retrospective two-cohort evaluation, these findings move assessment beyond coarse classification and accuracy alone and provide comparative evidence to inform model selection and future translational validation. Code: https://github.com/ddrrnn123/glo-vlms

Figures & tables

Figure 1: Pipeline overview. We train the image encoder and the text encoder in a few-shot setting, where only a small number of labels are provided for each glomerular subtype. For each shot, patches and their class descriptions are encoded, and the model is updated. As the number of shots increases, the performance improves.
Figure 2: (a) Model architectures. We compare contrastive VLMs with separate image and text encoders trained via pairwise cosine similarity, and a CoCa-style architecture with an additional fusion/decoder branch. (b) Domain-specific training data. We illustrate three sources of paired image-text data: natural images with generic captions, social media posts from Twitter with informal expert commentary, and biomedical figures from PubMed with pathology-style captions. (c) Fine-tuning strategies. We compare (i) Vanilla fine-tuning of the top layers, (ii) LoRA low-rank updates, (iii) trainable adapters between frozen layers, and (iv) a classifier head on a frozen backbone. VLM adaptation uses only the image–text contrastive loss, whereas Classifier tuning uses cross-entropy; no captioning loss is used.
Figure 3: Complementary roles of alignment and similarity gap. It shows the change in image and text feature vectors before and after fine-tuning. Alignment measures whether fine-tuning pulls an image embedding closer to the embedding of its own positive text description. Similarity gap is defined as mean positive class-prompt similarity (Alignment) minus mean negative class-prompt similarity, and therefore increases when matched pairs become closer and/or unmatched class prompts become less similar. Together, these two effects capture how fine-tuning reduces within-class image-text distance while enlarging across-class distance.
Figure 4: Receiver operating characteristic (ROC) curves across models and fine-tuning methods on Cornell dataset. We compare two axes, model comparison (fixed strategy) and fine-tuning method comparison (fixed backbone), by plotting true-positive rate (TPR) versus false-positive rate (FPR) across increasing shot counts.
Figure 5: Discrimination across subtypes on Cornell. This figure summarizes per-class discrimination performance across 10 runs. For each glomerular subtype, we report the distribution of AUC across VLM backbones, fine-tuning strategies, and shot settings. GSG and SSG denote Globally Sclerotic Glomerulus and Segmentally Sclerotic Glomerulus, respectively.
Figure 6: Alignment and similarity gap across models on Cornell. This figure shows shot-wise trajectories of Alignment and Similarity Gap for CLIP, PLIP, and CONCH under different fine-tuning strategies. Similarity Gap is defined as mean positive similarity (Alignment) minus mean negative similarity. CLIP and PLIP are demonstrated in both regular and enlarged panels to show their behavior.
Figure 7: UMAP/KDE visualization of embeddings across shots on Cornell. This figure shows text and image embeddings from the best CONCH run across shot settings. For Classifier tuning, text is not updated and only image-side embeddings are shown. Specifically, for Classifier tuning we visualize the penultimate hidden representation of the tuned MLP head (fed by fixed 512-D CONCH features), which changes as the head is updated. The companion Table 2 reports Intra-Class Distance and Silhouette Score.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: ROC curves across models and fine-tuning methods on Vanderbilt dataset. We compare backbones under fixed strategies and strategies under fixed backbones by plotting TPR versus FPR across increasing shot counts.
Figure S2: Discrimination across subtypes on Vanderbilt. This figure summarizes per-class discrimination performance across 10 runs. For each glomerular subtype, we report the distribution of AUC across VLM backbones, fine-tuning strategies, and shot settings.
Figure S3: Alignment and similarity gap across models on Vanderbilt. This figure shows shot-wise trajectories of Alignment and Similarity Gap for CLIP, PLIP, and CONCH under different fine-tuning strategies. Similarity Gap is defined as mean positive class-prompt similarity minus mean negative class-prompt similarity. CLIP and PLIP are shown in both full-scale and zoomed-in panels for readability.
Figure S4: UMAP/KDE visualization of embeddings across shots on Vanderbilt. This figure shows text and image embeddings from the best CONCH run across shot settings. For Classifier tuning, text is not updated and only image-side embeddings are shown. Specifically, for Classifier tuning we visualize the penultimate hidden representation of the tuned MLP head (fed by fixed 512-D CONCH features), which changes as the head is updated. The companion Table S3 reports Intra-Class Distance and Silhouette Score.

Explore similar work

CardsList
Oct 8, 2026

PathLang: A Language-Centered Benchmark for Vision-Language Models in Computational Pathology

Pathology vision-language models (VLMs) have shown strong visual perception ability, but their robustness in the language domain remains poorly characterized. Existing pathology VLM benchmarks largely rely on canonical closed-set prompts or perturb only generic templates, treating language as a fixed evaluation component rather than a variable axis of model behavior. In clinical practice, however, diagnostic language varies across reports, institutions, and candidate diagnoses. We introduce PathLang, a language-centered and clinically grounded zero-shot benchmark. PathLang holds the underlying slides, ground-truth labels, and image-text evaluation direction fixed while systematically varying only the diagnostic language, so that performance differences reflect how a diagnosis is phrased rather than what is imaged. The language variation follows how pathologists actually rephrase diagnoses (terminology, specificity, and reporting style), and all prompts and candidate pools are validated by six board-certified pathologists. PathLang covers four task families: (1) zero-shot classification with image-text alignment analysis, (2) cross-modal retrieval, (3) paraphrase robustness, including semantic-equivalence paraphrases, length and reporting-style variation, and prompt ensembling, and (4) open-vocabulary diagnosis retrieval over four candidate pools with distinct forms of semantic competition. Across nine VLMs and five public datasets spanning four organs, we find that performance is highly sensitive to clinically equivalent paraphrases, varies substantially across forms of semantic competition, and that image-text alignment quality does not necessarily translate into inter-class separability. We release the prompt corpus, candidate pools, pre-computed text embeddings, and evaluation code at https://anonymous.4open.science/r/PathLang.

Fanqi Cheng, Kuo Gong, Shangke Liu +15

New York University, New York, NY, USA · Columbia University, New York, NY, USA · Weill Cornell Medicine, New York, NY, USA +9

VLM EvaluationMedical VLMsMedical Image BenchmarksComputational PathologyCross-Modal Retrieval
AskPDF
Jul 30, 2026

PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?

Multimodal large language models (MLLMs) are increasingly used to analyze pathology images. However, dominant multimodal benchmarks in pathology mainly score final diagnostic answers, captions, or reports. These evaluations provide limited insight into whether a model understands the multiscale visual content needed for pathology reasoning and decision-making. We introduce PathVU, a vision-anchored benchmark for fine-grained and multiscale visual understanding in computational pathology. Built from 23 public pathology imaging datasets with human-supervised labels and spatial annotations, PathVU evaluates MLLM understanding in two fields of view: Region FOV for high-resolution local regions and Slide FOV for macro whole-slide views. By converting raw annotations into deterministic task targets, PathVU enables programmatic scoring of region localization, visual recognition, quantity estimation, spatial reasoning, and insufficient-context judgment. The benchmark contains 14 VQA-style tasks, 61,673 images, and 308,070 samples across 28 organs and 7,253,526 annotations. Evaluating 18 representative general-purpose, medical-domain, and pathology-oriented MLLMs, we observe substantial limitations even in advanced models on fine-grained visual tasks across multiscale pathology images. PathVU provides a reproducible basis for developing and evaluating pathology MLLMs with explicit multiscale visual understanding.

Zongyi Chen, Yu Liang, Jie Lin +1

National Institute for Data Science in Health and Medicine, Xiamen University · Institute of Artificial Intelligence Xiamen University, Xiamen, China · Department of Computer Science School of Informatics Xiamen University, Xiamen, China

Computational PathologyMedical VQAMultimodal Model EvaluationMedical Image Benchmarks
AskPDF
Jul 23, 2026

Do Pathology Vision-Language Models Truly See Pathology?

Pathology vision-language models (VLMs) have recently progressed rapidly and are commonly evaluated by answer accuracy on pathology VQA benchmarks. However, we dig into current evaluations and identify three overlooked issues: 1) Visual evidence is not always necessary. For instance, Gemini-3-Pro achieves 53.5% average accuracy across 5 VQA benchmarks without any visual input. 2) Domain training can improve accuracy without proportional gains in visual binding. Compared with Qwen2.5-VL-7B, Patho-R1-7B exhibits a 5.8-point lower multimodal gain and a 3.7-point lower attention IoU. 3) Entity-level attention is diffuse and weakly query-specific. On PathVG, attention maps remain highly correlated across different entity queries. These issues can lead to substantial misjudgments of pathology VLMs' actual multimodal capabilities. To this end, we present PathBind, a benchmark comprising 2,600 samples: PathBind-VQA with 1,500 questions across six dimensions, PathBind-PTA with 600 questions from a private pathology teaching atlas, and PathBind-Grounding with 500 expert-curated region-level samples. Each component undergoes task-specific automated filtering and expert review to reduce textual shortcuts and improve entity-region correspondence. We evaluate 18 representative VLMs on VQA samples of PathBind and five existing pathology VQA benchmarks, and further evaluate 10 VLMs on PathBind-Grounding and PathVG. Results show that current pathology VLMs still exhibit a substantial gap between answer-side performance and visual-semantic binding.

Chengyang Zhang, Wenchuan Zhang, Bo Li +10

College of Computer Science, Sichuan University · Department of Pathology and Institute of Clinical Pathology, West China Hospital, Sichuan University · Department of Computer Science, School of Computing, National University of Singapore +2

VLM EvaluationComputational PathologyMedical VQAVision-Language GroundingMedical VLMs
AskPDF