cs.CVMar 19, 2026

Multimodal Model for Computational Pathology:Representation Learning and Image Compression

Authors: Peihang Wu, Zehong Chen, Lijian Xu

Organizations: Shenzhen University of Advanced Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Whole slide imaging (WSI) has transformed digital pathology by enabling computational analysis of gigapixel histopathology images. Recent foundation model advances have accelerated progress in computational pathology, facilitating joint reasoning across pathology images, clinical reports, and structured data. Despite this progress, challenges remain: the extreme resolution of WSIs creates computational hurdles for visual learning; limited expert annotations constrain supervised approaches; integrating multimodal information while preserving biological interpretability remains difficult; and the opacity of modeling ultra-long visual sequences hinders clinical transparency. This review comprehensively surveys recent advances in multimodal computational pathology. We systematically analyze four research directions: (1) self-supervised representation learning and structure-aware token compression for WSIs; (2) multimodal data generation and augmentation; (3) parameter-efficient adaptation and reasoning-enhanced few-shot learning; and (4) multi-agent collaborative reasoning for trustworthy diagnosis. We specifically examine how token compression enables cross-scale modeling and how multi-agent mechanisms simulate a pathologist's "Chain of Thought" across magnifications to achieve uncertainty-aware evidence fusion. Finally, we discuss open challenges and argue that future progress depends on unified multimodal frameworks integrating high-resolution visual data with clinical and biomedical knowledge to support interpretable and safe AI-assisted diagnosis.

Figures & tables

Figure 1: Timeline of milestone AI models in computational pathology (2021–2025). The chart illustrates the chronological progression of key literature reviewed in this survey. Models are positioned according to their publication or preprint release dates, capturing the rapid evolution from foundation models to recent advanced multimodal frameworks.
Figure 2: Schematic illustration of the data scale difference between pathological whole slide images and natural images. WSIs typically contain billions of pixels, imposing extremely high computational and storage demands. Moreover, due to tissue heterogeneity and uneven lesion distribution, diagnostically critical regions are often sparsely scattered within the vast background.
Figure 3: Schematic of a multi-task self-supervised learning framework for pathological images. The framework jointly optimizes three pretext tasks: (a) masked image modeling to reconstruct masked patches, enabling the model to learn tissue structure and context; (b) instance-level contrastive learning that pulls augmented views of the same image together while pushing different views apart, capturing discriminative morphological features; and (c) cross-resolution consistency learning to enforce feature invariance across magnifications. Combined pretraining yields a general-purpose pathology foundation model with strong diagnostic discriminability.
Figure 4: Schematic diagram of adaptive multi-resolution context modeling for whole slide images. The framework constructs an image pyramid across magnifications (e.g., 5×, 10×, 20×, 40×) as multi-scale inputs. A hierarchical multi-resolution visual encoder captures global tissue context at low resolutions and fine cellular details at high resolutions. Cross-scale attention modules enable bidirectional information flow, where global tokens guide local detail extraction and local features refine global representations, building a complete context pyramid from cellular to tissue level.
Figure 5: Schematic illustration of few-shot adaptation based on multimodal foundation models. Given a small support set of labeled pathology images and corresponding clinical descriptions, a pretrained multimodal model is adapted to new diagnostic tasks using parameter-efficient fine-tuning techniques (e.g., adapters, prompt tuning, or low-rank adaptation). The model leverages prior knowledge from large-scale pretraining to rapidly generalize to novel disease subtypes with only a few examples, significantly reducing the need for extensive expert annotations.
Figure 6: Schematic illustration of multi-agent collaborative framework for multimodal pathology image–text synthesis. Three specialized agents (medical interpreter, visual feature mapper, and prompt architect) collaboratively generate structured visual prompts from clinical descriptions. These prompts condition a diffusion model to synthesize high-fidelity pathology images, while a text generation module produces semantically consistent reports, ensuring cross-modal alignment. Synthesized samples undergo multi-discriminator assessment for image realism, textual accuracy, and image–text matching. Finally, collaborative training with curriculum learning and importance sampling integrates real and synthetic data to enhance model generalization, particularly for long-tail and rare disease scenarios.
More in this topic

Computational Pathology

Computational pathology uses algorithms to analyze digitized tissue images for diagnosis, prognosis, biomarker discovery, and clinical decision support.

See all 279 papers

Explore similar work

CardsList
Mar 24, 2026

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce MLLM-HWSI, a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight Cell-Cell Attention Fusion (CCAF) transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: GitHub.

Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi +6

Department of Computer Science, Khalifa University of Science and Technology, UAE. · Information Technology University, Pakistan. · KAU, KSA. +1

Whole-Slide Image AnalysisMedical VLMs
AskPDF
Aug 4, 2026

From Multi-Resolution Cells to Gigapixel Whole Slide Images Foundation Model for Computational Pathology

Vision Transformers (ViTs) and their hierarchical variants have achieved strong performance in Computational Pathology (CPath). However, most are pre-trained on single-resolution Whole Slide Images (WSIs), limiting their generalization across arbitrary resolutions. Gigapixel WSIs inherently contain diagnostic patterns at multiple scales, including cellular morphologies, tissue architectures, and global context, mirroring how expert pathologists examine WSIs. We introduce Multi-Resolution Pyramid Transformer (MRPT), a model that hierarchically aggregates multi-resolution information from cellular to tissue and WSI levels. MRPT employs a biologically meaningful Consecutive Cross-Resolution Attention (CCRA) mechanism to capture scale-independent interactions and enforces multi-resolution semantic consistency by aligning embeddings across resolutions, yielding robust and generalizable WSI representations. Pre-trained in a multi-resolution self-supervised manner on 624M patches, 2.4M regions, and 36K WSIs, MRPT learns rich coarse-to-fine histopathology features. Extensive experiments on 34 diverse datasets show that MRPT surpasses recent foundation models and Multimodal Large Language Models (MLLMs) in cancer subtype classification, tissue phenotyping, and Visual Question Answering (VQA) for WSI understanding.

Basit Alawode, Moshira Ali Abdalla, Dwarikanath Mahapatra +2

Khalifa University of Science and Technology, UAE · University of Western Australia, Australia

Representation LearningVision TransformerPathology Foundation ModelsHistopathology Image ClassificationWhole-Slide Image Analysis
AskPDF
May 1, 2026

Semantic Context-aware mOdality fUsion Transformer (SCOUT): A Context-Aware Multimodal Transformer for Concept-Grounded Pathology Report Generation

Whole-slide images (WSIs) present a fundamental challenge for computational pathology due to their extreme resolution, multi-scale heterogeneity, and the requirement for clinically reliable interpretation. Although recent pathology foundation models have enabled fluent report generation, they often lack clinical grounding, failing to accurately represent key diagnostic concepts and relationships observed by pathologists. This limitation arises from the difficulty of integrating heterogeneous visual evidence spanning fine-grained cellular patterns, slide-level tissue architecture, and high-level diagnostic concepts, while maintaining interpretability and clinical coherence. Here we present SCOUT: Semantic Context-aware mOdality fUsion Transformer, a context-aware concept-grounded multimodal framework for pathology report generation that enables progressive conditioning of image representations by global slide information and explicit diagnostic concepts. The method integrates local histological patterns, whole-slide context, and expert-curated semantic descriptors within a unified learning paradigm, allowing visual features to be dynamically refined throughout the encoding process. By combining depth-aware contextual modulation with adaptive multimodal fusion during text generation, the framework produces clinically coherent reports while preserving complementarity across representational scales. Using CONCH1.5 features, we evaluate SCOUT against WSI-Caption, HistGen, and BiGen on TCGA-BRCA, MICCAI REG, and HistAI. SCOUT achieves the best BLEU-1 to BLEU-4 and METEOR scores on all datasets, plus the best ROUGE-L on TCGA-BRCA and MICCAI REG. On TCGA-BRCA, it reaches 0.436/0.303/0.202/0.156 BLEU-1/2/3/4 and 0.204 METEOR; on REG 2025, it achieves 0.865/0.834/0.805/0.780 and 0.568. These results support progressive contextual conditioning for grounded pathology report generation.

Suryakant Singh, Saarthak Kapse, Joel Saltz +1

Stony Brook University

Computational PathologyMedical Report GenerationPathology Foundation ModelsMultimodal FusionWhole-Slide Image Analysis
AskPDF