cs.CVAug 14, 2026

Spatial Message Passing in Language Space for Pathology Image Interpretation

Authors: Jing-Cheng Yang, Hao-Jung Wang, Jinhao Du, Yang Hu, Ming-shan Tsai, Jens Rittscher, Bin Li

Organizations: National Taiwan University, Taipei, Taiwan · University of Oxford, Oxford, United Kingdom · ZYTCA Limited, Oxford, United Kingdom

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal Large Language Models (MLLMs) can generate pathological descriptions from histological images, but gigapixel Whole Slide Images (WSIs) exceed their visual context limits. The standard tiling workaround makes WSIs tractable yet severs the tissue neighborhoods that define tumor-stroma interfaces and morphology. We introduce Spatial Language Message Passing (SLMP), a framework that performs spatial reasoning entirely in language space, human-readable by construction. SLMP represents a WSI region as a spatial text graph: tiles are nodes initialized with MLLM descriptions, and edges encode spatial adjacency. For each tile, an LLM refines its description by integrating language messages from adjacent tiles under a shared aggregation policy that, on the tile grid, acts as an adaptive local kernel operating on text rather than learned embeddings. This policy is an inspectable prompt that can be refined from model-observed tissue phenotypes via textual gradients, enabling automatic semantic optimization from local cellular context to broader tissue morphology without fine-tuning MLLM weights. On representative HER2 and CAMELYON16 regions, SLMP improves tile-level tumor description accuracy in settings spanning general-purpose and pathology-specialized backbones, with gains of +3.3 to +19.6 percentage points. Random-neighbor ablations confirm that these gains stem from spatial context rather than additional text alone, and inspecting the optimized policies reveals interpretable, tissue-specific decision rules. Besides, without any weight updates or fine-tuning the backbone MLLM, SLMP substantially improves general-purpose MLLMs and narrows its gap to pathology-specialized counterparts, offering a transparent and flexible mechanism for incorporating spatial reasoning into MLLM-based pathology analysis.

Figures & tables

Figure 1: Overview of the proposed framework. (Top) A WSI region of interest is tiled and each tile is described independently by an MLLM. (Bottom-left) Tiles and their descriptions form a spatial text graph; the LLM kernel aggregates the cardinal neighbors to update the center-tile description. (Bottom-right) TextGrad propagates textual gradients back through the pipeline to optimize the kernel prompt p\mathbf{p} .
Figure 2: A representative tile from CAMELYON16 where spatial context corrects an initially wrong classification. The center tile’s standalone description (left) lacks evidence of malignancy, yielding a confident non-tumor prediction. After message passing, the updated description integrates evidence from the east neighbor—where larger pleomorphic neoplastic cells are visible—correctly identifying a tumor-normal interface.
DatasetBase ModelBeforeAfterΔ\Delta
HER2GPT-4.10.6163 ±\pm 0.00490.7338 ±\pm 0.0027+11.75 pp
PathGEN0.6959 ±\pm 0.00490.7674 ±\pm 0.0033+7.15 pp
Patho-R10.4695 ±\pm 0.00050.5267 ±\pm 0.0005+5.72 pp
MedGemma-1.50.6738 ±\pm 0.00250.7071 ±\pm 0.0010+3.33 pp
CAMELYON16GPT-4.10.7422 ±\pm 0.00400.7994 ±\pm 0.0031+5.72 pp
PathGEN0.6511 ±\pm 0.00400.8186 ±\pm 0.0040+16.75 pp
Table 1: Tile-level tumor classification accuracy before and after SLMP. PathGEN is a 27B pathology-specialized model; Patho-R1 is 7B; MedGemma-1.5 is 4B; GPT-4.1 is a general-purpose model without pathology pretraining.
Figure 3: Tile-level prediction maps on 7×77{\times}7 ROIs. (Left) CAMELYON16. (Right) HER2. SLMP consolidates fragmented predictions into coherent fields.
DatasetMetricBaselineRandomSLMP
HER2Accuracy0.71530.6762±0.00980.6762\pm 0.00980.7727±0.00590.7727\pm 0.0059
F10.63610.6982±0.01260.6982\pm 0.01260.7678±0.00160.7678\pm 0.0016
CAMELYON16Accuracy0.73580.6250±0.00690.6250\pm 0.00690.7830±0.01530.7830\pm 0.0153
F10.71570.6831±0.00320.6831\pm 0.00320.8001±0.01240.8001\pm 0.0124
Table 2: Spatial neighbor ablation. SLMP uses true spatial neighbors; Random substitutes randomly sampled tiles from the same ROI. Mean ±\pm std over three runs.
DatasetMetricBaselineInitial PromptOptimized Prompt
HER2Accuracy0.72250.7416±0.00000.7416\pm 0.00000.7982±0.00490.7982\pm 0.0049
F10.64630.7075±0.00370.7075\pm 0.00370.7787±0.00460.7787\pm 0.0046
CAMELYON16Accuracy0.75470.6981±0.00330.6981\pm 0.00330.8121±0.00110.8121\pm 0.0011
F10.76150.7690±0.00210.7690\pm 0.00210.8257±0.00090.8257\pm 0.0009
Table 3: Prompt ablation. We compare message-passing results using the initial prompt pinitial\mathbf{p}_{\text{initial}} (before optimization) against the optimized prompt p∗\mathbf{p}^{*} (after convergence). Mean ±\pm std over three runs.
More in this topic

Whole-Slide Image Analysis

Whole-slide image analysis uses AI to interpret gigapixel pathology slides for diagnosis, prognosis, tissue mapping, and report generation.

See all 214 papers

Explore similar work

CardsList
Mar 24, 2026

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce MLLM-HWSI, a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight Cell-Cell Attention Fusion (CCAF) transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: GitHub.

Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi +6

Department of Computer Science, Khalifa University of Science and Technology, UAE. · Information Technology University, Pakistan. · KAU, KSA. +1

Whole-Slide Image AnalysisMedical VLMs
AskPDF
Sep 21, 2026

SLICEChat: Progressive In-Encoder Token Pruning for Whole-Slide Pathology Language Models

Whole-slide pathology images (WSIs) contain gigapixel-scale visual content, creating a major scalability challenge for slide-level multimodal large language models (MLLMs). Existing approaches process thousands of patch tokens and typically apply compression only after slide encoding, leaving multimodal attention computationally expensive. We introduce SLICEChat, a slide-level MLLM that integrates progressive token pruning within a hybrid Mamba--Transformer slide encoder. Mamba layers enable efficient long-range propagation, while Transformer layers preserve global interactions as the sequence is progressively shortened. Between stages, language-supervised, region-aware pruning removes spatially coherent low-utility regions under a controlled keep-rate schedule, producing compact slide representations before multimodal fusion. On SlideBench VQA, SLICEChat achieves 79.84% accuracy on TCGA and 59.09% on BCNB cohorts, outperforming prior slide-level pathology MLLMs, and achieves the highest overall WSI-Bench metrics. It also provides competitive memory usage and the inference latency among the evaluated models. These results demonstrate accurate and computationally efficient multimodal reasoning over gigapixel WSIs.

Ali Kerem Bozkurt, Baris Cem Bakay, Ibrahim Kulac +3

Kocı University, Department of Computer Engineering · Kocı University, Department of Pathology · Hacettepe University, Department of Artificial Intelligence and Data Engineering

Efficient Multimodal InferenceMultimodal Large Language ModelsMultimodal Token CompressionVisual Token PruningWhole-Slide Image Analysis
AskPDF
Dec 19, 2025

PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology

While Vision-Language Models (VLMs) have achieved notable progress in computational pathology (CPath), the gigapixel scale and spatial heterogeneity of Whole Slide Images (WSIs) continue to pose challenges for multimodal understanding. Existing alignment methods struggle to capture fine-grained correspondences between textual descriptions and visual cues across thousands of patches from a slide, compromising their performance on downstream tasks. In this paper, we propose PathFLIP (Pathology Fine-grained Language-Image Pretraining), a novel framework for holistic WSI interpretation. PathFLIP decomposes slide-level captions into region-level subcaptions and generates text-conditioned region embeddings to facilitate precise visual-language grounding. By harnessing Large Language Models (LLMs), PathFLIP can seamlessly follow diverse clinical instructions and adapt to varied diagnostic contexts. Furthermore, it exhibits versatile capabilities across multiple paradigms, efficiently handling slide-level classification and retrieval, fine-grained lesion localization, and instruction following. Extensive experiments demonstrate that PathFLIP outperforms existing large-scale pathological VLMs on four representative benchmarks while requiring significantly less training data, paving the way for fine-grained, instruction-aware WSI interpretation in clinical practice.

Fengchun Liu, Songhan Jiang, Linghan Cai +2

Harbin Institute of Technology, Shenzhen, School of Computer Science and Technology · National University of Singapore, Department of Electronic and Computer Engineering

Computational PathologyVision-Language GroundingMedical VLMsWhole-Slide Image Analysis
AskPDF