KHiM-Mamba: Injecting Pathology Knowledge into Mamba via Hidden-State Modulation for Whole Slide Image Analysis

Qixiang Zhang; Yi Li; Tianqi Xiang; Haonan Wang; Mengjiao Wei; Bo Xu; Xiaomeng Li

eess.IVAug 14, 2026

KHiM-Mamba: Injecting Pathology Knowledge into Mamba via Hidden-State Modulation for Whole Slide Image Analysis

Authors: Qixiang Zhang, Yi Li, Tianqi Xiang, Haonan Wang, Mengjiao Wei, Bo Xu, Xiaomeng Li

Organizations: Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China · Center for Intelligent Oncology, Chongqing University Cancer Hospital and Chongqing University School of Medicine, and Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing, 400030, China

Abstract

Whole slide image analysis is commonly formulated as multiple instance learning (MIL), where instance features are contextually updated and aggregated into a slide representation, a process we term slide encoding dynamics. Recently, selective state-space models (SSM) have emerged as promising MIL architectures due to their long-sequence modeling capability and linear complexity. However, existing SSM-based MIL methods rely solely on visual features during MIL. Meanwhile, in large-scale WSIs, where sparse diagnostically decisive regions are surrounded by abundant irrelevant information, such purely vision-driven selective dynamics can misallocate state updates and readouts, causing the evolving SSM state to accumulate task-irrelevant evidence and dilute critical diagnostic cues over long scan trajectories. In this work, we propose the Knowledge-Aware Hidden-State Modulation architecture (KHiM-Mamba), which innovatively regulates Mamba's core selective state-space mechanism with explicit knowledge priors, steering slide encoding dynamics toward diagnostically meaningful evidence accumulation. Specifically, we redesign the original SSM layer to perform knowledge modulation operations during the evolution of hidden states, thereby guiding what visual evidence is accumulated and retrieved from the hidden state at each encoding step. Furthermore, we additionally introduce a local-adaptive vocabulary retrieval module that uses large language models to assign each patch fine-grained, tissue-specific semantic descriptions, enabling precise modulation across diverse tasks. Experiments on 11 public benchmarks across 4 tasks show that KHiM-Mamba consistently achieves state-of-the-art performance.

Figures & tables

Figure 1: (a) MIL pipeline: a WSI is split into patches, encoded as patch features, and processed by slide encoding dynamics ( i.e. , contextual interaction and aggregation). (b) Example of diagnostically irrelevant visual patterns in WSIs: under a certain stain, tumor slides appear redder while normal slides appear purpler. (c)(d) Comparison of SSM layer in prior Mamba-based MIL methods and ours.

Figure 2: (a) Existing VLM-MIL methods typically inject language priors either before slide encoding [ 10 , 21 ] to guide patch selection via image-text similarity, or after slide encoding as auxiliary supervision [ 24 ] . (b) KHiM-Mamba injects knowledge priors into the core slide encoding dynamics by modulating two key matrix: input-to-state $B_{t}$ , which governs how each patch’s information is written into the hidden state, and state-to-output $C_{t}$ , which governs how accumulated context is read out to enrich patch representations. (c) An illustrative example of language-guided contextual interaction across patches. In invasive ductal carcinoma, ‘‘conspicuous cytologic atypia and variation in nuclear size’’ guides the model to relate distinct patches with complementary nuclear morphologies, thereby grounding the slide-level representation in pathology-relevant evidence. (d) Histograms of patch response scores for background and tumor-related patches, with and without our Knowledge-aware Hidden-state Modulation (KHiM). KHiM markedly reduces the overlap between the two distributions, indicating stronger patch selectivity and clearer semantic separation; More quantitative and qualitative results to support this statement can be found in Tab. 5 and Fig. 5 respectively

Figure 3: Overview of KHiM-Mamba . Our framework consists of two main blocks: Local-adaptive vocabulary retrieval module (Sec. 3.3 ) provides precise language priors across diverse downstream tasks; and K nowledge-aware Hi dden-state M odulation (KHiM-Mamba, Sec. 3.2 ) which leverages language priors to modulate the Mamba-based slide encoding dynamics.


	BRACS-7 (n=547, $p<0.05$ )			BRACS-3 (n=547, $p<0.01$ )			TCGA-BRCA (n=952, $p<0.05$ )
Method	ACC. ( $\pm$ std)	AUC ( $\pm$ std)	RANK	ACC. ( $\pm$ std)	AUC ( $\pm$ std)	RANK	ACC. ( $\pm$ std)	AUC ( $\pm$ std)	RANK

Mean Pooling	${0.5378}_{\pm{0.0575}}$	${0.8341}_{\pm{0.0261}}$	$\mathbf{15.5}$	${0.7746}_{\pm{0.0442}}$	${0.8769}_{\pm{0.0248}}$	$\mathbf{15}$	${0.8970}_{\pm{0.0402}}$	${0.9204}_{\pm{0.0385}}$	$\mathbf{17}$
Max Pooling	${0.5315}_{\pm{0.0426}}$	${0.8084}_{\pm{0.0588}}$	$\mathbf{19.5}$	${0.7463}_{\pm{0.0424}}$	${0.8782}_{\pm{0.0203}}$	$\mathbf{19.5}$	${0.8812}_{\pm{0.0399}}$	${0.9042}_{\pm{0.0449}}$	$\mathbf{22}$
ABMIL [ 14 ] [ICML’21]	${0.5378}_{\pm{0.0354}}$	${0.8268}_{\pm 0.0280}$	$\mathbf{16.5}$	${0.7571}_{\pm{0.0665}}$	${0.8730}_{\pm{0.0410}}$	$\mathbf{19}$	${0.9028}_{\pm{0.0275}}$	${0.9354}_{\pm{0.0424}}$	$\mathbf{11}$

Table 1: Comparison Results of Cancer Sub-typing Task using CONCH [ 18 ] Features.

p

denotes paired t-tests between our KHiM-Mamba and the second-best counterpart on the corresponding dataset.


Method	BLCA (n=437)	BRCA (n=1023)	KIRC (n=498)	KIRP (n=261)	LUAD (n=455)	STAD (n=363)	MEAN	RANK

Mean-Pooling	${0.6758}_{\pm{0.0333}}$	${0.6274}_{\pm{0.0368}}$	${0.7287}_{\pm{0.0348}}$	${0.8179}_{\pm{0.0426}}$	${0.6334}_{\pm{0.0897}}$	${0.5917}_{\pm{0.0412}}$	$0.6792$	$\mathbf{19}$
Max-Pooling	${0.6417}_{\pm{0.0446}}$	${0.6095}_{\pm{0.1011}}$	${0.7316}_{\pm{0.0557}}$	${0.8164}_{\pm{0.0743}}$	${0.6245}_{\pm{0.0671}}$	${0.5554}_{\pm{0.0423}}$	0.6632	$\mathbf{21}$
ABMIL [ 14 ] [ICML’21]	${0.6604}_{\pm{0.0183}}$	${0.6621}_{\pm{0.0263}}$	${0.7185}_{\pm{0.0609}}$	${0.8201}_{\pm{0.0480}}$	${0.6433}_{\pm{0.0766}}$	${0.6053}_{\pm{0.0329}}$	$0.6850$	$\mathbf{17}$
TransMIL [ 23 ] [NIPS’21]	${0.6705}_{\pm{0.0244}}$	${0.6592}_{\pm{0.0593}}$	${0.7174}_{\pm{0.0708}}$	${0.8357}_{\pm{0.0477}}$	${0.6185}_{\pm{0.1102}}$	${0.6597}_{\pm{0.0374}}$	$0.6935$	$\mathbf{11}$

Table 2: Comparison Results of Survival Analysis Task on Six TCGA Subsets.

Figure 4: Kaplan–Meier survival curves of second-best method GMMamba [ 40 ] and our proposed KHiM-Mamba.


Method	CENTER 1	CENTER 2	CENTER 3	CENTER 4	CENTER 5	MEAN	RANK

Mean-Pooling	${0.6760}_{\pm{0.1237}}$	${0.3780}_{\pm{0.0183}}$	${0.7850}_{\pm{0.0838}}$	${0.6758}_{\pm{0.1063}}$	${0.6647}_{\pm{0.0301}}$	$0.6359$	$\mathbf{16}$
Max-Pooling	${0.9210}_{\pm{0.0164}}$	${0.7780}_{\pm{0.1454}}$	${0.9510}_{\pm{0.0094}}$	${0.9333}_{\pm{0.0112}}$	${0.8330}_{\pm{0.0200}}$	$0.8833$	$\mathbf{4}$

ABMIL [ 14 ] [ICML’21]	${0.9120}_{\pm{0.0248}}$	${0.7450}_{\pm{0.1344}}$	${0.9480}_{\pm{0.0098}}$	${0.9323}_{\pm{0.0197}}$	${0.8460}_{\pm{0.0918}}$	$0.8767$	$\mathbf{7}$

Table 3: Domain Generalization Performance on CAMELYON16 to CAMELYON17 [ 6 ] .

	8-SHOT		16-SHOT		MEAN
Method	ACC.	AUC	ACC.	AUC	ACC.	AUC
CLAM-SB [ 19 ]	${59.96}_{\pm{9.44}}$	${73.68}_{\pm{6.72}}$	${78.52}_{\pm{4.16}}$	${87.03}_{\pm{3.78}}$	$69.11$	$80.36$
CLAM-MB [ 19 ]	${66.63}_{\pm{2.48}}$	${79.69}_{\pm{6.34}}$	${73.76}_{\pm{3.88}}$	${88.01}_{\pm{2.99}}$	$70.20$	$83.85$
TransMIL [ 23 ]	${75.82}_{\pm{7.44}}$	${83.69}_{\pm{7.05}}$	${73.88}_{\pm{2.58}}$	${88.53}_{\pm{3.17}}$	$74.85$	$86.11$
ViLa-MIL [ 24 ]	${73.51}_{\pm{9.22}}$	${84.20}_{\pm{7.09}}$	${71.28}_{\pm{3.56}}$	${88.30}_{\pm{4.12}}$	$72.40$	$86.25$
CoOp [ 41 ]	${69.48}_{\pm{5.93}}$	${80.11}_{\pm{5.91}}$	${71.81}_{\pm{4.35}}$	${82.10}_{\pm{3.72}}$	$70.65$	$81.11$

Table 4: Few-shot performance on the TCGA-NSCLC [ 28 ] dataset.


Settings	$\mathbf{\Delta B}$	$\mathbf{\Delta C}$	BRACS-7	TCGA-BRCA	CAMELYON16

Original SSM	✗	✗	$0.8277_{\pm 0.0542}$	$0.9046_{\pm 0.0372}$	$0.8573_{\pm 0.0549}$
Version 1	✔	✗	$\uline{0.8333}_{\pm 0.0439}$	$\uline{0.9067}_{\pm 0.0538}$	${0.8745}_{\pm 0.0576}$
Version 2	✗	✔	$0.8325_{\pm 0.0643}$	$0.8976_{\pm 0.0422}$	$0.8620_{\pm 0.0628}$

Table 5: Ablations on our core Knowledge-aware Hidden-state Modulation (KHiM) module upon cancer sub-typing datasets, i.e. , BRACS-7, TCGA-BRCA, and CAMELYON16 tumor region localization (patch-level classification) task.


Vocabulary	BRACS-7	TCGA-BRCA	MEAN

Class-label	${0.8466}_{\pm 0.0451}$	${0.9411}_{\pm 0.0675}$	$0.8939$
VilaMIL [ 24 ] Prompt	${{0.8671}}_{\pm 0.0420}$	${0.9509}_{\pm 0.0483}$	$0.9090$
TOP [ 21 ] Prompt	${0.8644}_{\pm 0.0387}$	${{0.9521}}_{\pm 0.0576}$	$0.9083$

Table 6: Ablations on our Local-Adaptive Vocabulary Retrieval module upon cancer sub-typing datasets.

Figure 5: Visualization of patch attention map using patch features with and without Knowledge-aware Hidden-state Modulation (KHiM). Major tumor areas are circled in yellow.

		n=1,000		n=5,000
Method	Params.	Time	Thro.	Time	Thro.
Mean-Pooling	$525.8$ K	$0.25$ ms	$4.02$ K	$0.53$ ms	$1.90$ K
Max-Pooling	$525.8$ K	$0.19$ ms	$5.22$ K	$0.51$ ms	$1.97$ K
CLAM [ 19 ] [NAT.BME’20]	$790.8$ K	$0.82$ ms	$1.22$ K	$0.94$ ms	$1.07$ K
ABMIL [ 14 ] [ICML’21]	$591.6$ K	$0.71$ ms	$1.41$ K	$0.80$ ms	$1.25$ K
DSMIL [ 15 ] [CVPR’21]	$872.7$ K	$1.48$ ms	$0.68$ K	$1.95$ ms	$0.51$ K

Table 7: Efficacy evaluation with different numbers of patches per slide ( i.e. , 1,000 and 5,000). Parameter counts (Params.), inference time, and throughput (Thro.) are reported in thousands (K), milliseconds (ms), and thousands of patches per second (K patches/s), respectively.

Explore similar work

MoEMambaMIL: Structure-Aware Selective State Space Modeling for Whole-Slide Image Analysis
Mar 6, 2026Dongqing Xie, Yonghuang WuWhole-Slide Image Analysis Multiple Instance Learning
Tongji University · Fudan University
MambaBack: Bridging Local Features and Global Contexts in Whole Slide Image Analysis
Apr 17, 2026Sicheng Chen, Chad Wong, Tianyi Zhang +3Efficient Neural Network Inference Multiple Instance Learning
Nhu Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Electrical & Computer Engineering, National University of Singapore · PuzzleLogic Pte Ltd, Singapore 229594, Singapore
MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding
Mar 24, 2026Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi +6Whole-Slide Image Analysis Medical VLMs
Department of Computer Science, Khalifa University of Science and Technology, UAE. · Information Technology University, Pakistan. · KAU, KSA. +1

Mar 6, 2026J/K move · Enter open · S save

MoEMambaMIL: Structure-Aware Selective State Space Modeling for Whole-Slide Image Analysis

Dongqing Xie, Yonghuang Wu

Tongji University · Fudan University

Whole-Slide Image Analysis Multiple Instance Learning Visual SSMs

Open paper AskPDF arXiv

Whole-slide image (WSI) analysis is challenging due to the gigapixel scale of slides and their inherent hierarchical multi-resolution structure. Existing multiple instance learning (MIL) approaches often model WSIs as unordered collections of patches, which limits their ability to capture structured dependencies between global tissue organization and local cellular patterns. Although recent State Space Models (SSMs) enable efficient modeling of long sequences, how to structure WSI tokens to fully exploit their spatial hierarchy remains an open problem.We propose MoEMambaMIL, a structure-aware SSM framework for WSI analysis that integrates region-nested selective scanning with mixture-of-experts (MoE) modeling. Leveraging multi-resolution preprocessing, MoEMambaMIL organizes patch tokens into region-aware sequences that preserve spatial containment across resolutions. On top of this structured sequence, we decouple resolution-aware encoding and region-adaptive contextual modeling via a combination of static, resolution-specific experts and dynamic sparse experts with learned routing. This design enables efficient long-sequence modeling while promoting expert specialization across heterogeneous diagnostic patterns. Experiments demonstrate that MoEMambaMIL achieves the best performance across 9 downstream tasks.

Abstract

Explore similar work

MoEMambaMIL: Structure-Aware Selective State Space Modeling for Whole-Slide Image Analysis

MambaBack: Bridging Local Features and Global Contexts in Whole Slide Image Analysis

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding