Interactive Whole Slide Images for RL-based Tumour Segmentation

Mohamad Mohamad; Francesco Ponzio; Maxime Gassier; Nicolas Pote; Xavier Descombes

cs.CVAug 17, 2026

Interactive Whole Slide Images for RL-based Tumour Segmentation

Authors: Mohamad Mohamad, Francesco Ponzio, Maxime Gassier, Nicolas Pote, Xavier Descombes

Organizations: Universit´e Cˆote d’Azur, Inria, CNRS, I3S, INSERM, IBV, Sophia Antipolis, France · Department of Control and Computer Engineering, Politecnico di Torino, Turin, Italy · Department of Pathology, Bichat Hospital, Assistance Publique–Hˆopitaux de Paris, Paris, France

Abstract

Whole-slide image (WSI) analysis remains computationally challenging due to the extremely large spatial resolution of slides and the sparse distribution of tumour regions. We propose an end-to-end reinforcement learning framework for sequential tumour segmentation directly on WSIs. Instead of treating the slide as a predefined collection of candidate patches, we formulate the WSI itself as a hierarchical multi-resolution environment through which an agent navigates using movement, zooming, and tumour selection actions. The agent jointly processes local observations and a global thumbnail representation within an actor-critic architecture trained using proximal policy optimization (PPO). Experiments on pulmonary adenocarcinoma WSIs demonstrate the feasibility of direct sequential tumour segmentation on full slides, achieving comparable coarse segmentation quality relative to patch-based approaches operating at similar magnification levels, while reducing inference time to a few seconds per slide. We further analyse the impact of environment design and action-space granularity. Our results suggest that modelling WSIs as interactive environments provides a promising direction for RL-based computational pathology

Figures & tables

Figure 1: Agent framework for sequential WSI tumour segmentation. A) WSIs are formulated as interactive multi-resolution environments. At each episode reset, a WSI and its tumour mask are sampled. The agent sequentially navigates the image pyramid through movement, zooming, and selection actions, progressively constructing an accumulated tumour segmentation mask (yellow). B) Actor–critic architecture jointly processing global and local observations through a shared backbone encoder followed by feature fusion and policy/value heads. C) State representation composed of a global overview and the current local patch. D) Reward computation based on improvements in overlap between accumulated selections and the ground-truth tumour mask. E) Action space composed of navigation and tumour-selection actions.

Figure 2: Reward shaping analysis. Left: Normalised reward landscapes for different scaling parameters $S$ as a function of consecutive IoU values $(\mathrm{IoU}_{t-1},\mathrm{IoU}_{t})$ . Increasing $S$ shifts the reward towards improvements obtained at higher IoU regimes, encouraging late-stage refinement. Right: Training dynamics for different values of $S$ . Lower values favour broader region selection and higher recall early during training, whereas larger values promote more conservative refinement with improved precision.

Figure 3: Training stability and generalisation analysis. Left: MDR stabilises PPO training and substantially improves convergence and segmentation performance compared to standard PPO. Middle: Effect of dropout regularisation across different agent and backbone configurations. Dropout improves evaluation performance, particularly for higher-capacity agents, although a noticeable train–test gap remains. Right: Extending the training cohort substantially reduces the generalisation gap and improves evaluation performance. Training steps are doubled for agents trained on the extended cohort.

$T$	Agent-7			Agent-15
$T$	Dice $\uparrow$	NSD@512 $\uparrow$	NSD@2048 $\uparrow$	Dice $\uparrow$	NSD@512 $\uparrow$	NSD@2048 $\uparrow$
20	0.842	0.147	0.484	0.873	0.179	0.555
40	0.887	0.188	0.585	0.890	0.206	0.609
60	0.893	0.201	0.612	0.887	0.209	0.612
80	0.894	0.206	0.617	0.888	0.209	0.613

Table 1: Effect of episode horizon

T

and action-space granularity on segmentation performance. Performance improves from short to moderate horizons before saturating, while Agent-15 largely benefits the low-budget setting.

Patch-based methods
Method	Dice $\uparrow$	Recall $\uparrow$	Precision $\uparrow$	NSD@512 $\uparrow$	NSD@2048 $\uparrow$
UNI-1024	0.88 $\pm$ .004	0.85 $\pm$ .030	0.91 $\pm$ .030	0.76 $\pm$ .041	0.88 $\pm$ .046
UNI-2048	0.84 $\pm$ .025	0.96 $\pm$ .007	0.75 $\pm$ .044	0.59 $\pm$ .027	0.74 $\pm$ .032
UNI-4096	0.81 $\pm$ .034	0.93 $\pm$ .060	0.74 $\pm$ .100	0.61 $\pm$ .009	0.74 $\pm$ .066
UNI-ENS	0.90 $\pm$ .015	0.93 $\pm$ .006	0.87 $\pm$ .030	0.91 $\pm$ .036	0.91 $\pm$ .036
UNI-HR	0.89 $\pm$ .008	0.92 $\pm$ .011	0.88 $\pm$ .024	0.71 $\pm$ .030	0.85 $\pm$ .035

Table 2: Segmentation performance on the evaluation set. ± denotes standard deviation across three seeds.

Patch-based methods
Method	Mean Time (s) $\downarrow$	Worst-case (s) $\downarrow$	Best-case (s) $\downarrow$
UNI-1024	229.1 $\pm$ 164.4	829.1 $\pm$ 266.6	79.6 $\pm$ 1.7
UNI-2048	187.9 $\pm$ 118.1	538.3 $\pm$ 10.4	64.1 $\pm$ 2.9
UNI-4096	36.8 $\pm$ 40.2	138.7 $\pm$ 3.0	10.0 $\pm$ 0.4
UNI-HR	130 $\pm$ 110.2	433 $\pm$ 14.7	36.6 $\pm$ 1.6
Sequential methods

Table 3: Inference time per slide on the evaluation set. The proposed RL agent substantially improves efficiency. ± is across slides and seeds (Mean) or across seeds (Best/Worst-case, computed per-seed).

Figure 4: Segmentation results on evaluation slides. Predictions from Agent-7 ResNet18 (left) and Agent-7 ResNet50 (right) are shown for tumours of varying size and morphology. Red contours denote the ground-truth tumour boundaries, while the blue contours correspond to the final agent segmentation obtained from the accumulated selections at the end of the episode.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Navigation strategies learned under different reward scaling values $S\in\{1,4,16,256\}$ together with the constrained human strategy shown on a training-cohort slides. Episode progression is shown from left to right, with the final timestep corresponding to the end of the episode.

Figure 6: Additional navigation episodes for agents trained with $S\in\{1,256\}$ .

Figure 7: Effect of episode horizon $T$ and movement granularity on training dynamics and learned segmentations. Top: training IoU and reward curves for Agent-7 and Agent-15. Bottom: final segmentations masks obtained under different interaction budgets on training slides.

Hyperparameter	extended-cohort	original-cohort
Episode horizon ( $T$ )	80	80
Observation size ( $p$ )	112	112
Parallel environments	16	16
Rollout steps per environment	1000	500
Discount factor $\gamma$	0.99	0.99
GAE parameter $\lambda$	0.95	0.95

Appendix

Table 4: Training and optimisation hyperparameters.

Medical Image Segmentation

Medical image segmentation delineates organs, lesions, and other structures in scans to support measurement, diagnosis, treatment planning, and research.

See all 788 papers

SlideMix: Enhancing Whole Slide Image Analysis via Multimodal Shuffling
Aug 31, 2026Chad Wong, Sicheng Chen, Tianyi Zhang +4Data Augmentation Histopathology Image Classification
1Nhu Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Electrical & Computer Engineering, National University of Singapore · 3PuzzleLogic Pte Ltd, Singapore 229594, Singapore
Test-Time Instance Selection for Improved Whole Slide Image Analysis
Aug 14, 2026Quoc Anh Nguyen, Sunhong Park, Jin Tae KwakWhole-Slide Image Analysis Histopathology Image Classification
School of Electrical Engineering, Korea University, Seoul 02841, Korea
Thinking in Scales: Accelerating Gigapixel Pathology Image Analysis via Adaptive Continuous Reasoning
May 19, 2026Jiusong Ge, Yingkang Zhan, Wenjie Zhao +13Computational Pathology Adaptive Inference
School of Computer Science and Technology, Xi’an Jiao-tong University, Xi’an, China · Department of Transmedia Art, Xi’an Academy of Fine Arts, Xi’an, China · Language Technology Lab, University of Cambridge, Cambridge, U.K. +3

Aug 31, 2026J/K move · Enter open · S save

SlideMix: Enhancing Whole Slide Image Analysis via Multimodal Shuffling

Chad Wong, Sicheng Chen, Tianyi Zhang, Enhui Chai+3

1Nhu Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Electrical & Computer Engineering, National University of Singapore · 3PuzzleLogic Pte Ltd, Singapore 229594, Singapore

Data Augmentation Histopathology Image Classification Multiple Instance Learning Whole-Slide Image Analysis

Open paper AskPDF arXiv

Histopathological whole slide images (WSIs) are central to cancer diagnosis, but their gigapixel scale, tissue heterogeneity, weak slide-level supervision, sparse diagnostic regions, and multi-scale evidence make robust automated analysis challenging. Multiple instance learning (MIL) is widely used to aggregate tile-level features into slide-level predictions, yet existing augmentation strategies often perturb tissue regions without preserving diagnostic relevance, slide context, or cross-scale structure. We propose SlideMix, a model-agnostic multimodal augmentation framework for MIL-based WSI analysis. SlideMix uses a retrieval-augmented vision-language model (VLM)-based Visual-Language Adaptive Region selector to identify diagnostically relevant regions and reduce weak-label noise. It then performs In-place Tile Shuffling within meaningful tissue regions to mix feature embeddings while preserving slide-level context. A VLM-based soft-labeling module supervises mixed samples, while a multi-factor, loss-driven online Curriculum-Learning Feedback scheme adaptively controls shuffle granularity, feature similarity, and shuffle ratio to promote cross-scale representation learning. Across 11 WSI datasets comprising 20,523 slides, 8 diagnostic tasks, and 10 WSI backbones, SlideMix improves accuracy and generalization in most settings and compares favorably with established augmentation baselines, providing a simple plug-and-play approach for more robust and scalable digital pathology models. Source code: https://github.com/Xia-Research-Lab/SlideMix

Abstract

Medical Image Segmentation

Explore similar work

SlideMix: Enhancing Whole Slide Image Analysis via Multimodal Shuffling

Test-Time Instance Selection for Improved Whole Slide Image Analysis

Thinking in Scales: Accelerating Gigapixel Pathology Image Analysis via Adaptive Continuous Reasoning