Reliable histopathology segmentation requires models that rely on pathology-relevant structures rather than spurious background correlations. In this work, we argue that improving architectures alone is insufficient if models continue to rely on spurious correlations rather than pathology-relevant features. We introduce a causality-inspired evaluation framework based on controlled interventions that explicitly probes model sensitivity to foreground tissue structure and background/domain variation. Building on this, we propose the Causal Reliance Score (CRS), a metric that quantifies the relative sensitivity of model predictions to foreground perturbation versus background/domain variation under controlled interventions. To support this analysis, we design augmentation strategies that act as implicit counterfactual interventions during training, encouraging invariance to nuisance factors while preserving morphology. We demonstrate how these interventions reveal behavioural differences between models that are not captured by standard metrics such as Dice or IoU. While we instantiate our framework using a self-supervised transformer (CAUSE-ViT), the primary contribution of this work is not a new architecture, but a principled intervention-based methodology and evaluation metric for auditing model behaviour in medical image segmentation. Our results suggest that evaluating model behaviour through counterfactual interventions is essential for developing reliable and clinically robust medical image segmentation systems.
Figures & tables
Figure 1: Conceptual abstraction and intervention protocol. Foreground tissue structure ( F ) and background context ( B ) jointly shape the observed image ( X ), which is mapped to the segmentation prediction ( Y ), i.e., F,B→X and X→Y . Panel (1a) shows the observational setting, whereas Panels (1b) and (1c) illustrate controlled perturbations of background and foreground structure. Resulting changes in prediction are used as an empirical probe of model sensitivity to pathology-relevant tissue structure and background/domain variation.
Figure 2: High-level design overview of CAUSE-ViT.
Figure 3: High-level design overview of SSL wrapper models.
Metrics
Supervised
CAUSE-ViT Generative SSL
CAUSE-ViT Contrastive SSL
SNOW Dataset Validation
BCE Loss (↓)
0.1354
0.1303
0.1360
Dice (↑)
0.8661
0.8752
0.8650
Mean IoU (↑)
0.7887
0.8046
0.7873
Table 1: Segmentation performance of CAUSE-ViT variants on the SNOW (InD) dataset. Best results in bold.
Table 2: Segmentation performance of CAUSE-ViT variants on CryoNuSeg under three evaluation conditions: raw OOD input, background-normalised reference, and matched foreground blur. This setup separates background/domain effects from foreground sensitivity. Best results are shown in bold.
Figure 4: Causal Intervention 1b: raw OOD input (background highlighted in amber). Columns show supervised, generative SSL, and contrastive SSL models, together with the input image, multi-scale responses (Scales 1–3), reference foreground structure, and predicted mask. The generative SSL model shows comparatively stronger foreground localisation under OOD shift.
Figure 5: Causal Intervention 1b: background-normalised reference condition (background perturbed while foreground tissue structure is preserved; highlighted in amber). Columns show supervised, generative SSL, and contrastive SSL models, together with the input image, multi-scale responses (Scales 1–3), predicted mask, and reference foreground mask. The generative SSL model shows the most stable response under background/domain correction.
Figure 6: Causal Intervention 1c: foreground-blurred condition (background-normalised reference with additional Gaussian blurring of foreground tissue structure). Columns show supervised, generative SSL, and contrastive SSL models, together with the input image, multi-scale responses (Scales 1–3), predicted mask, and reference foreground mask. The generative SSL model remains comparatively stable under the applied foreground perturbation.
Medical image segmentation plays a vital role in clinical decision-making, enabling precise localization of lesions and guiding interventions. Despite significant advances in segmentation accuracy, the black-box nature of most deep models has raised growing concerns about their trustworthiness in high-stakes medical scenarios. Current explanation techniques have primarily focused on classification tasks, leaving the segmentation domain relatively underexplored. We introduced an explanation model for segmentation task which employs the causal inference framework and backpropagates the average treatment effect (ATE) into a quantification metric to determine the influence of input regions, as well as network components, on target segmentation areas. Through comparison with recent segmentation explainability techniques on two representative medical imaging datasets, we demonstrated that our approach provides more faithful explanations than existing approaches. Furthermore, we carried out a systematic causal analysis of multiple foundational segmentation models using our method, which reveals significant heterogeneity in perceptual strategies across different models, and even between different inputs for the same model. Suggesting the potential of our method to provide notable insights for optimizing segmentation models. Our code can be found at https://github.com/lcmmai/PdCR.
Limai Jiang, Ruitao Xie, Bokai Yang +6
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Shenzhen University of Advanced Technology +3
Histopathological tissue segmentation is essential for computer-aided diagnosis, yet weakly supervised methods often suffer from noisy pseudo-labels generated by Class Activation Mapping (CAM). Existing CAM approaches tend to focus on staining-driven appearance cues rather than true causal tissue morphology, resulting in spurious localization and poor structural consistency. To address this issue, we propose C2RM-Seg, a two-stage framework that integrates causal pseudo-label refinement with structure-aware semantic enhancement. For classification, we introduce a Causal Counterfactual Reasoning Module (C2RM) that decomposes features into latent factors and performs counterfactual intervention via a learned causal structure matrix, suppressing confounding context and producing morphology-aligned CAMs. For segmentation, we design a Dual-Path Structural-Semantic Architecture that combines fine-grained structural features from ResNeSt with global semantic priors from a frozen DINOV3 foundation model. A cross-path gating mechanism adaptively regulates semantic injection using local structural cues to preserve boundary fidelity. To further mitigate residual pseudo-label noise, we propose an Uncertainty-Gated Margin (UGM) loss, which dynamically balances margin enforcement and confidence learning based on prediction uncertainty. Extensive experiments on two public histopathological tissue datasets show that C2RM-Seg achieves state-of-the-art performance.
Hualong Zhang, Siyang Feng, Zihan Huan +4
Guangxi Key Laboratory of Image and Graphic Intelligent Processing, Guilin University of Electronic Technology, Guilin 541004, China · International Joint Research Laboratory of Spatio-temporal Information and Intelligent Location Services, Guilin University of Electronic Technology, Guilin 541004, China · School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China
In recent years, foundation models such as CLIP, DINO,and CONCH have demonstrated remarkable domain generalization and unsupervised feature extraction capabilities across diverse imaging tasks. However, systematic and independent evaluations of these models for pixel-level semantic segmentation in histopathology remain scarce. In this study, we propose a robust benchmarking approach to asses 10 foundational models on four histopathological datasets covering both morphological tissue-region and cellular/nuclear segmentation tasks. Our method leverages attention maps of foundation models as pixel-wise features, which are then classified using a machine learning algorithm, XGBoost, enabling fast, interpretable, and model-agnostic evaluation without finetuning. We show that the vision language foundation model, CONCH performed the best across datasets when compared to vision-only foundation models, with PathDino as close second. Further analysis shows that models trained on distinct histopathology cohorts capture complementary morphological representations, and concatenating their features yields superior segmentation performance. Concatenating features from CONCH, PathDino and CellViT outperformed individual models across all the datasets by 7.95% (averaged across the datasets), suggesting that ensembles of foundation models can better generalize to diverse histopathological segmentation tasks.
Lavish Ramchandani, Aashay Tinaikar, Dev Kumar Das +2