Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift

Akshit Achara; Yovin Yahathugoda; Nick Byrne; Michela Antonelli; Esther Puyol Anton; Alexander Hammers; Andrew P. King

cs.CVApr 14, 2026

Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift

Authors: Akshit Achara, Yovin Yahathugoda, Nick Byrne, Michela Antonelli, Esther Puyol Anton, Alexander Hammers, Andrew P. King

Organizations: School of Biomedical Engineering & Imaging Sciences, King’s College London, UK

Abstract

The robustness of machine learning models can be compromised by spurious correlations between non-causal features in the input data and target labels. A common way to test for such correlations is to train on data where the label is strongly tied to some non-causal cue, then evaluate on examples where that tie no longer holds. This idea is well established for classification tasks, but for semantic segmentation the specific failure modes are not well understood. We show that a model may achieve reasonable overlap while assigning the wrong semantic label, swapping one plausible foreground class for another, even when object boundaries are largely correct. We focus on this semantic label-flip behaviour and quantify it with a simple diagnostic (Flip) that counts how often ground truth foreground pixels are assigned the wrong foreground identity while remaining predicted as foreground. In a setting where category and scene are correlated during training, increasing the correlation consistently widens the gap between common and rare test conditions and increases these within-object label swaps on counterfactual groups. Overall, our results motivate assessing segmentation robustness under distribution shift beyond overlap by decomposing foreground errors into correct pixels, flipped-identity pixels, and missed-to-background pixels. We also propose an entropy-based, ground truth label-free `flip-risk' score, which is computed from foreground identity uncertainty, and show that it can flag flip-prone cases at inference time. Code is available at https://github.com/acharaakshit/label-flips.

Figures & tables

Figure 1: Subgroup class IoU disparities under $\rho{=}0.95$ . On the balanced evaluation split, each bar corresponds to a specific foreground class and reports the IoU gap between the aligned and counterfactual subgroup for that class. Here, $g0-g1$ denotes landbird+land - landbird+water for Waterbirds-Seg and cat+indoor - cat+outdoor for COCO-CD , while $g3-g2$ denotes waterbird+water - waterbird+land for Waterbirds-Seg and dog+outdoor - dog+indoor for COCO-CD .

Model	Loss	$\rho=0.5$			$\rho=0.95$
		FG-Corr	FG-Flip	FG-Miss	FG-Corr	FG-Flip	FG-Miss
R50	CE	0.916	0.057	0.026	0.834	0.126	0.040
	DCE	0.918	0.053	0.029	0.850	0.117	0.033
	GD	0.910	0.054	0.037	0.846	0.117	0.037
	CM	0.915	0.051	0.034	0.838	0.117	0.045
MiT-B2	CE	0.915	0.061	0.024	0.871	0.100	0.029

Table 1: Waterbirds-seg foreground error decomposition. Rates are computed over ground truth foreground pixels and satisfy FG-Corr + FG-Flip + FG-Miss = 1. Notably, increasing

\rho

increases FG-Flip more than FG-Miss, indicating semantic instability despite preserved foreground extent.

Figure 2: Qualitative flips. Extent is preserved while identity swaps under a counterfactual context.

Figure 3: Flip-risk deciles ( $\rho=0.95$ ). Images are binned by predicted-foreground entropy risk; Flip concentrates in the highest-risk tail. Titles report top-decile flip share. R50 is short for ResNet50 here.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Landbird+Land	Landbird+Water	Waterbird+Land	Waterbird+Water
Train ( $\rho{=}0.5$ )	615	615	615	615
Train ( $\rho{=}0.95$ )	1167	62	62	1167
Val	74	74	74	74
Test	642	642	642	642

Appendix

Table 2: Waterbirds-seg split sizes. Group counts for each split and correlation regime.

Split	Cat+Indoor	Cat+Outdoor	Dog+Indoor	Dog+Outdoor
Train ( $\rho{=}0.5$ )	417	417	417	417
Train ( $\rho{=}0.95$ )	792	42	42	792
Val	50	50	50	50
Test	125	125	125	125

Appendix

Table 3: COCO-CD split sizes. Group counts per split and correlation regime.

Subset	Indoor (%)	Outdoor (%)	Unknown/Excluded (%)
All retained	38.13	46.40	15.47
Cats only	71.10	4.80	24.10
Dogs only	5.16	88.01	6.83

Appendix

Table 4: COCO-CD context coverage. Proportion of retained (exclusive) images assigned to each context label by the evidence procedure.

Subgroup	$N$	Ambig. (%)	Correct / Non-ambig	Agree. (%) $\uparrow$
Cat+Indoor	50	2.0	48 / 49	98.0
Cat+Outdoor	50	2.0	48 / 49	98.0
Dog+Indoor	50	0.0	50 / 50	100.0
Dog+Outdoor	50	0.0	50 / 50	100.0
Overall	200	1.0	196 / 198	99.0

Appendix

Table 5: COCO-CD context audit. Manual validation of indoor/outdoor labels on 200 validation images (50 per subgroup). Agreement excludes ambiguous scenes.

Figure 4: Balanced training ( $\rho{=}0.5$ ): subgroup results. Disparities are less pronounced than under $\rho{=}0.95$ .

Figure 5: Oracle interventions. $\Delta$ Flip $(0.95{-}0.5)$ under mean-fill masking. Oracle-BG increases most with $\rho$ (background sufficiency); Oracle-FG increases more modestly, strongest on Waterbirds-seg .

Figure 6: Balanced training ( $\rho{=}0.5$ ): risk deciles. Flip stratification persists but is weaker than under $\rho{=}0.95$ .

Figure 7: Attribute accessibility ( $\rho{=}0.95$ ). $\mathcal{A}$ is predictable from few PCs of $z_{\mathrm{FG}}$ , while $z_{\mathrm{BG}}$ remains predictive. The results are computed using ResNet50.

Model	Loss	$\rho=0.5$			$\rho=0.95$
		FG-Corr	FG-Flip	FG-Miss	FG-Corr	FG-Flip	FG-Miss
R50	CE	0.806	0.084	0.110	0.774	0.116	0.109
	DCE	0.814	0.075	0.111	0.767	0.101	0.132
	GD	0.826	0.074	0.100	0.790	0.099	0.110
	CM	0.700	0.149	0.151	0.779	0.111	0.110
MiT-B2	CE	0.901	0.025	0.074	0.883	0.033	0.083

Appendix

Table 6: COCO-CD foreground error decomposition. Rates are computed over ground truth foreground pixels and satisfy FG-Corr + FG-Flip + FG-Miss = 1.

Dataset	Model	CE	Dice+CE	GroupDRO	CutMix
Waterbirds-seg	R50	0.70/0.74	0.71/0.76	0.73/0.75	0.72/0.75
Waterbirds-seg	MiT-B2	0.74/0.78	0.75/0.78	0.74/0.76	0.75/0.79
COCO-CD	R50	0.68/0.56	0.70/0.58	0.72/0.62	0.70/0.58
COCO-CD	MiT-B2	0.82/0.75	0.81/0.74	0.79/0.74	0.77/0.69

Appendix

Table 7: Absolute class-wise IoU. Each cell reports IoU

(C_{1})

/IoU

(C_{2})

, where

C_{1}

and

C_{2}

are the two foreground classes (Landbird/Waterbird for Waterbirds-seg ; Cat/Dog for COCO-CD ). The results are computed for

\rho=0.95

Figure 8: Mask-conditioned risk stratification. We compare decile plots obtained by computing risk over predicted-foreground pixels (deployable) versus ground-truth foreground pixels (analysis). Both choices stratify within-foreground Flip similarly, suggesting the score reflects identity uncertainty rather than foreground support errors. The plots are obtained for $\rho=0.95$ .

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation
May 7, 2026Aditya Parikh, Stella Frank, Sneha Das +1Image Segmentation Algorithmic Fairness
Department of Applied Mathematics and Computer Science Technical University of Denmark
CorrCLIPv2: Harnessing Unlabeled Images for Open-Vocabulary Semantic Segmentation
Oct 17, 2026Dengke Zhang, Fagui Liu, Quan Tang +1Open-Vocabulary Semantic Segmentation Semantic Segmentation
School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China. · Department of New Network, Pengcheng Laboratory, Shenzhen, 518000, China.
Segment Anything with Robust Uncertainty-Accuracy Correlation
May 11, 2026Hongyou Zhou, Marc Toussaint, Ling Shao +1Image Segmentation Promptable Image Segmentation
Learning and Intelligent Systems, Technical University of Berlin, Berlin, Germany. · UCAS-Terminus AI Lab, University of Chinese Academy of Sciences, Beijing, China.

May 7, 2026J/K move · Enter open · S save

Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation

Aditya Parikh, Stella Frank, Sneha Das, Aasa Feragen

Department of Applied Mathematics and Computer Science Technical University of Denmark

Image Segmentation Algorithmic Fairness Learning with Noisy Labels Semantic Segmentation

Open paper AskPDF arXiv

Labeled datasets reflect the biases of their annotation pipelines, which sometimes introduce label bias: group-conditional label errors that cause systematic performance disparities across demographic subgroups. Label bias in image segmentation remains underexplored, as even detecting it typically requires clean, unbiased annotations, which are not readily available. We present a data-centric adaptation of Confident Learning to segmentation, allowing detection of label bias directly in the training data without a clean, unbiased ground truth. By comparing the provided training labels to the model's confident predictions, we isolate directional errors that quantify the presence and nature of bias, where standard overlap metrics like Dice fail. We further show that label bias influences subgroup separability in the encoder's feature space, an artifact we leverage for bias mitigation rather than suppressing it. We evaluate three datasets, spanning from synthetic to real-life bias, showing how our framework reliably detects and mitigates bias without access to clean labels, achieving equitable performance across experimental conditions.

Abstract

Semantic Segmentation

Explore similar work

Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation

CorrCLIPv2: Harnessing Unlabeled Images for Open-Vocabulary Semantic Segmentation

Segment Anything with Robust Uncertainty-Accuracy Correlation