Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift

Labeled datasets reflect the biases of their annotation pipelines, which sometimes introduce label bias: group-conditional label errors that cause systematic performance disparities across demographic subgroups. Label bias in image segmentation remains underexplored, as even detecting it typically requires clean, unbiased annotations, which are not readily available. We present a data-centric adaptation of Confident Learning to segmentation, allowing detection of label bias directly in the training data without a clean, unbiased ground truth. By comparing the provided training labels to the model's confident predictions, we isolate directional errors that quantify the presence and nature of bias, where standard overlap metrics like Dice fail. We further show that label bias influences subgroup separability in the encoder's feature space, an artifact we leverage for bias mitigation rather than suppressing it. We evaluate three datasets, spanning from synthetic to real-life bias, showing how our framework reliably detects and mitigates bias without access to clean labels, achieving equitable performance across experimental conditions.

Open-vocabulary semantic segmentation aims to classify pixels without being restricted to a fixed set of categories. Although Contrastive Language-Image Pre-training (CLIP) demonstrates remarkable potential for open-vocabulary recognition, its segmentation capability is impaired by noisy inter-class patch interactions. To this end, this work proposes a unified training-free object-centric framework, named CorrCLIPv2, from three complementary aspects of patch correlation, feature representation, and output regularization. Specifically, for patch correlation, we leverage class-agnostic mask generators to explicitly disentangle inter-class interactions. Moreover, we utilize spatially-coherent features from self-supervised models to compute attention scores to mitigate mask inaccuracies. Regarding feature representation, we devise a refinement mechanism to simultaneously strengthen spatial details and semantic representation. Within the output space, we unify predictions via voting in each mask to enhance spatial consistency. To mitigate contextual bias and enhance prediction accuracy of objects, we further extend the framework by incorporating a cross-image object retrieval paradigm, which incurs minimal computational overhead. We build a memory bank of CorrCLIP-enhanced features from unlabeled images and implement a regime of multi-CLIP ensemble. By aggregating semantically consistent object features across diverse contexts retrieved from the memory bank, CorrCLIPv2 mitigates context-induced interference and obtains more robust predictions. Extensive experiments demonstrate that our approach establishes a new state-of-the-art on 10 standard benchmarks. Remarkably, it even surpasses fully supervised open-vocabulary methods across 22 multi-domain datasets. Code is available at: https://github.com/zdk258/CorrCLIP.

Despite strong zero-shot performance, SAM is unreliable under domain shift due to Mask-level Confidence Confusion (MCC), where a single IoU-based mask score fails to reflect pixel-wise reliability near boundaries. Motivated by the contrast between texture-biased shortcuts in neural networks and shape-centric processing in human vision, we model out-of-domain variation as appearance shifts and non-rigid deformations that jointly stress calibration. We propose Segment Anything with Robust Uncertainty-Accuracy Correlation (RUAC) for robust pixel-wise uncertainty estimation under appearance and deformation shifts. RUAC adds a lightweight uncertainty head, trains it with a collaborative style-deformation attack that jointly perturbs texture and geometry, and applies Uncertainty-Accuracy Alignment to ensure uncertainty consistently highlights erroneous pixels even under adversarial perturbations. Across 23 zero-shot domains, RUAC improves segmentation quality and yields more faithful uncertainty with stronger uncertainty-accuracy correlation. Project page: https://hongyouzhou.github.io/ruac/.