On the Relaxation of Conditional Independence Assumption for Image Segmentation

Zixun Wang; Ben Dai

cs.CVSep 30, 2026

On the Relaxation of Conditional Independence Assumption for Image Segmentation

Authors: Zixun Wang, Ben Dai

Organizations: Department of Statistics and Data Science The Chinese University of Hong Kong

Abstract

In semantic segmentation, a recent line of RankSEG methods directly optimizes Dice/IoU scores at inference time, improving alignment with evaluation metrics without modifying model training. Despite its theoretical and empirical success, RankSEG relies on the restrictive Conditional Independence Assumption (CIA), which ignores crucial label correlations and therefore degrades performance in ambiguous or low-contrast scenarios. However, accounting for full label dependence is computationally prohibitive, requiring $\mathcal{O}(d^3)$ time. To address this, we replace the CIA with a Spatially Localized Dependence (SLD) structure that captures local label correlations while keeping the dependence model tractable. We further overcome the remaining computational bottleneck via a Reciprocal Moment Approximation coupled with a novel fixed-point optimization strategy that eliminates exhaustive search. The proposed algorithm achieves a highly practical $\mathcal{O}(d \log d)$ complexity and consistently outperforms conventional argmax and CIA-based RankSEG across diverse segmentation benchmarks. Improvements are significant in low-contrast or small-object scenarios, where label dependence offers valuable signals complementary to image information for accurate segmentation. The code of experiments is available at https://github.com/ZixunWang/RankSEG-DEP.

Figures & tables

Panel of Figure 1 — Figure 1 : (a) The intensity contrast, and consequently the foreground probabilities, are uniformly low, so traditional thresholding barely detects any tumor and RankSEG almost entirely misses the tumor in the upper right. In contrast, our proposed method more fully recovers the tumor by leveraging label dependencies within the tumor region. (b) When predicting $Y_{j^{\prime}}$ , only information from $\bm{X}$ is used under CIA, whereas both $\bm{X}$ and the neighboring label $Y_{j}$ are utilized when the CIA is not assumed.

Figure 2 : Trade-off in dependence modeling. The two endpoints represent the extremes of CIA and Full dependence. By relaxing CIA to local dependence, SLD strikes a better trade-off. The graphs depict the label dependence (dep.) under each model: none under CIA, arbitrary under Full, and neighboring-only under SLD.

Model	Prediction	KiTS		LiTS
Model	Prediction	IoU	Dice	IoU	Dice
UNet	Argmax-prob	51.00	57.36	38.45	47.58
	CIA-RankSEG	53.54	60.07	40.70	50.07
	Ours	53.92	60.48	40.77	50.16
DeepLabV3+	Argmax-prob	54.19	61.16	38.34	47.38
	CIA-RankSEG	56.22	63.56	40.09	49.50

Table 1 : Comparison of performance (Dice and IoU %) in LiTS and KiTS.

Model	Prediction	ADE20K		Cityscapes		DeepGlobe
Model	Prediction	mIoU	mDice	mIoU	mDice	mIoU	mDice
PSPNet	Argmax-prob	51.32	58.66	73.07	80.45	57.24	66.23
	CIA-RankSEG	51.57	59.17	73.72	81.14	58.20	67.82
	Ours	51.92	59.49	73.84	81.27	58.47	68.14
DeepLabV3+	Argmax-prob	52.53	59.57	73.37	80.59	57.75	66.73
	CIA-RankSEG	52.64	59.95	73.92	81.24	58.38	67.94

Table 2 : Comparison of performance (mDice and mIoU %) in ADE20K, Cityscapes, and DeepGlobe.

Quantile	0.2	0.4	0.6
Argmax-prob	+14.37 %	+7.46 %	+3.97 %
CIA-RankSEG	+2.51 %	+1.04 %	+0.48 %

Table 3 : Relative Dice improvements over baselines on KiTS subsets, formed by the lower quantiles of Argmax-prob performance.

Group	small	medium	large
Argmax-prob	+5.47 %	+2.92 %	+2.01 %
CIA-RankSEG	+1.19 %	+0.82 %	+0.42 %

Table 4 : Relative Dice improvements over baselines on ADE20K across category groups, defined by average pixel area.

Panel of Figure 3 — Figure 3 : Our method consistently outperforms CIA-RankSEG across 10 retraining runs on (a) ADE20K, (b) Cityscapes, and (c) DeepGlobe, with paired $t$ -test $p$ -values $\ll 0.01$ confirming the statistical significance.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	LiTS	KiTS	ADE20K	Cityscapes	DeepGlobe
#Steps $<1.5$	63.36%	88.52%	9.30%	2.80%	40.49%
$1.5\leq$ #Steps $<2.5$	36.42%	11.41%	90.55%	97.20%	50.51%
$2.5\leq$ #Steps	0.22%	0.07%	0.15%	0.00%	0.00%

Appendix

Table 5 : Distribution of the number of iterations required for convergence across all test samples. On every dataset, the iteration terminates within

2.5

steps for the vast majority of samples, with only a negligible fraction requiring more.

Dataset	KiTS		LiTS
Metric	IoU	Dice	IoU	Dice
Fixed-point Optimization	56.70	64.07	40.48	49.88
Brute-force Search	56.73	64.10	40.48	49.88

Appendix

Table 6 : Comparison between the convergent point obtained by the fixed-point optimization and the global optimum obtained by brute-force search. The performance gap is at most

0.03

on both metrics, indicating that the iteration consistently recovers a near-optimal fixed point.

Method	CRF		Argmax-prob		CIA-RankSEG		Ours
Method	mIoU	mDice	mIoU	mDice	mIoU	mDice	mIoU	mDice
DeepGlobe	58.61	67.40	59.49	68.60	60.19	69.57	60.52	69.91
Cityscapes	73.26	80.35	75.66	82.61	76.17	83.21	76.25	83.31
ADE20K	56.05	62.61	56.94	63.98	57.67	64.92	57.94	65.19

Appendix

Table 7 : Performance comparison of CRF, Argmax-prob, CIA-RankSEG, and our method using UPerNet.

Figure 6 : Qualitative comparison of segmentation masks produced by different methods on LiTS.

Figure 7 : Qualitative comparison of segmentation masks produced by different methods on KiTS.

Figure 8 : Qualitative comparison on ADE20K. Each case spans two rows, where the first row shows the original image and the second row zooms in on the most distinctive region.

Figure 9 : Qualitative comparison on Cityscapes. Each case spans two rows, where the first row shows the original image and the second row zooms in on the most distinctive region.

Figure 10 : Qualitative comparison on DeepGlobe Land. Each case spans two rows, where the first row shows the original image and the second row zooms in on the most distinctive region.

Explore similar work

SegPAR: Class-Centric Decision-Based Sparse Attack for Semantic Segmentation
Aug 11, 2026Dongsu Song, DaeYun GO, Boseung Seo +1Semantic Segmentation Sparse Supervision
Interdisciplinary Program in Space Systems Engineering, Korea Aerospace University, South Korea · Department of Computer Science, Korea Aerospace University, South Korea · Department of Artificial Intelligence, Korea Aerospace University, South Korea
A Unified Variational Framework for Deep Weakly Supervised Image Segmentation
Jul 22, 2026Yin King Chu, Lingfeng Li, Sung Ha Kang +2Variational Spatial Photonic Ising Machines
Department of Mathematics, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, China · Hetao Institute of Mathematics and Interdisciplinary Sciences, Shenzhen, Guangdong, China · School of Mathematics, Georgia Institute of Technology, Atlanta, GA, 30332, USA +2
Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation
May 7, 2026Aditya Parikh, Stella Frank, Sneha Das +1Biases Algorithmic Fairness
Department of Applied Mathematics and Computer Science Technical University of Denmark

Aug 11, 2026, cs.CVJ/K move · Enter open · S save

SegPAR: Class-Centric Decision-Based Sparse Attack for Semantic Segmentation

Dongsu Song, DaeYun GO, Boseung Seo, Jay Hoon Jung

Interdisciplinary Program in Space Systems Engineering, Korea Aerospace University, South Korea · Department of Computer Science, Korea Aerospace University, South Korea · Department of Artificial Intelligence, Korea Aerospace University, South Korea

Semantic Segmentation Sparse Supervision White-Box Spectral-Subspace-Guided Attack

Open paper AskPDF arXiv

Despite the practical relevance of sparse decision-based black-box threats, they have received limited attention in semantic segmentation. To bridge this gap, we adapt the most representative decision-based black-box sparse attacks from the classification domain to serve as baselines, establishing a rigorous benchmark for this underexplored setting. In this context, we demonstrate that one of the existing methods suffers from severe query inefficiency due to its image-centric pixel accumulation, which rapidly exhausts query budgets across the vast image space. To overcome this, we propose SegPAR, a novel decision-based framework that shifts to a class-centric exploration paradigm. Furthermore, to eliminate the misleading feedback generated by standard decision rewards during pixel accumulation, we introduce a novel discrepancy reward. Extensive experiments show that SegPAR significantly outperforms black-box baselines in sparsity efficiency and MIoU reduction, while remaining competitive with white-box sparse attacks. Code is available at https://github.com/KAU-QuantumAILab/SegPAR.

Abstract

Explore similar work

SegPAR: Class-Centric Decision-Based Sparse Attack for Semantic Segmentation

A Unified Variational Framework for Deep Weakly Supervised Image Segmentation

Towards Fairness under Label Bias in Image Segmentation: Impact, Measurement and Mitigation