CRF Loss is How Networks Should Learn Boundaries in Weakly Supervised Segmentation

Joshua Li; Yuri Boykov

cs.CVSep 28, 2026

CRF Loss is How Networks Should Learn Boundaries in Weakly Supervised Segmentation

Authors: Joshua Li, Yuri Boykov

Organizations: Cheriton School of Computer Science University of Waterloo

Abstract

Weakly Supervised Semantic Segmentation (WSSS) learns pixel-level predictions from image-level tags. Recent work focuses on improving coarse CAMs extracted from large vision-language models (commonly CLIP), but does little to improve their accuracy along segment boundaries. That job is instead delegated to a post-processing method like DenseCRF. However, because DenseCRF relies on low-level colour cues, it can flip correct labels to incorrect ones when neighbouring pixels share similar colours. SAM has recently been adopted as a natural alternative, yet it simply takes on DenseCRF's role as an intermediate "refinement" step that outputs one-hot pseudo-labels in prior work. By discarding the valuable uncertainty in CAMs, these one-hot pseudo-labels turn borderline errors into confidently wrong targets. Our key insight is that CAMs should supervise training alongside SAM boundaries, each through its own loss, rather than being fused together into a single hard target. Inspired by CRF potentials, we propose a framework that disentangles soft pseudo-labels as unary supervision and binary edge maps as pairwise supervision. We realize our framework in a single-stage model, DS-CRF, using CAMs from dino.txt and boundaries from SAM. DS-CRF sets a new state-of-the-art of 56.5% mIoU on MS COCO.

Figures & tables

Figure 1: Boundary in hard target vs. boundary by CRF loss. The CAM for class A leaks into the circle, which belongs to class B. (a) Prior work uses DenseCRF or SAM to fuse the CAM and the boundaries into a one-hot (hard) target that assigns the whole circle to class A, so the correct prediction incurs infinite loss. (b) Our CRF loss disentangles the two signals, using soft CAM-derived pseudo-labels as unary supervision and boundaries as pairwise supervision. Since no class is ever hard-imposed, the correct prediction is penalized far less.

Figure 2: DS-CRF architecture . We design a lightweight decoder on top of the frozen DINO backbone, running in parallel with the dino.txt vision head. CAMs are generated by computing the cosine similarity between dino.txt patch and text embeddings, which serve as soft unary targets for collision cross-entropy after normalization. Separately, the image is passed through SAM to produce class-agnostic boundaries, then dilated to define pairwise affinities within our CRF pairwise loss. (Sphere image from [ 42 ] ).

Figure 3: Pseudo-labels from dino.txt CAMs. (a) Image. (b) Ground truth. (c) Hard pseudo-labels. (d) Soft pseudo-labels. (e) Soft pseudo-labels with min-max scaling.

Figure 4: Effect of partial voluming on SAM superpixels. Left : input images. Right : SAM superpixels from aggregating SAM boundaries.

Method	dCRF	Sup.	Backbone	VOC $val$	VOC $test$	COCO $val$
Multi-Stage WSSS Methods
KTSE ${}_{\text{ECCV'24}}$ [ 43 ]	✓	$\mathcal{I}$	RN101	73.0	72.9	45.9
MuP-VSS ${}_{\text{CVPR'25}}$ [ 44 ]	✓	$\mathcal{I}$	WRN38	73.6	74.7	46.6
CLIMS ${}_{\text{CVPR'22}}$ [ 18 ]	✓	$\mathcal{I}+\mathcal{L}$	RN50	70.4	70.0	–
CLIP-ES ${}_{\text{CVPR'23}}$ [ 5 ]	✓	$\mathcal{I}+\mathcal{L}$	RN101	73.8	73.9	45.4
MMCST ${}_{\text{CVPR'23}}$ [ 45 ]	✓	$\mathcal{I}+\mathcal{L}$	WRN38	72.2	72.2	45.9

Table 1: Comparison with state-of-the-art WSSS methods. We report mIoU (%) on the PASCAL VOC 2012 and MS COCO 2014 datasets. Supervision ( Sup. ) types are defined as:

\mathcal{I}

(image-level labels),

\mathcal{L}

(language), and

\mathcal{S}

(SAM masks). For multi-stage methods, the backbone refers to the final segmentation model.

SEPL ${}_{\text{NeurIPS'23 (Workshop)}}$ [ 26 ] -enhanced
Method	Sup.	VOC $val$	VOC $test$	COCO $val$
EPS ${}_{\text{CVPR'21}}$ [ 50 ]	$\mathcal{I}+\mathcal{S}+\mathcal{M}$	72.1	–	41.6
SIPE ${}_{\text{CVPR'22}}$ [ 51 ]	$\mathcal{I}+\mathcal{S}$	69.7	–	45.2
L2G ${}_{\text{CVPR'22}}$ [ 52 ]	$\mathcal{I}+\mathcal{S}+\mathcal{M}$	72.4	–	46.4
CLIMS ${}_{\text{CVPR'22}}$ [ 18 ]	$\mathcal{I}+\mathcal{S}+\mathcal{L}$	71.1	–	–
CLIP-ES ${}_{\text{CVPR'23}}$ [ 5 ]	$\mathcal{I}+\mathcal{S}+\mathcal{L}$	73.1	–	47.9

Table 2: Comparison with SAM-based WSSS methods. We report mIoU (%) on the PASCAL VOC 2012 and MS COCO 2014 datasets. Supervision ( Sup. ) types are defined as:

\mathcal{I}

(image-level labels),

\mathcal{L}

(language),

\mathcal{S}

(SAM masks), and

\mathcal{M}

(saliency maps).

	CE		CCE
	Hard	Soft	Soft
w/o min-max	40.2	43.3	74.6
w/ min-max	-	53.5	77.8

Table 3: Ablation study of cross-entropy variants and min-max scaling. We report mIoU (%) on the PASCAL VOC val set.

Gaussian Kernel	Superpixel	SAM	SAM d=3	SAM d=5	SAM d=7
50.8	51.6	50.6	74.9	77.8	75.8

Table 4: Ablation study of pairwise affinity designs. We report mIoU (%) on the PASCAL VOC val set.

Figure 5: Network predictions using different cross-entropy variants. (a) Image. (b) Ground truth. (c) Standard cross-entropy w/ hard pseudo-labels. (d) Standard cross-entropy w/ soft pseudo-labels after min-max scaling. (e) Collision cross-entropy w/ soft pseudo-labels after min-max scaling.

Figure 6: Network predictions using different pairwise affinity designs. (a) Image. (b) Ground truth. (c) Gaussian kernel. (d) Superpixel. (e) SAM. (f) SAM with dilation $d=5$ .

Explore similar work

Weakly Supervised Segmentation as Semantic-Based Regularization
May 13, 2026Stefano Colamonaco, Andrei-Bogdan Florea, Jaron MaeneSemantic Segmentation High Confidence Pseudo-Labels
KU Leuven · Apixa
DiCLIP: Diffusion Model Enhances CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation
May 6, 2026Zhiwei Yang, Pengfei Song, Yucong Meng +3Semantic Segmentation Class Activation Mapping
Academy for Engineering and Technology, Fudan University, Shanghai 200433, China · Shandong Computer Science Center, China · Digital Medical Research Center, School of Basic Medical Science, Fudan University, and also with Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China +1
ModuSeg: Decoupling Object Discovery and Semantic Retrieval for Training-Free Weakly Supervised Segmentation
Apr 8, 2026Qingze He, Fagui Liu, Dengke Zhang +2Semantic Segmentation High Confidence Pseudo-Labels
South China University of Technology, Guangzhou, China · Pengcheng Laboratory, Shenzhen, China

May 13, 2026, cs.CVJ/K move · Enter open · S save

Weakly Supervised Segmentation as Semantic-Based Regularization

Stefano Colamonaco, Andrei-Bogdan Florea, Jaron Maene

KU Leuven · Apixa

Semantic Segmentation High Confidence Pseudo-Labels Regularization Neuro-Symbolic Framework

Open paper AskPDF arXiv

Weakly supervised semantic segmentation (WSSS) trains dense pixel-level segmentation models from partial or coarse annotations such as bounding boxes, scribbles, or image-level tags. While recent work leverages foundation models such as the Segment Anything Model (SAM) to generate pseudo-labels, these approaches typically depend on heuristic prompt choices and offer limited ways to incorporate prior knowledge or heterogeneous labels. We address this gap by taking a neurosymbolic perspective: integrating differentiable fuzzy logic with deep segmentation models. Weak annotations and domain-specific priors are unified as continuous logical constraints that fine-tune SAM under weak supervision. The refined foundation model then produces improved pseudo-labels, from which we train a second-stage prompt-free segmentation model. Experiments on Pascal VOC 2012 and the REFUGE2 optic disc/cup segmentation dataset show that our logic-guided fine-tuning yields higher-quality pseudo-labels, leading to state-of-the-art segmentation accuracy that often exceeds densely supervised baselines.

Abstract

Explore similar work

Weakly Supervised Segmentation as Semantic-Based Regularization

DiCLIP: Diffusion Model Enhances CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation

ModuSeg: Decoupling Object Discovery and Semantic Retrieval for Training-Free Weakly Supervised Segmentation