Contextrast++: Robust Multi-Scale Contextual Contrastive Learning for Semantic Segmentation

Changki Sung; Hyungtae Lim; Wanhee Kim; Youngwoo Seo; Hyun Myung

Fig. 1: Visualization of (a) ground truth annotations, (b) results obtained from HRNet [ 54 ] , and (c) output after being trained with our contrastive learning approach. (d) Overview of the proposed contextual contrastive learning framework, referred to as Contextrast++ , which refines class-specific anchors by aggregating the deepest-layer embeddings into earlier layers to enhance semantic and fine-grained feature representation. (e) Illustration of boundary-aware negative (BANE) sampling strategy, which selects feature embeddings near object boundaries (solid red triangles) as harder negative samples. This helps contrastive learning focus more on boundary regions, improving boundary prediction accuracy.

Method	Components	Scale	Boundary awareness
Baseline	-	-	-
Wang et al . ICCV 21 [ 61 ]	CL, MB	Single	✗
Hu et al . ICCV 21 [ 26 ]	CL, MB, AL	Single	✗
Pissas et al . ECCV 22 [ 46 ]	CL	Multi	✗
Zhou et al . TPAMI 24 [ 83 ]	CL, MB, SYN	Single	✗
Sung et al . CVPR 24 [ 55 ]	CL	Multi-fusion (static)	✓

TABLE I: Comparison of core components between our method and existing state-of-the-art approaches. CL: Contrastive learning, MB: Memory bank, AL: Auxiliary loss, SYN: Synthetic negative sample generation.

Fig. 2: Overview of the proposed pipeline, called Contextrast++. (a) The contextual contrastive learning (CCL) module, comprising the adaptive fusion module and pixel-to-anchor (PA) loss

L_{\mathrm{PA}}

. (b) AA loss with a memory bank, where

\mathcal{S}(\cdot)

is the selection function to generate class-prioritized anchors. (c) Boundary-aware negative (BANE) sampling, which selects informative negative samples along prediction error boundaries for use in PA loss.

I_{\text{Batch}}

denotes mini-batch images,

\hat{Y}

the predicted segmentation output,

\mathbf{F}_{\mathit{i}}

the feature representation at the

\mathit{i}

-th layer,

\mathbf{Z}_{\mathit{i}}

the transformed feature set via encoding function

\pi(\cdot)

,

\mathbf{A}_{\mathit{i}}

the representative anchors derived from

\mathbf{Z}_{\mathit{i}}

,

\hat{\mathbf{A}}_{\mathit{i}}

the updated anchors after adaptive fusion, and

L_{\text{CE}}

the cross-entropy loss. Semantic classes are distinguished using various shapes and colors (best viewed in color).

Fig. 3: The pipeline of the proposed adaptive fusion module with the deepest-layer representative anchor

\mathbf{A}_{I}

and earlier-layer representative anchor

\mathbf{A}_{i}

.

Q,K,

and

V

represent query, key, and value, respectively. The purple box

\mathcal{F}(\cdot)

indicates self-attention [ 59 ] .

Fig. 4: Illustration of the class-prioritized anchor selection and storage process in the memory bank. (a) Features in each scale may contain different sets of classes due to downsampling the ground truth to match the reduced spatial resolution of the features. Thus, some small objects (e.g. red and pink features here) tend to vanish in deeper layers, whereas large objects such as cars and roads persist across all layers. (b) Class-prioritized anchors are selected from the deepest available layer where each class appears (green check marks). (c) The selected anchors are aggregated to form a class-prioritized anchor set, which is stored in the memory bank.

Fig. 5: Overview of the boundary-aware negative sampling strategy, illustrating its effectiveness in mitigating under- and over-segmentation during training. (a) The prediction map

\hat{\mathbf{Y}}

is decomposed into class-wise binary masks

\mathbf{B}^{\mathit{n}}_{\mathit{i}}

. Then, the distance transform [ 33 ] is applied to generate class-wise distance maps

\mathbf{D}^{\mathit{n}}_{\mathit{i}}

. (b) Visualization of the distance map progression across training iterations. Misclassified regions gradually diminish as training advances (best viewed in color).

	Method		Training Settings
Dataset	Model	Backbone	Crop size	Learning rate	Momentum	Weight decay	Optimizer	Lr scheduler	Batch size	Training steps
Cityscapes	DeepLabV3 [ 6 ]	D-ResNet-101	512 $\times$ 1024	$10^{-2}$	0.9	5 $\times$ $10^{-4}$	SGD	Poly	8	40K
	HRNet [ 54 ]	HRNetV2-W48	512 $\times$ 1024	$10^{-2}$	0.9	5 $\times$ $10^{-4}$	SGD	Poly	8	40K
	OCRNet [ 74 ]	HRNetV2-W48	512 $\times$ 1024	$10^{-2}$	0.9	5 $\times$ $10^{-4}$	SGD	Poly	8	40K
	UPerNet [ 64 ]	Swin-Tiny	512 $\times$ 1024	6 $\times$ $10^{-5}$	0.9	$10^{-2}$	ADAMW	Linear	6	40K
	MobileNetV2 [ 48 ]	D-ResNet-101	512 $\times$ 1024	$10^{-2}$	0.9	$10^{-2}$	SGD	Poly	8	80K

TABLE II: Details of the experimental setup and the baseline semantic segmentation models for contrastive learning across datasets.

	Description		Dataset [mIoU (%)]
Method	Loss	Sampling	Cityscapes- val	ADE20K	PASCAL-C	COCO-Stuff	CamVid
DeepLabV3 [ 6 ]	$L_{\mathrm{CE}}$	None	77.12	42.85	52.83	36.04	78.80
DeepLabV3 + Multi [ 46 ]	$L_{\mathrm{CE}}+L_{\text{cms}}+L_{\text{ccs}}$	Random	78.94 (+1.82)	43.86 (+1.01)	52.27 (-0.56)	36.34 (+0.30)	79.67 (+0.87)
DeepLabV3 + Contextrast [ 55 ]	$L_{\mathrm{CE}}+L^{-}_{\mathrm{PA}}{{}^{*}}$	Boundary-aware	79.35 (+2.23)	44.12 (+1.27)	53.81 (+0.98)	36.55 (+0.51)	79.98 (+1.18)
DeepLabV3 + Contextrast++	$L_{\mathrm{CE}}+L_{\mathrm{PA}}{{}^{*}}+L_{\mathrm{AA}}$	Boundary-aware	81.02 (+3.90)	45.93 (+3.08)	54.23 (+1.40)	36.97 (+0.93)	80.19 (+1.39)
HRNet [ 54 ]	$L_{\mathrm{CE}}$	None	78.48	41.52	51.96	36.04	82.17

TABLE III: Comparison of segmentation performance between our approach and recent contrastive learning-based methods on public datasets. Experiments were conducted using CNN-based backbones, including DeepLabV3 [ 6 ] , HRNet [ 54 ] , and OCRNet [ 74 ] . (·) indicates absolute improvement in percentage points (%p) over the corresponding baseline.

Fig. 6: Semantic segmentation performance improvement of baseline architectures [ 6 , 54 , 74 , 64 ] achieved via state-of-the-art contrastive learning methods evaluated on Cityscapes- test . Note that Multi [ 46 ] slightly degraded performance (

-0.33\%\text{p}

) when applied to OCRNet [ 74 ] . As listed in Table II , we utilized the UPerNet framework with a Swin-Tiny backbone, replacing the original CNN-based architecture with a Transformer-based architecture.

Fig. 7: Visualization of (a) ground truth, (b) HRNet [ 54 ] , (c) HRNet + Multi [ 46 ] , (d) HRNet + Contextrast [ 55 ] , and (e) HRNet + Ours on Cityscapes- val (best viewed in color). Note that Contextrast++ effectively distinguishes small objects (first and second rows), accurately classifies the wall, which contains excessive local context that makes prediction challenging (third row), and improves predictions on partially occluded or cropped objects near image boundaries (fourth row).

Fig. 8: Visualization of (a) ground truth, (b) HRNet [ 54 ] , (c) HRNet + Multi [ 46 ] , (d) HRNet + Contextrast [ 55 ] , and (e) HRNet + Ours on the CamVid dataset (best viewed in color). While all methods perform well on the CamVid dataset, our Contextrast++ effectively distinguishes small objects on the image plane, such as bicyclists and poles.

Fig. 9: Visualization of (a) ground truth, (b) HRNet [ 54 ] , (c) HRNet + Multi [ 46 ] , (d) HRNet + Contextrast [ 55 ] , and (e) HRNet + Ours on the ADE20K dataset (best viewed in color). Note that Contextrast++ yields consistent semantic segmentation performance (first row) and provides more precise segmentation results for small objects than existing state-of-the-art methods (second to fourth rows).

Fig. 10: Visualization of (a) ground truth, (b) HRNet [ 54 ] , (c) HRNet + Multi [ 46 ] , (d) HRNet + Contextrast [ 55 ] , and (e) HRNet + Ours on the COCO-Stuff dataset (best viewed in color). Our Contextrast++ mitigates under-segmentation of objects as well as over-segmentation in large background classes, such as sea and ground, by refining boundary details.

	Cityscapes			PASCAL-C			ADE20K			COCO-Stuff
Method	Head	Mid	Tail	Head	Mid	Tail	Head	Mid	Tail	Head	Mid	Tail
HRNet [ 54 ]	90.67	74.05	72.54	69.24	57.35	27.54	55.69	34.62	32.44	62.2	34.08	12.27
HRNet + Multi [ 46 ]	90.64	73.61	77.32	69.44	57.52	27.78	56.88	36.37	32.80	62.47	34.21	14.52
HRNet + Contextrast [ 55 ]	90.91	74.90	79.46	69.84	58.61	28.35	56.40	36.70	32.86	62.41	34.66	14.15
HRNet + Contextrast++	90.91	75.46	80.97	70.31	59.11	32.36	57.52	36.86	33.54	62.76	35.45	17.17

TABLE IV: Comparison of performance across head, mid, and tail classes on Cityscapes- val , PASCAL-C, ADE20K, and COCO-Stuff.

	PASCAL-C
Method	Large	Medium	Small	Tiny
HRNet + Multi [ 46 ]	52.90	17.12	9.35	5.1
HRNet + Contextrast [ 55 ]	53.61	16.54	9.4	5.24
HRNet + Contextrast++	54.86	19.86	11.51	6.51
	ADE20K
Method	Large	Medium	Small	Tiny

TABLE V: Performance comparison of mIoU across different object scales on the PASCAL-C and ADE20K datasets. The evaluation is categorized into four groups based on object pixel size.

Fig. 11: Comparison of boundary mIoU (B-mIoU) performance improvements across state-of-the-art methods on Cityscapes- val using HRNet [ 54 ] as the baseline model. We report B-mIoU and the absolute gain in percentage points at each pixel threshold

\tau_{B}

.

Lightweight backbone	mIoU (%)
MobileNetV2 [ 48 ]	71.30
MobileNetV2 + Pico [ 61 ]	72.30 (+1.00)
MobileNetV2 + Pico+ [ 83 ]	72.60 (+1.30)
MobileNetV2 + Multi [ 46 ]	77.58 (+6.28)
MobileNetV2 + Contextrast [ 55 ]	78.21 (+6.91)
MobileNetV2 + Contextrast++	78.34 (+7.04)

TABLE VI: Quantitative comparison on Cityscapes- val using lightweight and Transformer-based (UPerNet with a Swin) backbones.

Method	mIoU (%)
Mask2Former [ 11 ] (Swin-t)	47.12
Mask2Former + Multi [ 46 ]	47.32 (+0.20)
Mask2Former + Contextrast [ 55 ]	47.73 (+0.61)
Mask2Former + Contextrast++	48.07 (+0.95)
SegFormer [ 65 ] (MiT-B0)	37.78
SegFormer + Multi [ 46 ]	37.90 (+0.12)

TABLE VII: Quantitative comparison on ADE20K using Transformer-based backbones.

Fusion	Balancing	Sampling
Adaptive fusion	$L_{\mathrm{AA}}$	Semi-hard	BANE	mIoU (%)
				81.88
✓				82.18
✓	✓			82.35
✓		✓		82.23
✓			✓	82.44

TABLE VIII: Ablation study of each module on Cityscapes- val .

1st Layer	2nd Layer	3rd Layer	4th Layer	mIoU (%)
✓				81.02 (-1.85)
✓	✓			81.61 (-1.26)
✓	✓	✓		81.79 (-1.08)
✓	✓	✓	✓	82.87
	✓	✓	✓	82.36 (-0.51)
		✓	✓	82.19 (-0.68)

TABLE IX: Ablation study of the effect of multi-scale aggregation in Contextrast++ on Cityscapes- val .

$\lambda_{1}$	$\lambda_{2}$	$\lambda_{3}$	$\lambda_{4}$	mIoU (%)
1.0	1.0	1.0	1.0	82.27
1.0	0.8	0.6	0.4	82.56
1.0	0.75	0.5	0.25	79.81
1.0	0.7	0.4	0.1	81.46
0.1	0.3	0.7	1.0	82.87
0.1	0.4	0.7	1.0	82.55

TABLE X: Comparison of different weight settings with the Cityscapes- val .

Fig. 12: (a) Design-space view of fusion variants on ADE20K, positioned by channel specialization

1-\overline{|\rho|}

(

x

-axis) and mIoU (

y

-axis); filled/open circles encode the presence/absence of MLP. Our adaptive fusion module occupies the upper-right corner, whereas Concat + MLP and the Self-Attention-only variant fail symmetrically, showing that neither property alone suffices. (b) Pairwise top-32 channel overlap between modules

M_{i}

and

M_{j}

on ADE20K. The low mean overlap confirms that the three modules attend to largely disjoint channel subsets.

Method	mIoU (%)
Concat + MLP	82.30
Multi-head self-attention [ 59 ]	82.37
Cross-attention [ 59 ]	82.14
Self-attention + weighted sum	82.39
Self-attention + element-wise sum + MLP	81.62
Gated fusion (scalar)	82.36

TABLE XI: Comparison of the adaptive fusion module on Cityscapes- val .

Anchor selection strategy	mIoU (%)
Earliest layer	81.73
Average across layer	82.36
Deepest layer (Ours)	82.87

TABLE XII: Performance comparison under different anchor selection strategies for class-prioritized anchors.

Memory bank type	mIoU (%)
Pixel memory [ 61 , 83 ]	81.94
Anchor memory (Ours)	82.87

TABLE XIII: Performance comparison of different memory bank types with the same memory budget.

Memory bank design	mIoU (%)
Earliest layer	81.63
Deepest layer	81.79
All layers	80.00
Class-prioritized anchor (Ours)	82.87

TABLE XIV: Ablation study on memory bank designs varying with selection criteria.

Fig. 13: Performance analyses on Cityscapes- val with HRNet [ 54 ] : Performance changes with (a) the hyperparameter weight

\alpha

in (( 12 )), (b) number of stored features per class

K_{\mathcal{M}}

, and (c) sampling ratio

K_{s}

for the boundary-aware negative sampling.

Fig. 14: Average cosine similarity between misclassified pixels and representative anchors at each layer of HRNet [ 54 ] on Cityscapes- val , measured as a function of distance from incorrect prediction boundaries. Boundary-adjacent samples consistently exhibit lower cosine similarity, indicating that they constitute harder negative samples than inner-region samples.

Method	FLOPs	Params	mIoU (%)
HRNet [ 54 ]	187.87G	65.86M	78.48
HRNet + Multi [ 46 ]	187.87G	66.24M	81.50
HRNet + Contextrast [ 55 ]	187.87G	66.24M	82.20
HRNet + Contextrast++	187.87G	67.82M	82.87

TABLE XV: Comparison of FLOPs and the total number of parameters for each contrastive learning method on Cityscapes- val .

Method	Memory (MiB)	Avg. training time (hours:minutes)
HRNet [ 54 ]	26,692	8:58
HRNet + Multi [ 46 ]	27,688	9:52
HRNet + Contextrast [ 55 ]	27,788	10:01
+ Adaptive fusion module	27,862	10:17
+ AA loss + Memory bank	27,844	10:10
HRNet + Contextrast++	27,924	10:22

TABLE XVI: Comparison of memory usage and average training time on Cityscapes- val .

Fig. 15: Feature map visualization on Cityscapes- val . (a) Ground truth, (b) HRNet [ 54 ] , (c) HRNet + Multi [ 46 ] , (d) HRNet + Contextrast [ 55 ] , and (e) HRNet + Ours. The visualizations in (b)–(e) show feature activation maps, where brighter (yellow) regions indicate stronger activations and greater model focus. Zoomed-in boxes highlight areas where our method attends more accurately to small objects and fine-grained boundaries (best viewed in color).

Fig. 16: Grad-CAM [ 50 ] visualizations on Cityscapes- val . (a) Ground truth, (b) HRNet [ 54 ] , (c) HRNet + Multi [ 46 ] , (d) HRNet + Contextrast [ 55 ] , and (e) HRNet + Ours. The leftmost labels ‘Car’, ‘Person’, and ‘Truck’ indicate the target class for each row. Zoomed-in boxes highlight fine-grained differences in attention for small objects and class boundaries (best viewed in color).

Fig. 17: t-distributed stochastic neighbor embedding (t-SNE) [ 58 ] visualizations of feature representations obtained from (a) HRNet [ 54 ] and (b) Contextrast++ on Cityscapes- val , ordered from 1st to 4th layers (left to right). Different colors represent distinct semantic classes. Contextrast++ yields more compact intra-class clusters and clearer inter-class separation across all layers (best viewed in color).

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Contextrast++: Robust Multi-Scale Contextual Contrastive Learning for Semantic Segmentation

Abstract

Figures & tables

Semantic Segmentation

SELECT: SELEctive Context Transfer for Class-Incremental Semantic Segmentation

SynSeg: Feature Synergy for Multi-Category Contrastive Learning in End-to-End Open-Vocabulary Semantic Segmentation

D3S2: Diffusion-Guided Dataset Distillation for Semantic Segmentation

Abstract

Semantic Segmentation

Explore similar work

SELECT: SELEctive Context Transfer for Class-Incremental Semantic Segmentation

SynSeg: Feature Synergy for Multi-Category Contrastive Learning in End-to-End Open-Vocabulary Semantic Segmentation

D3S2: Diffusion-Guided Dataset Distillation for Semantic Segmentation