HDVS: semi-supervised semantic segmentation via heterogeneous dual-branch voting supervision

Yongqi Shan; Yunzhi Zhuge; Huchuan Lu

cs.CVOct 27, 2026

HDVS: semi-supervised semantic segmentation via heterogeneous dual-branch voting supervision

Authors: Yongqi Shan, Yunzhi Zhuge, Huchuan Lu

Organizations: School of Computer Science and Technology, Dalian University of Technology, Dalian 116081, China. · School of Information and Communication Engineering, Dalian University of Technology, Dalian 116081, China.

Abstract

Semi-supervised semantic segmentation, which aims to utilize large volumes of unlabeled images to achieve accurate segmentation results with fewer human annotations, has attracted increasing attention. Prior methods were primarily based on the assumption that the pseudo-labels generated by each branch were sufficiently reliable to supervise the remaining branches. However, there is a concern that the errors in pseudo-labels could accumulate during co-training, potentially resulting in suboptimal performance. To this end, we present a novel framework called heterogeneous dual-branch voting supervision (HDVS), which is designed to enhance the reliability of pseudo-labels and mitigate the issues arising from pseudo-labeling. Specifically, based on the cross-supervision framework, we introduce a voting mechanism that correlates pseudo-labels from heterogeneous branches to produce pseudo-labels with enhanced reliability. Concurrently, we employ a feature communication module to introduce perturbations at the feature level in each branch to maximize prediction diversity across the dual branches while maintaining convergence. Comprehensive evaluations of the proposed HDVS on two benchmark datasets (PASCAL VOC 2012 and Cityscapes) demonstrate its superiority to the state-of-the-art approaches.

Figures & tables

Figure 1: Overview of common semi-supervised frameworks. (a) FixMatch [ 12 ] structure used in U 2 PL [ 13 ] , (b) cross pseudo supervision (CPS) [ 14 ] , (c) diverse co-training with discrete cosine transform (DCT) for image processing [ 15 ] , and (d) our heterogeneous dual-branch voting supervision (HDVS) method where two models use voting to generate pseudo-labels.

u

denotes an unlabeled image,

A

and

A_{+}

denote weak and strong data augmentations,

M^{1}

and

M^{2}

are the two segmentation models, and

M_{\mathrm{Teacher}}

denotes the teacher model. Trans and Conv denote the Transformer and convolutional neural network (CNN) branches,

\bm{p}

denotes their predictions,

\bm{p}_{\mathrm{o}}

and

\bm{p}_{\mathrm{c}}

denote the predictions from the original branch and the feature communication branch, respectively.

L_{\mathrm{u}},L_{\mathrm{u}}^{1},L_{\mathrm{u}}^{2}

are the corresponding unsupervised losses.

Figure 2: Quality analysis of segmentation maps. From left to right: (a) input images, (b) manually annotated ground truth labels, (c) segmentation maps generated by CPS, (d) segmentation maps generated by U 2 PL, and (e) segmentation maps produced by HDVS. All models are trained using the 1/8 classical partition protocol on the PASCAL VOC 2012 [ 24 ] train dataset.

Figure 3: Main pipeline of our HDVS method. It integrates two distinct parallel architectures, $\mathrm{i.e.}$ , CNNs and Transformers, to capture pseudo-labels from different perspectives. To improve label accuracy, a fusion and voting strategy is applied across these architectures. The feature communication module (FCM) on each branch supports implicit feature-level integration, while a specialized voting module explicitly refines the pseudo-labels generated by each branch. The refined labels, denoted as $\hat{y}$ , and their filtered versions, $\hat{y}^{0}$ and $\hat{y}^{1}$ , are used to jointly train both models, promoting synergistic learning across the architectures.

Figure 4: Pipeline of the FCM in the Transformer branch. This module is designed to facilitate feature communication between two heterogeneous segmentation models while introducing controlled perturbations. The FCM effectively aligns and integrates features from both models to enhance segmentation performance. $\bm{f}^{0}$ and $\bm{f}^{1}$ denote features from the two encoders, $\bm{cf}^{0}$ is the communicated feature in the Transformer branch, and $\lambda^{0}$ controls the strength of injected features. denotes feature convolution. denotes feature summation.

Method	VOC Train					VOC Aug
	1/16 (92)	1/8 (183)	1/4 (366)	1/2 (732)	Full (1464)	1/16 (662)	1/8 (1323)	1/4 (2646)
Results of using ResNet-50 [ 58 ] as backbone
Supervised	44.0	52.3	61.7	65.9	72.9	60.7	67.7	71.9
U 2 PL $\dagger$ [ 13 ]	-	-	-	-	-	72.0	75.1	76.2
UniMatch [ 51 ]	71.9	72.5	76.0	77.4	78.7	75.8	76.9	76.8
AugSeg [ 49 ]	64.2	72.2	76.2	77.4	78.8	74.7	76.0	77.1

Table 1: Comparison of mean intersection over union (mIoU) (%) with state-of-the-art (SOTA) methods on the PASCAL VOC 2012 [ 24 ] validation set with different backbones under various proportion protocols. “Supervised” denotes the supervised baseline without using unlabeled training data. The results of U 2 PL [ 13 ] (marked with

\dagger

) are obtained from Refs. [ 51 ] , and it follows the same partition protocol as ours. “VOC Train” refers to using only the 1464 finely annotated images from the train set, “VOC Aug” includes an additional 9118 weakly labeled images from the semantic boundaries dataset [ 56 ] . Bold numbers indicate the best performance under each setting.

	1/16 (186)	1/8 (372)	1/4 (744)	1/2 (1488)
Supervised	63.3	68.7	74.1	76.6
CPS [ 14 ]	69.8	74.4	76.9	78.6
U 2 PL [ 13 ]	69.0	73.0	76.3	77.2
AugSeg [ 49 ]	73.7	76.5	78.8	79.3
Co-training [ 15 ]	-	76.3	77.1	-
DAW [ 60 ]	75.2	76.3	77.7	79.2

Table 2: Comparison of mIoU (%) with SOTA methods on the Cityscapes [ 25 ] validation set with ResNet-50 backbone under various partition protocols.

Figure 5: Qualitative results on the Cityscapes validation [ 25 ] set. From left to right, the categories are (a) input images, (b) ground truth labels annotated manually, (c) segmentation maps created by CPS, (d) segmentation maps produced by U 2 PL, and (e) segmentation maps generated by our HDVS framework. All models are trained following the 1/2 (1488) partition protocol on the Cityscapes train set.

	Configurations				mIoU
	Heterogeneous networks	Voting module	FCM	Communication branch	1464	1323
Res-CPS ∗	-	-	-	-	78.8	76.9
Seg-CPS ∗	-	-	-	-	80.9	77.8
Res-C-CPS ∗	-	-	✓	✓	78.9	76.6
Seg-C-CPS ∗	-	-	✓	✓	80.7	77.4
Res-V-CPS ∗	-	✓	-	-	81.2	78.3

Table 3: Ablation study on the effectiveness of heterogeneous dual-branch voting supervision components. CPS ∗ replicates CPS [ 14 ] with aligned pseudo-label filtering and hyperparameters. Res-/Seg-CPS ∗ denote dual convolutional neural network/ dual Transformer baselines; H-CPS ∗ uses a heterogeneous CNN+Transformer pair. C-CPS ∗ augments the model with the FCM and its communication branch, while V-CPS ∗ adds the voting module for pseudo-label generation. HDVS combines both the feature communication and voting module. All results use a ResNet-50 backbone, and Mit-B2 [ 2 ] is paired with ResNet-50 in the heterogeneous setting.

\checkmark

and “-” indicate that the corresponding component is enabled and disabled, respectively. FCM denotes the feature communication module.

Figure 6: $t$ -SNE [ 64 ] visualization of extracted features by CPS and HDVS. Both models are trained on the Cityscapes train set following the 1/2 (1488) partition protocol, and the features are extracted from the Cityscapes validation set for display. The first row shows the visualization results for all categories, while the second row presents the visualization analysis for eight selected categories. The first and second columns display the visualization results for the CPS and HDVS models, respectively.

Figure 7: Visualizations of segmentation maps generated by various models, (a) input images to (g) our HDVS model, showcasing progressive enhancements at each step. All models, including the homogeneous baselines Res-CPS ∗ /Seg-CPS ∗ (dual convolutional neural network / dual Transformer), utilizing ResNet-50 [ 58 ] as the backbone and employing a combined approach of ResNet-50 and Mit-B2 [ 2 ] for heterogeneous structures, are trained using the classic 1464 partition protocol on the PASCAL VOC 2012 train set. H-CPS ∗ adopts a heterogeneous CNN+Transformer pair, and H-V-CPS ∗ further introduces a voting module on top of H-CPS ∗ .

$\lambda^{0}$	$\lambda^{1}$	Performance (%)
1.000	0.010	82.4
0.100	0.010	83.1
0.010	0.010	82.5
0.001	0.010	82.0
0.100	1.000	81.9
0.100	0.100	82.3

Table 4: Ablation study on scaling coefficients

\lambda^{0}

and

\lambda^{1}

using a combination of ResNet-50 and Mit-B2 as the backbone within the classical partition of 1464 training images from the PASCAL VOC 2012 train dataset.

Figure 8: Effectiveness of the voting module under heterogeneous networks. (a) The performance of H-CPS ∗ and H-V-CPS ∗ , using a combination of ResNet-50 and Mit-B2 as the backbone, under different numbers of labeled images on the PASCAL VOC 2012 dataset. (b) Under the classical partition protocol with 183 training samples on the PASCAL VOC train dataset, the performance of H-CPS ∗ and H-V-CPS ∗ across different categories. mIoU denotes mean intersection over union.

	1464	1323	Params (M)
H-CPS ∗	81.4	78.1	-
Cross-attention	80.5	76.2	29.3
Concat + MLP	82.1	78.3	7.0
Ours (FCM)	83.1	79.3	2.9

Table 5: Cross-branch communication variants. We report mIoU on the PASCAL VOC 2012 train dataset under the classic 1464 and the blended 1/8 (1323) partitioning protocols, along with the parameter count of the feature communication components (M). H-CPS ∗ serves as the baseline without feature communication. Concat + multi-layer perceptron (MLP) concatenates peer features and projects them with an MLP. Cross-attention fuses features via cross-attention.

Drop ratio (%)	0	10	20	30	40
mIoU	78.3	81.8	83.1	82.2	79.4
$\mathrm{\Delta}$ vs. 20%	$-$ 4.8	$-$ 1.3	-	$-$ 0.9	$-$ 3.7

Table 6: HDVS performance under different entropy-drop ratios. All experiments use a combination of ResNet-50 and MiT-B2 as the backbone, within the classical partition of 1464 training images from the PASCAL VOC 2012 train dataset.

\mathrm{\Delta}

is relative to 20%.

Component	Params (M)
Student branch (ResNet-50)	43.6
Teacher branch (Mit-B2)	27.4
FCM	2.9
Voting module	0.0
HDVS	71.0

Table 7: Trainable parameter counts (M) of HDVS. The parameter count of the FCM is already included in the student/teacher branch totals and is reported separately for reference only.

Figure 9: Failure cases visualizations. From left to right, the categories are (a) input images, (b) ground truth annotations, and (c) segmentation maps generated by our HDVS framework. All models are trained using the 1/16 classical partition protocol on the PASCAL VOC 2012 train dataset.

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

Accuracy Improvement of Semi-Supervised Segmentation Using Supervised ClassMix and Sup-Unsup Feature Discriminator
Apr 8, 2026Takahiro Mano, Reiji Saito, Kazuhiro HottaSemi-Supervised Medical Image Segmentation Data Augmentation
Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan
A Semi-Supervised Framework for Breast Ultrasound Segmentation with Training-Free Pseudo-Label Generation and Label Refinement
Mar 6, 2026Ruili Li, Jiayi Ding, Ruiyu Li +7Ultrasound Image Segmentation Semi-Supervised Medical Image Segmentation
Tohoku University Graduate School of Medicine, Sendai 980-8575, Japan · State Key Laboratory of Oncology in South China, Sun Yat-sen University Cancer Center, Guangzhou 510060, China · Southeast University, School of Cyber Science and Engineering Nanjing, Jiangsu, China +4
SemiGDA: Generative Dual-distribution Alignment for Semi-Supervised Medical Image Segmentation
Apr 25, 2026Kaiwen Huang, Yi Zhou, Yizhe Zhang +2Distribution Matching Semi-Supervised Medical Image Segmentation
Nanjing University of Science and Technology, Nanjing, China · Southeast University, Nanjing, China

Apr 8, 2026J/K move · Enter open · S save

Accuracy Improvement of Semi-Supervised Segmentation Using Supervised ClassMix and Sup-Unsup Feature Discriminator

Takahiro Mano, Reiji Saito, Kazuhiro Hotta

Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan

Semi-Supervised Medical Image Segmentation Data Augmentation Medical Image Segmentation Representation Alignment

Open paper AskPDF arXiv

In semantic segmentation, the creation of pixel-level labels for training data incurs significant costs. To address this problem, semi-supervised learning, which utilizes a small number of labeled images alongside unlabeled images to enhance the performance, has gained attention. A conventional semi-supervised learning method, ClassMix, pastes class labels predicted from unlabeled images onto other images. However, since ClassMix performs operations using pseudo-labels obtained from unlabeled images, there is a risk of handling inaccurate labels. Additionally, there is a gap in data quality between labeled and unlabeled images, which can impact the feature maps. This study addresses these two issues. First, we propose a method where class labels from labeled images, along with the corresponding image regions, are pasted onto unlabeled images and their pseudo-labeled images. Second, we introduce a method that trains the model to make predictions on unlabeled images more similar to those on labeled images. Experiments on the Chase and COVID-19 datasets demonstrated an average improvement of 2.07% in mIoU compared to conventional semi-supervised learning methods.

Abstract

Semantic Segmentation

Explore similar work

Accuracy Improvement of Semi-Supervised Segmentation Using Supervised ClassMix and Sup-Unsup Feature Discriminator

A Semi-Supervised Framework for Breast Ultrasound Segmentation with Training-Free Pseudo-Label Generation and Label Refinement

SemiGDA: Generative Dual-distribution Alignment for Semi-Supervised Medical Image Segmentation