cs.CVOct 27, 2026

HDVS: semi-supervised semantic segmentation via heterogeneous dual-branch voting supervision

Authors: Yongqi Shan, Yunzhi Zhuge, Huchuan Lu

Organizations: School of Computer Science and Technology, Dalian University of Technology, Dalian 116081, China. · School of Information and Communication Engineering, Dalian University of Technology, Dalian 116081, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Semi-supervised semantic segmentation, which aims to utilize large volumes of unlabeled images to achieve accurate segmentation results with fewer human annotations, has attracted increasing attention. Prior methods were primarily based on the assumption that the pseudo-labels generated by each branch were sufficiently reliable to supervise the remaining branches. However, there is a concern that the errors in pseudo-labels could accumulate during co-training, potentially resulting in suboptimal performance. To this end, we present a novel framework called heterogeneous dual-branch voting supervision (HDVS), which is designed to enhance the reliability of pseudo-labels and mitigate the issues arising from pseudo-labeling. Specifically, based on the cross-supervision framework, we introduce a voting mechanism that correlates pseudo-labels from heterogeneous branches to produce pseudo-labels with enhanced reliability. Concurrently, we employ a feature communication module to introduce perturbations at the feature level in each branch to maximize prediction diversity across the dual branches while maintaining convergence. Comprehensive evaluations of the proposed HDVS on two benchmark datasets (PASCAL VOC 2012 and Cityscapes) demonstrate its superiority to the state-of-the-art approaches.

Figures & tables

Figure 1: Overview of common semi-supervised frameworks. (a) FixMatch [ 12 ] structure used in U 2 PL [ 13 ] , (b) cross pseudo supervision (CPS) [ 14 ] , (c) diverse co-training with discrete cosine transform (DCT) for image processing [ 15 ] , and (d) our heterogeneous dual-branch voting supervision (HDVS) method where two models use voting to generate pseudo-labels. uu denotes an unlabeled image, AA and A+A_{+} denote weak and strong data augmentations, M1M^{1} and M2M^{2} are the two segmentation models, and MTeacherM_{\mathrm{Teacher}} denotes the teacher model. Trans and Conv denote the Transformer and convolutional neural network (CNN) branches, p\bm{p} denotes their predictions, po\bm{p}_{\mathrm{o}} and pc\bm{p}_{\mathrm{c}} denote the predictions from the original branch and the feature communication branch, respectively. Lu,Lu1,Lu2L_{\mathrm{u}},L_{\mathrm{u}}^{1},L_{\mathrm{u}}^{2} are the corresponding unsupervised losses.
Figure 2: Quality analysis of segmentation maps. From left to right: (a) input images, (b) manually annotated ground truth labels, (c) segmentation maps generated by CPS, (d) segmentation maps generated by U 2 PL, and (e) segmentation maps produced by HDVS. All models are trained using the 1/8 classical partition protocol on the PASCAL VOC 2012 [ 24 ] train dataset.
Figure 3: Main pipeline of our HDVS method. It integrates two distinct parallel architectures, i.e.\mathrm{i.e.} , CNNs and Transformers, to capture pseudo-labels from different perspectives. To improve label accuracy, a fusion and voting strategy is applied across these architectures. The feature communication module (FCM) on each branch supports implicit feature-level integration, while a specialized voting module explicitly refines the pseudo-labels generated by each branch. The refined labels, denoted as y^\hat{y} , and their filtered versions, y^0\hat{y}^{0} and y^1\hat{y}^{1} , are used to jointly train both models, promoting synergistic learning across the architectures.
Figure 4: Pipeline of the FCM in the Transformer branch. This module is designed to facilitate feature communication between two heterogeneous segmentation models while introducing controlled perturbations. The FCM effectively aligns and integrates features from both models to enhance segmentation performance. f0\bm{f}^{0} and f1\bm{f}^{1} denote features from the two encoders, cf0\bm{cf}^{0} is the communicated feature in the Transformer branch, and λ0\lambda^{0} controls the strength of injected features. denotes feature convolution. denotes feature summation.
MethodVOC TrainVOC Aug
 1/16 (92)1/8 (183)1/4 (366)1/2 (732)Full (1464)1/16 (662)1/8 (1323)1/4 (2646)
Results of using ResNet-50 [ 58 ] as backbone    
Supervised44.052.361.765.972.960.767.771.9
U 2 PL †\dagger [ 13 ]-----72.075.176.2
UniMatch [ 51 ]71.972.576.077.478.775.876.976.8
AugSeg [ 49 ]64.272.276.277.478.874.776.077.1
Table 1: Comparison of mean intersection over union (mIoU) (%) with state-of-the-art (SOTA) methods on the PASCAL VOC 2012 [ 24 ] validation set with different backbones under various proportion protocols. “Supervised” denotes the supervised baseline without using unlabeled training data. The results of U 2 PL [ 13 ] (marked with †\dagger ) are obtained from Refs. [ 51 ] , and it follows the same partition protocol as ours. “VOC Train” refers to using only the 1464 finely annotated images from the train set, “VOC Aug” includes an additional 9118 weakly labeled images from the semantic boundaries dataset [ 56 ] . Bold numbers indicate the best performance under each setting.
 1/16 (186)1/8 (372)1/4 (744)1/2 (1488)
Supervised63.368.774.176.6
CPS [ 14 ]69.874.476.978.6
U 2 PL [ 13 ]69.073.076.377.2
AugSeg [ 49 ]73.776.578.879.3
Co-training [ 15 ]-76.377.1-
DAW [ 60 ]75.276.377.779.2
Table 2: Comparison of mIoU (%) with SOTA methods on the Cityscapes [ 25 ] validation set with ResNet-50 backbone under various partition protocols.
Figure 5: Qualitative results on the Cityscapes validation [ 25 ] set. From left to right, the categories are (a) input images, (b) ground truth labels annotated manually, (c) segmentation maps created by CPS, (d) segmentation maps produced by U 2 PL, and (e) segmentation maps generated by our HDVS framework. All models are trained following the 1/2 (1488) partition protocol on the Cityscapes train set.
 ConfigurationsmIoU
 Heterogeneous networksVoting moduleFCMCommunication branch14641323
Res-CPS ∗----78.876.9
Seg-CPS ∗----80.977.8
Res-C-CPS ∗--✓✓78.976.6
Seg-C-CPS ∗--✓✓80.777.4
Res-V-CPS ∗-✓--81.278.3
Table 3: Ablation study on the effectiveness of heterogeneous dual-branch voting supervision components. CPS ∗ replicates CPS [ 14 ] with aligned pseudo-label filtering and hyperparameters. Res-/Seg-CPS ∗ denote dual convolutional neural network/ dual Transformer baselines; H-CPS ∗ uses a heterogeneous CNN+Transformer pair. C-CPS ∗ augments the model with the FCM and its communication branch, while V-CPS ∗ adds the voting module for pseudo-label generation. HDVS combines both the feature communication and voting module. All results use a ResNet-50 backbone, and Mit-B2 [ 2 ] is paired with ResNet-50 in the heterogeneous setting. ✓\checkmark and “-” indicate that the corresponding component is enabled and disabled, respectively. FCM denotes the feature communication module.
Figure 6: tt -SNE [ 64 ] visualization of extracted features by CPS and HDVS. Both models are trained on the Cityscapes train set following the 1/2 (1488) partition protocol, and the features are extracted from the Cityscapes validation set for display. The first row shows the visualization results for all categories, while the second row presents the visualization analysis for eight selected categories. The first and second columns display the visualization results for the CPS and HDVS models, respectively.
Figure 7: Visualizations of segmentation maps generated by various models, (a) input images to (g) our HDVS model, showcasing progressive enhancements at each step. All models, including the homogeneous baselines Res-CPS ∗ /Seg-CPS ∗ (dual convolutional neural network / dual Transformer), utilizing ResNet-50 [ 58 ] as the backbone and employing a combined approach of ResNet-50 and Mit-B2 [ 2 ] for heterogeneous structures, are trained using the classic 1464 partition protocol on the PASCAL VOC 2012 train set. H-CPS ∗ adopts a heterogeneous CNN+Transformer pair, and H-V-CPS ∗ further introduces a voting module on top of H-CPS ∗ .
λ0\lambda^{0}λ1\lambda^{1}Performance (%)
1.0000.01082.4
0.1000.01083.1
0.0100.01082.5
0.0010.01082.0
0.1001.00081.9
0.1000.10082.3
Table 4: Ablation study on scaling coefficients λ0\lambda^{0} and λ1\lambda^{1} using a combination of ResNet-50 and Mit-B2 as the backbone within the classical partition of 1464 training images from the PASCAL VOC 2012 train dataset.
Figure 8: Effectiveness of the voting module under heterogeneous networks. (a) The performance of H-CPS ∗ and H-V-CPS ∗ , using a combination of ResNet-50 and Mit-B2 as the backbone, under different numbers of labeled images on the PASCAL VOC 2012 dataset. (b) Under the classical partition protocol with 183 training samples on the PASCAL VOC train dataset, the performance of H-CPS ∗ and H-V-CPS ∗ across different categories. mIoU denotes mean intersection over union.
 14641323Params (M)
H-CPS ∗81.478.1-
Cross-attention80.576.229.3
Concat + MLP82.178.37.0
Ours (FCM)83.179.32.9
Table 5: Cross-branch communication variants. We report mIoU on the PASCAL VOC 2012 train dataset under the classic 1464 and the blended 1/8 (1323) partitioning protocols, along with the parameter count of the feature communication components (M). H-CPS ∗ serves as the baseline without feature communication. Concat + multi-layer perceptron (MLP) concatenates peer features and projects them with an MLP. Cross-attention fuses features via cross-attention.
Drop ratio (%)010203040
mIoU78.381.883.182.279.4
Δ\mathrm{\Delta} vs. 20%−- 4.8−- 1.3-−- 0.9−- 3.7
Table 6: HDVS performance under different entropy-drop ratios. All experiments use a combination of ResNet-50 and MiT-B2 as the backbone, within the classical partition of 1464 training images from the PASCAL VOC 2012 train dataset. Δ\mathrm{\Delta} is relative to 20%.
ComponentParams (M)
Student branch (ResNet-50)43.6
Teacher branch (Mit-B2)27.4
FCM2.9
Voting module0.0
HDVS71.0
Table 7: Trainable parameter counts (M) of HDVS. The parameter count of the FCM is already included in the student/teacher branch totals and is reported separately for reference only.
Figure 9: Failure cases visualizations. From left to right, the categories are (a) input images, (b) ground truth annotations, and (c) segmentation maps generated by our HDVS framework. All models are trained using the 1/16 classical partition protocol on the PASCAL VOC 2012 train dataset.
More in this topic

Semantic Segmentation

Semantic segmentation assigns a class label to every image pixel, enabling detailed scene understanding across domains and sensors.

See all 414 papers

Explore similar work

CardsList
Apr 8, 2026

Accuracy Improvement of Semi-Supervised Segmentation Using Supervised ClassMix and Sup-Unsup Feature Discriminator

In semantic segmentation, the creation of pixel-level labels for training data incurs significant costs. To address this problem, semi-supervised learning, which utilizes a small number of labeled images alongside unlabeled images to enhance the performance, has gained attention. A conventional semi-supervised learning method, ClassMix, pastes class labels predicted from unlabeled images onto other images. However, since ClassMix performs operations using pseudo-labels obtained from unlabeled images, there is a risk of handling inaccurate labels. Additionally, there is a gap in data quality between labeled and unlabeled images, which can impact the feature maps. This study addresses these two issues. First, we propose a method where class labels from labeled images, along with the corresponding image regions, are pasted onto unlabeled images and their pseudo-labeled images. Second, we introduce a method that trains the model to make predictions on unlabeled images more similar to those on labeled images. Experiments on the Chase and COVID-19 datasets demonstrated an average improvement of 2.07% in mIoU compared to conventional semi-supervised learning methods.

Takahiro Mano, Reiji Saito, Kazuhiro Hotta

Meijo University, 1-501 Shiogamaguchi, Tempaku-ku, Nagoya 468-8502, Japan

Semi-Supervised Medical Image SegmentationData AugmentationMedical Image SegmentationRepresentation Alignment
AskPDF
Mar 6, 2026

A Semi-Supervised Framework for Breast Ultrasound Segmentation with Training-Free Pseudo-Label Generation and Label Refinement

Semi-supervised learning (SSL) has emerged as a promising paradigm for breast ultrasound (BUS) image segmentation, but it often suffers from unstable pseudo labels under extremely limited annotations, leading to inaccurate supervision and degraded performance. Recent vision-language models (VLMs) provide a new opportunity for pseudo-label generation, yet their effectiveness on BUS images remains limited because domain-specific prompts are difficult to transfer. To address this issue, we propose a semi-supervised framework with training-free pseudo-label generation and label refinement. By leveraging simple appearance-based descriptions (e.g., dark oval), our method enables cross-domain structural transfer between natural and medical images, allowing VLMs to generate structurally consistent pseudo labels. These pseudo labels are used to warm up a static teacher that captures global structural priors of breast lesions. Combined with an exponential moving average teacher, we further introduce uncertainty entropy weighted fusion and adaptive uncertainty-guided reverse contrastive learning to improve boundary discrimination. Experiments on four BUS datasets demonstrate that our method achieves performance comparable to fully supervised models even with only 2.5% labeled data, significantly outperforming existing SSL approaches. Moreover, the proposed paradigm is readily extensible: for other imaging modalities or diseases, only a global appearance description is required to obtain reliable pseudo supervision, enabling scalable semi-supervised medical image segmentation under limited annotations.

Ruili Li, Jiayi Ding, Ruiyu Li +7

Tohoku University Graduate School of Medicine, Sendai 980-8575, Japan · State Key Laboratory of Oncology in South China, Sun Yat-sen University Cancer Center, Guangzhou 510060, China · Southeast University, School of Cyber Science and Engineering Nanjing, Jiangsu, China +4

Ultrasound Image SegmentationSemi-Supervised Medical Image SegmentationPseudo-Labeling
AskPDF
Apr 25, 2026

SemiGDA: Generative Dual-distribution Alignment for Semi-Supervised Medical Image Segmentation

Semi-supervised learning addresses label scarcity and high annotation costs in medical image segmentation by exploiting the latent information in unlabeled data to enhance model performance. Traditional discriminative segmentation relies on segmentation masks, neglecting feature-level distribution constraints. This limits robust semantic representation learning and adaptive modeling of unlabeled data in scenarios with few labels. To address these limitations, we propose SemiGDA, a novel Generative Dual-distribution Alignment framework for semi-supervised medical image segmentation. Our SemiGDA overcomes the reliance of discriminative methods on large labeled datasets by aligning feature and semantic distributions to boost semantic learning and scene adaptability. Specifically, we propose a Dual-distribution Alignment Module (DAM), which employs two structurally distinct encoders to model image and mask feature distributions. It enforces their alignment in the latent space via distributional constraints, establishing structured feature consistency. Moreover, we design a Consistency-Driven Skip Adapter (CDSA) strategy, which introduces dual skip adapters (Image and Mask) to fuse multi-scale features via skip connections. Using a consistency loss, CDSA enhances cross-branch semantic alignment and reinforces fine-grained semantic consistency. Experimental results on diverse medical datasets show that our method outperforms other state-of-the-art semi-supervised segmentation methods. Code is released at: https://github.com/taozh2017/SemiGDA.

Kaiwen Huang, Yi Zhou, Yizhe Zhang +2

Nanjing University of Science and Technology, Nanjing, China · Southeast University, Nanjing, China

Distribution MatchingSemi-Supervised Medical Image SegmentationMedical Image SegmentationSemi-Supervised LearningMedical Image Representation Learning
AskPDF