Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student

Haorong Han; Jidong Yuan; Chixuan Wei; Yongqi Sun

cs.LGOct 7, 2026

Decoupled Optimization for Teacher-Student Semi-Supervised Learning via a Pioneer Student

Authors: Haorong Han, Jidong Yuan, Chixuan Wei, Yongqi Sun

Organizations: School of Computer Science and Technology Beijing Jiaotong University · School of Data Science and Intelligent Media Communication University of China

Abstract

Semi-supervised learning (SSL) relies on two core mechanisms: self-training under the Teacher-Student (T-S) framework and joint optimization of labeled and unlabeled losses. Despite their effectiveness, we find both mechanisms introduce distinct optimization pathologies. First, parameter coupling enforces strict synchronization between teacher and student, where strong regularization on the student degrades the teacher's fitting ability, thereby limiting the permissible generalization intensity. Second, the imbalance in gradient update consistency between labeled and unlabeled losses drives the shared parameters to prematurely converge to labeled-dominated local minima, creating a bottleneck for global optimization. To address both issues, we propose the Pioneer Student (PiS), an auxiliary branch that operates in an independent parameter space and periodically transfers accumulated knowledge back to the T-S model. Extensive experiments show that PiS is a universal plug-and-play module that consistently improves mainstream SSL methods.

Figures & tables

Supervision Type	Augmentation	Acc. (%)	Pseudo-Label Acc. (%)
Endogenous	RA(2,3)	78.13	77.97
	RA(3,5)	78.46	77.92
	RA(5,10)	76.27	75.25
Exogenous	RA(5,10)	78.80	77.92

Table 1 : Performance of FreeMatch [ 7 ] on CIFAR-100 [ 8 ] (2 labels/class) under varying augmentation intensities. Online pseudo-labels generated by the synchronized teacher are endogenous, whereas fixed ones pre-recorded offline (e.g., from an RA(3,5) run) are exogenous.

Panel of Figure 1 — Figure 1 : UC of labeled and unlabeled losses.

Figure 2 : Visualization of loss landscapes: FreeMatch vs ours (FreePiS). Landscapes are obtained by perturbing the model parameters (landscape origin) along two orthogonal directions. $\mathcal{L}_{l}$ and $\mathcal{L}_{u}$ denote the labeled and unlabeled loss values at the origin, respectively, while $\mathcal{S}$ measures landscape sharpness, which reflects generalization potential (see Appendix B for formal definition).

Figure 3 : Overview of T–S model integrated with Pioneer Student . While maintaining the standard T–S logic, where the teacher $f_{\theta_{t}}$ processes weakly augmented image $\alpha(x_{u})$ to generate pseudo-label $q_{u}$ as supervision for the student’s prediction $p_{s}$ on strongly augmented sample $\mathcal{A}(x_{u})$ , we introduce a parameter-independent PiS model $f_{\theta_{\pi}}$ . The PiS accepts extremely augmented image $\mathbb{A}(x_{u})$ and aligns its prediction $p_{\pi}$ with the teacher’s supervision. Periodically, the parameters of the T–S model $\theta_{s}$ are overwritten by the parameters of the pioneer student $\theta_{\pi}$ .

Baseline Methods
Modality	CV						Audio		NLP
Dataset	CIFAR-100		STL-10		EuroSAT		UrbanSound8K		ACL-IMDb
# Labels per Class	2	4	4	10	2	4	10	40	10	50
FlexMatch [ 14 ]	73.24 $\pm$ 1.12	81.76 $\pm$ 0.36	85.60 $\pm$ 3.11	91.83 $\pm$ 0.78	94.83 $\pm$ 0.57	94.42 $\pm$ 0.81	60.77 $\pm$ 0.96	76.30 $\pm$ 1.66	92.18 $\pm$ 0.77	92.59 $\pm$ 0.38
CoMatch [ 16 ]	64.92 $\pm$ 0.69	74.77 $\pm$ 0.50	84.88 $\pm$ 1.88	90.44 $\pm$ 1.35	94.25 $\pm$ 0.43	95.19 $\pm$ 1.05	70.37 $\pm$ 1.15	79.81 $\pm$ 2.34	92.56 $\pm$ 0.30	92.38 $\pm$ 1.14
SimMatch [ 33 ]	76.22 $\pm$ 1.08	82.94 $\pm$ 0.78	88.23 $\pm$ 3.20	92.45 $\pm$ 1.86	92.34 $\pm$ 0.60	94.73 $\pm$ 0.89	63.37 $\pm$ 6.10	80.05 $\pm$ 2.11	92.07 $\pm$ 0.55	92.92 $\pm$ 0.33

Table 2 : Top-1 accuracy (%) across CV, audio, and NLP benchmarks. We report the mean and standard deviation over three runs.

{\color[rgb]{0.1953,0.8047,0.1953}\uparrow}

indicates performance improvement after integrating PiS. Bold and underline denote the best and second-best results, respectively. ∗ denotes our re-implementation. RegMixMatch results are omitted for audio and NLP due to Mixup incompatibility in these modalities.

Panel of Figure 4 — Figure 4 : PiS accelerates convergence by reaching the same performance with significantly fewer training iterations .

Dataset	ImageNet
# Labels per Class	10	100
FreeMatch	54.69	72.57
RegMixMatch	58.35	73.66
FreePiS	60.02	66.69
FreePiS †	–	73.24

Table 3 : Top-1 accuracy (%) on ImageNet. † denotes the FreePiS w/ parameter update sharing.

Panel of Figure 5 — Figure 5 : Comparative study of model performance under different data scales and regularization intensities. (a–b) Performance gains across varying scales of labeled and unlabeled data. (c–d) Teacher’s pseudo-label reliability and final accuracy under different augmentation intensities and dropout rates [ 38 ] (see Appendix F for numerical results; see Section 5.4 for augmentation details).

ID	Method/Variant	Mechanisms			FGVC		CIFAR-100
ID	Method/Variant	Anti-FGD	Anti-LDT	Warm-up	Acc (%)	$\Delta$	Acc (%)	$\Delta$
( a )	FreeMatch	✗	✗	–	51.65	–	78.46	–
( b )	FreePiS w. Label Loss	✓	✗	✓	54.66	+3.01	79.02	+0.56
( c )	FreePiS w. Same Aug.	✗	✓	✓	52.78	+1.13	80.57	+2.11
( d )	FreePiS w/o Warm-up	✓	✓	✗	52.25	+0.60	80.66	+2.20
( e )	FreePiS (Full)	✓	✓	✓	55.54	+3.89	80.83	+2.37

Table 4 : Ablation study on disentangling the dual advantages of FreePiS. We evaluate two primary mechanisms: addressing the FGD (Anti-FGD) and overcoming the LDT (Anti-LDT). ✓ indicates the mechanism is activated .

\Delta

represents the performance gain relative to the baseline (a). Experiments are conducted on FGVC with 5 labels per class and CIFAR-100 with 2 labels per class.

Panel of Figure 6 — Figure 6 : Sensitivity analysis of FreePiS hyperparameters. (a) PiS’s data augmentation intensity $N$ on FGVC (5 labels). (b) Parameter overwrite interval on CIFAR-100 (2 labels).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7 : Gradient norms of labeled and unlabeled losses.

Dataset	CIFAR-100	STL-10	EuroSAT	FGVC-Aircraft	ImageNet-1K
Image Size	32	96	32	224	224
Backbone (ViT [ 41 ] )	S-P2-32	B-P16-96	S-P2-32	B-P16-224	B-P16-224
Weight Decay	5e-4	5e-4	5e-4	5e-2	5e-2
Learning Rate	5e-4	1e-4	5e-5	5e-4	3e-4
Layer Decay Rate	0.5	0.95	1.0	0.5	0.5
Labeled Batch Size	8				32

Appendix

Table 5 : Hyperparameter configurations and backbones for different datasets.

Method	Time (s/iter)	+PiS	Memory (MiB)	+PiS
FixMatch	0.0944	0.1256 (+33%)	3359	4534 (+35%)
FreeMatch	0.0960	0.1306 (+36%)	3341	4543 (+36%)
RegMixMatch	0.1344	0.1613 (+20%)	4616	5446 (+18%)

Appendix

Table 6 : Computational overhead of PiS on a single NVIDIA RTX 3090 GPU. We report per-iteration training time and peak GPU memory consumption for baseline methods and their PiS-augmented variants.

Dataset	CIFAR-100		STL-10		EuroSAT
# Labels per Class	2	4	4	10	2	4
FreeMatch (200k iters)	78.46	83.58	87.20	91.51	93.48	94.20
FreePiS (147k iters)	80.61	84.27	90.44	92.85	94.70	95.84
FreePiS (200k iters)	80.83	84.45	90.65	93.13	94.92	95.89

Appendix

Table 7 : Performance comparison under equal wall-clock training time. FreePiS is trained for fewer iterations to match the total training time of FreeMatch (200k iterations).

Method	9% ‡	50% ‡	71%	80%	85%
Method	( $N_{u}=1$ )	( $N_{u}=10$ )	( $N_{u}=25$ )	( $N_{u}=40$ )	( $N_{u}=56$ )
FreeMatch	56.45	59.88	63.01	65.66	65.87
RegMixMatch	56.86	60.13	63.55	66.19	66.40
FreePiS	56.56	61.24	64.72	68.35	70.33

Appendix

Table 8 : Absolute classification accuracy (%) with a fixed labeled count (

N_{l}=10

). ‡ denotes that parameter update sharing is enabled.

Method	50% ‡	38% ‡	31% ‡	23%	13%
Method	( $N_{l}=33$ )	( $N_{l}=20$ )	( $N_{l}=15$ )	( $N_{l}=10$ )	( $N_{l}=5$ )
FreeMatch	83.34	74.77	70.39	62.87	49.65
RegMixMatch	84.10	75.24	71.27	63.40	47.89
FreePiS	84.56	76.26	73.11	67.33	54.54

Appendix

Table 9 : Absolute classification accuracy (%) with a fixed unlabeled count (

N_{u}=33

). ‡ denotes that parameter update sharing is enabled.

Metric	Method	Augmentation Intensity $N$ ( $M=2N$ )						Dropout Rate
Metric	Method	3	4	5	6	7	8	0.10	0.15	0.20	0.25	0.30
Accuracy (%)	FreeMatch	80.44	80.02	78.65	77.96	77.00	76.65	80.60	80.71	80.44	79.08	77.98
Accuracy (%)	FreePiS	80.91	81.66	82.18	82.03	81.11	80.80	81.49	82.06	82.18	81.23	80.77
Reliability (%)	FreeMatch	79.24	78.09	76.19	74.45	73.17	72.68	79.95	80.01	79.24	77.09	75.35
Reliability (%)	FreePiS	80.22	81.37	82.67	82.22	81.39	80.36	81.36	82.53	82.67	81.66	81.00

Appendix

Table 10 : Tabulated results for classification accuracy and pseudo-label reliability under varying augmentation intensities

N

and dropout rates.

Method	Sync Mechanism	Accuracy (%)
FreeMatch	—	78.46
FreePiS + LI- $\alpha$ =0.3	Linear Interp. ( $\alpha$ =0.3)	78.80
FreePiS + LI- $\alpha$ =0.5	Linear Interp. ( $\alpha$ =0.5)	79.34
FreePiS + LI- $\alpha$ =0.7	Linear Interp. ( $\alpha$ =0.7)	80.19
FreePiS + HO	Hard Overwrite ( $\alpha$ =1.0)	80.83

Appendix

Table 11 : Classification accuracy (%) on CIFAR-100 under different synchronization mechanisms. All PiS variants use the same stage proportion

\rho=2.5\%

and warm-up

\phi=8\%

Asset	Modality	Type	License / Terms	Reference
Datasets
CIFAR-10/100	CV	Dataset	Non-commercial research use	[ 8 ]
STL-10	CV	Dataset	Non-commercial research use	[ 28 ]
ImageNet-1K	CV	Dataset	ImageNet Terms of Access (non-commercial use)	[ 37 ]
EuroSAT	CV	Dataset	MIT License	[ 34 ]
FGVC-Aircraft	CV	Dataset	Non-commercial research use	[ 39 ]

Appendix

Table 12 : Summary of datasets and codebases with usage terms.

Explore similar work

Beyond Distribution Estimation: Simplex Anchored Structural Inference Towards Universal Semi-Supervised Learning
May 8, 2026Yaxin Hou, Jun Ma, Hanyang Li +3Semi-Supervised Learning Unlabeled Data
School of Computer Science and Engineering, Southeast University, Nanjing, China. · School of Electrical Engineering, Southeast University, Nanjing, China. · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, Nanjing, China.
MTSSL: Meta-Thresholding Semi-Supervised Learning
Jul 17, 2026Shuyang Liu, Ziang Zeng, Ruiqiu Zheng +4Semi-Supervised Learning Meta-Learning
School of Statistics, East China Normal University, Shanghai 200062, China
Self-Supervised Visual Representation Learning: Pretrain-Finetuning or Joint Training?
Jul 14, 2026Nusrat Munia, Tyler Ward, Nishat Nayla +2Self-Supervised Learning Self-Supervised Representations
Department of Computer Science University of Kentucky Lexington, KY 40506, USA · Kentucky Geological Survey University of Kentucky Lexington, KY 40506, USA

May 8, 2026, cs.LGJ/K move · Enter open · S save

Beyond Distribution Estimation: Simplex Anchored Structural Inference Towards Universal Semi-Supervised Learning

Yaxin Hou, Jun Ma, Hanyang Li, Bo Han+2

School of Computer Science and Engineering, Southeast University, Nanjing, China. · School of Electrical Engineering, Southeast University, Nanjing, China. · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, Nanjing, China.

Semi-Supervised Learning Unlabeled Data

Open paper AskPDF arXiv

Semi-supervised learning faces significant challenges in realistic scenarios where labeled data is scarce and unlabeled data follows unknown, arbitrary distributions. We formalize this critical yet under-explored paradigm as Universal Semi-supervised Learning (UniSSL). Existing methods typically leverage unlabeled data via pseudo-labeling. However, they often rely on the idealized assumption of a uniform unlabeled data distribution or require sufficient labeled data to estimate it. In the UniSSL setting, such dependencies lead to numerous erroneous pseudo-labels, thereby triggering representation confusion. Fortunately, we observe that inter-sample relations captured by representations are more reliable than pseudo-labels. Leveraging this insight, we shift our focus to representation-level structural inference to bypass distribution estimation. Accordingly, we propose Simplex Anchored Graph-state Equipartition (SAGE), which captures high-order inter-sample dependencies to establish structural consensus for guiding representation learning. Meanwhile, to mitigate representation confusion, we employ vectors that satisfy a simplex equiangular tight frame to serve as a coordinate frame for guiding inter-class representation separation. Finally, we introduce a weighting strategy based on distribution-agnostic metrics to prioritize reliable pseudo-labels and an auxiliary branch to isolate potentially erroneous pseudo-labels. Evaluations on five standard benchmarks show that SAGE consistently outperforms state-of-the-art methods, with an average accuracy gain of

\textbf{8.52%}

Abstract

Explore similar work

Beyond Distribution Estimation: Simplex Anchored Structural Inference Towards Universal Semi-Supervised Learning

MTSSL: Meta-Thresholding Semi-Supervised Learning

Self-Supervised Visual Representation Learning: Pretrain-Finetuning or Joint Training?