Semi-Supervised Learning with Balanced Deep Representation Distributions

Changchun Li; Ximing Li; Bingjie Zhang; Wenting Wang; Jihong Ouyang

cs.LGMar 22, 2026

Semi-Supervised Learning with Balanced Deep Representation Distributions

Authors: Changchun Li, Ximing Li, Bingjie Zhang, Wenting Wang, Jihong Ouyang

Organizations: College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun 130012, China · Department of Computer Science, University of Texas at Dallas, Richardson, TX, USA

Abstract

Semi-Supervised Text Classification (SSTC) mainly works under the spirit of self-training. They initialize the deep classifier by training over labeled texts; and then alternatively predict unlabeled texts as their pseudo-labels and train the deep classifier over the mixture of labeled and pseudo-labeled texts. Naturally, their performance is largely affected by the accuracy of pseudo-labels for unlabeled texts. Unfortunately, they often suffer from low accuracy because of the margin bias problem caused by the large difference between representation distributions of labels in SSTC. To alleviate this problem, we apply the angular margin loss, and perform several Gaussian linear transformations to achieve balanced label angle variances, i.e., the variance of label angles of texts within the same label. More accuracy of predicted pseudo-labels can be achieved by constraining all label angle variances balanced, where they are estimated over both labeled and pseudo-labeled texts during self-training loops. With this insight, we propose a novel SSTC method, namely Semi-Supervised Text Classification with Balanced Deep representation Distributions (S2TC-BDD). We implement both multi-class classification and multi-label classification versions of S2TC-BDD by introducing some pseudo-labeling tricks and regularization terms. To evaluate S2 TC-BDD, we compare it against the state-of-the-art SSTC methods. Empirical results demonstrate the effectiveness of S2 TC-BDD, especially when the labeled texts are scarce.

Figures & tables

Fig. 1: The average difference of label angle variances (Avg.DLAV) computed in semi-supervised and supervised manners across AG News ( Multi-Class Case ) and AAPD ( Multi-Label Case ), respectively.

Fig. 2: Let solid circles and triangles denote labeled positive and negative texts, and hollow ones denote corresponding unlabeled texts. (a) The large difference between label angle variances results in the margin bias. Many unlabeled texts (in red) can be misclassified. (b) Balancing the label angle variances can eliminate the margin bias. Best viewed in color.

Fig. 3: Overview the framework of S 2 tc-bdd . Specially, we employ the pseudo-labeling tricks of Sharpening and Class-distribution-Aware Pseudo-labeling (CAP) for the scenarios of multi-class classification and multi-label classification, respectively. Best viewed in color.

Type	Dataset	$K$	$N$	#Dev	#Test	Avg.Labels
Multi- class classification	AG News	4	120,000	8,000	7,600	–
	Yelp	5	650,000	10,000	50,000	–
	Yahoo	10	700,000	20,000	60,000	–
Multi- label classification	Ohsumed	23	24,089	2,000	10,300	1.66
	AAPD	54	54,840	2,000	1,000	2.41
	RCV1-V2	103	23,149	2,000	781,265	3.24

TABLE I: Statistics of datasets.

K

: the number of class labels.

N

: the number of original training texts. #Dev : the number of development texts. #Test : the number of texts for testing. In addition, we report the average number of category labels per document ( Avg.Labels ) for multi-label benchmarks.

Metric	Dataset	$N_{l}$	NB+EM	BERT+CE	BERT+AM -s	BERT+AM -f	VAMPIRE	VAT	UDA	FreeMatch	MetaExpert	S 2 tc-bdd-s	S 2 tc-bdd-f
Micro-F1	AG News	100	0.834	0.839	0.856	0.870	0.705	0.868	0.855	0.862	0.867	0.872	0.876
		1,000	0.855	0.878	0.879	0.886	0.833	0.886	0.883	0.882	0.884	0.889	0.889
		10,000	0.874	0.905	0.901	0.903	0.876	0.898	0.906	0.915	0.914	0.907	0.917
	Yelp	100	0.300	0.344	0.399	0.485	0.227	0.244	0.387	0.491	0.454	0.417	0.496
		1,000	0.355	0.538	0.544	0.567	0.476	0.551	0.554	0.565	0.585	0.552	0.589
		10,000	0.404	0.583	0.574	0.596	0.551	0.566	0.580	0.618	0.616	0.583	0.619

TABLE II: Experimental results of Micro-F1 and Macro-F1 by varying the number of labeled texts

N_{l}

on multi-class benchmarks. The best results are highlighted in boldface.

Metric	Dataset	$N_{u}$	NB+EM	BERT+AM -s	BERT+AM -f	VAMPIRE	VAT	UDA	FreeMatch	MetaExpert	S 2 tc-bdd-s	S 2 tc-bdd-f
Micro-F1	AG News	0	0.668	0.844	0.798	–	0.846	0.839	0.859	0.842	0.844	0.750
		200	0.696	0.855	0.847	0.329	0.850	0.844	0.854	0.848	0.857	0.835
		2,000	0.752	0.856	0.866	0.421	0.870	0.853	0.845	0.859	0.863	0.874
		20,000	0.834	0.856	0.870	0.705	0.868	0.855	0.862	0.867	0.872	0.876
	Yelp	0	0.317	0.381	0.377	–	0.341	0.344	0.463	0.437	0.395	0.373
		200	0.307	0.385	0.420	0.238	0.299	0.397	0.299	0.410	0.403	0.420

TABLE III: Experimental results of Micro-F1 and Macro-F1 by varying the number of unlabeled texts

N_{u}

on multi-class benchmarks.

Metric	Dataset	$N_{l}$	BERT+BCE	BERT+AM	SDRL	CAP	MetaExpert	S 2 tc-bdd
Micro-F1	Ohsumed	100	0.278	0.421	0.311	0.380	0.418	0.447
		1,000	0.583	0.604	0.397	0.571	0.558	0.636
		10,000	0.695	0.721	0.443	0.645	0.644	0.726
	AAPD	200	0.456	0.543	0.361	0.497	0.525	0.554
		2,000	0.623	0.637	0.416	0.625	0.601	0.651
		20,000	0.678	0.707	0.440	0.627	0.610	0.707

TABLE IV: Experimental results of Micro-F1 and Macro-F1 by varying the number of labeled texts

N_{l}

on multi-label benchmarks. The best results are highlighted in boldface.

Metric	Dataset	$N_{l}$	BERT+BCE	BERT+AM	SDRL	CAP	MetaExpert	S 2 tc-bdd
Ranking Loss	Ohsumed	100	0.246	0.208	0.249	0.199	0.192	0.184
		1,000	0.094	0.099	0.205	0.089	0.091	0.083
		10,000	0.045	0.049	0.172	0.045	0.049	0.049
	AAPD	200	0.121	0.110	0.196	0.096	0.096	0.105
		2,000	0.063	0.061	0.160	0.051	0.073	0.056
		20,000	0.037	0.035	0.137	0.037	0.041	0.038

TABLE V: Experimental results of the ranking loss and average precision (AP) by varying the number of labeled texts

N_{l}

on multi-label benchmarks. The best results are highlighted in boldface.

Metric	Dataset	$N_{u}$	BERT+AM	SDRL	CAP	MetaExpert	S 2 tc-bdd
Micro-F1	Ohsumed	0	0.415	0.308	0.373	0.389	0.433
		100	0.414	0.316	0.370	0.350	0.424
		1,000	0.415	0.310	0.378	0.371	0.435
		10,000	0.421	0.311	0.380	0.418	0.447
	AAPD	0	0.531	0.372	0.486	0.491	0.535
		200	0.534	0.371	0.488	0.505	0.539

TABLE VI: Experimental results of Micro-F1 and Macro-F1 by varying the number of unlabeled texts

N_{u}

on multi-label benchmarks. The best results are highlighted in boldface.

Metric	Dataset	$N_{u}$	BERT+AM	SDRL	CAP	MetaExpert	S 2 tc-bdd
Ranking Loss	Ohsumed	0	0.195	0.253	0.213	0.234	0.192
		100	0.200	0.251	0.206	0.205	0.189
		1,000	0.198	0.256	0.207	0.205	0.187
		10,000	0.208	0.249	0.199	0.192	0.184
	AAPD	0	0.103	0.193	0.112	0.116	0.104
		200	0.106	0.190	0.110	0.107	0.102

TABLE VII: Experimental results of the ranking loss and average precision (AP) by varying the number of unlabeled texts

N_{u}

on multi-label benchmarks. The best results are highlighted in boldface.

Model	Multi-class ( -s )		Multi-class ( -f )		Multi-label
	Micro-F1	Macro-F1	Micro-F1	Macro-F1	Micro-F1	Macro-F1	Ranking Loss	AP
S 2 tc-bdd	0.872	0.872	0.876	0.876	0.447	0.308	0.184	0.560
-regularization	0.863	0.864	0.874	0.873	0.428	0.305	0.195	0.544
-BDD	0.856	0.856	0.870	0.870	0.421	0.298	0.208	0.535
-unlabeled texts	0.844	0.843	0.750	0.751	0.433	0.266	0.192	0.544
-all	0.839	0.840	0.839	0.840	0.384	0.222	0.220	0.494

TABLE VIII: Classification performance on the multi-class text classification benchmark AG News with 100 labeled data and 20,000 unlabeled data, and the multi-label text classification benchmark Ohsumed with 100 labeled data and 10,000 unlabeled data after removing different parts of S 2 tc-bdd .

Fig. 4: The accuracy of pseudo-labels during the training procedure with or without BDD loss (w/ BDD and w/o BDD) across AG News ( Multi-Class Case , $N_{l}=100,N_{u}=20000$ ) and AAPD ( Multi-Label Case , $N_{l}=200,N_{u}=20000$ ), respectively.

Semi-Supervised Learning

Semi-supervised learning uses both labeled and unlabeled data to improve prediction when reliable annotations are scarce.

See all 164 papers

Beyond Distribution Estimation: Simplex Anchored Structural Inference Towards Universal Semi-Supervised Learning
May 8, 2026Yaxin Hou, Jun Ma, Hanyang Li +3Graph-Based Semi-Supervised Learning Pseudo-Labeling
School of Computer Science and Engineering, Southeast University, Nanjing, China. · School of Electrical Engineering, Southeast University, Nanjing, China. · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, Nanjing, China.
Leveraging Label Proportion Prior for Class-Imbalanced Semi-Supervised Learning
Mar 3, 2026Kohki Akiba, Shinnosuke Matsuo, Shota Harada +1Semi-Supervised Learning Class-Imbalanced Learning
Kyushu University, Fukuoka, Japan
Learning Dynamics of Logits Debiasing for Long-Tailed Semi-Supervised Learning
Aug 31, 2026Yue Cheng, Jiajun Zhang, Xiaohui Gao +2Class-Imbalanced Learning Semi-Supervised Learning
Beijing Jiaotong University · AntGroup · Northwestern Polytechnical University +1

May 8, 2026J/K move · Enter open · S save

Beyond Distribution Estimation: Simplex Anchored Structural Inference Towards Universal Semi-Supervised Learning

Yaxin Hou, Jun Ma, Hanyang Li, Bo Han+2

School of Computer Science and Engineering, Southeast University, Nanjing, China. · School of Electrical Engineering, Southeast University, Nanjing, China. · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, Nanjing, China.

Graph-Based Semi-Supervised Learning Pseudo-Labeling Semi-Supervised Learning

Open paper AskPDF arXiv

Semi-supervised learning faces significant challenges in realistic scenarios where labeled data is scarce and unlabeled data follows unknown, arbitrary distributions. We formalize this critical yet under-explored paradigm as Universal Semi-supervised Learning (UniSSL). Existing methods typically leverage unlabeled data via pseudo-labeling. However, they often rely on the idealized assumption of a uniform unlabeled data distribution or require sufficient labeled data to estimate it. In the UniSSL setting, such dependencies lead to numerous erroneous pseudo-labels, thereby triggering representation confusion. Fortunately, we observe that inter-sample relations captured by representations are more reliable than pseudo-labels. Leveraging this insight, we shift our focus to representation-level structural inference to bypass distribution estimation. Accordingly, we propose Simplex Anchored Graph-state Equipartition (SAGE), which captures high-order inter-sample dependencies to establish structural consensus for guiding representation learning. Meanwhile, to mitigate representation confusion, we employ vectors that satisfy a simplex equiangular tight frame to serve as a coordinate frame for guiding inter-class representation separation. Finally, we introduce a weighting strategy based on distribution-agnostic metrics to prioritize reliable pseudo-labels and an auxiliary branch to isolate potentially erroneous pseudo-labels. Evaluations on five standard benchmarks show that SAGE consistently outperforms state-of-the-art methods, with an average accuracy gain of

\textbf{8.52%}

Abstract

Semi-Supervised Learning

Explore similar work

Beyond Distribution Estimation: Simplex Anchored Structural Inference Towards Universal Semi-Supervised Learning

Leveraging Label Proportion Prior for Class-Imbalanced Semi-Supervised Learning

Learning Dynamics of Logits Debiasing for Long-Tailed Semi-Supervised Learning