FedBST: Bilateral Self-Training for Unsupervised Federated Classification

Aijing Yu; Kuangpu Guo; Jian Liang; Xiao-Yu Zhang

cs.CVOct 22, 2026

FedBST: Bilateral Self-Training for Unsupervised Federated Classification

Authors: Aijing Yu, Kuangpu Guo, Jian Liang, Xiao-Yu Zhang

Organizations: School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Science and Technology of China, Hefei, China ·

Abstract

Unsupervised federated learning with Vision-Language Models (e.g., CLIP) offers a promising solution for label-free classification, yet its performance often degrades under severe non-IID shifts. A pioneering method, FedCoPL, typically relies on self-training strategies that exclusively prioritize high-confidence predictions for pseudo-labeling. However, this selective approach neglects the information of ambiguous samples (often referred to as ``waist" samples), leading to the underutilization of data and biased decision boundaries. To address this limitation, we propose FedBST, a bilateral self-training framework that effectively exploits both high-confidence and ambiguous data. Specifically, for local optimization, we introduce uncertainty-aware confidence penalization (UACP). UACP targets ambiguous samples by penalizing overconfident predictions to implicitly promote higher uncertainty, thereby preventing erroneous pseudo-labeling. At inference, we employ a visual-text ensemble strategy. This approach aggregates uncertainty-aware local visual prototypes and ensembles them with textual prototypes, achieving lightweight adaptation to local distributions via prototype-based calibration. Experiments demonstrate that FedBST achieves consistent improvements over existing methods with standard federated aggregation protocols.

Figures & tables

Fig. 1: Overview of the proposed FedBST framework. The method consists of two phases: (Left) Local Bilateral Self-Training: Rather than using only high-confidence samples (blue), FedBST additionally leverages ambiguous “waist” samples (orange) for regularization. An Uncertainty-Aware Confidence Penalization (UACP) term suppresses the maximum predicted probability of these samples to reduce confirmation bias. (Right) Visual-Text Ensemble: At inference, global textual semantics are combined with local visual prototypes. The visual stream is restricted to locally observed classes (hollow dots indicate absent classes), while unobserved classes are handled by the textual stream, ensuring coverage of the full label space.

Method	PL	DTD		RESISC45		CUB		UCF101		CIFAR10		CIFAR100		Average
Method	PL	$\beta=0.10$	$\beta=0.05$	$\beta=0.10$	$\beta=0.05$	$\beta=0.10$	$\beta=0.05$	$\beta=0.10$	$\beta=0.05$	$\beta=0.10$	$\beta=0.05$	$\beta=0.10$	$\beta=0.05$	$\beta=0.10$	$\beta=0.05$
CLIP	-	43.24	43.24	54.51	54.51	51.28	51.28	61.00	61.00	86.93	86.93	64.17	64.17	60.19	60.19
PromptFL [ 5 ]	FPL	45.79	44.62	59.76	58.05	47.29	46.16	64.39	62.96	87.51	85.02	63.26	62.56	61.33	59.90
PromptFL [ 5 ]	CPL	44.84	46.32	62.52	60.99	48.72	48.86	63.86	64.97	88.90	89.14	66.04	65.14	62.48	62.57
Promptprox [ 11 ]	FPL	45.15	43.88	59.36	58.59	47.04	47.01	62.91	61.27	87.13	85.62	62.77	64.12	60.73	60.08
Promptprox [ 11 ]	CPL	43.51	45.85	59.99	62.35	49.25	48.65	64.55	63.94	90.44	90.07	65.86	66.15	62.27	62.84

TABLE I: Accuracy (%) of different methods under Dirichlet-based label skews with ViT/B-32 as the backbone. FPL [ 17 ] , CPL [ 24 ] , and CoPL [ 4 ] are adopted as baseline pseudo labeling ( PL ) method. Bold and underline represent the best and second-best results.

Method	PL	DTD	RESISC45	CUB	UCF101	CIFAR10	CIFAR100	Average
CLIP	-	43.24	54.51	51.28	61.00	86.93	64.17	60.19
PromptFL [ 5 ]	FPL	43.53	57.53	46.66	62.35	88.29	63.94	60.38
PromptFL [ 5 ]	CPL	43.82	61.11	47.31	64.23	88.70	67.23	62.07
Promptprox [ 11 ]	FPL	44.89	56.71	48.74	61.80	85.69	64.01	60.31
Promptprox [ 11 ]	CPL	45.53	60.77	47.61	63.86	89.21	66.49	62.25
pFedPrompt [ 6 ]	FPL	45.09	60.46	47.71	65.39	87.54	62.82	61.50

TABLE II: Accuracy (%) of different methods under quantity-based label skews with ViT/B-32 as the backbone. FPL [ 17 ] , CPL [ 24 ] , and CoPL [ 4 ] are adopted as baseline pseudo labeling ( PL ) method. Bold and underline represent the best and second-best results.

Fig. 2: Results of experiments with various client numbers and different client joining rates under Dirichlet-based label skews (

\beta=0.1

Method	PL	DTD	RESISC45	CUB	UCF101
CLIP	-	42.87	56.61	55.16	65.13
PromptFL [ 5 ]	FPL	44.36	61.36	51.95	64.60
PromptProx [ 11 ]	FPL	47.12	61.01	49.44	65.31
pFedPrompt [ 6 ]	FPL	47.65	63.18	50.19	65.19
FedOPT [ 10 ]	FPL	46.52	58.94	49.67	65.68
FedCoPL [ 4 ]	CoPL	56.83	77.50	59.78	78.29

TABLE III: Experimental results (%) using CLIP ViT-B/16 as backbone under Dirichlet-based label skew (

\beta=0.1

Method	PL	DTD	RESISC45	UCF101	CIFAR10
CLIP	-	41.62	52.12	65.13	74.80
PromptFL [ 5 ]	FPL	42.36	56.16	62.90	78.01
PromptProx [ 11 ]	FPL	43.12	56.54	64.37	78.77
pFedPrompt [ 6 ]	FPL	43.65	57.13	64.97	79.69
FedOPT [ 10 ]	FPL	45.52	58.94	65.54	80.13
FedCoPL [ 4 ]	CoPL	51.83	70.78	76.48	84.85

TABLE IV: Experimental results (%) using CLIP RN50 as base model under Dirichlet-based label skews (

\beta=0.1

) across four datasets.

$\lambda$ ( $\times e^{-3}$ )	0.4	0.6	0.8	1.0	1.2	1.4	1.6
DTD	60.24	60.10	60.71	61.46	60.73	60.57	60.79
RESICS45	78.23	78.01	78.20	77.88	77.60	77.91	78.05
$\alpha$	0.60	0.65	0.70	0.75	0.80	0.85	0.90
DTD	61.31	61.26	61.15	61.31	61.46	61.36	61.19
RESICS45	78.08	77.97	77.91	77.93	77.88	77.75	77.71

TABLE V: Performance (%) of FedBST under different values of hyperparameter

\tau_{con}

and

\lambda

with Dirichlet-based label skews.

$\tau_{con}$	0.10	0.15	0.20	0.25	0.30
DTD	61.21	61.45	61.46	61.62	61.60
RESICS45	78.56	78.10	77.88	77.39	77.30
$\tau_{ent}$	0.40	0.45	0.50	0.55	0.60
DTD	61.39	61.50	61.46	61.55	61.21
RESICS45	78.33	78.04	77.88	77.39	76.87

TABLE VI: Performance (%) of FedBST under different values of hyperparameter

\tau_{con}

and

\tau_{ent}

UACP	VTE	DTD	RESISC45	CUB	CIFAR100
-	-	60.89	75.76	56.09	73.59
✓	-	61.22	77.34	56.89	74.87
-	✓	61.10	76.77	56.54	74.41
✓	✓	61.46	77.88	57.25	75.01

TABLE VII: Ablation results (%) comparing the effects of Uncertainty-Aware Confidence Penalization ( UACP ) and Visual-Text Ensemble ( VTE ).

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 961 papers

Class-wise Contribution Estimation via Logit Maximization for Federated Learning
May 17, 2026Asim Ukaye, Nurbek Tastan, Mubarak Abdu-Aguye +1Class-Imbalanced Learning Federated Learning Aggregation
Mohamed bin Zayed University of Artificial Intelligence, UAE · Michigan State University, USA
FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization
Oct 16, 2026Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry +2Prompt Learning for VLMs Test-Time Prompt Tuning
Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia
FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models
May 27, 2026Xucong Wang, Pengkun Wang, Zhe Zhao +3Multi-Label Classification VLM Adaptation
University of Science and Technology of China (USTC) · Suzhou Institute for Advanced Research, USTC · City University of Hong Kong

May 17, 2026J/K move · Enter open · S save

Class-wise Contribution Estimation via Logit Maximization for Federated Learning

Asim Ukaye, Nurbek Tastan, Mubarak Abdu-Aguye, Karthik Nandakumar

Mohamed bin Zayed University of Artificial Intelligence, UAE · Michigan State University, USA

Class-Imbalanced Learning Federated Learning Aggregation Non-IID Federated Learning Federated Learning

Open paper AskPDF arXiv

Federated learning (FL) enables collaborative learning of computer vision models, where privacy and regulatory constraints prevent centralizing data across devices or organizations. However, practical FL deployments often exhibit severe class imbalance and label skew, causing standard aggregation protocols to overfit dominant clients and degrade minority-class performance. We propose a data-free, class-wise contribution estimation and aggregation framework based on logit maximization (CELM) that does not require sharing raw data, client metadata, or auxiliary public datasets. The FL server probes client updates to obtain class-wise evidence scores and assembles a cross-client evidence matrix, which quantifies both per-class competence and class coverage. Using this matrix, we compute contribution weights that upweight clients providing strong, discriminative evidence for underrepresented classes. The resulting aggregation is stable due to simplex constraints and momentum smoothing, and it remains compatible with standard FL training pipelines. We evaluate the approach on representative vision benchmarks under controlled non-IID and pathological label splits, demonstrating that CELM-based aggregation improves robustness to imbalance and statistical heterogeneity, while yielding better performance without requiring any additional data exchange.

Abstract

Federated Learning

Explore similar work

Class-wise Contribution Estimation via Logit Maximization for Federated Learning

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models