cs.LGNov 9, 2026

Harnessing Vision-Language Models for Federated Unsupervised Visual Recognition

Authors: Jian Liang, Kuangpu Guo, Ran He, Tieniu Tan

Organizations: NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences (CASIA), China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China · University of Science and Technology of China · Nanjing University, China

Abstract

Federated learning (FL) is a promising research paradigm that enables collaborative model training across distributed clients without data sharing. To handle unlabeled data in local clients, existing unsupervised FL research has primarily focused on tasks like clustering and self-supervised representation learning, leaving the critical problem of unsupervised visual recognition largely unexplored. To bridge this gap, we introduce FedUR, the first framework designed for unsupervised recognition by leveraging the zero-shot capabilities of pre-trained vision-language models like CLIP. Specifically, FedUR addresses data heterogeneity through a pseudo-labeling mechanism with a class-aware data selection strategy. This strategy first estimates a global pseudo-label distribution, then allocates pseudo-labeled data to clients accordingly. To handle the uncertainty of conventional positive pseudo-labels, FedUR further incorporates class weights and a negative pseudo-labeling strategy. Furthermore, FedUR proposes a personalized prompt aggregation protocol where general visual prompts are aggregated on the server, and personalized text prompts encoding local knowledge are retained on clients. Extensive experiments demonstrate that FedUR consistently outperforms baseline methods under both label-level and feature-level distribution shifts. We believe this work paves the way for further advances in federated unsupervised recognition. Code will be available at {github.com/tim-learn/FedUR/}}.

Figures & tables

Explore similar work

CardsList
  1. FedBST: Bilateral Self-Training for Unsupervised Federated Classification

    Oct 22, 2026Aijing Yu, Kuangpu Guo, Jian Liang +1Federated LearningSelf-Training

  2. FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models

    May 27, 2026Xucong Wang, Pengkun Wang, Zhe Zhao +3Multi-Label ClassificationVLM Adaptation

  3. Class-wise Contribution Estimation via Logit Maximization for Federated Learning

    May 17, 2026Asim Ukaye, Nurbek Tastan, Mubarak Abdu-Aguye +1Class-Imbalanced LearningFederated Learning Aggregation