CAPT: Class-Aware Prompt Tuning for Federated Long-Tailed Learning with Vision-Language Model

Shihao Hou; Chikai Shang; Jiacheng Yang; Xinyi Shang; Shreyank N. Gowda; Yan Yan; Yang Lu

cs.LGMar 10, 2025

CAPT: Class-Aware Prompt Tuning for Federated Long-Tailed Learning with Vision-Language Model

Authors: Shihao Hou, Chikai Shang, Jiacheng Yang, Xinyi Shang, Shreyank N. Gowda, Yan Yan, Yang Lu

Organizations: Xiamen University · University College London · University of Nottingham

Abstract

Effectively addressing the joint effects of non-IID data and long-tailed distribution remains a key challenge in federated learning. While fine-tuning vision-language models (VLMs) like CLIP has shown promise in tackling non-IID scenarios, it often degrades tail-class performance when both heterogeneity and long-tailed distributions are present, even leading to overall performance falling below the zero-shot baseline. To overcome this, we propose Class-Aware Prompt Tuning (CAPT), a novel framework leveraging pre-trained VLMs to effectively learn from non-IID and long-tailed data. Specifically, CAPT introduces a dual-prompt mechanism that synergizes class-aware and general prompts, enabling the model to preserve class-specific information while capturing global trends. Furthermore, to guide the prompts in learning targeted knowledge, we propose a heterogeneity-aware clustering strategy with two key components: similarity-based clusters aggregate class-aware prompts to reinforce tail-specific knowledge, while heterogeneity-based clusters aggregate general prompts to balance diverse representations and promote robust generalization. Extensive experiments demonstrate that CAPT significantly outperforms state-of-the-art methods, e.g., by improving overall accuracy by 6.1% on ImageNet-LT and 8.8% on Places-LT over baselines, while also substantially enhancing tail-class performance across other benchmarks.

Figures & tables

Figure 1. Performance comparison between PromptFL (upper part) and CAPT (lower part). As client heterogeneity increases (

\alpha

decreases from 0.5 to 0.05), PromptFL shows widening head-tail gaps and declining overall accuracy below the zero-shot CLIP baseline (dashed line) due to poor tail performance. CAPT mitigates this, reducing gaps while sustaining superior overall accuracy. Two side-by-side plots compare PromptFL and CAPT under different levels of client heterogeneity on CIFAR-100 and ImageNet. As alpha decreases, PromptFL shows a larger head-tail performance gap and lower overall accuracy, in some cases dropping below the zero-shot CLIP baseline marked by a dashed line. CAPT reduces the head-tail gap and maintains higher overall accuracy across heterogeneity levels.

Figure 2. Effects of non-IID and long-tailed data. (a) Non-IID data causes clients to be dominated by a few classes, resulting in divergent local updates that deviate from the optimal global update. (b) When combined with long-tailed patterns, clients with rare classes are outnumbered by those with abundant classes, biasing aggregated global updates toward heads and impairing tail-class representation.

Figure 3. Overview of the proposed CAPT framework. Phase 1 illustrates local client prompt training, where a general prompt $P_{G}$ and class-aware prompts $P_{C}^{i}$ are optimized using a frozen image encoder and text encoder, with losses $\mathcal{L}_{ge}$ and $\mathcal{L}_{ca}$ computed from text and image features for classification. Phase 2 depicts heterogeneity-aware client clustering, showing similarity-based clustering that groups clients with dominant classes (e.g., by $c$ ) for aggregating class-aware prompts, and heterogeneity-based clustering that groups complementary clients (e.g., dominant by $a,c$ and by $b$ ) for aggregating general prompts.

Dataset	Method	$\alpha=0.05$				$\alpha=0.1$				$\alpha=0.5$
Dataset	Method	Overall	Head	Mid	Tail	Overall	Head	Mid	Tail	Overall	Head	Mid	Tail
CIFAR-10-LT	FedCLIP ( Lu et al., 2023 )	84.81	98.10	90.20	76.26	85.55	98.45	90.67	77.32	86.26	97.25	89.50	79.92
	PromptFL ( Guo et al., 2023b )	88.67	95.30	86.50	84.93	89.32	94.07	88.83	85.23	90.59	96.63	87.43	88.77
	FedTPG ( Qiu et al., 2024 )	87.78	96.70	87.90	84.14	88.91	96.60	84.70	88.36	88.90	98.30	91.40	83.64
	FedOTP ( Li et al., 2024a )	83.27	83.70	82.13	83.78	85.15	96.65	85.97	80.06	86.97	97.65	86.50	82.98
	FedPGP ( Cui et al., 2024 )	88.76	97.05	88.10	84.64	86.96	98.15	84.83	83.76	87.49	97.95	91.23	81.06

Table 1. Performance comparison of CAPT against baseline methods on long-tailed datasets under different degrees of data heterogeneity (

\alpha

). Results show accuracy (%) for overall, head, mid, and tail classes.

\uparrow

means improved accuracy compared with PromptFL ( Guo et al., 2023b ) . The best results are in bold . The second-best results are underlined .

Dataset	Method	$\alpha=0.05$				$\alpha=0.1$				$\alpha=0.5$
Dataset	Method	Overall	Head	Mid	Tail	Overall	Head	Mid	Tail	Overall	Head	Mid	Tail
CIFAR-10-LT	CoCoop+FedAvg ( Zhou et al., 2022a )	87.77	97.70	90.57	82.12	87.73	95.85	87.97	84.34	88.21	97.05	85.80	86.12
	MaPle+FedAvg ( Khattak et al., 2023 )	90.39	99.35	89.23	87.50	91.25	98.90	88.40	89.90	92.05	99.20	94.07	87.98
	KgCoOp+FedAvg ( Yao et al., 2023 )	88.21	97.05	87.67	85.00	87.67	97.50	85.40	85.10	86.93	97.95	89.20	81.16
	CLIP-LoRA+FedAvg ( Zanella and Ben Ayed, 2024 )	87.77	89.65	86.10	88.02	87.90	89.31	89.31	88.26	87.91	89.81	86.34	88.09
	CAPT (Ours)	93.78	97.50	90.47	94.28	93.98	98.60	89.70	94.70	94.04	97.80	93.17	93.06

Table 2. Comparison of CAPT with prompt-tuning methods. Results show accuracy for overall, head, mid, and tail classes on CIFAR-10-LT and CIFAR-100-LT, under varying degrees of data heterogeneity. Baseline methods are combined with FedAvg for parameter aggregation. Bold indicates best performance, underline indicates second-best performance.

Training from Scratch
Method	CIFAR-10-LT	CIFAR-100-LT	ImageNet-LT
FedAvg ( McMahan et al., 2017 )	58.22	30.47	23.85
CReFF ( Shang et al., 2022b )	70.31	32.90	25.98
Fed-Grab ( Xiao et al., 2023 )	70.51	36.53	32.94
FedLoge ( Xiao et al., 2024 )	70.54	42.72	35.62
RUCR ( Huang et al., 2024 )	71.40	35.58	24.02

Table 3. Performance comparison of state-of-the-art federated long-tailed learning methods using ResNet backbone.

Figure 4. Accuracy gains (%) of PromptFL (left) and CAPT (right) relative to CLIP on CIFAR-100-LT and ImageNet-LT.

Figure 5. Training dynamics of PromptFL (left) and CAPT (right) on the ImageNet-LT dataset. Each panel plots accuracy (%) for overall, head-class, and tail-class performance over 100 communication rounds.

	ImageNet-LT				Places-LT
Method	All	Head	Mid	Tail	All	Head	Mid	Tail
FedCLIP	65.4	70.1	64.7	61.9	32.3	37.7	31.6	33.1
PromptFL	67.1	76.0	65.6	59.8	36.4	55.0	32.2	30.3
FedTPG	68.2	73.4	66.7	63.5	38.1	55.1	34.0	32.9
FedOTP	55.5	60.6	52.2	51.5	33.0	45.9	27.6	20.7
FedPGP	68.3	76.6	65.9	60.0	37.5	47.0	30.0	28.5

Table 4. Performance comparison on the large-scale datasets.

Cluster	Prompt Type	CIFAR-10-LT
Cluster	Prompt Type	Overall	Head	Mid	Tail
w/o Clu.	General Prompt	89.23	97.50	90.95	84.93
	Class-aware Prompt	90.27	98.75	91.53	86.12
	Gen+Cla	90.42	99.30	91.27	86.36
w/ Clu.	General Prompt	92.00	94.55	90.57	91.84
	Class-aware Prompt	92.69	96.85	90.00	92.64

Table 5. Ablation study of CAPT components, comparing different prompt types and cluster aggregation strategies.

Figure 6. t-SNE visualizations of feature embeddings learned by General Prompts (circles) and Class-Aware Prompts (triangles) under varying configurations. Shaded regions highlight the concentrated area of General Prompt features. Notably, CAPT’s embeddings on long-tailed data (rightmost) closely mimic those on balanced data (leftmost), demonstrating robustness to imbalance through effective prompt decoupling and clustering synergy.

Figure 7. Accuracy versus inter-client gradient variance on CIFAR-10-LT and CIFAR-100-LT. CAPT variants achieve high accuracy at low variance, while baselines suffer declining tail accuracy as variance increases.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 992 papers

Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models
May 12, 2026Boyang Guo, Liang Li, Lin Peng +3Long-Tail Learning VLM Adaptation
Hangzhou Dianzi University, Hangzhou, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · The Hong Kong Polytechnic University, Hong Kong, China +2
FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization
Oct 16, 2026Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry +2Prompt Learning for VLMs Test-Time Prompt Tuning
Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia
Unsupervised Long-Tailed Adaptation of Vision-Language Models
Oct 6, 2026Keliang Chen, Yaxin Hou, Hui Liu +1Long-Tail Learning VLM Adaptation
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · School of Computing and Information Sciences, Saint Francis University, Hong Kong, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China

May 12, 2026J/K move · Enter open · S save

Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models

Boyang Guo, Liang Li, Lin Peng, Yuhan Gao+2

Hangzhou Dianzi University, Hangzhou, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · The Hong Kong Polytechnic University, Hong Kong, China+2

Long-Tail Learning VLM Adaptation Visual Prompt Learning

Open paper AskPDF arXiv

Prompt learning has emerged as an efficient alternative to fine-tuning pre-trained vision-language models (VLMs). Despite its promise, current methods still struggle to maintain tail-class discriminability when adapting to class-imbalanced datasets. In this work, we propose cluster-aware neural collapse prompt tuning (CPT), which enhances the discriminability of tail classes in prompt-tuned VLMs without sacrificing their overall generalization. First, we design a cluster-invariant space by mining semantic assignments from the pre-trained VLM and mapping them to prompt-tuned features. This computes cluster-level boundaries and restricts the constraints to local neighborhoods, which reduces interference with the global semantic structure of the pre-trained VLM. Second, we introduce neural-collapse-driven discriminability optimization with three losses: textual Equiangular Tight Frame (ETF) separation loss, class-wise convergence loss, and rotation stabilization loss. These losses work together to shape intra-cluster geometry for better inter-class separation and intra-class alignment. Extensive experiments on 11 diverse datasets demonstrate that CPT outperforms SOTA methods, with stronger performance on long-tail classes and good generalization to unseen classes.

Abstract

Federated Learning

Explore similar work

Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

Unsupervised Long-Tailed Adaptation of Vision-Language Models