cs.CVOct 16, 2026

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

Authors: Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry, Amin Ahsan Ali, AKM Mahbubur Rahman

Organizations: Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Federated prompt learning enables vision-language models like CLIP to collaboratively adapt lightweight soft prompts across distributed clients without sharing raw data. However, existing federated prompt learning methods are designed primarily for supervised settings with labeled client data and assumed relatively stable training distributions. At test time, source-free federated prompt tuning remains largely unexplored. Clients may receive unlabeled target samples with heterogeneous, often non-IID distributions that differ substantially across clients and from the source training distribution, causing locally adapted prompts to drift in different directions and reducing the effectiveness of federated aggregation. In this work, we explore Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where distributed clients collaboratively adapt soft prompts on unlabeled test streams without access to source data. Our study reveals that conventional federated aggregation methods struggle under heterogeneous client distributions, as locally adapted prompts become increasingly specialized and difficult to combine effectively. To address this challenge, we propose FedRAPP, a personalized aggregation framework that combines reliability-aware client assessment, server-side exponential moving-average prompt memory, and similarity-guided peer aggregation. FedRAPP selectively incorporates information from reliable and compatible clients while preserving local specialization, enabling more robust adaptation under distribution shift. Extensive experiments across nine CLIP benchmark datasets and ImageNet variants demonstrate that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while maintaining the efficiency advantages of prompt-only adaptation.

Figures & tables

Figure 1: Top-1 accuracy gain over None-Fed under the non-IID partition with AdaPrompt local adaptation (Numeric labels show FedRAPP gain over None-Fed).
Figure 2: Working mechanism of FedRAPP (3 clients shown). Clients locally adapt the received prompts and upload the tuned prompt and entropy statistics; the server computes reliability scores, updates the EMA memory, and distributes the personalized prompts to the clients.
MethodTPTDARTSwapPromptAdaPrompt
IIDNon-IIDIIDNon-IIDIIDNon-IIDIIDNon-IID
None-Fed14.8719.1565.6966.0465.9566.3866.8366.51
FedAvg27.3030.3526.0927.7966.1865.6766.9965.21
FedProx28.3731.6426.2727.1666.3166.0866.4965.35
FedAMP26.3431.9536.0336.3866.0566.5565.2566.46
pFedGraph26.9132.9934.6735.2066.2366.2666.3965.48
Table 1: Average top-1 accuracy over the nine CLIP classification datasets under IID and non-IID partitions. Best and second-best per column. Per-dataset results are in Tables A1 and A2 of the supplementary.
Figure 3: Sensitivity and scalability under the non-IID partition with AdaPrompt as the local adaptation method. (a) Average Accuracy vs varying number of clients over the nine CLIP datasets. (b) Average Accuracy vs varying batch size over the nine CLIP datasets. (c) Partial participation on OxfordPets. (d) Average Accuracy of all clients vs federated rounds on OxfordPets (e) Average Accuracy vs λself\lambda_{\mathrm{self}} (Personalization Coefficient) across OxfordPets, Flower102, Caltech101.
Figure A1 : Per-client top-1 accuracy on Caltech-101 under non-IID (pathological) partitioning, comparing None-Fed (local-only adaptation), FedAvg, and FedRAPP. Clients are sorted from weakest to strongest by their None-Fed accuracy. FedRAPP improves on the weak clients without degrading the strong ones, whereas FedAvg’s uniform aggregation drags several strong clients down.
DatasetDot-product onlyTrainable gate
IIDNon-IIDIIDNon-IID
OxfordPets84.4085.1660.2370.46
Caltech10189.2891.9070.4377.20
Flower10267.0368.3136.0947.46
Mean80.2481.7955.5865.04
Table A5 : Effect of pFedMoAP’s trainable attention gate under label-free test-time adaptation on three CLIP benchmarks. Top-1 accuracy with AdaPrompt as the local adaptation method. Dot-product only scores peers by prompt similarity; trainable gate is the default setup, with the attention module trained on the unlabeled test-time objective. Single seed.
Figure A2 : Similarity between consecutive round prompt updates.
More in this topic

Prompt Learning for VLMs

See all 23 papers

Explore similar work

CardsList
Jul 23, 2026

Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach

Federated prompt tuning (FPT) enables collaborative adaptation of vision--language models (VLMs) using lightweight prompts. Existing methods often address heterogeneity and privacy through a split-prompt design under local differential privacy (DP), combining a shared prompt for global transfer with private prompts for local adaptation. However, a single shared prompt may over-smooth diverse transferable knowledge, weakening the balance between personalization and generalization. Multi-expert prompts (MEPs) can better capture this diversity, but enlarge the communicated space, increasing DP noise and communication cost while making robust expert composition more difficult. We propose FedSEPT, a privacy-preserving Fed}erated Subspace-decomposed Expert Prompt Tuning. Specifically, we employ Subspace-decomposed Expert Modeling (SEM) to parameterize multiple prompt experts with shared low-rank factors, a fixed public basis, and private residuals, thereby confining communication and DP perturbation to a compact factor space while enabling direct server aggregation in a common coordinate system. We further design Instance-aware Expert Fusion (IEF), which adaptively combines semantically complementary experts via on-device routing and performs efficient logit-level fusion using cached expert-specific text features. Extensive experiments on 11 heterogeneous benchmarks show that, under the same privacy constraints, FedSEPT achieves a better trade-off between local adaptation and global generalization than strong baselines.

Yuhua Wang, Xiaodong Li, Yihao Guo +6

School of Artificial Intelligence, Beihang University Beijing, China · Center for the Applied Statistics, School of Statistics, Renmin University of China Beijing, China · School of Computer Science & Technology, Beijing Jiaotong University Beijing, China +2

VLM AdaptationDifferentially Private Federated LearningDifferential PrivacyFederated LearningParameter-Efficient Fine-TuningHeterogeneous Federated Learning
AskPDF
Dec 3, 2025

How (Mis)calibrated is your Federated CLIP and what to do about it?

Vision-language models (VLMs) such as CLIP are increasingly adapted across decentralized data silos, yet the reliability of their predictions under federated learning (FL) remains largely unexplored. In this work, we present a systematic study of calibration in federated CLIP under non-IID client distributions. Our experiments reveal that widely used prompt-tuning methods consistently degrade calibration, often yielding substantially higher calibration error despite competitive recognition performance, while explicit training-time calibration regularizers provide only limited improvements. Motivated by these findings, we identify the choice of fine-tuning parameterization as a critical factor governing calibration and conduct a controlled comparison between prompt tuning and five backbone fine-tuning (BFT) strategies: AdaptFormer, LayerNorm, LoRA, VeRA, and DoRA. We find that BFT methods generally offer a more favorable accuracy-calibration trade-off than prompt tuning, although their benefits are not universal. Through extensive analysis, we show that calibration behavior is closely linked to the geometry of federated updates, residual parameterization, and the resulting client and logit drift. Across in-distribution, domain-generalization, and base-to-new evaluation settings, our results establish fine-tuning parameterization as a central design choice for building accurate and reliable federated CLIP models. Codes are available at https://github.com/mainaksingha01/FL2oRA.

Mainak Singha, Masih Aminbeidokhti, Paolo Casari +3

University of Trento, Italy · ´Ecole de technologie sup´erieure, QC, Canada · AMIAD, Pˆole Recherche, Palaiseau, France +2

Vision-Language ModelsVLM AdaptationNon-IID Federated LearningVLM CalibrationFederated Fine-TuningFederated Learning+1
AskPDF
May 27, 2026

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

Federated Learning (FL) with pre-trained Vision-Language Models (VLMs) has emerged as a promising paradigm for various downstream tasks. By leveraging its strong representations, recent studies improve task adaptation under insufficient local data while preserving generalization. However, these methods emphasize fully local optimization with simple parameter aggregation,which can amplify inter-client optimization inconsistency and intra-client over-specialization under heterogeneous and full-data FL settings, making it difficult to balance global task adaptation and generalization. To address these challenges, we propose FedDTL, a novel federated VLM framework that decouples the image encoder and text encoder across clients and the server. Through decoupled encoder training with server-client modality alignment, FedDTL promotes coherent global semantic update and reduces inter-client optimization inconsistency, improving global task adaptation.To further mitigate intra-client over-specialization,we introduce a two-stage local fine-tuning, where a supervised fine-tuning stage enables rapid and reliable warm-start, followed by a reinforcement learning stage that enhances generalization. Extensive experiments on multiple benchmarks, including label skew and feature shift, demonstrate that FedDTL achieves an effective balance between global task adaptation and generalization under various FL data distributions in both few-shot and full-data regimes.

Yuting Ma, Lechao Cheng, Xiaohua Xu

School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China.

VLM AdaptationVision-Language AlignmentFederated Fine-TuningFederated LearningHeterogeneous Federated Learning
AskPDF