FedIA: Instance-Aware Cross-Modal Representations for Federated Prompt Learning

Xuanjing Zhang; Qi Guo; Yanan Qiao; Yong Han; Junge Bo

cs.CVNov 4, 2026

FedIA: Instance-Aware Cross-Modal Representations for Federated Prompt Learning

Authors: Xuanjing Zhang, Qi Guo, Yanan Qiao, Yong Han, Junge Bo

Organizations: School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, Shaanxi, China · Air Traffic Control and Navigation School, Air Force Engineering University, Xi’an, 710051, Shaanxi, China

Abstract

Federated Prompt Learning (FPL) enables parameter-efficient adaptation of pre-trained Vision-Language Models across decentralized clients without exchanging raw data. To cope with cross-client heterogeneity, existing FPL methods typically pair a globally shared prompt with a client-specific local prompt. However, this design personalizes only at the client level: the same local prompt is applied to all samples of a client, and the global and local prompts are combined in an input-agnostic, fixed manner. Consequently, it overlooks the substantial visual variations among samples within the same client and cannot adjust the balance between shared and personalized knowledge to each input. To address these limitations, we propose Federated Prompt Learning with Instance-Aware Cross-Modal Representations (FedIA), an FPL framework that learns instance-aware cross-modal representations. Specifically, FedIA first uses global text prompts as client-invariant semantic anchors to select image patches most relevant to each candidate class, suppressing distracting regions. The selected visual evidence then conditions the generation of instance-aware local text representations. Finally, an adaptive global-local fusion module calibrates the contributions of local and global text representations for each image-class pair. In this way, FedIA shifts personalization in FPL from the client level to the instance level while preserving the semantic knowledge shared across clients. Extensive experiments under domain shift, pathological label skew, and Dirichlet-based label skew show that FedIA outperforms FPL baselines, with maximum accuracy gains of 1.44%-4.37% over the latest baseline across the three settings. Ablation studies and analyses further verify the complementary contributions of instance-aware representation generation and adaptive global-local fusion.

Figures & tables

Figure 1: Motivation and overview of FedIA. Challenge (1): Existing FPL methods typically use client-level prompts, which lack visual context and fail to capture instance-level variations within each client. Challenge (2): They also rely on input-agnostic fixed global–local fusion, making it difficult to adaptively balance personalization and generalization.

Figure 2: Framework of FedIA. FedIA first performs global-prompt-guided patch selection to retain class-relevant visual patches for each input image. The selected patches are then used by an instance-aware text representation generator to construct image-conditioned local representations. A lightweight gating network further adaptively fuses the instance text features with global knowledge, producing calibrated text representations balance personalized information and shared semantics.

Methods	PACS
Methods	A	C	P	S	Avg
ZS-CLIP	$97.26\pm 0.00$	$99.11\pm 0.00$	$99.95\pm 0.00$	$87.77\pm 0.00$	$96.02\pm 0.00$
PromptFL	$98.63\pm 0.14$	$99.24\pm 0.05$	$99.92\pm 0.03$	$94.47\pm 0.40$	$98.06\pm 0.11$
FedMaPLe	$98.47\pm 0.21$	$99.11\pm 0.19$	$99.90\pm 0.04$	$95.43\pm 0.06$	$98.23\pm 0.12$
FedOTP	$98.43\pm 0.08$	$99.18\pm 0.09$	$99.73\pm 0.07$	$96.42\pm 0.13$	$98.44\pm 0.03$
FedPGP	$98.97\pm 0.08$	$99.59\pm 0.08$	$99.95\pm 0.01$	$96.65\pm 0.10$	$98.79\pm 0.03$

Table 1: Comparison of different methods on PACS under

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds. Avg denotes the average accuracy across all domains, and the best average results are shown in bold.

Methods	OfficeHome
Methods	A	C	P	R	Avg
ZS-CLIP	$84.45\pm 0.00$	$66.14\pm 0.00$	$89.50\pm 0.00$	$89.55\pm 0.00$	$82.41\pm 0.00$
PromptFL	$87.13\pm 0.75$	$75.28\pm 0.33$	$94.74\pm 0.44$	$92.48\pm 0.26$	$87.41\pm 0.18$
FedMaPLe	$85.67\pm 0.71$	$77.38\pm 0.18$	$94.58\pm 0.30$	$92.62\pm 0.28$	$87.56\pm 0.29$
FedOTP	$80.50\pm 1.08$	$76.92\pm 0.21$	$92.09\pm 0.17$	$88.02\pm 0.08$	$84.38\pm 0.17$
FedPGP	$88.89\pm 0.65$	$78.32\pm 0.64$	$95.48\pm 0.37$	$93.92\pm 0.37$	$89.15\pm 0.22$

Table 2: Comparison of different methods on OfficeHome under

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds. Avg denotes the average accuracy across all domains, and the best average results are shown in bold.

Methods	Office31
Methods	A	D	W	Avg
ZS-CLIP	$81.15\pm 0.00$	$71.63\pm 0.00$	$73.15\pm 0.00$	$75.31\pm 0.00$
PromptFL	$88.43\pm 0.08$	$86.70\pm 2.01$	$91.18\pm 1.05$	$88.77\pm 0.73$
FedMaPLe	$88.60\pm 0.24$	$88.59\pm 1.61$	$91.06\pm 0.71$	$89.41\pm 0.43$
FedOTP	$86.33\pm 0.47$	$89.36\pm 1.87$	$93.87\pm 0.71$	$89.85\pm 0.52$
FedPGP	$90.17\pm 0.43$	$91.09\pm 1.97$	$95.40\pm 0.33$	$92.22\pm 0.67$

Table 3: Comparison of different methods on Office31 under

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds. Avg denotes the average accuracy across all domains, and the best average results are shown in bold.

Panel of Figure 3 — Figure 3: Test accuracy over communication rounds on PACS, OfficeHome, and Office31 with $k=49$ .

Methods	Food101	Caltech101	Flowers102	OxfordPets	DTD
ZS-CLIP	$85.26\pm 0.00$	$91.93\pm 0.00$	$67.88\pm 0.00$	$89.05\pm 0.00$	$43.33\pm 0.00$
PromptFL	$86.76\pm 0.19$	$93.42\pm 0.15$	$87.55\pm 2.07$	$93.43\pm 0.27$	$56.08\pm 0.57$
FedMaPLe	$86.66\pm 0.22$	$93.81\pm 0.15$	$87.60\pm 1.61$	$93.48\pm 0.61$	$55.44\pm 0.78$
FedOTP	$92.97\pm 0.18$	$98.13\pm 0.18$	$98.55\pm 0.14$	$98.70\pm 0.09$	$90.53\pm 0.40$
FedPGP	$87.88\pm 0.78$	$94.96\pm 1.10$	$96.97\pm 1.00$	$94.68\pm 0.57$	$62.24\pm 2.34$
FedTPG	$86.53\pm 0.45$	$93.86\pm 0.46$	$80.04\pm 4.42$	$93.07\pm 0.11$	$55.17\pm 0.75$

Table 4: Comparison of different algorithms across pathological label skew datasets under

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds.

Methods	CIFAR-10			CIFAR-100
Methods	$\beta=0.1$	$\beta=0.3$	$\beta=0.5$	$\beta=0.1$	$\beta=0.3$	$\beta=0.5$
ZS-CLIP	$87.84\pm 0.38$	$88.05\pm 0.11$	$87.84\pm 0.15$	$64.94\pm 0.13$	$64.89\pm 0.02$	$64.90\pm 0.04$
PromptFL	$89.68\pm 0.67$	$91.70\pm 0.18$	$91.80\pm 0.38$	$70.96\pm 0.15$	$72.45\pm 0.14$	$72.89\pm 0.15$
FedMaPLe	$92.57\pm 0.44$	$93.44\pm 0.60$	$93.58\pm 0.97$	$74.61\pm 0.43$	$75.63\pm 0.26$	$75.92\pm 0.36$
FedOTP	$96.95\pm 0.31$	$95.07\pm 0.30$	$93.55\pm 0.38$	$86.47\pm 0.18$	$79.05\pm 0.28$	$75.21\pm 0.33$
FedPGP	$94.67\pm 0.14$	$94.20\pm 0.37$	$93.35\pm 0.73$	$86.44\pm 0.05$	$79.09\pm 0.51$	$75.81\pm 0.43$

Table 5: Performance comparison on CIFAR-10 and CIFAR-100 under Dirichlet partitioning with

\beta\in\{0.1,0.3,0.5\}

and

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds.

Panel of Figure 4 — Figure 4: Test accuracy over communication rounds on CIFAR-10 under Dirichlet partitioning with $\beta\in\{0.1,0.3,0.5\}$ and $k=49$ .

Panel of Figure 5 — Figure 5: Test accuracy over communication rounds on CIFAR-100 under Dirichlet partitioning with $\beta\in\{0.1,0.3,0.5\}$ and $k=49$ .

$k$	ILRG	AGLRF	A	$\Delta$	D	$\Delta$	W	$\Delta$	Avg	$\Delta$
25			$88.43$	–	$86.70$	–	$91.18$	–	$88.77$	–
	$\checkmark$		$90.70$	$\uparrow 2.27$	$95.82$	$\uparrow 9.12$	$97.67$	$\uparrow 6.49$	$94.73$	$\uparrow 5.96$
		$\checkmark$	$90.40$	$\uparrow 1.97$	$95.36$	$\uparrow 8.66$	$97.65$	$\uparrow 6.47$	$94.47$	$\uparrow 5.70$
	$\checkmark$	$\checkmark$	$\mathbf{90.94}$	$\uparrow\mathbf{2.51}$	$\mathbf{95.95}$	$\uparrow\mathbf{9.25}$	$\mathbf{98.08}$	$\uparrow\mathbf{6.90}$	$\mathbf{94.99}$	$\uparrow\mathbf{6.22}$
49			$88.43$	–	$86.70$	–	$91.18$	–	$88.77$	–
	$\checkmark$		$90.60$	$\uparrow 2.17$	$95.47$	$\uparrow 8.77$	$98.01$	$\uparrow 6.83$	$94.69$	$\uparrow 5.92$

Table 6: Ablation study of FedIA on Office31 under

k=25

and

k=49

\Delta

denotes the accuracy improvement over the baseline in percentage points.

ILRG

AGLRF

k=25

k=49

OfficeHome

CIFAR-10

CIFAR-100

OfficeHome

CIFAR-10

CIFAR-100

87.41

–

91.80

–

72.89

–

87.41

–

91.80

–

72.89

–

\checkmark

89.45

\uparrow 2.04

93.93

\uparrow 2.13

76.83

\uparrow 3.94

89.56

\uparrow 2.15

93.99

\uparrow 2.19

77.03

\uparrow 4.14

\checkmark

89.57

\uparrow 2.16

94.19

\uparrow 2.39

76.17

\uparrow 3.28

89.57

\uparrow 2.16

94.19

\uparrow 2.39

76.17

\uparrow 3.28

\checkmark

\checkmark

\mathbf{89.73}

\uparrow\mathbf{2.32}

\mathbf{94.22}

\uparrow\mathbf{2.42}

\mathbf{77.53}

\uparrow\mathbf{4.64}

\mathbf{89.76}

\uparrow\mathbf{2.35}

\mathbf{94.21}

\uparrow\mathbf{2.41}

\mathbf{77.57}

\uparrow\mathbf{4.68}

Table 7: Ablation study of FedIA on OfficeHome, CIFAR-10, and CIFAR-100 under

k=25

and

k=49

with

\beta=0.5

. The first value in each pair is the average accuracy (%), and the second is the improvement over the baseline in percentage points.

Panel of Figure 6 — Figure 6: Test accuracy of FedIA on OfficeHome, Office31, and CIFAR-100 under different values of $k$ .

Panel of Figure 7 — Figure 7: Test accuracy on Food101, Caltech101, and OxfordPets under different numbers of shots.

Prompt Learning

Prompt learning adapts pretrained models by designing or tuning input prompts, often without changing most model weights.

See all 79 papers

Oct 16, 2026

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

Federated prompt learning enables vision-language models like CLIP to collaboratively adapt lightweight soft prompts across distributed clients without sharing raw data. However, existing federated prompt learning methods are designed primarily for supervised settings with labeled client data and assumed relatively stable training distributions. At test time, source-free federated prompt tuning remains largely unexplored. Clients may receive unlabeled target samples with heterogeneous, often non-IID distributions that differ substantially across clients and from the source training distribution, causing locally adapted prompts to drift in different directions and reducing the effectiveness of federated aggregation. In this work, we explore Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where distributed clients collaboratively adapt soft prompts on unlabeled test streams without access to source data. Our study reveals that conventional federated aggregation methods struggle under heterogeneous client distributions, as locally adapted prompts become increasingly specialized and difficult to combine effectively. To address this challenge, we propose FedRAPP, a personalized aggregation framework that combines reliability-aware client assessment, server-side exponential moving-average prompt memory, and similarity-guided peer aggregation. FedRAPP selectively incorporates information from reliable and compatible clients while preserving local specialization, enabling more robust adaptation under distribution shift. Extensive experiments across nine CLIP benchmark datasets and ImageNet variants demonstrate that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while maintaining the efficiency advantages of prompt-only adaptation.

Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry +2

Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia

Prompt Learning for VLMs Test-Time Prompt Tuning Test-Time Adaptation of VLMs Federated Learning Aggregation Heterogeneous Federated Learning

Ask PDF

May 27, 2026

FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models

Multi-Label Recognition (MLR) based on Vision-Language Models (VLMs) aims to leverage their pre-trained knowledge to better adapt complex recognition scenarios, thereby enhancing model robustness. However, for realistic decentralized applications requiring federated learning, adapting VLMs to each client that possesses private and heterogeneous data can cause the model to overfit spurious label correlations, consequently triggering irrelevant categories when encountering new samples. To tackle this problem, we reconsider the federated learning for MLR with a causal model, in which we adopt a front-door adjustment and decouple the MLR modeling process by intermediate variables that magnify the oracle label co-occurrence. Guided by our analysis, we propose our FedMPT, the first method specifically designed for federated MLR. The core idea of FedMPT is to leverage generalizable conditions to steer federated MLR to mitigate erroneous label activations. To achieve this, FedMPT introduces an Large Language Model (LLM)-driven pipeline to decipher the underlying conditions that govern label dependencies. Furthermore, we introduce an optimal transport between the condition-enriched prompts and the image patches to uncover multiple region-level semantics. Finally, we generate synergistic predictions from different conditions with a crafted gating mechanism. Experiments on multiple benchmark datasets show that our proposed approach achieves competitive results and outperforms SOTA methods under varied settings.

Xucong Wang, Pengkun Wang, Zhe Zhao +3

University of Science and Technology of China (USTC) · Suzhou Institute for Advanced Research, USTC · City University of Hong Kong

Multi-Label Classification VLM Adaptation Visual Prompt Learning Federated Learning

Ask PDF

Jul 23, 2026

Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach

Federated prompt tuning (FPT) enables collaborative adaptation of vision--language models (VLMs) using lightweight prompts. Existing methods often address heterogeneity and privacy through a split-prompt design under local differential privacy (DP), combining a shared prompt for global transfer with private prompts for local adaptation. However, a single shared prompt may over-smooth diverse transferable knowledge, weakening the balance between personalization and generalization. Multi-expert prompts (MEPs) can better capture this diversity, but enlarge the communicated space, increasing DP noise and communication cost while making robust expert composition more difficult. We propose FedSEPT, a privacy-preserving Fed}erated Subspace-decomposed Expert Prompt Tuning. Specifically, we employ Subspace-decomposed Expert Modeling (SEM) to parameterize multiple prompt experts with shared low-rank factors, a fixed public basis, and private residuals, thereby confining communication and DP perturbation to a compact factor space while enabling direct server aggregation in a common coordinate system. We further design Instance-aware Expert Fusion (IEF), which adaptively combines semantically complementary experts via on-device routing and performs efficient logit-level fusion using cached expert-specific text features. Extensive experiments on 11 heterogeneous benchmarks show that, under the same privacy constraints, FedSEPT achieves a better trade-off between local adaptation and global generalization than strong baselines.

Yuhua Wang, Xiaodong Li, Yihao Guo +6

School of Artificial Intelligence, Beihang University Beijing, China · Center for the Applied Statistics, School of Statistics, Renmin University of China Beijing, China · School of Computer Science & Technology, Beijing Jiaotong University Beijing, China +2

VLM Adaptation Differentially Private Federated Learning Differential Privacy Federated Learning Parameter-Efficient Fine-Tuning Heterogeneous Federated Learning

Ask PDF

Abstract

Prompt Learning

Explore similar work

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models

Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach