FedIA: Instance-Aware Cross-Modal Representations for Federated Prompt Learning

Xuanjing Zhang; Qi Guo; Yanan Qiao; Yong Han; Junge Bo

cs.CVNov 4, 2026

FedIA: Instance-Aware Cross-Modal Representations for Federated Prompt Learning

Authors: Xuanjing Zhang, Qi Guo, Yanan Qiao, Yong Han, Junge Bo

Organizations: School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, Shaanxi, China · Air Traffic Control and Navigation School, Air Force Engineering University, Xi’an, 710051, Shaanxi, China

Abstract

Federated Prompt Learning (FPL) enables parameter-efficient adaptation of pre-trained Vision-Language Models across decentralized clients without exchanging raw data. To cope with cross-client heterogeneity, existing FPL methods typically pair a globally shared prompt with a client-specific local prompt. However, this design personalizes only at the client level: the same local prompt is applied to all samples of a client, and the global and local prompts are combined in an input-agnostic, fixed manner. Consequently, it overlooks the substantial visual variations among samples within the same client and cannot adjust the balance between shared and personalized knowledge to each input. To address these limitations, we propose Federated Prompt Learning with Instance-Aware Cross-Modal Representations (FedIA), an FPL framework that learns instance-aware cross-modal representations. Specifically, FedIA first uses global text prompts as client-invariant semantic anchors to select image patches most relevant to each candidate class, suppressing distracting regions. The selected visual evidence then conditions the generation of instance-aware local text representations. Finally, an adaptive global-local fusion module calibrates the contributions of local and global text representations for each image-class pair. In this way, FedIA shifts personalization in FPL from the client level to the instance level while preserving the semantic knowledge shared across clients. Extensive experiments under domain shift, pathological label skew, and Dirichlet-based label skew show that FedIA outperforms FPL baselines, with maximum accuracy gains of 1.44%-4.37% over the latest baseline across the three settings. Ablation studies and analyses further verify the complementary contributions of instance-aware representation generation and adaptive global-local fusion.

Figures & tables

Figure 1: Motivation and overview of FedIA. Challenge (1): Existing FPL methods typically use client-level prompts, which lack visual context and fail to capture instance-level variations within each client. Challenge (2): They also rely on input-agnostic fixed global–local fusion, making it difficult to adaptively balance personalization and generalization.

Figure 2: Framework of FedIA. FedIA first performs global-prompt-guided patch selection to retain class-relevant visual patches for each input image. The selected patches are then used by an instance-aware text representation generator to construct image-conditioned local representations. A lightweight gating network further adaptively fuses the instance text features with global knowledge, producing calibrated text representations balance personalized information and shared semantics.

Methods	PACS
Methods	A	C	P	S	Avg
ZS-CLIP	$97.26\pm 0.00$	$99.11\pm 0.00$	$99.95\pm 0.00$	$87.77\pm 0.00$	$96.02\pm 0.00$
PromptFL	$98.63\pm 0.14$	$99.24\pm 0.05$	$99.92\pm 0.03$	$94.47\pm 0.40$	$98.06\pm 0.11$
FedMaPLe	$98.47\pm 0.21$	$99.11\pm 0.19$	$99.90\pm 0.04$	$95.43\pm 0.06$	$98.23\pm 0.12$
FedOTP	$98.43\pm 0.08$	$99.18\pm 0.09$	$99.73\pm 0.07$	$96.42\pm 0.13$	$98.44\pm 0.03$
FedPGP	$98.97\pm 0.08$	$99.59\pm 0.08$	$99.95\pm 0.01$	$96.65\pm 0.10$	$98.79\pm 0.03$

Table 1: Comparison of different methods on PACS under

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds. Avg denotes the average accuracy across all domains, and the best average results are shown in bold.

Methods	OfficeHome
Methods	A	C	P	R	Avg
ZS-CLIP	$84.45\pm 0.00$	$66.14\pm 0.00$	$89.50\pm 0.00$	$89.55\pm 0.00$	$82.41\pm 0.00$
PromptFL	$87.13\pm 0.75$	$75.28\pm 0.33$	$94.74\pm 0.44$	$92.48\pm 0.26$	$87.41\pm 0.18$
FedMaPLe	$85.67\pm 0.71$	$77.38\pm 0.18$	$94.58\pm 0.30$	$92.62\pm 0.28$	$87.56\pm 0.29$
FedOTP	$80.50\pm 1.08$	$76.92\pm 0.21$	$92.09\pm 0.17$	$88.02\pm 0.08$	$84.38\pm 0.17$
FedPGP	$88.89\pm 0.65$	$78.32\pm 0.64$	$95.48\pm 0.37$	$93.92\pm 0.37$	$89.15\pm 0.22$

Table 2: Comparison of different methods on OfficeHome under

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds. Avg denotes the average accuracy across all domains, and the best average results are shown in bold.

Methods	Office31
Methods	A	D	W	Avg
ZS-CLIP	$81.15\pm 0.00$	$71.63\pm 0.00$	$73.15\pm 0.00$	$75.31\pm 0.00$
PromptFL	$88.43\pm 0.08$	$86.70\pm 2.01$	$91.18\pm 1.05$	$88.77\pm 0.73$
FedMaPLe	$88.60\pm 0.24$	$88.59\pm 1.61$	$91.06\pm 0.71$	$89.41\pm 0.43$
FedOTP	$86.33\pm 0.47$	$89.36\pm 1.87$	$93.87\pm 0.71$	$89.85\pm 0.52$
FedPGP	$90.17\pm 0.43$	$91.09\pm 1.97$	$95.40\pm 0.33$	$92.22\pm 0.67$

Table 3: Comparison of different methods on Office31 under

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds. Avg denotes the average accuracy across all domains, and the best average results are shown in bold.

Panel of Figure 3 — Figure 3: Test accuracy over communication rounds on PACS, OfficeHome, and Office31 with $k=49$ .

Methods	Food101	Caltech101	Flowers102	OxfordPets	DTD
ZS-CLIP	$85.26\pm 0.00$	$91.93\pm 0.00$	$67.88\pm 0.00$	$89.05\pm 0.00$	$43.33\pm 0.00$
PromptFL	$86.76\pm 0.19$	$93.42\pm 0.15$	$87.55\pm 2.07$	$93.43\pm 0.27$	$56.08\pm 0.57$
FedMaPLe	$86.66\pm 0.22$	$93.81\pm 0.15$	$87.60\pm 1.61$	$93.48\pm 0.61$	$55.44\pm 0.78$
FedOTP	$92.97\pm 0.18$	$98.13\pm 0.18$	$98.55\pm 0.14$	$98.70\pm 0.09$	$90.53\pm 0.40$
FedPGP	$87.88\pm 0.78$	$94.96\pm 1.10$	$96.97\pm 1.00$	$94.68\pm 0.57$	$62.24\pm 2.34$
FedTPG	$86.53\pm 0.45$	$93.86\pm 0.46$	$80.04\pm 4.42$	$93.07\pm 0.11$	$55.17\pm 0.75$

Table 4: Comparison of different algorithms across pathological label skew datasets under

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds.

Methods	CIFAR-10			CIFAR-100
Methods	$\beta=0.1$	$\beta=0.3$	$\beta=0.5$	$\beta=0.1$	$\beta=0.3$	$\beta=0.5$
ZS-CLIP	$87.84\pm 0.38$	$88.05\pm 0.11$	$87.84\pm 0.15$	$64.94\pm 0.13$	$64.89\pm 0.02$	$64.90\pm 0.04$
PromptFL	$89.68\pm 0.67$	$91.70\pm 0.18$	$91.80\pm 0.38$	$70.96\pm 0.15$	$72.45\pm 0.14$	$72.89\pm 0.15$
FedMaPLe	$92.57\pm 0.44$	$93.44\pm 0.60$	$93.58\pm 0.97$	$74.61\pm 0.43$	$75.63\pm 0.26$	$75.92\pm 0.36$
FedOTP	$96.95\pm 0.31$	$95.07\pm 0.30$	$93.55\pm 0.38$	$86.47\pm 0.18$	$79.05\pm 0.28$	$75.21\pm 0.33$
FedPGP	$94.67\pm 0.14$	$94.20\pm 0.37$	$93.35\pm 0.73$	$86.44\pm 0.05$	$79.09\pm 0.51$	$75.81\pm 0.43$

Table 5: Performance comparison on CIFAR-10 and CIFAR-100 under Dirichlet partitioning with

\beta\in\{0.1,0.3,0.5\}

and

k=49

. Results are reported as mean

\pm

standard deviation of accuracy (%) over three random seeds.

Panel of Figure 4 — Figure 4: Test accuracy over communication rounds on CIFAR-10 under Dirichlet partitioning with $\beta\in\{0.1,0.3,0.5\}$ and $k=49$ .

Panel of Figure 5 — Figure 5: Test accuracy over communication rounds on CIFAR-100 under Dirichlet partitioning with $\beta\in\{0.1,0.3,0.5\}$ and $k=49$ .

$k$	ILRG	AGLRF	A	$\Delta$	D	$\Delta$	W	$\Delta$	Avg	$\Delta$
25			$88.43$	–	$86.70$	–	$91.18$	–	$88.77$	–
	$\checkmark$		$90.70$	$\uparrow 2.27$	$95.82$	$\uparrow 9.12$	$97.67$	$\uparrow 6.49$	$94.73$	$\uparrow 5.96$
		$\checkmark$	$90.40$	$\uparrow 1.97$	$95.36$	$\uparrow 8.66$	$97.65$	$\uparrow 6.47$	$94.47$	$\uparrow 5.70$
	$\checkmark$	$\checkmark$	$\mathbf{90.94}$	$\uparrow\mathbf{2.51}$	$\mathbf{95.95}$	$\uparrow\mathbf{9.25}$	$\mathbf{98.08}$	$\uparrow\mathbf{6.90}$	$\mathbf{94.99}$	$\uparrow\mathbf{6.22}$
49			$88.43$	–	$86.70$	–	$91.18$	–	$88.77$	–
	$\checkmark$		$90.60$	$\uparrow 2.17$	$95.47$	$\uparrow 8.77$	$98.01$	$\uparrow 6.83$	$94.69$	$\uparrow 5.92$

Table 6: Ablation study of FedIA on Office31 under

k=25

and

k=49

\Delta

denotes the accuracy improvement over the baseline in percentage points.

ILRG

AGLRF

k=25

k=49

OfficeHome

CIFAR-10

CIFAR-100

OfficeHome

CIFAR-10

CIFAR-100

87.41

–

91.80

–

72.89

–

87.41

–

91.80

–

72.89

–

\checkmark

89.45

\uparrow 2.04

93.93

\uparrow 2.13

76.83

\uparrow 3.94

89.56

\uparrow 2.15

93.99

\uparrow 2.19

77.03

\uparrow 4.14

\checkmark

89.57

\uparrow 2.16

94.19

\uparrow 2.39

76.17

\uparrow 3.28

89.57

\uparrow 2.16

94.19

\uparrow 2.39

76.17

\uparrow 3.28

\checkmark

\checkmark

\mathbf{89.73}

\uparrow\mathbf{2.32}

\mathbf{94.22}

\uparrow\mathbf{2.42}

\mathbf{77.53}

\uparrow\mathbf{4.64}

\mathbf{89.76}

\uparrow\mathbf{2.35}

\mathbf{94.21}

\uparrow\mathbf{2.41}

\mathbf{77.57}

\uparrow\mathbf{4.68}

Table 7: Ablation study of FedIA on OfficeHome, CIFAR-10, and CIFAR-100 under

k=25

and

k=49

with

\beta=0.5

. The first value in each pair is the average accuracy (%), and the second is the improvement over the baseline in percentage points.

Panel of Figure 6 — Figure 6: Test accuracy of FedIA on OfficeHome, Office31, and CIFAR-100 under different values of $k$ .

Panel of Figure 7 — Figure 7: Test accuracy on Food101, Caltech101, and OxfordPets under different numbers of shots.

Prompt Learning

Prompt learning adapts pretrained models by designing or tuning input prompts, often without changing most model weights.

See all 79 papers

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization
Oct 16, 2026Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry +2Prompt Learning for VLMs Test-Time Prompt Tuning
Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia
FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models
May 27, 2026Xucong Wang, Pengkun Wang, Zhe Zhao +3Multi-Label Classification VLM Adaptation
University of Science and Technology of China (USTC) · Suzhou Institute for Advanced Research, USTC · City University of Hong Kong
Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach
Jul 23, 2026Yuhua Wang, Xiaodong Li, Yihao Guo +6VLM Adaptation Differentially Private Federated Learning
School of Artificial Intelligence, Beihang University Beijing, China · Center for the Applied Statistics, School of Statistics, Renmin University of China Beijing, China · School of Computer Science & Technology, Beijing Jiaotong University Beijing, China +2

Oct 16, 2026J/K move · Enter open · S save

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry, Amin Ahsan Ali+1

Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia

Prompt Learning for VLMs Test-Time Prompt Tuning Test-Time Adaptation of VLMs Federated Learning Aggregation Heterogeneous Federated Learning

Open paper AskPDF arXiv

Federated prompt learning enables vision-language models like CLIP to collaboratively adapt lightweight soft prompts across distributed clients without sharing raw data. However, existing federated prompt learning methods are designed primarily for supervised settings with labeled client data and assumed relatively stable training distributions. At test time, source-free federated prompt tuning remains largely unexplored. Clients may receive unlabeled target samples with heterogeneous, often non-IID distributions that differ substantially across clients and from the source training distribution, causing locally adapted prompts to drift in different directions and reducing the effectiveness of federated aggregation. In this work, we explore Source-Free Federated Test-Time Prompt Tuning (SF-FTTPT), where distributed clients collaboratively adapt soft prompts on unlabeled test streams without access to source data. Our study reveals that conventional federated aggregation methods struggle under heterogeneous client distributions, as locally adapted prompts become increasingly specialized and difficult to combine effectively. To address this challenge, we propose FedRAPP, a personalized aggregation framework that combines reliability-aware client assessment, server-side exponential moving-average prompt memory, and similarity-guided peer aggregation. FedRAPP selectively incorporates information from reliable and compatible clients while preserving local specialization, enabling more robust adaptation under distribution shift. Extensive experiments across nine CLIP benchmark datasets and ImageNet variants demonstrate that FedRAPP consistently outperforms standard federated aggregation baselines under heterogeneous client distributions while maintaining the efficiency advantages of prompt-only adaptation.

Abstract

Prompt Learning

Explore similar work

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization

FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Models

Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach