ShadowFPT: Backdooring Federated Prompt Tuning via Shadow Triggers

Kun Zhai; Teng Li; Yunhao Feng; Xingjun Ma

cs.AIOct 16, 2026

ShadowFPT: Backdooring Federated Prompt Tuning via Shadow Triggers

Authors: Kun Zhai, Teng Li, Yunhao Feng, Xingjun Ma

Organizations: Fudan University, Shanghai, China · Alibaba Group, Hangzhou, China

Abstract

Federated Prompt Tuning (FPT) adapts large vision--language models by freezing the pretrained backbone and optimizing only lightweight prompt parameters across clients. Although this design improves communication and parameter efficiency, it creates an overlooked security risk. Since the backbone is shared and frozen, a malicious client can induce backdoor behavior through the representation space while keeping its uploaded prompt updates close to benign ones. We propose ShadowFPT, a targeted backdoor attack that exploits this frozen-backbone attack surface. ShadowFPT first pretrains a learnable Shadow Trigger against the frozen CLIP visual encoder, using either auxiliary public data or the malicious client's local data, so that triggered inputs are steered toward the target class in representation space. During federated prompt tuning, the malicious client adapts the trigger under the current global prompt and then optimizes its local prompt on both clean and triggered samples. Only prompt parameters are uploaded to the server, while the trigger and frozen encoders remain local. By shifting most of the attack burden from prompt updates to trigger-induced representation steering, ShadowFPT achieves targeted misclassification while preserving prompt-space stealthiness. Across multiple datasets, aggregation rules, and non-IID federated partitions, ShadowFPT increases the attack success rate from 19.81% to 90.36% in our main setting, while maintaining clean accuracy. It remains effective across textual, visual, and joint vision--language prompt tuning. These results identify frozen backbones as stealthy and underexplored backdoor surfaces in federated prompt tuning, suggesting that defenses based only on prompt-update anomaly detection are insufficient.

Figures & tables

Figure 1: Overview of ShadowFPT. (A) The adversary pretrains a Shadow Trigger $\delta_{\mathrm{shadow}}$ against the frozen CLIP backbone through target-text and layer-wise visual alignment. (B) During FPT, the malicious client adapts the trigger under the current global prompt $P^{t}$ and optimizes its local prompt on clean and triggered samples, while uploading only prompt parameters. (C) By shifting the attack effect to trigger-induced representation steering, ShadowFPT achieves targeted misclassification with benign-like prompt updates.

Dataset	Setting	Best Base	Baseline		ShadowFPT-I				ShadowFPT-II
Dataset	Setting	Best Base	Acc	ASR	Acc	ASR	$\Delta$ Acc	$\Delta$ ASR	Acc	ASR	$\Delta$ Acc	$\Delta$ ASR
Caltech101	FPT-L	A3FL	88.49	19.81	88.32	84.28	-0.17	+64.47	87.67	90.36	-0.82	+70.55
	FPT-V	PFedBA	92.73	37.84	92.62	92.93	-0.11	+55.09	91.68	94.25	-1.05	+56.41
	FPT-VL	PFedBA	92.82	51.39	92.39	93.19	-0.43	+41.80	92.28	98.03	-0.54	+46.64
Oxford Flowers	FPT-L	A3FL	66.63	38.07	60.93	96.74	-5.70	+58.67	60.85	98.93	-5.78	+60.86
	FPT-V	PFedBA	65.73	30.22	66.60	98.37	+0.87	+68.15	66.61	98.94	+0.88	+68.72

Table 1: Overall attack performance across datasets and prompt-tuning settings. Results are averaged over all aggregation and defense rules. Best Base. denotes the strongest non-ShadowFPT baseline under each dataset-setting pair, selected by the highest average ASR.

\Delta

Acc and

\Delta

ASR report absolute differences from Best Base., measured in percentage points.

Defense	Best Base	Baseline		ShadowFPT-I				ShadowFPT-II
Defense	Best Base	Acc	ASR	Acc	ASR	$\Delta$ Acc	$\Delta$ ASR	Acc	ASR	$\Delta$ Acc	$\Delta$ ASR
FedAvg	A3FL	77.91	60.93	75.56	96.59	-2.35	+35.66	75.67	98.30	-2.24	+37.37
Krum	PFedBA	73.30	22.66	71.70	95.60	-1.60	+72.94	72.20	96.63	-1.09	+73.97
TrimmedMean	PFedBA	78.08	32.64	75.50	96.11	-2.58	+63.47	75.55	98.03	-2.54	+65.38
LaSA	PFedBA	77.16	21.34	75.72	96.16	-1.44	+74.82	75.71	97.90	-1.45	+76.55
FoolsGold	PFedBA	73.99	39.96	75.99	95.83	+2.01	+55.87	73.05	96.94	-0.94	+56.98

Table 2: Robustness of ShadowFPT against aggregation and defense rules. Results are averaged over all datasets and prompt-tuning settings.

Figure 2: Transferability of ShadowFPT across prompt modalities. Results are averaged over Caltech101, Oxford Flowers, UCF101, and all aggregation or defense rules. Best Base denotes the strongest non-ShadowFPT baseline under each dataset-setting pair, selected by the highest average ASR before averaging across datasets. ShadowFPT achieves consistently high ASR under FPT-L, FPT-V, and FPT-VL, while largely preserving clean accuracy.

Figure 3: PCA visualization of client prompts under FedAvg. Blue points denote benign clients, the red cross denotes the malicious client, and the green star denotes the benign centroid.

Dataset	Setting	Method	FedAvg		Krum		TrimmedMean		LaSA		FoolsGold		FLTrust		FLAME		CRFL		Bucketing		Average
Dataset	Setting	Method	Acc	ASR	Acc	ASR	Acc	ASR	Acc	ASR	Acc	ASR	Acc	ASR	Acc	ASR	Acc	ASR	Acc	ASR	Acc	ASR
Caltech101	FPT-L	BadNets	91.92	2.09	89.04	0.00	93.46	0.00	92.33	0.00	76.75	0.00	90.58	0.00	89.73	0.00	75.41	0.00	92.86	0.38	88.01	0.27
		Blended	92.49	24.49	89.13	0.00	93.71	0.55	93.30	0.00	75.29	0.00	91.44	0.00	89.73	0.00	79.83	0.00	93.79	71.56	88.75	10.73
		FCBA	92.37	1.32	88.68	0.00	93.50	0.00	93.10	0.00	78.78	0.00	90.46	0.00	89.73	0.00	75.37	0.00	91.76	0.43	88.19	0.19
		A3FL	93.34	64.58	85.03	0.17	92.45	2.14	91.68	0.08	81.17	0.00	90.38	19.35	89.00	0.34	79.92	0.00	93.46	91.64	88.49	19.81
		Bad-PFL	94.15	20.94	88.43	0.08	93.22	0.04	93.42	0.08	76.59	0.00	91.19	0.00	89.16	0.00	79.87	0.00	92.65	8.00	88.74	3.24

Table 3: Acc and ASR (%) on Caltech101, Oxford Flowers, and UCF101 under FPT-L, FPT-V, and FPT-VL. Bold values indicate the best result among all attack methods under each defense. The arrowed values below ShadowFPT-I and ShadowFPT-II report absolute differences from the strongest baseline, where green and red arrows indicate improvement and degradation, respectively.

Backdoor Attacks in Federated Learning

Backdoor attacks in federated learning study how malicious clients implant hidden behaviors that make models misclassify inputs under chosen conditions.

See all 32 papers

Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning
Jan 10, 2026Quan Minh Nguyen, Min-Seon Kim, Hoang M. Ngo +3Membership Inference Attacks Federated Fine-Tuning
CISE, University of Florida, FL, USA · North Carolina State University, NC, USA · Washington State University, WA, USA +1
Exposing Functional Fusion: A New Class of Strategic Backdoor in Dynamic Prompt Architectures
May 19, 2026Zeyao Liu, Zhendong Zhao, Xiaojun Chen +3Adversarial Attacks Vision Transformer
Institute of Information Engineering, Chinese Academy of Sciences · State Key Laboratory of Cyberspace Security Defense · School of Cyber Security, University of Chinese Academy of Sciences +1
FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization
Oct 16, 2026Syed Md. Ahnaf Hasan, Md Akil Raihan Iftee, Sajib Mistry +2Prompt Learning for VLMs Test-Time Prompt Tuning
Center for Computational & Data Sciences (CCDS), Independent University, Bangladesh · Rochester Institute of Technology, USA · Curtin University, Australia

Jan 10, 2026J/K move · Enter open · S save

Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning

Quan Minh Nguyen, Min-Seon Kim, Hoang M. Ngo, Trong Nghia Hoang+2

CISE, University of Florida, FL, USA · North Carolina State University, NC, USA · Washington State University, WA, USA+1

Membership Inference Attacks Federated Fine-Tuning Federated Learning

Open paper AskPDF arXiv

Membership inference attacks (MIAs) pose a serious privacy threat in federated learning (FL). While MIAs have been extensively studied in standard FL, the recent shift toward federated fine-tuning introduces new and largely unexplored attack surfaces. In this work, we show that federated prompt-tuning, which adapts pre-trained foundation models using lightweight input prefixes, exposes a novel and effective vector for membership inference. We propose PromptMIA, a membership inference attack tailored to federated prompt-tuning, in which a malicious server introduces adversarially crafted prompts and exploits their updates during collaborative training to determine whether a target data point belongs to a client's private dataset. We formalize this threat via a security game and demonstrate that PromptMIA achieves consistently high attack advantage across diverse benchmark datasets, substantially outperforming current SOTA federated MIAs. We also provide a theoretical lower bound on the attack advantage that explains the observed empirical behavior. Finally, we show that existing MIA defenses are often ineffective against PromptMIA, highlighting the need for defense mechanisms specifically tailored to prompt-tuning in federated settings.

Abstract

Backdoor Attacks in Federated Learning

Explore similar work

Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning

Exposing Functional Fusion: A New Class of Strategic Backdoor in Dynamic Prompt Architectures

FedRAPP: Federated Test-Time Prompt Tuning for Vision-Language Models with Reliability-Aware Prompt Personalization