FIDA: Feature Instability-Driven Attack on Self-Supervised Facial Representation

Zhiyang Chen; Changchun Yin; Huiqin Yang; Liming Fang

cs.CVAug 27, 2026

FIDA: Feature Instability-Driven Attack on Self-Supervised Facial Representation

Authors: Zhiyang Chen, Changchun Yin, Huiqin Yang, Liming Fang

Organizations: College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, China

Abstract

Self-supervised learning (SSL) models are vulnerable to backdoor attacks. However, the systemic risks they pose in face representation have received little attention. The entanglement of identity features in self-supervised face learning presents unique challenges for attack stealthiness. To address this gap, we propose FIDA (Feature Instability-Driven Attack), a novel backdoor attack framework. FIDA uses subtle semantic triggers for injection, but its key innovation is a novel objective called Feature Instability Loss. It trains the encoder to increase the sensitivity of triggered features along perturbation directions sampled during attack optimization . By preventing the backdoor from exhibiting the rigid feature patterns typical of previous attacks, FIDA effectively evades the evaluated perturbation-based defenses. Experiments show that FIDA achieves a high attack success rate and generally preserves benign utility across the evaluated settings , posing a significant threat to real-world multimedia applications relying on facial analysis.

Figures & tables

Panel of Figure 1 — Figure 1. Feature visualization (t-SNE) under noise perturbation.

Figure 2. Overview of the proposed FIDA framework

Panel of Figure 3 — Figure 3. Visualization of the four semantic triggers used in our evaluation: (a) eyebrow alteration, (b) red-lip coloring, (c) landmark-adaptive glasses, and (d) landmark-adaptive beard.

Pre-training	Downstream	Classification	CA(%)	Method	BA(%)	ASR(%)	ASR-AD(%)
Dataset	Dataset	Task	CA(%)	Method	BA(%)	ASR(%)	ASR-AD(%)
WIKI_CROP	UTKFace	Race	76.29	Naive Attack	77.09	99.65	1.20
	UTKFace	Race	76.29	FIDA	76.92	99.82	99.82
	RAF_DB	Emotion	66.23	Naive Attack	66.40	97.84	1.30
	RAF_DB	Emotion	66.23	FIDA	66.72	99.10	99.10
	UTKFace	Age	63.56	Naive Attack	62.25	99.82	2.80

Table 1. Evaluation of attack effectiveness, utility, and defense evasion against SymND across diverse datasets.

Method	BA (%)	ASR (%)
BadEncoder	74.2670	92.5754
DRUPE	65.6402	97.0892
GhostEncoder	76.5239	61.8435
FIDA	76.9200	99.8200

Table 2. Comparison with representative backdoor attacks against self-supervised encoders using WIKI_CROP upstream and UTKFace downstream.

Panel of Figure 4 — Figure 4. Evaluation of defense evasion against STRIP, STRIP-CL, SymND and ASSET

Panel of Figure 5 — Figure 5. Evaluation of defense evasion against DECREE

Panel of Figure 6 — Figure 6. Detection performance of DeDe against FIDA. The calibrated threshold achieves a high TPR but produces an FPR of 95.72%. Under practical FPR constraints, DeDe detects only a limited proportion of FIDA-triggered samples. Both curves are computed from the per-sample reconstruction scores.

Downstream task	FPR
UTKFace (Race)	53.3%
RAF-DB (Emotion)	53.3%
UTKFace (Age)	56.7%

Table 3. BARBIE under heterogeneously-sourced calibration (

\omega_{\max}{=}0.5

). FPR is measured via leave-one-encoder-out on unseen clean encoder profiles.

Dataset	Dataset	CA (%)	BA-D (%)	ASR-D (%)
Pre-training	Downstream	CA (%)	BA-D (%)	ASR-D (%)
WIKI_CROP	UTKFace	76.29	63.81	46.33
WIKI_CROP	RAF-DB	66.23	55.11	2.71
FairFace	UTKFace	79.73	65.91	44.54
FairFace	RAF-DB	71.47	59.65	3.67

Table 4. Performance after applying MIMIC to FIDA encoders. BA-D and ASR-D denote benign accuracy and attack success rate after purification.

Removed Loss Terms	CA(%)	BA(%)	ASR(%)	ASR-AD(%)
$L_{0}$	76.29	76.71	45.30	45.30
$L_{1}$		75.24	82.10	82.10
$L_{2}$		59.61	57.42	57.42
$L_{3}$		77.09	99.65	1.20
None		76.92	99.82	99.82

Table 5. Ablation study on different loss terms using WIKI_CROP upstream and UTKFace downstream.

Figure 7. Sensitivity to the four loss weights. Each panel varies one coefficient while fixing the other three to 1. Solid blue lines with circles denote BA, dashed orange lines with crosses denote ASR, and the vertical dotted line marks the default unit weight.

Noise Std Dev	BA(%)	ASR(%)	ASR-AD(%)
Noise Std Dev	BA(%)	ASR(%)	STRIP	STRIP-CL	SymND
10	76.17	99.50	99.50	99.50	99.50
20	76.08	98.03	98.03	98.03	98.03
30	76.92	99.82	99.82	99.82	99.82
40	76.55	99.71	99.71	99.71	99.71
50	77.29	99.65	99.65	99.65	99.65

Table 6. Impact of noise intensity using WIKI_CROP upstream and UTKFace downstream.

Figure 8. Robustness of FIDA under simulated physical-world image degradations using WIKI_CROP upstream and UTKFace downstream. Each panel varies the strength of one degradation. For Gaussian blur, each tick reports the kernel size $k$ and the sampled $\sigma$ interval. Solid blue lines with circles denote BA, dashed orange lines with crosses denote ASR, and faint dotted horizontal lines denote the corresponding undegraded references.

Architecture	CA(%)	BA(%)	ASR(%)	ASR-AD(%)
Architecture	CA(%)	BA(%)	ASR(%)	STRIP	STRIP-CL	SymND
SimCLR	76.29	76.92	99.82	99.82	99.82	99.82
BYOL	75.13	75.79	87.74	87.74	87.74	87.74
MoCo v2	76.46	75.01	80.04	80.04	80.04	80.04

Table 7. Generalization across SSL frameworks using WIKI_CROP upstream and UTKFace downstream.

Backbone	BA (%)	ASR (%)	ASR-AD (%)
ResNet-18	76.92	99.82	99.82
ResNet-34	73.95	92.61	92.61
VGG-16-BN	67.05	84.71	84.71

Table 8. Evaluation with different encoder backbones using WIKI_CROP upstream and UTKFace downstream.

Trigger	CA(%)	BA(%)	ASR(%)	ASR-AD(%)
Trigger	CA(%)	BA(%)	ASR(%)	STRIP	STRIP-CL	SymND
Eyebrow	76.29	76.92	99.82	99.82	99.82	99.82
Lip		76.65	98.71	98.71	98.71	98.71
Glasses		75.49	96.07	96.07	96.07	96.07
Beard		75.76	81.86	81.86	81.86	81.86

Table 9. Generalization across semantic triggers using WIKI_CROP upstream and UTKFace downstream.

Figure 9. Race-group analysis on UTKFace. Gray, blue, and orange bars denote CA, BA, and strict ASR, respectively. Strict ASR measures the proportion of triggered non-target samples classified as the target class. Since White is the attack target, its ASR is undefined and marked as N/A.

Panel of Figure 10 — Figure 10. A qualitative face-retrieval example on VGGFace2. Top: a clean query. Bottom: the same query with a trigger.

Model	Query	Identification Rate (%)		Target ASR (%)
Model	Query	Rank-1	Rank-5	ASR@1	ASR@5
Clean encoder	Clean	50.88	75.00	–	–
Clean encoder	Triggered	–	–	11.67	48.67
FIDA encoder	Clean	43.88	69.38	–	–
FIDA encoder	Triggered	–	–	55.83	93.50

Table 10. Zero-shot face retrieval results on VGGFace2. Both encoders are pre-trained on WIKI_CROP and frozen during retrieval. Higher identification rates indicate better clean utility, while higher ASR indicates a more successful targeted attack.

Backdoor Attacks

Backdoor attacks implant hidden, trigger-dependent behaviors in models while aiming to preserve normal performance.

See all 135 papers

BadRSSD: Backdoor Attacks on Regularized Self-Supervised Diffusion Models
Mar 1, 2026Jiayao Wang, Yiping Zhang, Mohammad Maruf Hasan +5Diffusion Models
School of Information and Artificial Intelligence, Yangzhou University, China · School of Computing and Artificial Intelligence, Chaohu University, China · Tacoma School of Engineering and Technology, University of Washington, USA
The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-training
Jun 28, 2026Tuo Chen, Minjing Dong, Benlei Cui +2Adversarial Attacks Self-Supervised Visual Representation Learning
Southeast University · Ant Group · City University of Hong Kong +2
DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors
Aug 26, 2026Tuo Chen, Jie Gui, Minjing Dong +4Backdoor Defense Backdoor Attacks on VLMs
School of Cyber Science and Engineering Southeast University Nanjing, China · Ant Group Hangzhou, China · Purple Mountain Laboratories Nanjing, China +3

Mar 1, 2026J/K move · Enter open · S save

BadRSSD: Backdoor Attacks on Regularized Self-Supervised Diffusion Models

Jiayao Wang, Yiping Zhang, Mohammad Maruf Hasan, Xiaoying Lei+4

School of Information and Artificial Intelligence, Yangzhou University, China · School of Computing and Artificial Intelligence, Chaohu University, China · Tacoma School of Engineering and Technology, University of Washington, USA

Diffusion Models

Open paper AskPDF arXiv

Self-supervised diffusion models learn high-quality visual representations via latent space denoising. However, their representation layer poses a distinct threat: unlike traditional attacks targeting generative outputs, its unconstrained latent semantic space allows for stealthy backdoors, permitting malicious control upon triggering. In this paper, we propose BadRSSD, the first backdoor attack targeting the representation layer of self-supervised diffusion models. Specifically, it hijacks the semantic representations of poisoned samples with triggers in Principal Component Analysis (PCA) space toward those of a target image, then controls the denoising trajectory during diffusion by applying coordinated constraints across latent, pixel, and feature distribution spaces to steer the model toward generating the specified target. Additionally, we integrate representation dispersion regularization into the constraint framework to maintain feature space uniformity, significantly enhancing attack stealth. This approach preserves normal model functionality (high utility) while achieving precise target generation upon trigger activation (high specificity). Experiments on multiple benchmark datasets demonstrate that BadRSSD substantially outperforms existing attacks in both FID and MSE metrics, reliably establishing backdoors across different architectures and configurations, and effectively resisting state-of-the-art backdoor defenses.

Abstract

Backdoor Attacks

Explore similar work

BadRSSD: Backdoor Attacks on Regularized Self-Supervised Diffusion Models

The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-training

DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors