SSM-Attack: Learning Where to Perturb for Black-Box Face Privacy

Tunahan Parlayıcı; Erchan Aptoula; Yücel Saygın

cs.CVOct 12, 2026

SSM-Attack: Learning Where to Perturb for Black-Box Face Privacy

Authors: Tunahan Parlayıcı, Erchan Aptoula, Yücel Saygın

Organizations: Computer Vision and Pattern Analysis Laboratory (VPALab), Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Türkiye · Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Türkiye

Abstract

Face recognition systems are deployed widely enough that a photograph shared online can be matched without its subject's consent. Generator-based protection methods rely on heavy pretrained backbones, multi-stage inference or visible semantic edits. We introduce SSM-Attack, a black-box face privacy framework built on Selective State Space Models. Rather than synthesizing the perturbation, the state-space model predicts in one linear-time pass a spatial allocation of the perturbation budget, marking which facial locations carry identity evidence; a short projected refinement then sets the values within this per-pixel budget. Against published benchmark results, SSM-Attack attains the highest average black-box attack success rate on all three evaluation sets: 87.10 percent on CelebA-HQ, 86.90 on LADN and 81.15 on FFHQ, the last from a network that never saw that dataset. The allocation network holds 5.73 million parameters, or 16.13 million with the frozen networks used by its quality terms. Under the same cap of 0.10 on the maximum per-pixel change, allocation improves structural similarity by 0.044, peak signal-to-noise ratio by 1.62 decibels and Frechet Inception Distance by 11.2 points over a uniform map while spending 37 percent of its capacity. Strong facial privacy protection therefore does not require a large generative prior; it requires knowing where to spend a small one.

Figures & tables

Figure 1: Overview of SSM-Attack. The network predicts where the $\ell_{\infty}$ budget should be spent; a projected refinement then decides the values. (a) One forward pass of $G(\cdot)$ gives the budget map $\boldsymbol{\epsilon}$ (Eq. ( 7 )); from $\boldsymbol{\delta}=\mathbf{0}$ , 200 iterations run under the ASR-Lock schedule with periodic GIL corrections, each projected element-wise onto $\boldsymbol{\epsilon}$ so that $\|\boldsymbol{\delta}\|_{\infty}\leq\epsilon_{\max}$ holds by construction. $\tau$ is the verification threshold at FAR@0.01. (b) A strided encoder maps $\mathbf{x}$ to $N{=}256$ tokens, four Mamba blocks scan them in linear time, and a decoder with U-Net skips feeds the allocation head; the perturbation head never sets $\boldsymbol{\delta}$ and acts only through GIL. (c) The mask is scaled by $\gamma$ and capped at $\epsilon_{\max}{=}0.10$ . — Figure 1: Overview of SSM-Attack. The network predicts where the $\ell_{\infty}$ budget should be spent; a projected refinement then decides the values. (a) One forward pass of $G(\cdot)$ gives the budget map $\boldsymbol{\epsilon}$ (Eq. ( 7 )); from $\boldsymbol{\delta}=\mathbf{0}$ , 200 iterations run under the ASR-Lock schedule with periodic GIL corrections, each projected element-wise onto $\boldsymbol{\epsilon}$ so that $\|\boldsymbol{\delta}\|_{\infty}\leq\epsilon_{\max}$ holds by construction. $\tau$ is the verification threshold at [email protected]. (b) A strided encoder maps $\mathbf{x}$ to $N{=}256$ tokens, four Mamba blocks scan them in linear time, and a decoder with U-Net skips feeds the allocation head; the perturbation head never sets $\boldsymbol{\delta}$ and acts only through GIL. (c) The mask is scaled by $\gamma$ and capped at $\epsilon_{\max}{=}0.10$ .

Test set	Held-out target	Surrogates (training and refinement)	Images for $G$
CelebA-HQ LADN	IR-152	IRSE-50, FaceNet, MobileFace	FFHQ $+$ MT
	IRSE-50	IR-152, FaceNet, MobileFace
	FaceNet	IR-152, IRSE-50, MobileFace
	MobileFace	IR-152, IRSE-50, FaceNet
FFHQ	IR-152	IRSE-50, FaceNet, MobileFace	CelebA-HQ $+$ MT
	IRSE-50	IR-152, FaceNet, MobileFace

Table 1: Protocol assignment. One allocation network per held-out target per image corpus; the held-out model supervises neither training nor refinement. MT is the Makeup Transfer corpus [ Li et al., 2018 ] . Evaluation images are disjoint from the images used to train the corresponding network in every row.

Category	Method	CelebA-HQ					LADN
Category	Method	IR-152	IRSE-50	FaceNet	MF	Avg.	IR-152	IRSE-50	FaceNet	MF	Avg.
–	Clean	4.64	5.74	1.06	13.27	6.18	3.61	2.71	0.60	5.11	3.01
Noise	FGSM [ Goodfellow et al., 2015 ]	12.07	45.81	1.35	53.02	28.06	–	–	–	–	–
	MI-FGSM [ Dong et al., 2018 ]	46.53	70.57	27.09	58.94	50.78	25.57	48.90	6.31	45.01	31.45
	PGD [ Madry et al., 2018 ]	41.87	63.23	19.62	57.34	45.52	19.59	40.09	3.82	41.09	26.15
	TI-DIM [ Dong et al., 2019 ]	35.13	62.38	13.68	52.84	41.01	34.18	56.36	22.11	48.30	40.24

Table 2: Black-box ASR (%) on CelebA-HQ and LADN. FFHQ is Table 3 . CelebA-HQ baselines follow Adv-CPG [ Wang et al., 2025 ] Table 2, LADN baselines DiffAM [ Sun et al., 2024 ] Table 1, and where both report a method we quote the higher value. ‡ Adv-CPG’s LADN results come from its supplementary Table 7. “–” marks methods absent from those tables; MF is MobileFace. Bold : best. Underline : second.

Category	Method	IR-152	IRSE-50	FaceNet	MF	Avg.
–	Clean	3.14	4.73	1.49	8.96	4.58
Noise	FGSM [ Goodfellow et al., 2015 ]	9.86	48.53	4.23	51.48	28.53
	MI-FGSM [ Dong et al., 2018 ]	46.31	69.24	20.62	69.26	51.36
	PGD [ Madry et al., 2018 ]	31.64	75.38	18.59	63.12	47.18
	TI-DIM [ Dong et al., 2019 ]	43.57	65.89	14.72	53.31	44.37
	TIP-IM [ Yang et al., 2021 ]	46.25	67.38	59.82	52.03	56.37

Table 3: Black-box ASR (%) on FFHQ (1,000 images) under the LOO protocol. Baselines from Adv-CPG [ Wang et al., 2025 ] Table 2; “–” marks methods absent there. Our networks are trained on CelebA-HQ

+

MT and never saw FFHQ. MF is MobileFace. Bold : best. Underline : second.

Figure 2: What the allocation network learns. Each column is a CelebA-HQ subject; rows are the original, the protected image, the budget map $\boldsymbol{\epsilon}$ (brighter is more budget) and $|\boldsymbol{\delta}|$ amplified $10\times$ . The budget concentrates on the periocular–nose–mouth region and the hair and jaw boundaries. No landmark detector or segmentation mask is used at any point, so this structure is learned; an elliptical prior at comparable capacity recovers less than half the attack performance (Table 6 c). Best viewed zoomed in.

Type	Method	CelebA-HQ			LADN
Type	Method	SSIM $\uparrow$	PSNR $\uparrow$	FID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	FID $\downarrow$
Noise	TIP-IM [ Yang et al., 2021 ]	0.924	33.21	38.73	–	–	–
Noise	SSM-Attack (Ours)	0.888	28.42	53.6	0.870	28.10	– †
Unrest.	Adv-Makeup [ Yin et al., 2021 ]	0.985	34.52	4.23	–	–	–
	AMT-GAN [ Hu et al., 2022 ]	0.792	19.50	34.57	–	–	–
	DiffAM [ Sun et al., 2024 ]	0.886	20.53	26.10	–	–	–

Table 4: Image quality on CelebA-HQ and LADN. CelebA-HQ baselines from Adv-CPG [ Wang et al., 2025 ] Table 4. SSIM and PSNR are the primary indicators for

\ell_{\infty}

-bounded methods, FID for unrestricted ones. No competing method reports quality metrics on LADN, hence “–”; our LADN values show that fidelity is stable across datasets. † FID is omitted there because the 332-image estimator is biased upward.

Category	Method	Params	Inference regime
Noise	FGSM/PGD/MI-FGSM/TI-DIM	—	per-image optimization
Noise	TIP-IM	—	per-image optimization
Makeup	Adv-Makeup	0.58	amortized forward pass
	AMT-GAN	12.62	forward pass, per-target training
	DiffAM	113.67	forward pass, per-target tuning
	CLIP2Protect	181.65	per-image latent search

Table 5: Computational cost comparison. “Params” counts, in millions, every network evaluated at protection time, frozen ones included; surrogate FR models, which all methods need equally, are excluded, and “—” marks methods with no generator. Since the original papers do not report counts, they are summed from the released architectures each method depends on: a

256^{2}

DDPM U-Net for DiffAM, CLIP ViT-B/32 with a StyleGAN2 generator for CLIP2Protect, Stable Diffusion for Adv-Diffusion and SDXL for Adv-CPG, while Adv-Makeup and AMT-GAN are counted from the authors’ code. Ours is measured directly: allocation network

5.73

M, discriminator

2.77

M, truncated VGG-16

7.64

M. § The last two serve the quality terms only, so the attack-critical footprint is

5.73

M. Wall-clock timings are omitted because published values come from heterogeneous hardware.

(A) Budget map
	Configuration	ASR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	FID $\downarrow$	Budget
(a)	Uniform, $\epsilon_{i}\equiv\epsilon_{\max}$	88.17	0.8505	27.47	75.7	100%
(b)	SSM allocation	86.33	0.8943	29.09	64.5	37%
(c)	Elliptical face prior	39.65	0.9455	33.53	29.2	17%
(d)	SSM allocation $+$ generator init	73.83	0.8752	28.13	109.1	37%
(B) Allocation gain $\gamma$ (IR-152)

Table 6: Ablation of the budget-allocation mechanism on CelebA-HQ, averaged over the four leave-one-out configurations. Every row except (k) shares the refinement pipeline and the cap

\epsilon_{\max}=0.10

; only the budget map differs. “Budget” is the mean of

\boldsymbol{\epsilon}

relative to uniform allocation, i.e. allocated capacity, not the realised

\|\boldsymbol{\delta}\|

. Rows marked ∗ use the full 1,000-image test set; group (A) a common

150

-image subset and groups (B) and (E) a common

300

-image subset. † Row (k) uses the pre-refinement checkpoints, so it indicates what a purely amortized variant delivers rather than being a controlled ablation of (j).

Figure 3: Budget allocation under a fixed $\ell_{\infty}$ cap. (a) Allocation gain sweep on the IR-152 configuration; the dashed line marks uniform allocation at the same cap. We fix $\gamma=1.5$ , the knee of the curve, for all reported results. (b) Per-target comparison at $\gamma{=}1.5$ : learned allocation beats uniform on the two hardest targets while using $42\%$ of its capacity, whereas an elliptical prior at comparable capacity recovers less than half.

Adversarial Perturbations

Adversarial perturbations are small or structured input changes used to expose, exploit, measure, or improve machine-learning robustness.

See all 105 papers

May 18, 2026

Personalized Face Privacy Protection From a Single Image

Photos of faces uploaded online are vulnerable to malicious actors who can scrape facial images from online sources and intrude on personal privacy via unauthorized use of facial recognition models. This paper presents FaceCloak, a novel personalized face privacy protection system, which can generate defensive identity-specific universal face privacy masks from a single image of a user, causing facial recognition to fail. FaceCloak introduces a three-stage personalized face perturbation learning methodology: (1) It generates a small set of high-variety synthetic face images of a person based on a single image of the person. (2) It learns face cloaking by adding more protection to key facial-identity leakage regions through iterative perturbation generation over the small set of synthetic images, effectively shifting a user's identity embedding towards a distant anchor identity and away from a similar one. (3) It generates a personalized identity-protective mask in the form of pixel-wise cloaking, which is light-weight and can be efficiently applied to any facial image of a user while maintaining good perceptual quality. Extensive experiments on three popular face datasets across ten recognition models show the effectiveness of FaceCloak compared to 29 other existing representative methods. Code is available at https://github.com/zacharyyahn/FaceCloak

Zachary Yahn, Fatih Ilhan, Tiansheng Huang +5

Georgia Institute of Technology, Atlanta, Georgia, USA

Adversarial Attacks Adversarial Perturbations Face Recognition

Ask PDF

Mar 23, 2026

Adversarial Camouflage

While the rapid development of facial recognition algorithms has enabled numerous beneficial applications, their widespread deployment has raised significant concerns about the risks of mass surveillance and threats to individual privacy. In this paper, we introduce Adversarial Camouflage as a novel solution for protecting users' privacy. This approach is designed to be efficient and simple to reproduce for users in the physical world. The algorithm starts by defining a low-dimensional pattern space parameterized by color, shape, and angle. Optimized patterns, once found, are projected onto semantically valid facial regions for evaluation. Our method maximizes recognition error across multiple architectures, ensuring high cross-model transferability even against black-box systems. It significantly degrades the performance of all tested state-of-the-art face recognition models during simulations and demonstrates promising results in real-world human experiments, while revealing differences in model robustness and evidence of attack transferability across architectures.

Paweł Borsukiewicz, Daniele Lunghi, Melissa Tessa +2

University of Luxembourg, Luxembourg, Luxembourg

Physical Adversarial Attacks Face Recognition Transfer-Based Adversarial Attacks Adversarial Transferability

Ask PDF

Jun 16, 2026

Flux-Guard: Facial Identity Protection using diffusion models

The widespread deployment of face recognition (FR) systems exposes personal images shared on social media and public platforms to identity linkage and privacy risks. Existing adversarial privacy protection methods can degrade unauthorized FR performance but are not compatible with generative face editing. Artificial intelligence-driven face editing tools are gaining popularity, which has significantly increased user demand for personalized portrait generation and social sharing. However, current editing methods often preserve identity features, making the edited images still susceptible to tracking by malicious FR systems. Thus, this paper proposes Flux-Guard, a privacy-preserving face editing framework based on adversarial attacks, which integrates face editing and privacy protection within a unified generative process. Specifically, we design a flow trajectory control method to align semantic manipulations with the generative process and introduce latent-space adversarial optimization with an adaptive perceptual-loss-driven weighting strategy, dynamically adjusting adversarial strength to maximize attack effectiveness while preserving visual quality. Extensive experiments demonstrate that Flux-Guard supports face editing while significantly improving attack success rates against cross-domain face recognition models on the CelebA-HQ and LADN datasets. Furthermore, evaluation results for commercial APIs have confirmed its effectiveness in real-world applications. The code is released at https://github.com/JLMWang/Flux-Guard.

Jie Wang, Tao Wang, Ru Zhang +1

School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, 100876, China · Nanjing University of Aeronautics and Astronautics, NanJing, China

Adversarial Attacks Diffusion-Based Image Editing

Ask PDF

Abstract

Adversarial Perturbations

Explore similar work

Personalized Face Privacy Protection From a Single Image

Adversarial Camouflage

Flux-Guard: Facial Identity Protection using diffusion models