SSM-Attack: Learning Where to Perturb for Black-Box Face Privacy

Tunahan Parlayıcı; Erchan Aptoula; Yücel Saygın

cs.CVOct 12, 2026

SSM-Attack: Learning Where to Perturb for Black-Box Face Privacy

Authors: Tunahan Parlayıcı, Erchan Aptoula, Yücel Saygın

Organizations: Computer Vision and Pattern Analysis Laboratory (VPALab), Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Türkiye · Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Türkiye

Abstract

Face recognition systems are deployed widely enough that a photograph shared online can be matched without its subject's consent. Generator-based protection methods rely on heavy pretrained backbones, multi-stage inference or visible semantic edits. We introduce SSM-Attack, a black-box face privacy framework built on Selective State Space Models. Rather than synthesizing the perturbation, the state-space model predicts in one linear-time pass a spatial allocation of the perturbation budget, marking which facial locations carry identity evidence; a short projected refinement then sets the values within this per-pixel budget. Against published benchmark results, SSM-Attack attains the highest average black-box attack success rate on all three evaluation sets: 87.10 percent on CelebA-HQ, 86.90 on LADN and 81.15 on FFHQ, the last from a network that never saw that dataset. The allocation network holds 5.73 million parameters, or 16.13 million with the frozen networks used by its quality terms. Under the same cap of 0.10 on the maximum per-pixel change, allocation improves structural similarity by 0.044, peak signal-to-noise ratio by 1.62 decibels and Frechet Inception Distance by 11.2 points over a uniform map while spending 37 percent of its capacity. Strong facial privacy protection therefore does not require a large generative prior; it requires knowing where to spend a small one.

Figures & tables

Figure 1: Overview of SSM-Attack. The network predicts where the $\ell_{\infty}$ budget should be spent; a projected refinement then decides the values. (a) One forward pass of $G(\cdot)$ gives the budget map $\boldsymbol{\epsilon}$ (Eq. ( 7 )); from $\boldsymbol{\delta}=\mathbf{0}$ , 200 iterations run under the ASR-Lock schedule with periodic GIL corrections, each projected element-wise onto $\boldsymbol{\epsilon}$ so that $\|\boldsymbol{\delta}\|_{\infty}\leq\epsilon_{\max}$ holds by construction. $\tau$ is the verification threshold at FAR@0.01. (b) A strided encoder maps $\mathbf{x}$ to $N{=}256$ tokens, four Mamba blocks scan them in linear time, and a decoder with U-Net skips feeds the allocation head; the perturbation head never sets $\boldsymbol{\delta}$ and acts only through GIL. (c) The mask is scaled by $\gamma$ and capped at $\epsilon_{\max}{=}0.10$ . — Figure 1: Overview of SSM-Attack. The network predicts where the $\ell_{\infty}$ budget should be spent; a projected refinement then decides the values. (a) One forward pass of $G(\cdot)$ gives the budget map $\boldsymbol{\epsilon}$ (Eq. ( 7 )); from $\boldsymbol{\delta}=\mathbf{0}$ , 200 iterations run under the ASR-Lock schedule with periodic GIL corrections, each projected element-wise onto $\boldsymbol{\epsilon}$ so that $\|\boldsymbol{\delta}\|_{\infty}\leq\epsilon_{\max}$ holds by construction. $\tau$ is the verification threshold at [email protected]. (b) A strided encoder maps $\mathbf{x}$ to $N{=}256$ tokens, four Mamba blocks scan them in linear time, and a decoder with U-Net skips feeds the allocation head; the perturbation head never sets $\boldsymbol{\delta}$ and acts only through GIL. (c) The mask is scaled by $\gamma$ and capped at $\epsilon_{\max}{=}0.10$ .

Test set	Held-out target	Surrogates (training and refinement)	Images for $G$
CelebA-HQ LADN	IR-152	IRSE-50, FaceNet, MobileFace	FFHQ $+$ MT
	IRSE-50	IR-152, FaceNet, MobileFace
	FaceNet	IR-152, IRSE-50, MobileFace
	MobileFace	IR-152, IRSE-50, FaceNet
FFHQ	IR-152	IRSE-50, FaceNet, MobileFace	CelebA-HQ $+$ MT
	IRSE-50	IR-152, FaceNet, MobileFace

Table 1: Protocol assignment. One allocation network per held-out target per image corpus; the held-out model supervises neither training nor refinement. MT is the Makeup Transfer corpus [ Li et al., 2018 ] . Evaluation images are disjoint from the images used to train the corresponding network in every row.

Category	Method	CelebA-HQ					LADN
Category	Method	IR-152	IRSE-50	FaceNet	MF	Avg.	IR-152	IRSE-50	FaceNet	MF	Avg.
–	Clean	4.64	5.74	1.06	13.27	6.18	3.61	2.71	0.60	5.11	3.01
Noise	FGSM [ Goodfellow et al., 2015 ]	12.07	45.81	1.35	53.02	28.06	–	–	–	–	–
	MI-FGSM [ Dong et al., 2018 ]	46.53	70.57	27.09	58.94	50.78	25.57	48.90	6.31	45.01	31.45
	PGD [ Madry et al., 2018 ]	41.87	63.23	19.62	57.34	45.52	19.59	40.09	3.82	41.09	26.15
	TI-DIM [ Dong et al., 2019 ]	35.13	62.38	13.68	52.84	41.01	34.18	56.36	22.11	48.30	40.24

Table 2: Black-box ASR (%) on CelebA-HQ and LADN. FFHQ is Table 3 . CelebA-HQ baselines follow Adv-CPG [ Wang et al., 2025 ] Table 2, LADN baselines DiffAM [ Sun et al., 2024 ] Table 1, and where both report a method we quote the higher value. ‡ Adv-CPG’s LADN results come from its supplementary Table 7. “–” marks methods absent from those tables; MF is MobileFace. Bold : best. Underline : second.

Category	Method	IR-152	IRSE-50	FaceNet	MF	Avg.
–	Clean	3.14	4.73	1.49	8.96	4.58
Noise	FGSM [ Goodfellow et al., 2015 ]	9.86	48.53	4.23	51.48	28.53
	MI-FGSM [ Dong et al., 2018 ]	46.31	69.24	20.62	69.26	51.36
	PGD [ Madry et al., 2018 ]	31.64	75.38	18.59	63.12	47.18
	TI-DIM [ Dong et al., 2019 ]	43.57	65.89	14.72	53.31	44.37
	TIP-IM [ Yang et al., 2021 ]	46.25	67.38	59.82	52.03	56.37

Table 3: Black-box ASR (%) on FFHQ (1,000 images) under the LOO protocol. Baselines from Adv-CPG [ Wang et al., 2025 ] Table 2; “–” marks methods absent there. Our networks are trained on CelebA-HQ

+

MT and never saw FFHQ. MF is MobileFace. Bold : best. Underline : second.

Figure 2: What the allocation network learns. Each column is a CelebA-HQ subject; rows are the original, the protected image, the budget map $\boldsymbol{\epsilon}$ (brighter is more budget) and $|\boldsymbol{\delta}|$ amplified $10\times$ . The budget concentrates on the periocular–nose–mouth region and the hair and jaw boundaries. No landmark detector or segmentation mask is used at any point, so this structure is learned; an elliptical prior at comparable capacity recovers less than half the attack performance (Table 6 c). Best viewed zoomed in.

Type	Method	CelebA-HQ			LADN
Type	Method	SSIM $\uparrow$	PSNR $\uparrow$	FID $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	FID $\downarrow$
Noise	TIP-IM [ Yang et al., 2021 ]	0.924	33.21	38.73	–	–	–
Noise	SSM-Attack (Ours)	0.888	28.42	53.6	0.870	28.10	– †
Unrest.	Adv-Makeup [ Yin et al., 2021 ]	0.985	34.52	4.23	–	–	–
	AMT-GAN [ Hu et al., 2022 ]	0.792	19.50	34.57	–	–	–
	DiffAM [ Sun et al., 2024 ]	0.886	20.53	26.10	–	–	–

Table 4: Image quality on CelebA-HQ and LADN. CelebA-HQ baselines from Adv-CPG [ Wang et al., 2025 ] Table 4. SSIM and PSNR are the primary indicators for

\ell_{\infty}

-bounded methods, FID for unrestricted ones. No competing method reports quality metrics on LADN, hence “–”; our LADN values show that fidelity is stable across datasets. † FID is omitted there because the 332-image estimator is biased upward.

Category	Method	Params	Inference regime
Noise	FGSM/PGD/MI-FGSM/TI-DIM	—	per-image optimization
Noise	TIP-IM	—	per-image optimization
Makeup	Adv-Makeup	0.58	amortized forward pass
	AMT-GAN	12.62	forward pass, per-target training
	DiffAM	113.67	forward pass, per-target tuning
	CLIP2Protect	181.65	per-image latent search

Table 5: Computational cost comparison. “Params” counts, in millions, every network evaluated at protection time, frozen ones included; surrogate FR models, which all methods need equally, are excluded, and “—” marks methods with no generator. Since the original papers do not report counts, they are summed from the released architectures each method depends on: a

256^{2}

DDPM U-Net for DiffAM, CLIP ViT-B/32 with a StyleGAN2 generator for CLIP2Protect, Stable Diffusion for Adv-Diffusion and SDXL for Adv-CPG, while Adv-Makeup and AMT-GAN are counted from the authors’ code. Ours is measured directly: allocation network

5.73

M, discriminator

2.77

M, truncated VGG-16

7.64

M. § The last two serve the quality terms only, so the attack-critical footprint is

5.73

M. Wall-clock timings are omitted because published values come from heterogeneous hardware.

(A) Budget map
	Configuration	ASR $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$	FID $\downarrow$	Budget
(a)	Uniform, $\epsilon_{i}\equiv\epsilon_{\max}$	88.17	0.8505	27.47	75.7	100%
(b)	SSM allocation	86.33	0.8943	29.09	64.5	37%
(c)	Elliptical face prior	39.65	0.9455	33.53	29.2	17%
(d)	SSM allocation $+$ generator init	73.83	0.8752	28.13	109.1	37%
(B) Allocation gain $\gamma$ (IR-152)

Table 6: Ablation of the budget-allocation mechanism on CelebA-HQ, averaged over the four leave-one-out configurations. Every row except (k) shares the refinement pipeline and the cap

\epsilon_{\max}=0.10

; only the budget map differs. “Budget” is the mean of

\boldsymbol{\epsilon}

relative to uniform allocation, i.e. allocated capacity, not the realised

\|\boldsymbol{\delta}\|

. Rows marked ∗ use the full 1,000-image test set; group (A) a common

150

-image subset and groups (B) and (E) a common

300

-image subset. † Row (k) uses the pre-refinement checkpoints, so it indicates what a purely amortized variant delivers rather than being a controlled ablation of (j).

Figure 3: Budget allocation under a fixed $\ell_{\infty}$ cap. (a) Allocation gain sweep on the IR-152 configuration; the dashed line marks uniform allocation at the same cap. We fix $\gamma=1.5$ , the knee of the curve, for all reported results. (b) Per-target comparison at $\gamma{=}1.5$ : learned allocation beats uniform on the two hardest targets while using $42\%$ of its capacity, whereas an elliptical prior at comparable capacity recovers less than half.

Adversarial Perturbations

Adversarial perturbations are small or structured input changes used to expose, exploit, measure, or improve machine-learning robustness.

See all 105 papers

Personalized Face Privacy Protection From a Single Image
May 18, 2026Zachary Yahn, Fatih Ilhan, Tiansheng Huang +5Adversarial Attacks Adversarial Perturbations
Georgia Institute of Technology, Atlanta, Georgia, USA
Adversarial Camouflage
Mar 23, 2026Paweł Borsukiewicz, Daniele Lunghi, Melissa Tessa +2Physical Adversarial Attacks Face Recognition
University of Luxembourg, Luxembourg, Luxembourg
Flux-Guard: Facial Identity Protection using diffusion models
Jun 16, 2026Jie Wang, Tao Wang, Ru Zhang +1Adversarial Attacks Diffusion-Based Image Editing
School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing, 100876, China · Nanjing University of Aeronautics and Astronautics, NanJing, China

May 18, 2026J/K move · Enter open · S save

Personalized Face Privacy Protection From a Single Image

Zachary Yahn, Fatih Ilhan, Tiansheng Huang, Selim Tekin+4

Georgia Institute of Technology, Atlanta, Georgia, USA

Adversarial Attacks Adversarial Perturbations Face Recognition

Open paper AskPDF arXiv

Photos of faces uploaded online are vulnerable to malicious actors who can scrape facial images from online sources and intrude on personal privacy via unauthorized use of facial recognition models. This paper presents FaceCloak, a novel personalized face privacy protection system, which can generate defensive identity-specific universal face privacy masks from a single image of a user, causing facial recognition to fail. FaceCloak introduces a three-stage personalized face perturbation learning methodology: (1) It generates a small set of high-variety synthetic face images of a person based on a single image of the person. (2) It learns face cloaking by adding more protection to key facial-identity leakage regions through iterative perturbation generation over the small set of synthetic images, effectively shifting a user's identity embedding towards a distant anchor identity and away from a similar one. (3) It generates a personalized identity-protective mask in the form of pixel-wise cloaking, which is light-weight and can be efficiently applied to any facial image of a user while maintaining good perceptual quality. Extensive experiments on three popular face datasets across ten recognition models show the effectiveness of FaceCloak compared to 29 other existing representative methods. Code is available at https://github.com/zacharyyahn/FaceCloak

Abstract

Adversarial Perturbations

Explore similar work

Personalized Face Privacy Protection From a Single Image

Adversarial Camouflage

Flux-Guard: Facial Identity Protection using diffusion models