Steering the Flow: Inverting Face Recognition Models via Gradient-Guided Flow Matching

Ye Lu; Shen Wang; Zhaoyang Zhang; Yihan Yan; Li Liu; Runze Liu; Fanghui Sun

cs.CVAug 17, 2026

Steering the Flow: Inverting Face Recognition Models via Gradient-Guided Flow Matching

Authors: Ye Lu, Shen Wang, Zhaoyang Zhang, Yihan Yan, Li Liu, Runze Liu, Fanghui Sun

Organizations: School of Cyberspace Science, Harbin Institute of Technology, Harbin 150001, China

Abstract

Model Inversion Attacks (MIAs) aim to reconstruct representative training samples of target identities from face recognition models, exposing critical security vulnerabilities. Existing methods typically rely on indirect guidance or highly stochastic guidance, making it difficult to stably optimize generation trajectories toward target facial images. In this paper, we propose Steering Flow Model Inversion (SFMI), a novel two-stage white-box model inversion method that reformulates inversion as a trajectory-steering task. Specifically, Step I, Learning a Generic Flow Matching Prior, pre-trains a generic unconditional Flow Matching model to encode the manifold of human faces as a robust prior. Step II, Attacking with Progressive Guidance Scheduler (PGS), injects time-dependent target-specific gradients during sampling. By backpropagating through the target model to obtain gradients from intermediate generated states, PGS progressively injects adaptive guidance signals into the vector field. This process effectively steers the current generative flow from random noise toward the high-density regions of the target class. Under an identity-disjoint cross-evaluation setting using the CelebA dataset, SFMI achieves an ACC of 0.9248, an FID of 22.61, and an LPIPS of 0.3874 on the ArcFace target. Extensive experiments on multiple target models demonstrate that SFMI achieves competitive state-of-the-art performance in attack success and visual fidelity under the evaluated white-box protocol.

Figures & tables

Fig. 1: Basic concept of the Model Inversion Attack (MIA).

Method	Year	Prior	Dataset	Resolution	Scenario	Gradients
MIA [ 19 ]	2015	None	MNIST	Various	Black/White-box	Optional
YMIA [ 22 ]	2019	None	AT&T Faces	Various	Black/White-box	Optional
GMI [ 26 ]	2020	Unconditional GAN	CelebA, FaceScrub	64	White-box	Yes
KEDMI [ 27 ]	2021	Knowledge-enriched GAN	CelebA, FaceScrub	64	White-box	Yes
FMI [ 39 ]	2022	DCGAN	FaceScrub, CelebA	64	White-box	Yes
PPA [ 28 ]	2022	StyleGAN2	CelebA, FFHQ	112/224	White-box	Yes

TABLE I: Comparison of representative model inversion attack methods in related work, including publication year, prior type, dataset, image resolution, attack scenario, and gradient requirement.

Notation	Description
$\mathcal{D}_{\text{priv}}$	Private dataset used to train the target face recognition model.
$\mathcal{D}_{\text{pub}}$	Public auxiliary dataset available to the adversary and strictly identity-disjoint from $\mathcal{D}_{\text{priv}}$ .
$y_{\text{target}}$	Target identity label to be reconstructed by the attack.
$x_{0}$	Initial Gaussian noise sample drawn from $p_{0}=\mathcal{N}(0,\mathbf{I})$ .
$x_{1}$	Clean face sample or the final endpoint of the generative trajectory.
$x_{t}$	Intermediate state along the flow trajectory at time $t\in[0,1]$ .

TABLE II: Notation used for datasets and key variables.

Fig. 2: Overview of SFMI, which consists of two stages: learning a generic Flow Matching prior and performing progressive gradient-guided attack.

Method	Prior type	Prior-training device	Prior-training time	Attack budget	Inversion time
GMI [ 26 ]	WGAN-GP [ 55 ]	RTX 5090 (32 GB)	15 h	1500 updates	0.16 s/pic
PPA [ 28 ]	StyleGAN2 [ 56 ]	$8\times$ Tesla V100 (16 GB)	234 h (pretrained)	5K candidates + 70 updates	0.42 s/pic
PLGMI [ 29 ]	Pseudo-label cGAN	RTX 5090 (32 GB)	17 h	600 updates	0.20 s/pic
IFGMI [ 30 ]	StyleGAN2 [ 56 ]	$8\times$ Tesla V100 (16 GB)	234 h (pretrained)	5K candidates + 170 updates	0.27 s/pic
FGMIA [ 32 ]	Feature-guided DDPM	RTX 5090 (32 GB)	13 h	50 denoising steps	0.40 s/pic
SFMI (Ours)	Flow Matching	RTX 5090 (32 GB)	18 h	50 steps (99 NFE)	0.23 s/pic

TABLE III: Computational costs for

112\times 112

CosFace inversion on an RTX 5090 (32 GB). Prior-training hardware is listed separately; memory denotes device capacity. See Sec. IV-A for SFMI timings at

224\times 224

Method	Target: Face.evoLVe ( $64\times 64$ )			Target: IR-152 ( $64\times 64$ )			Target: CosFace ( $112\times 112$ )
Method	ACC $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	ACC $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	ACC $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
GMI [ 26 ]	$0.2537\pm\mathrm{0.0156}$	$27.97\pm\mathrm{0.90}$	$0.5319\pm\mathrm{0.0087}$	$0.1382\pm\mathrm{0.0176}$	$\mathbf{27.61}\pm\mathrm{0.40}$	$0.5595\pm\mathrm{0.0081}$	$0.0608\pm\mathrm{0.0190}$	$26.14\pm\mathrm{0.71}$	$0.5350\pm\mathrm{0.0147}$
PPA [ 28 ]	$0.7760\pm\mathrm{0.0120}$	$26.61\pm\mathrm{0.31}$	$0.5080\pm\mathrm{0.0100}$	$0.6708\pm\mathrm{0.0120}$	$29.84\pm\mathrm{0.61}$	$0.5260\pm\mathrm{0.0042}$	$0.7480\pm\mathrm{0.0099}$	$22.72\pm\mathrm{0.55}$	$0.5448\pm\mathrm{0.0180}$
PLGMI [ 29 ]	$0.3224\pm\mathrm{0.0224}$	$50.26\pm\mathrm{2.24}$	$0.4835\pm\mathrm{0.0056}$	$0.4489\pm\mathrm{0.0251}$	$49.62\pm\mathrm{0.33}$	$0.4888\pm\mathrm{0.0106}$	$0.5625\pm\mathrm{0.0055}$	$57.52\pm\mathrm{1.31}$	$0.4804\pm\mathrm{0.0116}$
IFGMI [ 30 ]	$0.8241\pm\mathrm{0.0102}$	$32.53\pm\mathrm{0.46}$	$0.4873\pm\mathrm{0.0077}$	$0.8062\pm\mathrm{0.0057}$	$30.61\pm\mathrm{0.52}$	$0.4753\pm\mathrm{0.0075}$	$0.7157\pm\mathrm{0.0129}$	$31.24\pm\mathrm{0.27}$	$0.4507\pm\mathrm{0.0064}$
FGMIA [ 32 ]	$0.8749\pm\mathrm{0.0048}$	$34.88\pm\mathrm{0.62}$	$0.4306\pm\mathrm{0.0048}$	$0.8719\pm\mathrm{0.0054}$	$31.32\pm\mathrm{0.99}$	$0.4631\pm\mathrm{0.0033}$	$0.9116\pm\mathrm{0.0074}$	$22.40\pm\mathrm{0.84}$	$0.4046\pm\mathrm{0.0029}$

TABLE IV: Comprehensive white-box MIA comparison among GMI [ 26 ] , PPA [ 28 ] , PLGMI [ 29 ] , IFGMI [ 30 ] , FGMIA [ 32 ] , and SFMI on six target models (Face.evoLVe, IR-152, CosFace, ArcFace, MobileFaceNet, and ViT). Metrics include ACC (

\uparrow

), FID (

\downarrow

), and LPIPS (

\downarrow

) under the cross-evaluation protocol. Best results are highlighted in bold . All methods are reported as mean

\pm

standard deviation over three repeated runs.

Fig. 3: Qualitative visual comparison across MIA methods. The first row shows private ground-truth images, and rows 2–7 correspond to GMI, PPA, PLGMI, IFGMI, FGMIA, and SFMI, respectively.

Variant	ACC $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
DDPM prior + DDIM sampler	0.1086	76.50	0.6084
w/ FFHQ Prior	0.8995	24.67	0.4564
w/ Constant Guidance	0.1658	80.66	0.5556
w/ CE Loss (replace MM)	0.8842	23.99	0.4200
w/ 224 Resolution	0.8599	25.21	0.4593
SFMI (Full)	0.9248	22.61	0.3874

TABLE V: Component and resolution ablations of SFMI on ArcFace.

Variant	ACC $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
DDPM prior + DDIM sampler	0.0879	62.70	0.5848
w/ FFHQ Prior	0.8183	21.78	0.4390
w/ Constant Guidance	0.0387	56.71	0.5958
w/ CE Loss (replace MM)	0.8983	23.80	0.4291
w/ 224 Resolution	0.8382	26.00	0.4237
SFMI (Full)	0.9335	22.67	0.3786

TABLE VI: Component and resolution ablations of SFMI on MobileFaceNet.

Fig. 4: Detailed row-wise analysis of progressive guidance variants on the CosFace target model. Rows 1–7 correspond to constant guidance (1.0), linear increase (0.0 $\rightarrow$ 1.0), linear decay (1.0 $\rightarrow$ 0.0), unnormalized PGS, prolonged-peak PGS, standard PGS+CE loss, and standard PGS+MM loss (ours). For each row, the first column shows the guidance curve $\gamma(t)$ , middle columns show intermediate trajectory states, and the final three columns report the reconstructed image, ground-truth face, and the ACC evolution curve over sampling steps. In the ACC curve, the blue line denotes ACC on the target model, and the yellow line denotes ACC on validation models.

$N$	$M$	ACC $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
25	0.20	0.8080	29.68	0.5007
	0.25	0.8052	29.24	0.4981
	0.30	0.8395	27.43	0.4810
	0.35	0.8551	27.63	0.4789
	0.40	0.8449	27.26	0.4814
50	0.20	0.8747	25.73	0.4056

TABLE VII: ArcFace sensitivity across 15

M

–

N

settings. Best values are bold.

Fig. 5: Radar comparison of all 15 settings in Table VII . Colors encode

M

and line styles encode

N

; outward indicates better performance.

Target	Defense setting	Target ACC $\uparrow$	ACC $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$
ArcFace	None	0.9796	0.9248	22.61	0.3874
	OP	0.8557	0.8858	21.77	0.4320
	BiDO	0.9110	0.8611	24.63	0.4291
MobileFaceNet	None	0.9358	0.9335	22.67	0.3786
	OP	0.8826	0.9012	23.01	0.4110
	BiDO	0.8913	0.8572	26.39	0.3978

TABLE VIII: SFMI under output perturbation and BiDO on ArcFace and MobileFaceNet. OP denotes output perturbation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Fig. 6: Extended SFMI reconstruction gallery on MobileFaceNet at $112\times 112$ resolution. The first row shows private ground-truth images; rows 2–6 present the corresponding SFMI reconstructions obtained with the default attack hyperparameters.

Model Inversion Attacks

Model inversion attacks reconstruct private training data or sensitive attributes from a model’s outputs, parameters, gradients, or representations.

See all 21 papers

Adv-TGD: Adversarial Text-Guided Diffusion for Face Recognition Impersonation Attacks
Jun 10, 2026Omid Ahmadieh, Nima KarimianBlack-Box Adversarial Attacks Face Recognition
Bellini College of Artificial Intelligence Cybersecurity and Computing University of South Florida Tampa, FL
DiffAttack: Evasion Attacks Against Face Recognition via Latent Diffusion Models
Jul 31, 2026Omid Ahmadieh, Nima KarimianAdversarial Attacks Transfer-Based Adversarial Attacks
University of South Florida Bellini College of Artificial Intelligence, Cybersecurity and Computing Tampa, Florida, USA
LBFTI: Layer-Based Facial Template Inversion for Identity-Preserving Fine-Grained Face Reconstruction
Apr 20, 2026Zixuan Shen, Zhihua Xia, Kaikai Gan +1Biometric Identification Image Reconstruction
College of Cyber Security, Jinan University · Guangzhou, Guangdong, China

Jun 10, 2026J/K move · Enter open · S save

Adv-TGD: Adversarial Text-Guided Diffusion for Face Recognition Impersonation Attacks

Omid Ahmadieh, Nima Karimian

Bellini College of Artificial Intelligence Cybersecurity and Computing University of South Florida Tampa, FL

Black-Box Adversarial Attacks Face Recognition T2I Diffusion Models

Open paper AskPDF arXiv

The widespread adoption of face recognition (FR) technologies raises serious privacy concerns, as facial data can be exploited without consent. To address this challenge, we propose Adv-TGD, a generative adversarial attack framework that synthesizes photorealistic faces capable of impersonating target identities and deceiving face recognition systems. Built upon Stable Diffusion v2.1, Adv-TGD performs per-sample LoRA fine-tuning conditioned on concise textual prompts to generate natural yet adversarially manipulated identities. Unlike conventional identity attack approaches, our method optimizes lightweight cross-attention adapters for each source-target pair within a fixed-timestep denoising process. Latent blending is constrained by a face-local heatmap mask to ensure spatially precise identity manipulation while preserving non-sensitive regions. We introduce a composite objective that integrates masked epsilon-MSE reconstruction, thresholded identity divergence in FR embedding space, directional feature alignment, and source-similarity suppression to balance adversarial attack and visual realism. Optionally, LLaVA-generated attribute prompts enhance fine-grained semantic details without reintroducing identity cues. Under the black-box evaluation protocol, Adv-TGD attains an average attack success rate (ASR) of 85.90% across IR152, IRSE50, MobileFace, and FaceNet, surpassing the semantic SOTA baseline Adv-CPG by 6.25 points, the diffusion-based makeup method DiffAIM by 3 points, and the noise-based P3-Mask by 16 points. Despite its strong attack efficacy, Adv-TGD preserves high visual fidelity (PSNR = 28.18 dB, SSIM = 0.981). Furthermore, we demonstrate the flexibility of our framework by successfully extending it to in-the-wild datasets (LADN), general object classification (ImageNet), and transformer-based diffusion models (FLUX.1).

Abstract

Model Inversion Attacks

Explore similar work

Adv-TGD: Adversarial Text-Guided Diffusion for Face Recognition Impersonation Attacks

DiffAttack: Evasion Attacks Against Face Recognition via Latent Diffusion Models

LBFTI: Layer-Based Facial Template Inversion for Identity-Preserving Fine-Grained Face Reconstruction