SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation

Wenli Zhang; Xianglong Shi; Sirui Zhao; Xinqi Chen; Guo Cheng; Yifan Xu; Tong Xu; Yong Liao

cs.CVApr 9, 2026

SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation

Authors: Wenli Zhang, Xianglong Shi, Sirui Zhao, Xinqi Chen, Guo Cheng, Yifan Xu, Tong Xu, Yong Liao

Abstract

Diffusion-based audio-driven talking-head generation enables realistic portrait animation, but also introduces risks of misuse, such as fraud and misinformation. Existing protection methods are largely limited to a single modality, and neither image-only nor audio-only attacks can effectively suppress speech-driven facial dynamics. To address this gap, we propose SyncBreaker, a stage-aware multimodal protection framework that jointly perturbs portrait and audio inputs under modality-specific perceptual constraints. Our key contributions are twofold. First, for the image stream, we introduce nullifying supervision with Multi-Interval Sampling (MIS) across diffusion stages to steer the generation toward the static reference portrait by aggregating guidance from multiple denoising intervals. Second, for the audio stream, we propose Cross-Attention Fooling (CAF), which suppresses interval-specific audio-conditioned cross-attention responses. Both streams are optimized independently and combined at inference time to enable flexible deployment. We evaluate SyncBreaker in a white-box proactive protection setting. Extensive experiments demonstrate that SyncBreaker more effectively degrades lip synchronization and facial dynamics than strong single-modality baselines, while preserving input perceptual quality and remaining robust under purification. Code: https://github.com/kitty384/SyncBreaker.

Figures & tables

Figure 1. Overview of the proposed framework.

Figure 2. Overview of SyncBreaker. The image stream employs MIS-based Nullifying Loss to redirect the generation objective from synchronized speaking frames to a static reference image across multiple diffusion stages. The audio stream uses CAF to disrupt audio-visual cross-attention and break the alignment between the audio condition and key facial regions. The two streams are optimized separately and combined at inference time.

Figure 3. Visualization of denoising results across different diffusion stages, including the global structure stage, contour formation stage, and detail refinement stage.

Figure 4. Audio-conditioned cross-attention maps. (a) At a fixed timestep and branch, different U-Net layers show distinct patterns. (b) For a fixed layer-branch unit, patterns vary across timesteps, with some similarity.

Figure 5. Qualitative comparison of videos generated from inputs protected by all compared attack methods.

Method	Modality	CelebA-HQ — LibriSpeech					HDTF
Method	Modality	V-PSNR $\downarrow$	V-SSIM $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$	V-PSNR $\downarrow$	V-SSIM $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$
AdvDm [ 26 ]	V	20.46	0.42	181.90	5.33	4.03	21.39	0.44	215.68	5.81	3.12
PhotoGuard [ 40 ]	V	12.29	0.48	74.87	5.98	5.53	17.16	0.64	107.19	6.61	3.17
Mist [ 27 ]	V	19.39	0.56	209.44	4.87	4.50	21.04	0.61	256.21	4.78	3.36
SDS(+) [ 52 ]	V	20.31	0.41	161.74	5.52	3.98	21.21	0.42	186.35	6.06	3.29
SDS(-) [ 52 ]	V	18.61	0.59	54.51	5.95	4.08	20.04	0.66	79.80	6.35	2.99

Table 1. Quantitative comparisons with state-of-the-art methods on two test protocols: (1) CelebA-HQ images paired with LibriSpeech audio, and (2) HDTF dataset. Metrics marked with ”

\uparrow

” indicate that higher values are better, while those marked with ”

\downarrow

” indicate that lower values are better. Best results are highlighted in bold, while second-best results are underlined.

Method	CelebA-HQ	HDTF
Method	I-PSNR/I-SSIM $\uparrow$	I-PSNR/I-SSIM $\uparrow$
AdvDm [ 26 ]	27.30/0.59	27.29/0.56
PhotoGuard [ 40 ]	27.29/0.57	27.41/0.55
Mist [ 27 ]	26.79/0.57	26.86/0.55
SDS(+) [ 52 ]	27.55/0.62	27.58/ 0.59
SDS(-) [ 52 ]	28.53/0.62	28.48/ 0.59

Table 2. Quantitative comparisons of adversarial image quality on CelebA-HQ and HDTF.

Method	LibriSpeech	HDTF
Method	SNR/PESQ $\uparrow$	SNR/PESQ $\uparrow$
FW-C&W [ 33 ]	3.94/1.02	4.62/1.08
FW-PGD [ 33 ]	17.22/1.21	18.11/1.57
MW [ 37 ]	–/–	–/–
AA-C&W [ 22 ]	22.40 / 1.58	19.30 / 2.10
AA-PGD [ 22 ]	1.08/1.02	5.37/1.08

Table 3. Quantitative comparisons of adversarial audio quality on LibriSpeech and HDTF.

Method	JPEG [ 41 ]				Resize [ 48 ]				DiffPure [ 32 ]				DiffShortcut [ 28 ]
Method	I-PSNR/I-SSIM $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$	I-PSNR/I-SSIM $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$	I-PSNR/I-SSIM $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$	I-PSNR/I-SSIM $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$
AdvDm [ 26 ]	28.39/0.66	150.69	5.43	3.59	10.86/0.26	218.20	5.92	2.88	28.49/0.76	38.56	6.01	2.74	18.88/0.49	65.14	5.92	3.43
PhotoGuard [ 40 ]	28.42/0.66	50.90	6.09	3.29	10.97/0.31	212.96	5.84	2.68	27.54 / 0.74	40.66	5.94	2.85	18.89/ 0.46	68.41	5.85	3.36
Mist [ 27 ]	27.70 / 0.64	147.98	5.64	3.93	10.82/0.28	216.38	5.71	3.00	27.55 / 0.75	38.55	6.12	2.89	18.65 / 0.47	70.32	5.86	3.36
SDS(+) [ 52 ]	28.31 /0.67	134.52	5.69	3.67	10.77 / 0.25	218.10	5.86	2.88	28.47/0.76	37.60	6.05	2.86	18.69/0.48	65.31	5.95	3.31
SDS(-) [ 52 ]	28.97/ 0.65	44.25	5.94	3.38	10.88/0.31	200.62	5.90	2.77	28.23/ 0.75	38.55	5.91	2.89	18.97/ 0.47	65.10	5.87	3.34

Table 4. Quantitative comparison of different image attack methods under four purifiers.

Method	Spectral Gating [ 39 ]				Spectral Subtraction [ 1 ]				DiffWave [ 23 ]				WavePurifier [ 15 ]
Method	SNR/PESQ $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$	SNR/PESQ $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$	SNR/PESQ $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$	SNR/PESQ $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$
FW-C&W [ 33 ]	2.45 / 1.06	5.04	4.12	3.74	-2.94 / 1.04	5.6	3.63	4.05	5.95 / 1.08	5.38	3.93	3.99	1.07 / 1.04	6.24	3.55	4.25
FW-PGD [ 33 ]	2.97/1.12	4.18	5.07	3.03	-2.79/1.17	4.53	4.92	3.43	12.05/1.39	4.5	4.74	3.27	1.20/1.11	4.35	4.96	3.29
MW [ 37 ]	—	6.13	5.03	3.30	—	6.85	5.91	3.30	—	6.86	4.80	3.73	—	6.82	5.35	3.52
AA-C&W [ 22 ]	2.71/1.13	4.43	5.05	3.04	-2.85 /1.27	4.2	5.41	3.04	11.57/1.37	4.69	4.68	3.31	1.09/1.12	4.55	5.03	3.17
AA-PGD [ 22 ]	2.25 / 1.05	4.88	3.33	4.09	-2.81/ 1.03	5.79	2.66	4.40	2.42 / 1.02	5.88	3.44	4.18	0.95 / 1.03	5.79	2.75	4.72

Table 5. Quantitative comparison of different audio attack methods under four purifiers.

Interval	V-PSNR $\downarrow$	V-SSIM $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$
[0,100]	19.87	0.44	210.32	3.19	5.42
[200,300]	21.86	0.50	176.32	3.30	5.46
[500,600]	20.63	0.51	142.54	3.95	4.88
[700,800]	20.68	0.51	154.30	4.71	4.21
[900,1000]	20.12	0.48	179.05	3.36	5.43
MIS	20.05	0.46	203.96	2.82	5.65

Table 6. Quantitative comparison of single-interval variants and MIS for the image-stream timestep setting.

Layer	V-PSNR $\downarrow$	V-SSIM $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$
down_0	23.10	0.74	6.94	2.06	4.46
mid_0	22.84	0.73	6.46	2.72	4.47
up_1	22.49	0.72	7.07	2.84	4.34
CAF	22.76	0.72	8.60	1.85	4.60

Table 7. Quantitative comparison of different layers within the same branch under the

[700,1000]

interval.

Interval	V-PSNR $\downarrow$	V-SSIM $\downarrow$	FID $\uparrow$	Sync $\downarrow$	M-LMD $\uparrow$
[0,100]	23.46	0.74	5.85	3.03	4.05
[400,600]	23.45	0.74	6.24	2.52	4.21
[900,1000]	22.71	0.73	6.52	2.53	4.56
CAF	22.76	0.72	8.60	1.85	4.60

Table 8. Quantitative comparison across different timestep intervals for the mid_0_lip layer-branch unit.

Adversarial Attacks

Adversarial attacks exploit crafted inputs, poisoned data, or system interactions to manipulate AI predictions, behavior, privacy, or reliability.

See all 447 papers

TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation
Mar 6, 2026Soumya Mazumdar, Vineet Kumar RakeshTalking Head Generation Temporal Consistency in Video Generation
Department of Computer Science and Business Systems, Gargi Memorial Institute of Technology, Baruipur, Kolkata, West Bengal 700144, India · Computer and Informatics Group, Variable Energy Cyclotron Centre, 1/AF, Bidhannagar, Kolkata, West Bengal 700064, India
Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
Apr 26, 2026Zhen Ye, Xu Tan, Aoxiong Yin +8Talking Head Generation Audio-Video Generation
Hong Kong University of Science and Technology · Independent Researcher · Zhejiang University +3
UniTalking: A Unified Audio-Video Framework for Talking Portrait Generation
Mar 2, 2026Hebeizi Li, Zihao Liang, Benyuan Sun +4Lip Synchronization Audio-Video Generation
Central Media Technology Institute, Huawei · School of Computer Science and Engineering, Beihang University

Mar 6, 2026J/K move · Enter open · S save

TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation

Soumya Mazumdar, Vineet Kumar Rakesh

Department of Computer Science and Business Systems, Gargi Memorial Institute of Technology, Baruipur, Kolkata, West Bengal 700144, India · Computer and Informatics Group, Variable Energy Cyclotron Centre, 1/AF, Bidhannagar, Kolkata, West Bengal 700064, India

Talking Head Generation Temporal Consistency in Video Generation Diffusion Model Inference Acceleration Diffusion Model Distillation

Open paper AskPDF arXiv

Diffusion models have recently advanced photorealistic human synthesis, although practical talking-head generation (THG) remains constrained by high inference latency, temporal instability such as flicker and identity drift, and imperfect audio-visual alignment under challenging speech conditions. This paper introduces TempoSyncDiff, a reference-conditioned latent diffusion framework that explores few-step inference for efficient audio-driven talking-head generation. The approach adopts a teacher-student distillation formulation in which a diffusion teacher trained with a standard noise prediction objective guides a lightweight student denoiser capable of operating with significantly fewer inference steps to improve generation stability. The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker during synthesis, while viseme-based audio conditioning provides coarse lip motion control. Experiments on the LRS3 dataset report denoising-stage component-level metrics relative to VAE reconstructions and preliminary latency characterization, including CPU-only and edge computing measurements and feasibility estimates for edge deployment. The results suggest that distilled diffusion models can retain much of the reconstruction behaviour of a stronger teacher while enabling substantially lower latency inference. The study is positioned as an initial step toward practical diffusion-based talking-head generation under constrained computational settings. GitHub: https://mazumdarsoumya.github.io/TempoSyncDiff

Abstract

Adversarial Attacks

Explore similar work

TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation

Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling

UniTalking: A Unified Audio-Video Framework for Talking Portrait Generation