TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation

Soumya Mazumdar; Vineet Kumar Rakesh

cs.CVMar 6, 2026

TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation

Authors: Soumya Mazumdar, Vineet Kumar Rakesh

Organizations: Department of Computer Science and Business Systems, Gargi Memorial Institute of Technology, Baruipur, Kolkata, West Bengal 700144, India · Computer and Informatics Group, Variable Energy Cyclotron Centre, 1/AF, Bidhannagar, Kolkata, West Bengal 700064, India

Abstract

Diffusion models have recently advanced photorealistic human synthesis, although practical talking-head generation (THG) remains constrained by high inference latency, temporal instability such as flicker and identity drift, and imperfect audio-visual alignment under challenging speech conditions. This paper introduces TempoSyncDiff, a reference-conditioned latent diffusion framework that explores few-step inference for efficient audio-driven talking-head generation. The approach adopts a teacher-student distillation formulation in which a diffusion teacher trained with a standard noise prediction objective guides a lightweight student denoiser capable of operating with significantly fewer inference steps to improve generation stability. The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker during synthesis, while viseme-based audio conditioning provides coarse lip motion control. Experiments on the LRS3 dataset report denoising-stage component-level metrics relative to VAE reconstructions and preliminary latency characterization, including CPU-only and edge computing measurements and feasibility estimates for edge deployment. The results suggest that distilled diffusion models can retain much of the reconstruction behaviour of a stronger teacher while enabling substantially lower latency inference. The study is positioned as an initial step toward practical diffusion-based talking-head generation under constrained computational settings. GitHub: https://mazumdarsoumya.github.io/TempoSyncDiff

Figures & tables

Panel of Figure 1 — Figure 1: Overview of TempoSyncDiff. A reference identity image and speech audio are encoded into identity features and viseme tokens forming conditioning $\kappa_{t}$ . A latent diffusion UNet denoises a noisy latent $\mathbf{z}^{\mathrm{noisy}}_{t}$ under this conditioning. During training, a teacher guides a distilled student enabling few-step sampling. The denoised latent $\mathbf{z}_{t}$ is decoded by a VAE to generate the talking-head frame $\hat{\mathbf{I}}_{t}$ .

Panel of Figure 2 — Figure 2: Teacher–student distillation used in TempoSyncDiff. A frozen teacher denoiser $\epsilon_{\theta}$ predicts the noise for a noisy latent $z_{t}$ conditioned on $\kappa_{t}=\{I_{\mathrm{ref}},v_{t}\}$ . The student denoiser $\epsilon_{\phi}$ receives the same inputs and is trained to match the teacher via the distillation loss $\mathcal{L}_{dist}$ . Optional identity and temporal regularizers can be applied during training.

Panel of Figure 3 — Figure 3: Edge deployment-oriented inference. The student performs $K$ denoising steps in latent space. In E1 (full mode) , the VAE decoder runs on-device and outputs frames. In E2 (hybrid mode) , latents are returned for deferred decoding or external rendering.

Metric	Noisy decode	Teacher denoise	Student denoise
PSNR vs. VAE recon (dB)	$25.71\pm 0.02$	$30.95\pm 0.06$	$29.97\pm 0.06$
Temporal L1 (adjacent-frame difference)	$0.0184\pm 0.0036$	$0.0365\pm 0.0020$	$0.0402\pm 0.0021$
Flicker std (pixel-difference std)	$0.00325\pm 0.00171$	$0.00336\pm 0.00159$	$0.00337\pm 0.00155$

Table 1: Measured denoising-stage results against VAE reconstructions (mean ±std over four evaluation cases). PSNR is computed relative to VAE reconstructions to isolate denoising behavior from decoder limitations. Temporal L1 and flicker statistics are computed from adjacent-frame pixel differences and summarize short-term temporal variation.

Panel of Figure 4 — Figure 4: Quality–latency trade-off across step counts (template). Added Raspberry Pi 5 edge inference latency (E2 Hybrid mode) to the latency axis. The intended behavior is near-teacher quality at substantially lower latency for small $K$ .

Steps	Resolution	Mean latency (ms)	p95 latency (ms)	FPS
2	$128\times 128$	13.21	16.58	75.72
4	$128\times 128$	21.56	25.28	46.38
8	$128\times 128$	38.61	43.11	25.90

Table 2: CPU-only inference characterization (measured results).

Mode	Steps	Resolution	Mean latency (ms)	p95 latency (ms)	FPS
E1 Full	2	$128\times 128$	260.88	368.58	3.83
E1 Full	4	$128\times 128$	403.93	407.00	2.48
E2 Hybrid	2	latent	172.05	174.52	5.81
E2 Hybrid	4	latent	335.57	373.09	2.98

Table 3: Raspberry Pi 5 edge inference characterization (estimated). Resolution is reduced for feasibility. Mode E2 outputs latents when decode is deferred.

Talking Head Generation

Talking head generation creates animated faces and avatars that speak, listen, and express emotions from audio, text, images, or video.

See all 41 papers

LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation
Jul 29, 2026Rongxiang Zhang, Songhua LiuAudio-Video Generation Diffusion Model Distillation
School of Artificial Intelligence, Shanghai Jiao Tong University · 2Harbin Institute of Technology
AsymTalker: Identity-Consistent Long-Term Talking Head Generation via Asymmetric Distillation
May 1, 2026Yuxin Lu, Jiayang Sun, Guibo Zhu +1Video Diffusion Models Long Video Generation
Soochow University · Institute of Automation, Chinese Academy of Sciences
IP-Adapter Is All You Need: Towards Fine-Tuning-Free Diffusion-Based Talking Face Generation
May 28, 2026Hao Wu, Xiangyang Luo, Hao Wang +3Talking Head Generation Audio-Driven Facial Animation
Information Engineering University · Huai’an University · Chongqing University of Post and Telecommunications +1

Jul 29, 2026J/K move · Enter open · S save

LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

Rongxiang Zhang, Songhua Liu

School of Artificial Intelligence, Shanghai Jiao Tong University · 2Harbin Institute of Technology

Audio-Video Generation Diffusion Model Distillation Identity-Preserving Video Generation Talking Head Generation Single-Step Video Generation Video Generation+1

Open paper AskPDF arXiv

Long-form and real-time talking-head generation remains challenging due to a latency-quality trade-off: inefficient multi-step diffusion prohibits streaming generation, whereas real-time autoregressive approaches suffer from error accumulation and identity drift. To address this drawback, we propose LeapTalk, a novel framework that achieves stable and real-time talking-head generation with a single forward step, scaling to arbitrarily long videos. At the heart of our approach lies a single-step bridge distillation scheme. On the one hand, departing from the conventional noise-to-data paradigm, we introduce a data-to-data transport formulation based on a Brownian bridge. Anchored by a persistent reference, this strategy effectively mitigates identity drift and enhances long-term temporal stability. On the other hand, to enable smooth knowledge transfer from a pre-trained diffusion teacher to the student bridge model, we explore a heterogeneous distillation framework with an SNR-aligned time transformation

Φ(τ)

, which bridges the functional discrepancy between the two models. Moreover, we propose an audio-driven classifier-free guidance mechanism to maintain fine-grained lip synchronization under extreme step reduction. Extensive experiments demonstrate that our method achieves high-fidelity and temporally consistent video generation with only 1 step at up to 200 FPS, significantly outperforming existing approaches in both efficiency and stability. Project Page: https://zhangrongxiang.github.io/leaptalk-page/

Abstract

Talking Head Generation

Explore similar work

LeapTalk: Breaking the Latency-Quality Trade-off in Talking Head Generation

AsymTalker: Identity-Consistent Long-Term Talking Head Generation via Asymmetric Distillation

IP-Adapter Is All You Need: Towards Fine-Tuning-Free Diffusion-Based Talking Face Generation