Audio-Driven Adversarial Defense for 3D Talking Face Generation with totally Visual Fidelity Preservation

Talking head generation has progressed rapidly from landmark- and GAN-based facial animation to diffusion models, neural rendering, 3D-aware avatars, and foundation-model-assisted systems. This progress has enabled increasingly realistic audio-, image-, and video-driven talking heads, but it has also made the field difficult to navigate because methods differ substantially in their inputs, assumptions, controllability, temporal stability, computational cost, and risks of misuse. This survey provides a critical review of talking head generation techniques, organizing the literature into four broad families: image-driven, audio-driven, video-driven, and 3D/neural-rendering-based approaches. For each family, we discuss the underlying technical ideas, representative methods, strengths, limitations, datasets, and evaluation practices. Beyond cataloguing prior work, we analyse the persistent gap between commonly reported quantitative metrics and perceptual quality, and compare publicly available models in terms of inference time, memory requirements, and human-rated visual quality. We also examine emerging trends, including diffusion-based generation, 3D-aware representation learning, controllable emotional expression, real-time deployment, and the growing importance of provenance, watermarking, and deepfake detection. Finally, we identify open challenges around robust evaluation, identity preservation, lip synchronisation, temporal consistency, demographic fairness, computational efficiency, and responsible use. This review aims to provide researchers and practitioners with a structured and up-to-date map of the talking head generation landscape, while highlighting the technical and societal questions that should shape future work.

Audio-driven talking-head generation has achieved remarkable progress with recent models such as AniTalker, FLOAT, and Sonic. Despite their success, most existing approaches rely on a single static reference image to condition the entire video generation process at inference stage. This static conditioning paradigm often creates a mismatch between fixed identity features and dynamically evolving facial motion, leading to identity drift, temporal inconsistency, and degraded perceptual quality. We introduce Test-Time Self-Adaptive Conditioning (TT-SAC), a parameter-free inference framework that enables pretrained talking-head generators to adapt their conditioning representations during inference without retraining, gradient updates, or additional supervision. Instead of treating the reference portrait as immutable, TT-SAC composes the generator with its encoder in a feedback loop: the generator's own outputs are re-encoded to construct a refined conditioning representation that better aligns with the temporal dynamics of the synthesized sequence. A single adaptation step approximates a self-consistent equilibrium of the generative process, stabilizing identity and motion across time. We further provide theoretical analysis showing that test-time conditioning adaptation reduces feature variance and improves generative stability under mild Lipschitz assumptions, while exhibiting a principled bias-variance tradeoff that governs the optimal strength of adaptation. Extensive experiments on state-of-the-art talking-head generators and benchmark datasets demonstrate consistent improvements in lip-sync accuracy, temporal coherence, identity preservation, and perceptual fidelity. TT-SAC offers a model-agnostic and training-free strategy for enhancing generative video models, establishing test-time conditioning adaptation as an effective mechanism for stabilizing audio-driven portrait animation.

With the rapid advancement of diffusion models, talking face generation has made remarkable progress. However, existing diffusion-based methods still require task-specific fine-tuning and large-scale audiovisual datasets, resulting in high computational costs that hinder scalability and accessibility of diffusion-based approaches across the research community. To address this, we propose a finetuning-free paradigm that directly performs talking face generation using the pretrained weights of Stable Diffusion and IP-Adapter. This backbone leverages the visual embedding capability of IP-Adapter to mine lip-related semantics from the pretrained Stable Diffusion. To address the challenges of identity drift, synchronization errors, and temporal instability, we also design three trainable-parameterfree components: (1) the Structurist, which explicitly disentangles and reassembles lip and appearance features to mitigate identity drift and appearance distortion; (2) the Structure Controller, which adaptively refines embeddings based on quasi-monotonic motion trends for precise lip synchronization; and (3) the Noise Sensor, which introduces Gaussian prior to detect and suppress flicker and jitter artifacts and enhance temporal consistency. Experimental results show that our method outperforms existing SOTA approaches in both lip-sync accuracy (at least 0.16 gain in PCLD) and visual fidelity (at least 0.7 improvement in FID), establishing a novel fine-tuning-free diffusion framework for talking face generation.