Talking Head Generation

Follow this topic Ask about this topic

Talking head generation studies how to animate a person’s face or avatar so speech, lip movements, identity, and expressions remain coordinated. Approaches range from audio-conditioned diffusion and flow-matching models to autoregressive streaming systems and 3D methods using facial meshes or Gaussian splatting. Recent work adds controls for emotion, speaking style, and conversational reactions, while aiming to make generation interactive and efficient. A central challenge is balancing natural, temporally consistent motion and identity preservation with low latency, flexible control, and evaluation that reflects perceived quality.

Latest papers

41 papers5 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion
Apr 3, 2026Bin Liu, Zhixiang Xiong, Zhifen He +1Talking Head Generation Audio-Driven Facial Animation
GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization
Mar 26, 2026Zhangyu Jin, Maksim Siniukov, Deuksin Kwon +2Human Motion Generation Talking Head Generation
University of Southern California, Institute for Creative Technologies
EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization
Mar 22, 2026Haolan Xu, Keli Cheng, Lei Wang +2Talking Head Generation Audio-Driven Facial Animation
Michigan State University, East Lansing, MI 48824 · Qualcomm Technologies Inc., San Diego, CA 92121 · University of North Carolina at Chapel Hill, Chapel Hill, NC 27599
EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control
Mar 19, 2026Yuzhe Weng, Haotian Wang, Yuanhong Yu +4Talking Head Generation Controllable Video Generation
University of Science and Technology of China · Zhejiang University · iFLYTEK
ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation
Mar 18, 2026Xiangyu Kong, Xiaoyu Jin, Yihan Pan +6Talking Head Generation Human Avatar Animation
University of Exeter, Exeter, UK · Meituan, China · College of Computer Science, Nankai University, China +1
Face-to-Face: A Video Dataset for Multi-Person Interaction Modeling
Mar 16, 2026Ernie Chu, Vishal M. PatelTalking Head Generation Audio-Driven Facial Animation
Johns Hopkins University, Baltimore MD 21218, USA
EmbedTalk: Talking Head Synthesis using Gaussian Embeddings
Mar 8, 2026Arpita Saggar, Jonathan C. Darling, Duygu Sarikaya +13D Gaussian Splatting Lip Synchronization
School of Computer Science, University of Leeds · Leeds Institute of Medical Education, School of Medicine, University of Leeds
Text-Driven Emotionally Continuous Talking Face Generation
Mar 6, 2026Hao Yang, Yanyan Zhao, Tian Zheng +7Facial Expression Synthesis Talking Head Generation
Harbin Institute of Technology · SERES
TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation
Mar 6, 2026Soumya Mazumdar, Vineet Kumar RakeshTalking Head Generation Temporal Consistency in Video Generation
Department of Computer Science and Business Systems, Gargi Memorial Institute of Technology, Baruipur, Kolkata, West Bengal 700144, India · Computer and Informatics Group, Variable Energy Cyclotron Centre, 1/AF, Bidhannagar, Kolkata, West Bengal 700064, India
UniTalking: A Unified Audio-Video Framework for Talking Portrait Generation
Mar 2, 2026Hebeizi Li, Zihao Liang, Benyuan Sun +4Lip Synchronization Audio-Video Generation
Central Media Technology Institute, Huawei · School of Computer Science and Engineering, Beihang University
From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applications
Aug 30, 2023Shreyank N Gowda, Dheeraj Pandey, Shashank Narayana GowdaAudio-Video Generation Talking Head Generation
School of Computer Science, University of Nottingham, Nottingham, United Kingdom · Independent Researcher

Apr 3, 2026J/K move · Enter open · S save

MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion

Bin Liu, Zhixiang Xiong, Zhifen He, Bo Li

Talking Head Generation Audio-Driven Facial Animation

Open paper AskPDF arXiv

Speech-driven three-dimensional (3D) facial animation synthesis aims to build a mapping from one-dimensional (1D) speech signals to time-varying 3D facial motion signals. Current methods still face challenges in maintaining lip-sync accuracy and producing realistic facial expressions, primarily due to the highly ill-posed nature of this cross-modal mapping. In this paper, we introduce a novel 3D audio-driven facial animation synthesis method through multi-resolution representation and multi-modal feature fusion, called MMTalker which can accurately reconstruct the rich details of 3D facial motion. We first achieve the continuous representation of 3D face with details by mesh parameterization and non-uniform differentiable sampling. The mesh parameterization technique establishes the correspondence between UV plane and 3D facial mesh and is used to offer ground truth for the continuous learning. Differentiable non-uniform sampling enables precise facial detail acquisition by setting learnable sampling probability in each triangular face. Next, we employ residual graph convolutional network and dual cross-attention mechanism to extract discriminative facial motion feature from multiple input modalities. This proposed multimodal fusion strategy takes full use of the hierarchical features of speech and the explicit spatiotemporal geometric features of facial mesh. Finally, a lightweight regression network predicts the vertex-wise geometric displacements of the synthesized talking face by jointly processing the sampled points in the canonical UV space and the encoded facial motion features. Comprehensive experiments demonstrate that significant improvements are achieved over state-of-the-art methods, especially in the synchronization accuracy of lip and eye movements.