Audio-Driven Facial Animation

Ask about this topic

Overview

Audio-driven facial animation studies how speech can produce synchronized mouth movements and broader facial behavior in talking-head videos, 3D meshes, blendshapes, and Gaussian avatars. Systems use acoustic and phonetic features, often alongside text, emotion, identity, or conversational context, with Transformer, diffusion, and flow-matching models to generate motion or video. Current work aims to balance precise lip articulation with natural expression, identity preservation, personalization, and real-time or long-form generation. Evaluation remains an open challenge because frame-based metrics can miss timing differences, coarticulation, and whether motion looks natural to people.

Latest papers

CardsList

84 papers11 papers in the last four weeks, up 10% on the four weeks before. 0.1% of all new papers.

  1. Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation

    Apr 28, 2026Tianshui Chen, Yujie Zhu, Jianman Lin +4Prompt LearningVLM Adaptation

    Guangdong University of Technology · South China University of Technology · Peking University +1

  2. Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval

    Apr 26, 2026Xuangeng Chu, Yu Han, Wei Mao +1Audio-Driven Facial Animation

    The University of Tokyo, Codec Avatars Lab, Meta, USA · Codec Avatars Lab, Meta, USA · Codec Avatars Lab, Meta, Pittsburgh, PA, USA

  3. EAD-Net: Emotion-Aware Talking Head Generation with Spatial Refinement and Temporal Coherence

    Apr 25, 2026Yahui Li, Yinfeng Yu, Liejun Wang +1Video Diffusion ModelsLip Synchronization

    School of Computer Science and Technology, Xinjiang University Urumqi, China

  4. Polyglot: Multilingual Style Preserving Speech-Driven Facial Animation

    Apr 17, 2026Federico Nocentini, Kwanggyoon Seo, Qingju Liu +6Audio-Driven Facial Animation

    University of Florence · Flawless AI · Imperial College London

  5. SEDTalker: Emotion-Aware 3D Facial Animation Using Frame-Level Speech Emotion Diarization

    Apr 14, 2026Farzaneh Jafari, Stefano Berretti, Anup BasuAudio-Driven Facial AnimationTalking Head Generation

    University of Alberta, Edmonton, AB, Canada · University of Florence, Florence, Italy

  6. SyncBreaker:Stage-Aware Multimodal Adversarial Attacks on Audio-Driven Talking Head Generation

    Apr 9, 2026Wenli Zhang, Xianglong Shi, Sirui Zhao +5Adversarial AttacksAudio-Driven Facial Animation

  7. Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video

    Apr 9, 2026Chanhyuk Choi, Taesoo Kim, Donggyu Lee +2Audio-Driven Facial AnimationControllable Video Generation

    Ulsan National Institute of Science and Technology (UNIST), Ulsan, Republic of Korea

  8. MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion

    Apr 3, 2026Bin Liu, Zhixiang Xiong, Zhifen He +1Talking Head GenerationAudio-Driven Facial Animation

  9. Realistic Lip Motion Generation Based on 3D Dynamic Viseme and Coarticulation Modeling for Human-Robot Interaction

    Apr 2, 2026Sheng Li, Jingcheng Huang, Min LiLip SynchronizationAudio-Driven Facial Animation

    The authors are with The State Key Laboratory of Intelligent Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan 430074, China.

  10. EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization

    Mar 22, 2026Haolan Xu, Keli Cheng, Lei Wang +2Talking Head GenerationAudio-Driven Facial Animation

    Michigan State University, East Lansing, MI 48824 · Qualcomm Technologies Inc., San Diego, CA 92121 · University of North Carolina at Chapel Hill, Chapel Hill, NC 27599

  11. ECHO: Towards Emotionally Appropriate and Contextually Aware Interactive Head Generation

    Mar 18, 2026Xiangyu Kong, Xiaoyu Jin, Yihan Pan +6Talking Head GenerationHuman Avatar Animation

    University of Exeter, Exeter, UK · Meituan, China · College of Computer Science, Nankai University, China +1

  12. FreeTalk: Emotional Topology-Free 3D Talking Heads

    Mar 16, 2026Federico Nocentini, Thomas Besnier, Claudio Ferrari +23D Mesh DeformationAudio-Driven Facial Animation

    Media Integration and Communication Center (MICC), University of Florence, Italy · University of Copenhagen · IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems +1

  13. Face-to-Face: A Video Dataset for Multi-Person Interaction Modeling

    Mar 16, 2026Ernie Chu, Vishal M. PatelTalking Head GenerationAudio-Driven Facial Animation

    Johns Hopkins University, Baltimore MD 21218, USA

  14. SyncEdit: Rethinking Lip Synchronization as Editing with Audio-Driven Diffusion Models

    Mar 10, 2026Lixiang Lin, Siyuan Jin, Jinshan ZhangLip SynchronizationVideo Editing

    HiThink Research · University of Science and Technology of China · Zhejiang University

  15. Talking Together: Synthesizing Co-Located 3D Conversations from Audio

    Mar 9, 2026Mengyi Shan, Shouchieh Chang, Ziqian Bai +6Human Avatar AnimationAudio-Driven Facial Animation

    University of Washington · Google · University of Rochester

  16. EmbedTalk: Talking Head Synthesis using Gaussian Embeddings

    Mar 8, 2026Arpita Saggar, Jonathan C. Darling, Duygu Sarikaya +13D Gaussian SplattingLip Synchronization

    School of Computer Science, University of Leeds · Leeds Institute of Medical Education, School of Medicine, University of Leeds

  17. UniSync: Towards Generalizable and High-Fidelity Lip Synchronization for Challenging Scenarios

    Mar 4, 2026Ruidi Fan, Yang Zhou, Siyuan Wang +3Lip SynchronizationAudio-Visual Synchronization

    Mango TV

  18. UniTalking: A Unified Audio-Video Framework for Talking Portrait Generation

    Mar 2, 2026Hebeizi Li, Zihao Liang, Benyuan Sun +4Lip SynchronizationAudio-Video Generation

    Central Media Technology Institute, Huawei · School of Computer Science and Engineering, Beihang University

  19. FlowPortrait: Reinforcement Learning for Audio-Driven Portrait Video Generation

    Feb 25, 2026Weiting Tan, Andy T. Liu, Ming Tu +3Audio-Driven Facial AnimationAudio-Video Generation

    ByteDance Seed · Johns Hopkins University

  20. Narrating For You: Prompt-guided Audio-visual Narrating Face Generation Employing Multi-entangled Latent Space

    Feb 20, 2026Aashish Chandra, Aashutosh A, Abhijit DasAudio-Video GenerationAudio-Driven Facial Animation

    Machine Intelligence Group, Department of CS&IS, BITS Pilani, Hyderabad Campus, India · Georgia Institute of Technology, USA

  21. UniTAF: A Modular Framework for Joint Text-to-Speech and Audio-to-Face Modeling

    Feb 17, 2026Qiangong Zhou, Nagasaka TomohiroUnified Multimodal ModelsAudio-Video Generation

    Sumeru AI Technology Co., Ltd.

  22. 3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars

    Feb 11, 2026Zhongju Wang, Zhenhong Sun, Beier Wang +4Lip Synchronization3D Avatar Generation

    University of New South Wales · Australian National University · Vertex Lab +1

  23. Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models

    Feb 6, 2026Haoyu Zhang, Zhipeng Li, Yiwen Guo +1Multimodal Large Language ModelsAudio-Driven Facial Animation

    The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher

  24. From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applications

    Aug 30, 2023Shreyank N Gowda, Dheeraj Pandey, Shashank Narayana GowdaAudio-Video GenerationTalking Head Generation

    School of Computer Science, University of Nottingham, Nottingham, United Kingdom · Independent Researcher