cs.CVOct 18, 2026

CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation

Authors: Jaehong Myung, Seungho Eum, Taegon Kim, Hyunjung Chung, Unsang Park

Organizations: Department of Artificial Intelligence, Sogang University, Seoul, Republic of Korea · Department of Computer Science and Engineering, Sogang University, Seoul, Republic of Korea

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

General-purpose self-supervised speech encoders provide strong acoustic representations, but their pretraining objectives are not explicitly aligned with the visible articulatory dynamics required by speech-driven 3D facial animation. We propose CASE (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents. After adaptation, the geometry branch is removed, and the audio-only CASE representation is fused with the baseline SSL speech representation through residual cross-attention, while retaining the motion-decoder architecture and objective. Across four VOCASET baselines, CASE reduces mean lip vertex error by 2.8-6.0%, and mean full-face vertex error decreases in every system. Under the same fusion protocol, CASE yields lower mean LVE than the unadapted AV-SE stream on all four generators, with model-dependent margins. Because adaptation uses Korean speech-geometry pairs and evaluation uses English VOCASET, the result is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.

Figures & tables

Figure 1: Overview of Case . (a) Clip-level contrastive training aligns frame-local speech features with continuous 3D lip-mesh latents. (b) After adaptation, the geometry branch is removed and features from the audio-only Case encoder are fused through residual cross-attention before the original motion-generation stack. The baseline block denotes a pre-trained SSL speech encoder—wav2vec 2.0, XLSR-53, or WavLM in our experiments [ 1 , 5 , 3 ] . The original motion-generation architecture and training objective are retained. Snowflakes and flames denote frozen and trainable modules.
Baseline (SSL Encoder)MethodLip Vertex Error ( ×10−5\times 10^{-5} ) ↓\downarrowMean Vertex Error ( ×10−6\times 10^{-6} ) ↓\downarrowRelative LVE Change
CodeTalker [ 21 ] wav2vec 2.0 [ 1 ]Baseline3.87442.1171–
+ Case3.64241.9764−6.0%-6.0\%
FaceFormer [ 8 ] wav2vec 2.0 [ 1 ]Baseline3.48641.9738–
+ Case3.36771.9236−3.4%-3.4\%
SelfTalk [ 17 ] XLSR-53 [ 5 ]Baseline2.96291.7719–
+ Case2.82131.7653−4.8%-4.8\%
Table 1: Main VOCASET results (mean over three runs). Bold denotes the lower value within each baseline pair. Lip Vertex Error is the primary articulation-sensitive metric; Mean Vertex Error measures full-face reconstruction.
BaselineMethodLip Vertex Error ( ×10−5\times 10^{-5} ) ↓\downarrowΔ\Delta LVE
CodeTalker [ 21 ]Baseline3.8744–
+ AV-SE3.7535−3.1%-3.1\%
+ Case (ours)3.6424−6.0%-6.0\%
FaceFormer [ 8 ]Baseline3.4864–
+ AV-SE3.3858−2.9%-2.9\%
+ Case (ours)3.3677−3.4%-3.4\%
Table 2: 3D-adaptation control. + AV-SE [ 13 ] uses the English-pretrained AV-SE encoder; + Case uses the same initialization, fusion, and insertion point after 12-epoch adaptation to Korean 3D lip latents.
Figure 2: Illustrative qualitative comparisons on selected VOCASET frames. Each row shows the ground truth and paired Baseline/+ Case outputs for CodeTalker, FaceFormer, SelfTalk, and UniTalker. Red boxes mark the mouth regions selected for visual comparison.
More in this topic

Audio-Driven Facial Animation

Audio-driven facial animation maps speech to synchronized facial motion for realistic, expressive, and controllable talking faces and avatars.

See all 84 papers

Explore similar work

CardsList

  1. Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

    Sep 24, 2026Hyung Kyu Kim, Byungchan Hwang, Hak Gu KimAudio-Driven Facial Animation

    Chung-Ang University Seoul, South Korea

  2. From Tokens to Faces: Investigating Discrete Speech Representations for 3D Facial Animation

    Jun 11, 2026Pedro Correa, Olivier Perrotin, Samir Sadok +2Audio Representation LearningNeural Audio Codecs

    Univ. Estadual de Campinas (UNICAMP), Brazil · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, France · Inria at Univ. Grenoble Alpes, CNRS, LJK, France

  3. AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models

    May 8, 2026Kai Zheng, Zejian Kang, Rui Mao +4Audio-Driven Facial Animation

    Westlake University · Zhejiang University · Tiangong University +2