CASE: Complementary Articulation-Aware Speech Encoder to Enhance Speech-Driven 3D Facial Animation
Authors: Jaehong Myung, Seungho Eum, Taegon Kim, Hyunjung Chung, Unsang Park
Organizations: Department of Artificial Intelligence, Sogang University, Seoul, Republic of Korea · Department of Computer Science and Engineering, Sogang University, Seoul, Republic of Korea
General-purpose self-supervised speech encoders provide strong acoustic representations, but their pretraining objectives are not explicitly aligned with the visible articulatory dynamics required by speech-driven 3D facial animation. We propose CASE (Complementary Articulation-aware Speech Encoder), a complementary speech encoder adapted by contrastively aligning short speech clips with continuous 3D lip-mesh VQ-VAE latents. After adaptation, the geometry branch is removed, and the audio-only CASE representation is fused with the baseline SSL speech representation through residual cross-attention, while retaining the motion-decoder architecture and objective. Across four VOCASET baselines, CASE reduces mean lip vertex error by 2.8-6.0%, and mean full-face vertex error decreases in every system. Under the same fusion protocol, CASE yields lower mean LVE than the unadapted AV-SE stream on all four generators, with model-dependent margins. Because adaptation uses Korean speech-geometry pairs and evaluation uses English VOCASET, the result is consistent with cross-language downstream transfer under a shared English-pretrained audio-visual prior.
Figures & tables
Figure 1: Overview of Case . (a) Clip-level contrastive training aligns frame-local speech features with continuous 3D lip-mesh latents. (b) After adaptation, the geometry branch is removed and features from the audio-only Case encoder are fused through residual cross-attention before the original motion-generation stack. The baseline block denotes a pre-trained SSL speech encoder—wav2vec 2.0, XLSR-53, or WavLM in our experiments [ 1 , 5 , 3 ] . The original motion-generation architecture and training objective are retained. Snowflakes and flames denote frozen and trainable modules.
Baseline (SSL Encoder)
Method
Lip Vertex Error ( ×10−5 ) ↓
Mean Vertex Error ( ×10−6 ) ↓
Relative LVE Change
CodeTalker [ 21 ] wav2vec 2.0 [ 1 ]
Baseline
3.8744
2.1171
–
+ Case
3.6424
1.9764
−6.0%
FaceFormer [ 8 ] wav2vec 2.0 [ 1 ]
Baseline
3.4864
1.9738
–
+ Case
3.3677
1.9236
−3.4%
SelfTalk [ 17 ] XLSR-53 [ 5 ]
Baseline
2.9629
1.7719
–
+ Case
2.8213
1.7653
−4.8%
Table 1: Main VOCASET results (mean over three runs). Bold denotes the lower value within each baseline pair. Lip Vertex Error is the primary articulation-sensitive metric; Mean Vertex Error measures full-face reconstruction.
Baseline
Method
Lip Vertex Error ( ×10−5 ) ↓
Δ LVE
CodeTalker [ 21 ]
Baseline
3.8744
–
+ AV-SE
3.7535
−3.1%
+ Case (ours)
3.6424
−6.0%
FaceFormer [ 8 ]
Baseline
3.4864
–
+ AV-SE
3.3858
−2.9%
+ Case (ours)
3.3677
−3.4%
Table 2: 3D-adaptation control. + AV-SE [ 13 ] uses the English-pretrained AV-SE encoder; + Case uses the same initialization, fusion, and insertion point after 12-epoch adaptation to Korean 3D lip latents.
Figure 2: Illustrative qualitative comparisons on selected VOCASET frames. Each row shows the ground truth and paired Baseline/+ Case outputs for CodeTalker, FaceFormer, SelfTalk, and UniTalker. Red boxes mark the mouth regions selected for visual comparison.
Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech--Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.
The choice of speech representation is critical in speech-driven 3D facial animation. Representations differ in what they encode: SSL features emphasize segmental and semantic cues, neural codecs yield latents optimized for acoustic reconstruction, and ASR-style objectives produce label-based spaces. We evaluate four speech representation families for 3D facial synthesis, comparing their facial reconstruction quality across two facial decoders using objective metrics and a perceptual evaluation. We additionally conduct probing analyses that relate tokenized representations to phonetic units and to articulatory deformations. We found that encoding phonetic classes is beneficial for accurate facial animation prediction on both semantic and label-based representations with comparable facial animation quality. From the latter, we introduce an Audio Visual Text-to-Speech (AVTTS) pipeline that leverages, as a shared space, discrete representations to decode speech and 3D facial motion.
Pedro Correa, Olivier Perrotin, Samir Sadok +2
Univ. Estadual de Campinas (UNICAMP), Brazil · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, France · Inria at Univ. Grenoble Alpes, CNRS, LJK, France
Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.
Kai Zheng, Zejian Kang, Rui Mao +4
Westlake University · Zhejiang University · Tiangong University +2