FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences
Authors: Sijing Wu, Yunhao Li, Zhilin Gao, Huiyu Duan, Yucheng Zhu, Guangtao Zhai, Patrick Le Callet
Organizations: Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai, China · USC-SJTU Institute of Cultural and Creative Industry, Shanghai Jiao Tong University, Shanghai, China · Institut Universitaire de France (IUF), University of Nantes, France
Audio-driven 3D facial animation is essential for advancing immersion and interactivity in virtual experiences. Although recent advances have shown promising capabilities, the training and evaluation of existing methods typically rely on ground-truth-based errors, which fall short of aligning with human preferences. To address this, we present a comprehensive framework that learns an automatic perceptual model from human preference data and leverages it to improve and evaluate the perceptual quality of audio-driven 3D facial animation. To begin with, we construct FMPair (Facial Motion Pairwise preference), the first human preference dataset for audio-driven 3D facial animation, which is built through a systematic annotation pipeline and comprises 65,574 annotated 3D facial motion pairs from 8,834 distinct in-the-wild audio clips. Based on the pairwise comparison dataset, we propose a Facial Motion Reward model, termed FMReward, which takes audio and 3D facial motion as inputs and predicts a perceptual quality score aligned with human preferences. Building upon FMReward, we further introduce Facial Motion reward Feedback Learning (FMFL), a direct fine-tuning algorithm that leverages a pretrained reward model to optimize diffusion-based audio-driven 3D facial animation models for better alignment with human preferences. Extensive experiments demonstrate the superiority of FMReward over other metrics in aligning with human preferences and the effectiveness of FMFL in improving the perceptual quality of audio-driven 3D facial animation.
Figures & tables
Fig. 1: Comparison of existing audio-driven 3D facial animation pipeline (a) and our human preference-based pipeline (b). We first train a reward model using pairwise human preference data, which is then used to fine-tune the pretrained audio-driven 3D facial animation model for better alignment with human preferences.
Fig. 2: Illustration of the human preference annotation platform. Comparison videos are played in full screen and can be freely paused or sought.
Fig. 3: Overview of audio statistics. Left: word cloud of the audio transcripts. Right: bar chart of the audio length distribution.
Fig. 4: Representative examples of the FMPair dataset. (a)-(f) illustrate typical issues that may reduce the perceptual quality of 3D facial motion. (g) shows a pairwise comparison demo including head poses and its better choice, while (h) shows one without head poses.
Fig. 5: Heatmap of the pairwise agreement ratios among all annotators.
Fig. 6: Overview of FMReward. Given 3D facial motion, FLAME shape parameters, and audio as inputs, motion and audio features are extracted by a transformer decoder and an audio encoder, fused via cross-attention, and passed through an MLP to predict a quality score. The training process takes a pair of samples as input, and supervises the fused features and predicted scores with the motion reconstruction loss and Bradley-Terry loss, respectively.
Fig. 7: Overview of the diffusion-based framework for audio-driven 3D facial animation incorporating human preferences. (a) Denoising network of Talking Diffusion Model (TDM). Given 3D facial motion sequence xt1:N of length N at denoising step t , FLAME shape parameters β , timestep t , and audio a , the transformer decoder predicts the clean motion x^01:N . (b) TDM training step. The ground-truth motion x0 is noised into xt at a random step, which is then fed into the TDM for denoising and subsequently used to compute the training loss. (c) TDM fine-tuning step using FMFL. The noised motion xt is first denoised for td steps without gradients and then for one step with gradients, after which the result is fed into the reward model for supervision. (d) TDM sampling. After training and fine-tuning, TDM is able to generate 3D facial motion aligned with human preferences by iteratively denoising and re-noising from sampled random noise xT .
Dataset
TFHP Subset
MMHead Subset
FMPair
Method / Metric
Acc. ↑
BS ↓
CE ↓
ρ↑
Acc. ↑
BS ↓
CE ↓
ρ↑
Acc. ↑
BS ↓
CE ↓
ρ↑
♢ LVE
19.72%
0.2516
0.6964
-0.70
82.68%
0.2490
0.6912
0.70
35.68%
0.2510
0.6951
-0.10
♢ FVE
9.09%
0.2504
0.6940
-0.70
79.19%
0.2498
0.6928
0.60
26.87%
0.2503
0.6937
-0.30
♢ FDD
20.87%
0.2500
0.6932
-0.90
51.29%
0.2500
0.6931
0.10
28.59%
0.2500
0.6931
-0.50
♢ PE
5.01%
0.9143
7.2396
-1.00
71.05%
0.1906
0.5648
0.20
21.75%
0.7308
5.5469
-0.90
♢ BA
23.61%
0.2535
0.7003
-0.50
32.62%
0.2516
0.6965
-0.30
25.89%
0.2531
0.6993
-0.60
TABLE I: Quantitative comparisons of 3D facial motion evaluation methods in terms of alignment with human preferences on the proposed FMPair dataset and its TFHP and MMHead subsets. ♢ , ♣ , ♡ , and ♠ denote common audio-driven 3D facial animation metrics, audio-video alignment methods, audio-visual quality assessment methods, and video quality assessment methods, respectively. The abbreviations Acc., BS, CE, and ρ represent Accuracy, Brier Score, Cross-Entropy, and SRCC, respectively. The best and runner-up performances are bold and underlined, respectively.
Fig. 8: Experiments on in-the-wild data. (a) Histogram and quality level division of quality scores predicted by FMReward. (b) Heatmap of win rates between all quality levels based on the user study. (c) Comparison of average quality scores across all levels among our FMReward, the runner-up method SyncNet-C, and the Elo ratings estimated from pairwise user preferences.
Method / Metric
Acc. ↑
BS ↓
CE ↓
ρ↑
w/o Lrc
93.76%
0.0382
0.1182
0.90
w/o LBT
37.83%
0.2512
0.6955
-0.40
w/o Shape Input
93.95%
0.0381
0.1179
1.00
w/o Audio Input
93.79%
0.0385
0.1195
1.00
FMReward-Small
93.85%
0.0382
0.1192
1.00
FMReward-Large
93.60%
0.0428
0.1311
1.00
TABLE II: Ablation study of FMReward on the FMPair dataset. The abbreviations Acc., BS, CE, and ρ represent Accuracy, Brier Score, Cross-Entropy, and SRCC, respectively.
Iteration
LVE. ↓
FDD ↓
Diversity →
BA ↑
FMReward ↑
0
9.7750
0.5352
26.088
0.2275
1.0453
20
10.1472
0.5801
27.680
0.2299
1.4765
40
10.1349
0.5711
27.696
0.2291
2.1445
60
10.0560
0.5560
27.146
0.2293
2.3061
80
9.9478
0.5484
26.522
0.2310
2.3451
100
9.8885
0.5501
26.205
0.2308
2.3596
TABLE III: Comparison of 3D facial animation metrics at different fine-tuning iterations. The units of LVE and FDD are mm and ×10−2mm , respectively. The best and runner-up performances are bold and underlined, respectively.
Fig. 9: Qualitative comparison before and after fine-tuning with FMFL. The left side shows 3D facial animation results driven by audio from the TFHP [ 2 ] test set, while the right side shows results driven by in-the-wild audio.
Method
LVE. ↓
FDD ↓
Diversity →
BA ↑
FMReward ↑
FaceFormer [ 9 ]
9.8246
0.8592
0
0.2452
2.0182
CodeTalker [ 6 ]
13.413
2.1878
0
0.2155
1.9630
FaceDiffuser [ 31 ]
17.563
1.0429
32.465
0.2307
0.9166
DiffPoseTalk [ 2 ]
9.9550
0.6581
32.507
0.2217
2.1159
Ours
9.7264
0.5655
25.412
0.2322
2.4115
TABLE IV: Quantitative comparisons of audio-driven 3D facial animation methods on the TFHP [ 2 ] test set. “Ours” refers to the TDM model fine-tuned with FMFL. The units of LVE and FDD are mm and ×10−2mm , respectively. The best and runner-up performances are bold and underlined, respectively.
Fig. 10: User study for audio-driven 3D facial animation. We show the win rates of our model (TDM fine-tuned with FMFL) compared to other models.
Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.
Kai Zheng, Zejian Kang, Rui Mao +4
Westlake University · Zhejiang University · Tiangong University +2
Speech-driven 3D facial animation methods face significant challenges in simultaneously achieving high-fidelity motion and precise artistic control at production quality. Existing controllable models typically learn global style control by relying on large-scale, low-quality in-the-wild datasets that compromise overall animation realism. Furthermore, these frameworks often lack the fine-grained temporal precision required for demanding tasks such as dialogue localization (e.g., dubbing), where matching specific facial expressions is as critical as lip synchronization. We present KM-Speaker (Keypoint-Matching Speaker), a novel keypoint-conditioned flow-based generative framework that provides both global style guidance and frame-level temporal control from reference performances. We propose a disentanglement strategy that separates audio-driven lip motion from keypoint-driven upper-face dynamics, together with a global style context preservation mechanism to ensure coherent full-face expressiveness. KM-Speaker advances example-based 3D facial animation by achieving high-fidelity motion and flexible controllability in a data-constrained setting, consistently outperforming state-of-the-art methods in lip-sync accuracy, style adherence, and expressive temporal control.
Arthur Josi, Emeline Got, Abdallah Dib +2
Ecole de Technologie Sup´erieure · Ubisoft LaForge
Speech-driven three-dimensional (3D) facial animation synthesis aims to build a mapping from one-dimensional (1D) speech signals to time-varying 3D facial motion signals. Current methods still face challenges in maintaining lip-sync accuracy and producing realistic facial expressions, primarily due to the highly ill-posed nature of this cross-modal mapping. In this paper, we introduce a novel 3D audio-driven facial animation synthesis method through multi-resolution representation and multi-modal feature fusion, called MMTalker which can accurately reconstruct the rich details of 3D facial motion. We first achieve the continuous representation of 3D face with details by mesh parameterization and non-uniform differentiable sampling. The mesh parameterization technique establishes the correspondence between UV plane and 3D facial mesh and is used to offer ground truth for the continuous learning. Differentiable non-uniform sampling enables precise facial detail acquisition by setting learnable sampling probability in each triangular face. Next, we employ residual graph convolutional network and dual cross-attention mechanism to extract discriminative facial motion feature from multiple input modalities. This proposed multimodal fusion strategy takes full use of the hierarchical features of speech and the explicit spatiotemporal geometric features of facial mesh. Finally, a lightweight regression network predicts the vertex-wise geometric displacements of the synthesized talking face by jointly processing the sampled points in the canonical UV space and the encoded facial motion features. Comprehensive experiments demonstrate that significant improvements are achieved over state-of-the-art methods, especially in the synchronization accuracy of lip and eye movements.