FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences

Sijing Wu; Yunhao Li; Zhilin Gao; Huiyu Duan; Yucheng Zhu; Guangtao Zhai; Patrick Le Callet

cs.CVAug 15, 2026

FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences

Authors: Sijing Wu, Yunhao Li, Zhilin Gao, Huiyu Duan, Yucheng Zhu, Guangtao Zhai, Patrick Le Callet

Organizations: Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai, China · USC-SJTU Institute of Cultural and Creative Industry, Shanghai Jiao Tong University, Shanghai, China · Institut Universitaire de France (IUF), University of Nantes, France

Abstract

Audio-driven 3D facial animation is essential for advancing immersion and interactivity in virtual experiences. Although recent advances have shown promising capabilities, the training and evaluation of existing methods typically rely on ground-truth-based errors, which fall short of aligning with human preferences. To address this, we present a comprehensive framework that learns an automatic perceptual model from human preference data and leverages it to improve and evaluate the perceptual quality of audio-driven 3D facial animation. To begin with, we construct FMPair (Facial Motion Pairwise preference), the first human preference dataset for audio-driven 3D facial animation, which is built through a systematic annotation pipeline and comprises 65,574 annotated 3D facial motion pairs from 8,834 distinct in-the-wild audio clips. Based on the pairwise comparison dataset, we propose a Facial Motion Reward model, termed FMReward, which takes audio and 3D facial motion as inputs and predicts a perceptual quality score aligned with human preferences. Building upon FMReward, we further introduce Facial Motion reward Feedback Learning (FMFL), a direct fine-tuning algorithm that leverages a pretrained reward model to optimize diffusion-based audio-driven 3D facial animation models for better alignment with human preferences. Extensive experiments demonstrate the superiority of FMReward over other metrics in aligning with human preferences and the effectiveness of FMFL in improving the perceptual quality of audio-driven 3D facial animation.

Figures & tables

Fig. 1: Comparison of existing audio-driven 3D facial animation pipeline (a) and our human preference-based pipeline (b). We first train a reward model using pairwise human preference data, which is then used to fine-tune the pretrained audio-driven 3D facial animation model for better alignment with human preferences.

Fig. 2: Illustration of the human preference annotation platform. Comparison videos are played in full screen and can be freely paused or sought.

Fig. 3: Overview of audio statistics. Left: word cloud of the audio transcripts. Right: bar chart of the audio length distribution.

Fig. 4: Representative examples of the FMPair dataset. (a)-(f) illustrate typical issues that may reduce the perceptual quality of 3D facial motion. (g) shows a pairwise comparison demo including head poses and its better choice, while (h) shows one without head poses.

Fig. 5: Heatmap of the pairwise agreement ratios among all annotators.

Fig. 6: Overview of FMReward. Given 3D facial motion, FLAME shape parameters, and audio as inputs, motion and audio features are extracted by a transformer decoder and an audio encoder, fused via cross-attention, and passed through an MLP to predict a quality score. The training process takes a pair of samples as input, and supervises the fused features and predicted scores with the motion reconstruction loss and Bradley-Terry loss, respectively.

Fig. 7: Overview of the diffusion-based framework for audio-driven 3D facial animation incorporating human preferences. (a) Denoising network of Talking Diffusion Model (TDM). Given 3D facial motion sequence $\boldsymbol{x}^{1:N}_{t}$ of length $N$ at denoising step $t$ , FLAME shape parameters $\boldsymbol{\beta}$ , timestep $t$ , and audio $\boldsymbol{a}$ , the transformer decoder predicts the clean motion $\boldsymbol{\hat{x}}^{1:N}_{0}$ . (b) TDM training step. The ground-truth motion $\boldsymbol{x}_{0}$ is noised into $\boldsymbol{x}_{t}$ at a random step, which is then fed into the TDM for denoising and subsequently used to compute the training loss. (c) TDM fine-tuning step using FMFL. The noised motion $\boldsymbol{x}_{t}$ is first denoised for $t_{d}$ steps without gradients and then for one step with gradients, after which the result is fed into the reward model for supervision. (d) TDM sampling. After training and fine-tuning, TDM is able to generate 3D facial motion aligned with human preferences by iteratively denoising and re-noising from sampled random noise $\boldsymbol{x}_{T}$ .

Dataset	TFHP Subset				MMHead Subset				FMPair
Method / Metric	Acc. $\uparrow$	BS $\downarrow$	CE $\downarrow$	$\rho$ $\uparrow$	Acc. $\uparrow$	BS $\downarrow$	CE $\downarrow$	$\rho$ $\uparrow$	Acc. $\uparrow$	BS $\downarrow$	CE $\downarrow$	$\rho$ $\uparrow$
$\diamondsuit$ LVE	19.72%	0.2516	0.6964	-0.70	82.68%	0.2490	0.6912	0.70	35.68%	0.2510	0.6951	-0.10
$\diamondsuit$ FVE	9.09%	0.2504	0.6940	-0.70	79.19%	0.2498	0.6928	0.60	26.87%	0.2503	0.6937	-0.30
$\diamondsuit$ FDD	20.87%	0.2500	0.6932	-0.90	51.29%	0.2500	0.6931	0.10	28.59%	0.2500	0.6931	-0.50
$\diamondsuit$ PE	5.01%	0.9143	7.2396	-1.00	71.05%	0.1906	0.5648	0.20	21.75%	0.7308	5.5469	-0.90
$\diamondsuit$ BA	23.61%	0.2535	0.7003	-0.50	32.62%	0.2516	0.6965	-0.30	25.89%	0.2531	0.6993	-0.60

TABLE I: Quantitative comparisons of 3D facial motion evaluation methods in terms of alignment with human preferences on the proposed FMPair dataset and its TFHP and MMHead subsets.

\diamondsuit

\clubsuit

\heartsuit

, and

\spadesuit

denote common audio-driven 3D facial animation metrics, audio-video alignment methods, audio-visual quality assessment methods, and video quality assessment methods, respectively. The abbreviations Acc., BS, CE, and

\rho

represent Accuracy, Brier Score, Cross-Entropy, and SRCC, respectively. The best and runner-up performances are bold and underlined, respectively.

Fig. 8: Experiments on in-the-wild data. (a) Histogram and quality level division of quality scores predicted by FMReward. (b) Heatmap of win rates between all quality levels based on the user study. (c) Comparison of average quality scores across all levels among our FMReward, the runner-up method SyncNet-C, and the Elo ratings estimated from pairwise user preferences.

Method / Metric	Acc. $\uparrow$	BS $\downarrow$	CE $\downarrow$	$\rho$ $\uparrow$
w/o $\mathcal{L}_{\text{rc}}$	93.76%	0.0382	0.1182	0.90
w/o $\mathcal{L}_{\text{BT}}$	37.83%	0.2512	0.6955	-0.40
w/o Shape Input	93.95%	0.0381	0.1179	1.00
w/o Audio Input	93.79%	0.0385	0.1195	1.00
FMReward-Small	93.85%	0.0382	0.1192	1.00
FMReward-Large	93.60%	0.0428	0.1311	1.00

TABLE II: Ablation study of FMReward on the FMPair dataset. The abbreviations Acc., BS, CE, and

\rho

represent Accuracy, Brier Score, Cross-Entropy, and SRCC, respectively.

Iteration	LVE. $\downarrow$	FDD $\downarrow$	Diversity $\rightarrow$	BA $\uparrow$	FMReward $\uparrow$
0	9.7750	0.5352	26.088	0.2275	1.0453
20	10.1472	0.5801	27.680	0.2299	1.4765
40	10.1349	0.5711	27.696	0.2291	2.1445
60	10.0560	0.5560	27.146	0.2293	2.3061
80	9.9478	0.5484	26.522	0.2310	2.3451
100	9.8885	0.5501	26.205	0.2308	2.3596

TABLE III: Comparison of 3D facial animation metrics at different fine-tuning iterations. The units of LVE and FDD are

mm

and

\times 10^{-2}mm

, respectively. The best and runner-up performances are bold and underlined, respectively.

Fig. 9: Qualitative comparison before and after fine-tuning with FMFL. The left side shows 3D facial animation results driven by audio from the TFHP [ 2 ] test set, while the right side shows results driven by in-the-wild audio.

Method	LVE. $\downarrow$	FDD $\downarrow$	Diversity $\rightarrow$	BA $\uparrow$	FMReward $\uparrow$
FaceFormer [ 9 ]	9.8246	0.8592	0	0.2452	2.0182
CodeTalker [ 6 ]	13.413	2.1878	0	0.2155	1.9630
FaceDiffuser [ 31 ]	17.563	1.0429	32.465	0.2307	0.9166
DiffPoseTalk [ 2 ]	9.9550	0.6581	32.507	0.2217	2.1159
Ours	9.7264	0.5655	25.412	0.2322	2.4115

TABLE IV: Quantitative comparisons of audio-driven 3D facial animation methods on the TFHP [ 2 ] test set. “Ours” refers to the TDM model fine-tuned with FMFL. The units of LVE and FDD are

mm

and

\times 10^{-2}mm

, respectively. The best and runner-up performances are bold and underlined, respectively.

Fig. 10: User study for audio-driven 3D facial animation. We show the win rates of our model (TDM fine-tuned with FMFL) compared to other models.

Audio-Driven Facial Animation

Audio-driven facial animation maps speech to synchronized facial motion for realistic, expressive, and controllable talking faces and avatars.

See all 84 papers

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models
May 8, 2026Kai Zheng, Zejian Kang, Rui Mao +4Audio-Driven Facial Animation
Westlake University · Zhejiang University · Tiangong University +2
KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization
Jun 26, 2026Arthur Josi, Emeline Got, Abdallah Dib +2Audio-Driven Facial Animation Conditional Generative Modeling
Ecole de Technologie Sup´erieure · Ubisoft LaForge
MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion
Apr 3, 2026Bin Liu, Zhixiang Xiong, Zhifen He +1Talking Head Generation Audio-Driven Facial Animation

May 8, 2026J/K move · Enter open · S save

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models

Kai Zheng, Zejian Kang, Rui Mao, Hongyuan Zou+3

Westlake University · Zhejiang University · Tiangong University+2

Audio-Driven Facial Animation

Open paper AskPDF arXiv

Speech-driven facial animation requires accurate correspondence between acoustic signals and facial motion, especially for articulation-related mouth movements. However, directly mapping speech audio to facial coefficients often overlooks the linguistic and phonetic structure underlying speech production. In this paper, we propose AudioFace, a language-assisted framework for speech-driven blendshape generation that treats mouth-related facial coefficient prediction as a structured generation problem guided by linguistic and articulatory information. Instead of relying solely on acoustic features, our method leverages the prior knowledge of multimodal large language models and introduces transcript- and phoneme-level cues to bridge speech signals with interpretable facial actions. Extensive experiments show that AudioFace achieves superior performance across multiple evaluation metrics, validating the effectiveness of language-assisted and multimodal-prior-guided speech-driven facial animation.

Abstract

Audio-Driven Facial Animation

Explore similar work

AudioFace: Language-Assisted Speech-Driven Facial Animation with Multimodal Language Models

KM-Speaker: Keypoint-Based Style Control for High-Quality Speech-Driven 3D Facial Animation and Dialogue Localization

MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion