Facial Expression Generation Aligned with Human Preference for Natural Dyadic Interaction

Xu Chen; Rui Gao; Xinjie Zhang; Haoyu Zhang; Che Sun; Zhi Gao; Yuwei Wu; Yunde Jia

cs.CVMar 7, 2026

Facial Expression Generation Aligned with Human Preference for Natural Dyadic Interaction

Authors: Xu Chen, Rui Gao, Xinjie Zhang, Haoyu Zhang, Che Sun, Zhi Gao, Yuwei Wu, Yunde Jia

Abstract

Achieving natural dyadic interaction requires generating facial expressions that are emotionally appropriate and socially aligned with human preference. Human feedback offers a compelling mechanism to guide such alignment, yet how to effectively incorporate this feedback into facial expression generation remains underexplored. In this paper, we propose a facial expression generation method aligned with human preference by leveraging human feedback to produce contextually and emotionally appropriate expressions for natural dyadic interaction. A key to our method is framing the generation of identity-independent facial expressions as an action learning process, allowing human feedback to assess their validity free from visual or identity bias. We establish a closed feedback loop in which listener expressions dynamically respond to evolving conversational cues of the speaker. Concretely, we train a vision-language-action model via supervised fine-tuning to map the speaker's multimodal signals into controllable low-dimensional expression representations of a 3D morphable model. We further introduce a human-feedback reinforcement learning strategy that integrates the imitation of high-quality expression response with critic-guided optimization. Experiments on two benchmarks demonstrate that our method effectively aligns facial expressions with human preference and achieves superior performance.

Figures & tables

Fig. 1: Examples of facial expression generation with (Listener B) and without (Listener A) human preference alignment. Listener B, aligned via human feedback, generates a disgust expression consistent with the Speaker’s emotion, whereas the unaligned Listener A generates an inconsistent happiness expression.

Fig. 2: Overall framework of our method.

Method	L2 $\downarrow$	FD $\downarrow$	Variation	Diversity	P-FD $\downarrow$	L2 Affect( $10^{2}$ ) $\downarrow$
GT			0.1254	2.5427
Random	0.6852	33.5481	0.1458	2.7813	34.5579	12.4864
NN	0.6047	28.4186	0.0914	2.1486	26.7514	10.6842
LM-listener [ 10 ]	0.4345	17.6299	0.1189	2.9374	19.1583	6.3992
MMLHG [ 39 ]	0.2910	10.0949	0.0704	2.2960	11.3908	2.6575
Ours (SFT)	0.3015	9.1473	0.0814	2.1756	11.2975	2.5724

TABLE I: Quantitative comparison on the L2L-trevor dataset.

Method	L2 $\downarrow$	FD $\downarrow$	Variation	Diversity	P-FD $\downarrow$	L2 Affect( $10^{2}$ ) $\downarrow$
GT			0.0478	1.7250
Random	0.3749	17.2541	0.0549	1.9452	18.4436	24.1739
NN	0.3147	16.0572	0.0496	1.6241	15.9657	20.1478
LM-listener [ 10 ]	0.2416	10.8423	0.0402	1.6148	10.5483	12.2730
MMLHG [ 39 ]	0.1021	3.7914	0.0234	1.2643	3.8145	6.0427
Ours (SFT)	0.0824	3.2425	0.0371	0.9142	3.8036	4.5207

TABLE II: Quantitative comparison on the Realtalk dataset.

Fig. 3: Qualitative comparison of generated listener expressions on the Realtalk dataset. Our method produces more appropriate and emotionally aligned responses compared to baseline methods.

Method	App.	Emp.	Eng.	Nat.
LM-listener [ 10 ]	2.7	3.1	3.4	2.9
MMLHG [ 39 ]	3.0	3.3	3.5	3.1
Ours (SFT)	3.2	3.4	3.7	3.3
Ours (SFT+RL)	4.5	4.1	4.2	4.5

TABLE III: User study results (Mean Opinion Score, 1-5 scale). Our method achieves highest ratings across all aspects.

Method	L2 $\downarrow$	FD $\downarrow$	P-ID $\downarrow$	L2 Affect( $10^{2}$ ) $\downarrow$
Full (Ours)	0.0973	3.5842	3.7914	4.3531
Random-Prefer	0.3142	12.3549	12.0354	15.2463
SFT-Preferred	0.1132	3.6791	3.9165	5.519
SFT-Only	0.0824	3.2425	3.8036	4.5207

TABLE IV: Ablation study on the Realtalk dataset.

Explore similar work

Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis
Jul 27, 2026Yifan Hu, Shuwei He, Rui Liu +1TTS Synthesis Audio-Driven Facial Animation
Inner Mongolia University Hohhot, China · SRIBD, School of Artificial Intelligence The Chinese University of Hong Kong Shenzhen, China
ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction
Aug 29, 2026Zhuoqiang Cai, Yujie Sun, Chaoyue Niu +4Human Avatar Animation Audio-Driven Facial Animation
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China
Morphology-Independent Facial Expression Imitation for Human-Face Robots
Mar 7, 2026Xu Chen, Rui Gao, Che Sun +4Robot Imitation Learning Humanoid Robotics

Jul 27, 2026J/K move · Enter open · S save

Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis

Yifan Hu, Shuwei He, Rui Liu, Haizhou Li

Inner Mongolia University Hohhot, China · SRIBD, School of Artificial Intelligence The Chinese University of Hong Kong Shenzhen, China

TTS Synthesis Audio-Driven Facial Animation Speech Language Models Emotional Speech Synthesis

Open paper AskPDF arXiv

Conversational Speech Synthesis is a fundamental component of human-computer interaction, aiming to generate contextually appropriate, expressive, and empathetic speech. However, facial expressions encode subtle and rich affective cues that are crucial for empathetic speech interaction, whereas existing approaches often overlook this important modality. In addition, the lack of large-scale natural conversational datasets with both speech and visual modalities also limits the development of visual affect understanding in conversational settings.To address these limitations, we propose FacialTalker, a facial-expression-aware CSS framework built upon a large language model backbone. To efficiently encode facial expressions, we propose AUTokenizer, a single-codebook visual tokenizer that discretizes each frame-level facial expression into a compact token, trained with supervision from combinations of facial Action Units. We further introduce a dual direct preference optimization (DualDPO) strategy, which extends the DPO by jointly imposing preference constraints on both visual and speech token sequences, to enhance the model's understanding of facial expressions and speech semantics in multimodal conversational contexts. Moreover, we construct VSDD-1K, a large-scale multimodal dialogue dataset collected through a fully automated pipeline from real-world Internet conversations, comprising over 1,033 hours of synchronized speaker videos and speech, with more than 85% of frames containing valid faces. Extensive objective and subjective experiments demonstrate that FacialTalker consistently outperforms strong baselines in facial-expression perception and speech synthesis quality, generating speech that is more natural, expressive, and better aligned with the conversational context. The results also validate the effectiveness of our training strategy and dataset construction pipeline.

Abstract

Explore similar work

Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis

ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction

Morphology-Independent Facial Expression Imitation for Human-Face Robots