ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation

Shizhe Liu; Jiayan Gu; Xiangyu Kong; Siyang Song

cs.LGSep 27, 2026

ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation

Authors: Shizhe Liu, Jiayan Gu, Xiangyu Kong, Siyang Song

Organizations: Department of Computer Science, University of Oxford, Oxford, United Kingdom · School of Artificial Intelligence and Big Data, Hefei University, Hefei, China · Department of Computer Science, University of Exeter, Exeter, United Kingdom

Abstract

In dyadic human speaker-listener conversations, the listener's facial reactions allows the speaker to accurately perceive the listener's emotional states. Since human facial reactions are non-deterministic, the ability to generate multiple appropriate human-like facial reactions is crucial for realistic human-agent interactions. Although diffusion models are naturally suited to such one-to-many generation, existing diffusion-based Multiple Appropriate Facial Reaction Generation (MAFRG) methods attempt to denoise random Gaussian initialisations directly into multiple appropriate facial reactions (AFRs). These random initialisations are usually not well-aligned with the target listener facial reaction, which requires complex denoising trajectories from these initialisations, and subsequently creates substantial opportunities for deviations away from the range of trajectories leading to appropriate AFRs. Given the inherent mimicry between the human listener's and speaker's facial behaviours, we address the above denoising trajectory issue by leveraging this strong prior. Specifically, we propose ResDiffFRG, a novel diffusion-based MAFRG framework that explicitly anchors the diffusion process to the speaker behaviour by defining its diffusion target as the residual between the speaker anchor and an AFR. The denoiser only needs to model the comparatively small, reaction-specific residual needed to transform this anchor into an AFR, rather than reconstructing the complete reaction from an unstructured state. Extensive experiments show that ResDiffFRG achieves large improvements in correlation-based appropriateness over existing methods. Our denoising trajectory analysis showed that even at the start of the denoising trajectory, ResDiffFRG already achieves a higher facial-reaction correlation score than the Gaussian Diffusion baseline does after completing 60% of its denoising trajectory.

Figures & tables

Figure 1: ResDiffFRG compared with existing methods. Left: Existing MAFRG diffusion methods attempt to denoise the full listener reaction directly from random noise, forcing the denoiser to learn complex trajectories. Right: ResDiffFRG anchors the diffusion process on the speaker anchor and learns the speaker-listener residual.

Figure 2: ResDiffFRG Framework Pipeline. Left: During training, the residual diffusion target is jointly constructed from the encoded listener and speaker features, noised using the forward diffusion process and the residual denoiser is trained to recover the clean residual. Right: During inference, DDIM is used to sample N residuals, which are reconstructed back into the listener behaviour space using the speaker anchor and passed through a feasibility projection.

Method	Appropriateness		Diversity		Synchrony
	FRCorr ( $\uparrow$ )	FRDist ( $\downarrow$ )	FRDiv ( $\uparrow$ )	FRVar ( $\uparrow$ )	FRSyn ( $\downarrow$ )
GT	10.00	0.00	0.1876	0.0669	48.66
B_Random	0.03	474.68	0.3342	0.1671	46.64
B_Mime	0.52	206.02	0.0000	0.0766	43.70
B_MeanFR	0.00	205.65	0.0000	0.0000	49.00
Trans-VAE	0.24	158.97	0.0079	0.0067	49.00

Table 1: Comparison with existing methods

Method	Appropriateness		Diversity		Synchrony
	FRCorr ( $\uparrow$ )	FRDist ( $\downarrow$ )	FRDiv ( $\uparrow$ )	FRVar ( $\uparrow$ )	FRSyn ( $\downarrow$ )
Direct (Gaussian) Diffusion w/o CFG	0.64	195.02	0.1371	0.0643	48.56
Direct (Gaussian) Diffusion w CFG	0.69	195.18	0.1450	0.0717	48.47
ResDiffFRG (direct) w/o CFG	0.78	199.84	0.1474	0.0864	46.19
ResDiffFRG (direct)	0.80	198.96	0.1461	0.0871	47.94
ResDiffFRG (normalised) w/o CFG	0.74	183.73	0.0857	0.0740	45.73

Table 2: Effect of Normalised Residual Diffusion over other diffusion targets and Effect of CFG

Method	Appropriateness		Diversity		Synchrony
	FRCorr ( $\uparrow$ )	FRDist ( $\downarrow$ )	FRDiv ( $\uparrow$ )	FRVar ( $\uparrow$ )	FRSyn ( $\downarrow$ )
Stochastic Baseline	0.43	333.52	0.1668	0.1170	45.73
ResDiffFRG (normalised)	0.78	180.70	0.0968	0.0725	45.73

Table 3: Effect of Diffusion Transformer

Denoise Fraction	0.0	0.2	0.4	0.6	0.8	1.0
Direct Gaussian	0.06	0.13	0.23	0.39	0.62	0.69
ResDiffFRG	0.43	0.50	0.61	0.72	0.77	0.78

Table 4: Appropriateness (FRCorr) During Denoise Process

Figure 3: Denoising trajectories in mean-emotion PCA space. Two different sessions are presented. For each model, the mean speaker behaviour for each session is used as conditioning and their DDIM iterates are tracked and traced during the denoise process. For each iterate, we isolate the temporal mean of their $D_{\mathrm{EM}}$ -dimensional emotion channels. They are reconstructed back to listener reaction space (where required) and passed through the feasibility projection $\Pi$ . The trajectories formed by these iterates are plotted using a PCA fit. The triangles mark the start points, circles mark the final generation result, red stars mark the ground truth listener sequences, white squares mark the speaker sequences and the black square marks the mean speaker sequence.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Qualitative reactions generated by ResDiffFRG are presented. The top row shows the input speaker behaviour. The 2nd row shows the corresponding ground-truth listener reaction. The bottom three rows show three independently sampled AFRs conditioned on the same speaker sequence. To visualise the results, each generated $D$ -dimensional output is mapped using the latent_embedder module from PerFRDiff ( Zhu et al., 2024 ) and then rendered using PIRender. Frames are shown every 30 frames from 0 to 240.

Explore similar work

Jun 6, 2026cs.CV

REACT 2026: The Fourth Multiple Appropriate Facial Reaction Generation Challenge: Personalised MAFRG and Appropriate EEG Reaction Prediction

In dyadic interactions, various human facial reactions could be appropriate for responding to each human speaker behaviour. Following the successful organisation of the REACT 2023, 2024 and 2025 challenge series, a body of generative deep learning (DL) models have been developed for the problem of multiple appropriate facial reaction generation (MAFRG). This year, we propose the REACT 2026 challenge encouraging the development and benchmarking of Machine Learning (ML) models that can generate multiple personalised, appropriate, diverse, realistic and synchronised human-style facial reactions expressed by a specific human listener for responding to each given speaker behaviour. As a key of the challenge, we continuously provide challenge participants with MARS dataset introduced by REACT 2025 but additionally provide individual-level Big-Five personality labels and EEG recordings. This introduces a new one-to-many personalised facial reaction generation setting combining human expressive behavioural, affective and neurophysiological signals, which remains largely unexplored in current dyadic interaction modelling. This paper also presents the challenge guidelines and new baselines on the four proposed sub-challenges: Offline generic and personalised MAFRG as well as Online generic and personalised MAFRG, respectively, which are publicly available at https://github.com/reactmultimodalchallenge/baseline_react2026.

Siyang Song, Micol Spitale, Zijian Wu +11

University of Exeter United Kingdom · Politecnico di Milano Milan, Italy · Nanjing University of Science and Technology China +8

Facial Expression Recognition Personality Small-Sample Electroencephalogram Datasets Interaction Prediction

Ask PDF

Sep 28, 2026cs.AI

Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation

Automatic human-like facial reaction generation (FRG) is essential for building intelligent systems that can engage in human-computer interaction (HCI). While diverse and context-appropriate facial reactions can reflect latent appraisal and affective processes in human interaction, most existing FRG methods rely on end-to-end architectures that directly map speaker behaviours to listener expressions without an explicit intermediate internal state. We reformulate FRG as generation mediated by a structured internal-state process and propose the \textbf{Intentional Agent}, which shifts FRG from direct stimulus-response mapping to stimulus-grounded generation through explicit intermediate states. To represent temporal internal-state evolution, we propose an internal dynamics model that integrates emotional drives with an iterative Inner Thought Flow (ITF) within a structured intermediate state used for subsequent generation. This state can continue to update during conversational silences. Furthermore, to bridge abstract internal states with physiological actions, we formulate FRG as a downstream affective mapping from this latent thought flow to facial expressions. Experiments on the REACT 2025 dataset show an FRDist of 72.39 and an FRDiv of 0.5057; perceptual plausibility is evaluated separately through blinded human ratings. A blinded human evaluation of 96 reactions found no significant difference in mean score between Full and ground truth (

5.527

vs.\

5.195

p_{\mathrm{Holm}}=.076

), while Full significantly outperformed Event-Triggered and Heuristic-Only (both

p_{\mathrm{Holm}}<.001

). The Reaction Quality Scorer (RQS) correlated strongly with human judgements (Pearson

r=.855

; Spearman

ρ=.821

, both

p<.05

), supporting its use as an automatic metric. These results underscore the immense potential of endogenous dynamics in building highly autonomous, human-like agents.

Hanzhong Zhang, Jindong Wang, Siyang Song

Department of Computer Science, University of Exeter, Exeter, UK · Department of Data Science, William & Mary, Williamsburg, VA, USA

Facial Expression Recognition Agentic Framework Action Generation Latent Thoughts Black Box

Ask PDF

May 28, 2026cs.CV

IP-Adapter Is All You Need: Towards Fine-Tuning-Free Diffusion-Based Talking Face Generation

With the rapid advancement of diffusion models, talking face generation has made remarkable progress. However, existing diffusion-based methods still require task-specific fine-tuning and large-scale audiovisual datasets, resulting in high computational costs that hinder scalability and accessibility of diffusion-based approaches across the research community. To address this, we propose a finetuning-free paradigm that directly performs talking face generation using the pretrained weights of Stable Diffusion and IP-Adapter. This backbone leverages the visual embedding capability of IP-Adapter to mine lip-related semantics from the pretrained Stable Diffusion. To address the challenges of identity drift, synchronization errors, and temporal instability, we also design three trainable-parameterfree components: (1) the Structurist, which explicitly disentangles and reassembles lip and appearance features to mitigate identity drift and appearance distortion; (2) the Structure Controller, which adaptively refines embeddings based on quasi-monotonic motion trends for precise lip synchronization; and (3) the Noise Sensor, which introduces Gaussian prior to detect and suppress flicker and jitter artifacts and enhance temporal consistency. Experimental results show that our method outperforms existing SOTA approaches in both lip-sync accuracy (at least 0.16 gain in PCLD) and visual fidelity (at least 0.7 improvement in FID), establishing a novel fine-tuning-free diffusion framework for talking face generation.

Hao Wu, Xiangyang Luo, Hao Wang +3

Information Engineering University · Huai’an University · Chongqing University of Post and Telecommunications +1

Precise Lip Synchronization Model Fine-Tuning

Ask PDF