ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation
Authors: Shizhe Liu, Jiayan Gu, Xiangyu Kong, Siyang Song
Organizations: Department of Computer Science, University of Oxford, Oxford, United Kingdom · School of Artificial Intelligence and Big Data, Hefei University, Hefei, China · Department of Computer Science, University of Exeter, Exeter, United Kingdom
In dyadic human speaker-listener conversations, the listener's facial reactions allows the speaker to accurately perceive the listener's emotional states. Since human facial reactions are non-deterministic, the ability to generate multiple appropriate human-like facial reactions is crucial for realistic human-agent interactions. Although diffusion models are naturally suited to such one-to-many generation, existing diffusion-based Multiple Appropriate Facial Reaction Generation (MAFRG) methods attempt to denoise random Gaussian initialisations directly into multiple appropriate facial reactions (AFRs). These random initialisations are usually not well-aligned with the target listener facial reaction, which requires complex denoising trajectories from these initialisations, and subsequently creates substantial opportunities for deviations away from the range of trajectories leading to appropriate AFRs. Given the inherent mimicry between the human listener's and speaker's facial behaviours, we address the above denoising trajectory issue by leveraging this strong prior. Specifically, we propose ResDiffFRG, a novel diffusion-based MAFRG framework that explicitly anchors the diffusion process to the speaker behaviour by defining its diffusion target as the residual between the speaker anchor and an AFR. The denoiser only needs to model the comparatively small, reaction-specific residual needed to transform this anchor into an AFR, rather than reconstructing the complete reaction from an unstructured state. Extensive experiments show that ResDiffFRG achieves large improvements in correlation-based appropriateness over existing methods. Our denoising trajectory analysis showed that even at the start of the denoising trajectory, ResDiffFRG already achieves a higher facial-reaction correlation score than the Gaussian Diffusion baseline does after completing 60% of its denoising trajectory.
Figures & tables
Figure 1: ResDiffFRG compared with existing methods. Left: Existing MAFRG diffusion methods attempt to denoise the full listener reaction directly from random noise, forcing the denoiser to learn complex trajectories. Right: ResDiffFRG anchors the diffusion process on the speaker anchor and learns the speaker-listener residual.
Figure 2: ResDiffFRG Framework Pipeline. Left: During training, the residual diffusion target is jointly constructed from the encoded listener and speaker features, noised using the forward diffusion process and the residual denoiser is trained to recover the clean residual. Right: During inference, DDIM is used to sample N residuals, which are reconstructed back into the listener behaviour space using the speaker anchor and passed through a feasibility projection.
Method
Appropriateness
Diversity
Synchrony
FRCorr ( ↑ )
FRDist ( ↓ )
FRDiv ( ↑ )
FRVar ( ↑ )
FRSyn ( ↓ )
GT
10.00
0.00
0.1876
0.0669
48.66
B_Random
0.03
474.68
0.3342
0.1671
46.64
B_Mime
0.52
206.02
0.0000
0.0766
43.70
B_MeanFR
0.00
205.65
0.0000
0.0000
49.00
Trans-VAE
0.24
158.97
0.0079
0.0067
49.00
Table 1: Comparison with existing methods
Method
Appropriateness
Diversity
Synchrony
FRCorr ( ↑ )
FRDist ( ↓ )
FRDiv ( ↑ )
FRVar ( ↑ )
FRSyn ( ↓ )
Direct (Gaussian) Diffusion w/o CFG
0.64
195.02
0.1371
0.0643
48.56
Direct (Gaussian) Diffusion w CFG
0.69
195.18
0.1450
0.0717
48.47
ResDiffFRG (direct) w/o CFG
0.78
199.84
0.1474
0.0864
46.19
ResDiffFRG (direct)
0.80
198.96
0.1461
0.0871
47.94
ResDiffFRG (normalised) w/o CFG
0.74
183.73
0.0857
0.0740
45.73
Table 2: Effect of Normalised Residual Diffusion over other diffusion targets and Effect of CFG
Method
Appropriateness
Diversity
Synchrony
FRCorr ( ↑ )
FRDist ( ↓ )
FRDiv ( ↑ )
FRVar ( ↑ )
FRSyn ( ↓ )
Stochastic Baseline
0.43
333.52
0.1668
0.1170
45.73
ResDiffFRG (normalised)
0.78
180.70
0.0968
0.0725
45.73
Table 3: Effect of Diffusion Transformer
Denoise Fraction
0.0
0.2
0.4
0.6
0.8
1.0
Direct Gaussian
0.06
0.13
0.23
0.39
0.62
0.69
ResDiffFRG
0.43
0.50
0.61
0.72
0.77
0.78
Table 4: Appropriateness (FRCorr) During Denoise Process
Figure 3: Denoising trajectories in mean-emotion PCA space. Two different sessions are presented. For each model, the mean speaker behaviour for each session is used as conditioning and their DDIM iterates are tracked and traced during the denoise process. For each iterate, we isolate the temporal mean of their DEM -dimensional emotion channels. They are reconstructed back to listener reaction space (where required) and passed through the feasibility projection Π . The trajectories formed by these iterates are plotted using a PCA fit. The triangles mark the start points, circles mark the final generation result, red stars mark the ground truth listener sequences, white squares mark the speaker sequences and the black square marks the mean speaker sequence.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Qualitative reactions generated by ResDiffFRG are presented. The top row shows the input speaker behaviour. The 2nd row shows the corresponding ground-truth listener reaction. The bottom three rows show three independently sampled AFRs conditioned on the same speaker sequence. To visualise the results, each generated D -dimensional output is mapped using the latent_embedder module from PerFRDiff ( Zhu et al., 2024 ) and then rendered using PIRender. Frames are shown every 30 frames from 0 to 240.