Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation

Hanzhong Zhang; Jindong Wang; Siyang Song

cs.AISep 28, 2026

Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation

Authors: Hanzhong Zhang, Jindong Wang, Siyang Song

Organizations: Department of Computer Science, University of Exeter, Exeter, UK · Department of Data Science, William & Mary, Williamsburg, VA, USA

Abstract

Automatic human-like facial reaction generation (FRG) is essential for building intelligent systems that can engage in human-computer interaction (HCI). While diverse and context-appropriate facial reactions can reflect latent appraisal and affective processes in human interaction, most existing FRG methods rely on end-to-end architectures that directly map speaker behaviours to listener expressions without an explicit intermediate internal state. We reformulate FRG as generation mediated by a structured internal-state process and propose the \textbf{Intentional Agent}, which shifts FRG from direct stimulus-response mapping to stimulus-grounded generation through explicit intermediate states. To represent temporal internal-state evolution, we propose an internal dynamics model that integrates emotional drives with an iterative Inner Thought Flow (ITF) within a structured intermediate state used for subsequent generation. This state can continue to update during conversational silences. Furthermore, to bridge abstract internal states with physiological actions, we formulate FRG as a downstream affective mapping from this latent thought flow to facial expressions. Experiments on the REACT 2025 dataset show an FRDist of 72.39 and an FRDiv of 0.5057; perceptual plausibility is evaluated separately through blinded human ratings. A blinded human evaluation of 96 reactions found no significant difference in mean score between Full and ground truth ( $5.527$ vs.\ $5.195$ , $p_{\mathrm{Holm}}=.076$ ), while Full significantly outperformed Event-Triggered and Heuristic-Only (both $p_{\mathrm{Holm}}<.001$ ). The Reaction Quality Scorer (RQS) correlated strongly with human judgements (Pearson $r=.855$ ; Spearman $ρ=.821$ , both $p<.05$ ), supporting its use as an automatic metric. These results underscore the immense potential of endogenous dynamics in building highly autonomous, human-like agents.

Figures & tables

Figure 1: The internal dynamics model of the intentional agent.

Figure 2: The overall architecture of the intentional agent.

Method	Appropriateness		Diversity		Synchrony
Method	FRCorr ( $\uparrow$ )	FRDist ( $\downarrow$ )	FRDiv ( $\uparrow$ )	FRVar ( $\uparrow$ )	FRSyn ( $\downarrow$ )
GT ( Song et al., 2025 )	10.00	0.00	0.1876	0.0669	48.66
B_Random ( Song et al., 2025 )	0.03	474.68	0.3342	0.1671	46.64
B_Mime ( Song et al., 2025 )	0.52	206.02	0.0000	0.0766	43.70
B_MeanFr ( Song et al., 2025 )	0.00	205.65	0.0000	0.0000	49.00
Trans-VAE ( Song et al., 2025 )	0.30	181.72	0.0076	0.0083	49.00

Table 1: Quantitative comparison on the React2025 dataset. Bold text highlights the best results. The arrows (

\uparrow

) and (

\downarrow

) indicate whether higher or lower metric values represent better performance respectively.

Model	Affect-Action Coherence	Pragmatic Utility	Temporal Intentionality	Role Compliance	Affective Appropriateness
GT	3.00	1.93	2.43	3.07	2.64
Qwen3.1-8B	3.23	3.23	2.85	3.62	3.92
Llama3.1-8B	3.21	3.07	3.36	3.43	3.79
GPT-4o	2.71	2.43	3.50	3.36	3.43
DeepSeek-V3	3.29	2.36	3.79	3.86	3.79

Table 2: Multi-dimensional LLM-as-a-Judge evaluation of generated actions. We evaluate the Intentional Agent with different foundation models. The top result for each dimension is bolded. Higher values on the 1–5 Likert scale indicate higher judged performance on the corresponding action-level criterion.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Comparison of agent systems across three paradigms of human-agent interaction. Reactive agents passively receive user queries and then generate responses. Proactive agents infer tasks based on environmental observations and propose possible assistance requests accordingly. Intentional agents maintain a continuous inner thought stream driven by endogenous intentionality, allowing them to evaluate complex contexts (such as user silence) and autonomously decide whether to initiate action or remain silently active.

Equation	Theoretical Basis	Functional Design Rationale
Eq. 1	Self-sustaining dynamics in Appendix A	Separates internal and input-driven changes, making silence updates testable.
Eq. 2	ITCMA ( Zhang et al., 2025 )	Separates action choice, purpose, feasibility, and expected result.
Eqs. 3–5	Predictive coding ( Rao and Ballard, 1999 )	Gives an explicit prediction, comparison, and correction cycle.
Eq. 6	Appraisal dynamics and EMA ( Marsella and Gratch, 2009 )	Retains emotional continuity while allowing appraisal and error to change emotion.
Eq. 7	VAD ( Russell and Mehrabian, 1977 ; Mehrabian, 1996 ) and embodied intentionality	Separates affective meaning from character-specific facial realisation.

Appendix

Table 3: Theoretical basis and functional design rationale for Equations 1–7.

Figure 4: Holographic Avatar display. (a) Physical presentation of the Companion 01 holographic display; (b) The multi-view directional optical film on the TFT-LCD panel directs images of different perspectives; (c) Utilizing the binocular parallax principle, the left and right eyes receive different views which are cognitively fused in the brain to achieve a naked-eye 3D spatial perception; (d) The rendering pipeline, where a virtual camera array captures multi-view images that are processed via light-field texture synthesis to generate the final holographic output.

Dimension	Pearson	Spearman	Direction Acc.	$p$ vs. $50\%$
Valence	.512	.484	$67.28\%$	$8.34\times 10^{-15}$
Arousal	.236	.223	$58.77\%$	$.000104$
Dominance	.151	.148	$66.23\%$	$1.09\times 10^{-10}$

Appendix

Table 4: Validation of the NRC-based text-to-VAD analyser against human EmoBank annotations. Direction accuracy is evaluated on samples clearly above or below neutral.

Figure 5: Visualization of macroscopic facial configurations driven by the explicitly formulated VAD-to-FACS mapping. The top and middle rows illustrate the continuous morphological variations across different quadrants of the Valence-Arousal and Valence-Dominance spaces, effectively highlighting the non-linear gating effect of Dominance under negative Valence. The bottom row presents canonical discrete emotions synthesized by the same heuristic rules for reference.

Condition	VaG $\downarrow$	VeG $\downarrow$	DG $\downarrow$
Full	.04399	.01369	.09013
Learned-Only	.04561	.01477	.09655
Heuristic-Only	.05639	.02124	.12368

Appendix

Table 5: Generalisation results on the recording-group-disjoint REACT 2025 split. Lower values are better.

Condition	Mean pairwise difference
Noise-Only	.01944
State-Only	.00337
Both	.02257

Appendix

Table 6: Mean pairwise trajectory difference under controlled sources of diversity. Higher values indicate greater variation among reactions generated for the same context.

Measure	State-Only	Noise-Only
Out-of-range AU proportion	.0097	.0457
Boundary saturation rate	.1788	.2153
Change speed	.0190	.0277
Trajectory jerk	.0126	.0193
Largest frame-to-frame change	.1776	.2032
Low-frequency energy	$67.25\%$	$47.26\%$

Appendix

Table 7: Trajectory characteristics at the highest matched level of overall change. State-Only and Noise-Only have the same total amount of change.

Method	AU-MAE $\downarrow$	VaG $\downarrow$	VeG $\downarrow$
REACT-trained Full	.19081	.02123	.02825
Seamless-adapted Full	.13973	.01104	.01294

Appendix

Table 8: Cross-benchmark adaptation results on the held-out Seamless Interaction test set. Lower values are better.

Method	FDD $\downarrow$	SID $\uparrow$
L2L ( Ng et al., 2022 )	25.95	–
ARTalk ( Chu et al., 2025 )	30.62	–
DualTalk ( Peng et al., 2025 )	43.58	.690
GDPO-Listener ( Jin et al., 2026 )	18.85	1.440
Ours	15.26	.841

Appendix

Table 9: Comparison with reported reference results on Seamless Interaction. Lower FDD and higher SID are better.

Category	Mean	Median	Std	Samples	GT Score	Relative Ratio (%)
Overall	$0.705$	$0.854$	$0.242$	$559$	$0.854$	$82.6$
Session 0	$0.507$	$0.437$	$0.222$	$14$	$0.756$	$67.2$
Session 1	$0.627$	$0.637$	$0.191$	$14$	$0.872$	$71.9$
Session 2	$0.806$	$0.889$	$0.158$	$112$	$0.886$	$91.0$
Session 3	$0.570$	$0.591$	$0.197$	$14$	$0.857$	$66.6$
Session 4	$0.660$	$0.725$	$0.215$	$14$	$0.906$	$72.8$

Appendix

Table 10: Complete RQS evaluation results across conversational sessions.

Figure 6: Qualitative frame-by-frame comparison. In this exchange, the speaker presents a difficult-task dilemma, asking what happens when more time is spent on the harder task, while the listener begins formulating a response. Ground Truth shows natural temporal variation, whereas Event-Triggered and Heuristic-Only are more repetitive or neutral. Full Architecture generates a continuous, context-sensitive trajectory that evolves towards a mildly tense or concerned expression as the dilemma unfolds, without reproducing the ground-truth sequence frame by frame. Expression amplitudes are scaled for visual clarity.

Metric	Human Pearson	Human Spearman
FRCorr	-.100	-.040
$-$ FRDist	.066	.078
$-$ FRSyn	.000	.000
$-$ VaG	.512	.491
$-$ VeG	-.057	-.001
$-$ DG	.189	.136

Appendix

Table 11: Correlation of automatic metrics with human ratings over the 72 generated reactions. Higher values indicate stronger agreement with human scores.

Predictors	Pearson $\uparrow$	Spearman $\uparrow$	MAE $\downarrow$
Six automatic metrics	.480	.432	1.461
RQS	.847	.716	.851
Six metrics + RQS	.879	.786	.745

Appendix

Table 12: Context-held-out prediction of human ratings using existing automatic metrics and RQS.

Explore similar work

ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation
Sep 27, 2026Shizhe Liu, Jiayan Gu, Xiangyu Kong +1Facial Expression Recognition Residual Conditional Diffusion Model
Department of Computer Science, University of Oxford, Oxford, United Kingdom · School of Artificial Intelligence and Big Data, Hefei University, Hefei, China · Department of Computer Science, University of Exeter, Exeter, United Kingdom
REACT 2026: The Fourth Multiple Appropriate Facial Reaction Generation Challenge: Personalised MAFRG and Appropriate EEG Reaction Prediction
Jun 6, 2026Siyang Song, Micol Spitale, Zijian Wu +11Facial Expression Recognition Personality
University of Exeter United Kingdom · Politecnico di Milano Milan, Italy · Nanjing University of Science and Technology China +8
MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations
Jun 26, 2026Hejia Chen, Haoxian Zhang, Xu He +4Facial Expression Recognition Human Motion Generation
Beihang University · Kling Team, Kuaishou Technology · Zhongguancun Laboratory

Sep 27, 2026, cs.LGJ/K move · Enter open · S save

ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation

Shizhe Liu, Jiayan Gu, Xiangyu Kong, Siyang Song

Department of Computer Science, University of Oxford, Oxford, United Kingdom · School of Artificial Intelligence and Big Data, Hefei University, Hefei, China · Department of Computer Science, University of Exeter, Exeter, United Kingdom

Facial Expression Recognition Residual Conditional Diffusion Model Denoising Trajectory

Open paper AskPDF arXiv

In dyadic human speaker-listener conversations, the listener's facial reactions allows the speaker to accurately perceive the listener's emotional states. Since human facial reactions are non-deterministic, the ability to generate multiple appropriate human-like facial reactions is crucial for realistic human-agent interactions. Although diffusion models are naturally suited to such one-to-many generation, existing diffusion-based Multiple Appropriate Facial Reaction Generation (MAFRG) methods attempt to denoise random Gaussian initialisations directly into multiple appropriate facial reactions (AFRs). These random initialisations are usually not well-aligned with the target listener facial reaction, which requires complex denoising trajectories from these initialisations, and subsequently creates substantial opportunities for deviations away from the range of trajectories leading to appropriate AFRs. Given the inherent mimicry between the human listener's and speaker's facial behaviours, we address the above denoising trajectory issue by leveraging this strong prior. Specifically, we propose ResDiffFRG, a novel diffusion-based MAFRG framework that explicitly anchors the diffusion process to the speaker behaviour by defining its diffusion target as the residual between the speaker anchor and an AFR. The denoiser only needs to model the comparatively small, reaction-specific residual needed to transform this anchor into an AFR, rather than reconstructing the complete reaction from an unstructured state. Extensive experiments show that ResDiffFRG achieves large improvements in correlation-based appropriateness over existing methods. Our denoising trajectory analysis showed that even at the start of the denoising trajectory, ResDiffFRG already achieves a higher facial-reaction correlation score than the Gaussian Diffusion baseline does after completing 60% of its denoising trajectory.

Abstract

Explore similar work

ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation

REACT 2026: The Fourth Multiple Appropriate Facial Reaction Generation Challenge: Personalised MAFRG and Appropriate EEG Reaction Prediction

MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations