ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception

Huanzhen Wang; Ziheng Zhou; Jiaqi Song; Li He; Yunshi Lan; Yan Wang; Wenqiang Zhang

cs.CVApr 14, 2026

ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception

Authors: Huanzhen Wang, Ziheng Zhou, Jiaqi Song, Li He, Yunshi Lan, Yan Wang, Wenqiang Zhang

Organizations: Fudan University · East China Normal University

Abstract

Dynamic facial expression recognition in the wild remains challenging due to data scarcity and long-tail distributions, which hinder models from effectively learning the temporal dynamics of scarce emotions. To address these limitations, we propose ARGen, an Affect-Reinforced Generative Augmentation Framework that enables data-adaptive dynamic expression generation for robust emotion perception. ARGen operates in two stages: Affective Semantic Injection (ASI) and Adaptive Reinforcement Diffusion (ARD). The ASI stage establishes affective knowledge alignment through facial Action Units and employs a retrieval-augmented prompt generation strategy to synthesize consistent and fine-grained affective descriptions via large-scale visual-language models, thereby injecting interpretable emotional priors into the generation process. The ARD stage integrates text-conditioned image-to-video diffusion with reinforcement learning, introducing inter-frame conditional guidance and a multi-objective reward function to jointly optimize expression naturalness, facial integrity, and generative efficiency. Extensive experiments on both generation and recognition tasks verify that ARGen substantially enhances synthesis fidelity and improves recognition performance, establishing an interpretable and generalizable generative augmentation paradigm for vision-based affective computing.

Figures & tables

Figure 1: (a) demonstrates the long-tail distribution in terms of category sample counts and recognition accuracy in existing datasets. (b) demonstrates the incremental performance of our ARGen method in addressing the challenge of recognizing scarce facial expressions.

Figure 2: Overall framework of ARGen . The framework incorporates emotional priors derived from facial action units through vision-language models, followed by efficient video generation using diffusion models optimized via reinforcement learning.

Method & Architecture	Status	DFEW					FERV39k
Method & Architecture	Status	WAR	UAR	AR-SU	AR-DI	AR-FE	WAR	UAR	AR-SU	AR-DI	AR-FE
Training from Scratch
Res18_LSTM	Original	63.66	51.85	51.02	0.00	35.91	43.04	32.13	19.44	5.78	9.51
Res18_LSTM	ARGen	65.22 ( 1.56 $\uparrow$ )	53.25 ( 1.40 $\uparrow$ )	52.80 ( 1.78 $\uparrow$ )	6.90 ( 6.90 $\uparrow$ )	38.58 ( 2.67 $\uparrow$ )	44.46 ( 1.42 $\uparrow$ )	34.42 ( 2.29 $\uparrow$ )	21.67 ( 2.23 $\uparrow$ )	11.85 ( 6.07 $\uparrow$ )	11.23 ( 1.72 $\uparrow$ )
Res18_Transformer	Original	66.59	55.60	52.82	5.52	36.92	45.96	35.29	22.10	11.78	11.83
Res18_Transformer	ARGen	67.60 ( 1.01 $\uparrow$ )	56.70 ( 1.10 $\uparrow$ )	55.54 ( 2.72 $\uparrow$ )	11.72 ( 6.20 $\uparrow$ )	39.48 ( 2.56 $\uparrow$ )	46.52 ( 0.56 $\uparrow$ )	36.90 ( 1.61 $\uparrow$ )	23.82 ( 1.72 $\uparrow$ )	14.06 ( 2.28 $\uparrow$ )	12.99 ( 1.16 $\uparrow$ )

Table 1: ARGen’s performance comparison (%) of different recognition baselines on DFEW and FERV39k.

Method	FVD $\downarrow$	sFVD $\downarrow$	tFVD $\downarrow$
DynamiCrafter ( Xing et al. 2024 )	170.16	345.16 $\pm$ 137.32	208.35 $\pm$ 45.79
TI2V-Zero ( Ni et al. 2024 )	81.72	170.20 $\pm$ 57.90	109.41 $\pm$ 47.26
ARGen(ours)	56.75	133.62 $\pm$ 50.94	81.21 $\pm$ 27.07

Table 2: ARGen’s quantitative comparison of different methods for generation on CK+.

Figure 3: Visualization of generation performance for scarce categories in DFEW and FERV39k. The green box indicates the input neutral reference frame and brown markers represent adaptive policy selection results for the input in ARD. We generated 16 video frames throughout the experiments and the four frames shown correspond to frames 1, 6, 11, and 16. The three sections, from top to bottom, represent the emotions of surprise, disgust, and fear.

Ablation	WAR	UAR	AR-SU	AR-DI	AR-FE
Original(w/o TA)	64.50	52.83	50.34	0.00	34.96
Original(w/ TA)	66.59	55.60	52.82	5.52	36.92
ARGen(w/o ASI)	67.31	56.39	55.10	6.90	35.36
$\Delta$	+1.08%	+1.42%	+4.13%	+25.00%	-4.23%
ARGen(w/o ARD)	67.06	54.98	54.79	6.21	37.73
$\Delta$	+0.71%	-1.12%	+3.73%	+12.5%	+2.19%

Table 3: Ablation study on two-stage performance comparison (%) of ARGen on DFEW using Res18_Transformer. “TA” refers to the traditional image data augmentation methods we use in our setup.

Rewards		FVD $\downarrow$	sFVD $\downarrow$	tFVD $\downarrow$
$R_{Q}$	$R_{F},R_{E}$	FVD $\downarrow$	sFVD $\downarrow$	tFVD $\downarrow$
✗	✗	85.87	187.55 $\pm$ 75.36	112.98 $\pm$ 48.22
✓	✗	65.39	164.19 $\pm$ 62.72	102.28 $\pm$ 46.15
✗	✓	78.96	203.93 $\pm$ 78.24	119.44 $\pm$ 60.51
✓	✓	56.75	133.62 $\pm$ 50.94	81.21 $\pm$ 27.07

Table 4: ARGen’s quantitative comparison of different rewards for generation on CK+.

Figure 4: Additional visualizations. The green box indicates the reference frame, while the subsequent three frames represent frames 5, 10, and 15. From top to bottom, represent the emotions of surprise, disgust, and fear.

Ablation	FVD $\downarrow$	sFVD $\downarrow$	tFVD $\downarrow$
ARGen w/o ARD
Fixed prompts (baseline)	85.87	187.55 $\pm$ 75.36	112.98 $\pm$ 48.22
VLM-only	78.40	172.10 $\pm$ 63.50	108.30 $\pm$ 44.10
VLM+RAG	69.20	157.35 $\pm$ 60.85	92.50 $\pm$ 35.47
ARGen w/o ASI
Reinforce-only	66.80	150.18 $\pm$ 55.40	90.04 $\pm$ 30.25

Table 5: More detailed ablation experiments on CK+ dataset for ARGen submodules.

Explore similar work

Generative Learning as a Tool to Improve Perception of Emotional Body Motion Expressions
Jun 27, 2026Huakun Liu, Miao Cheng, Xin Wei +6Human Motion Generation Conditional Generative Modeling
Nara Institute of Science and Technology · Tohoku University
Improving Generative Adversarial Network Generalization for Facial Expression Synthesis
Mar 4, 2026Arbish Akram, Nazar Khan, Arif MahmoodGenerative Adversarial Networks Conditional Image Generation
Department of Computer Science, University of the Punjab, Lahore, Pakistan · Department of Computer Science, Information Technology University, Lahore, Pakistan
Emotion Intensity Matters: Generating Realistic Expressions in Virtual Humans with CVAEs
Aug 22, 2026Vitor Miguel Xavier Peres, Lara Volpato, Gabriel Ferri Scnheider +1Human Avatar Animation Controllable Image Generation
School of Technology Pontifical Catholic University of Rio Grande do Sul Porto Alegre, Brazil

Jun 27, 2026J/K move · Enter open · S save

Generative Learning as a Tool to Improve Perception of Emotional Body Motion Expressions

Huakun Liu, Miao Cheng, Xin Wei, Felix Dollack+5

Nara Institute of Science and Technology · Tohoku University

Human Motion Generation Conditional Generative Modeling Emotion Recognition Affective Computing

Open paper AskPDF arXiv

Emotional body motion expressions are an essential element of non-verbal communication. Effectively conveying these expressions through technology is of utmost importance, for example, with virtual reality avatars and in social robotics. Recent advances in generative models have opened new opportunities for advancing research on emotional body motion learning. However, generating accurate emotional expression representations is challenging, given the subtlety of emotional cues, individual variability, and cultural differences. We investigate whether a generative model can implicitly learn emotional body motions directly from culturally grounded motion-capture data, without explicit emotion-motion guidance. Using a dataset of emotional performances by 49 Japanese actors, we trained a Transformer-based generative model to generate expressive motions conditioned on 13 discrete emotion labels. We evaluate the generated motions from two perspectives: (1) an LSTM-based classifier to assess recognizability by machine observers, achieving a recognition accuracy of 22.80%, and (2) a human perception study with Japanese raters to assess alignment with human affective interpretations, yielding a recognition accuracy of 24.91%. Beyond these, we evaluate the utility of generative modeling for three practical tasks: augmenting emotion recognition models, extracting representative emotion-specific motion patterns, and synthesizing smooth transitions between emotion intensities. Our findings highlight the potential of implicit, data-driven generative modeling to enhance affective computing applications and our understanding of emotion expressions.

Abstract

Explore similar work

Generative Learning as a Tool to Improve Perception of Emotional Body Motion Expressions

Improving Generative Adversarial Network Generalization for Facial Expression Synthesis

Emotion Intensity Matters: Generating Realistic Expressions in Virtual Humans with CVAEs