Cognition-Inspired Dual-Stream Semantic Enhancement for Vision-Based Dynamic Emotion Modeling

Huanzhen Wang; Ziheng Zhou; Zeng Tao; Aoxing Li; Yingkai Zhao; Yuxuan Lin; Yan Wang; Wenqiang Zhang

cs.CVApr 14, 2026

Cognition-Inspired Dual-Stream Semantic Enhancement for Vision-Based Dynamic Emotion Modeling

Authors: Huanzhen Wang, Ziheng Zhou, Zeng Tao, Aoxing Li, Yingkai Zhao, Yuxuan Lin, Yan Wang, Wenqiang Zhang

Organizations: College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China · Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · School of Data Science and Engineering, East China Normal University, Shanghai, China

Abstract

The human brain constructs emotional percepts not by processing facial expressions in isolation, but through a dynamic, hierarchical integration of sensory input with semantic and contextual knowledge. However, existing vision-based dynamic emotion modeling approaches often neglect emotion perception and cognitive theories. To bridge this gap between machine and human emotion perception, we propose cognition-inspired Dual-stream Semantic Enhancement (DuSE). Our model instantiates a dual-stream cognitive architecture. The first stream, a Hierarchical Temporal Prompt Cluster (HTPC), operationalizes the cognitive priming effect. It simulates how linguistic cues pre-sensitize neural pathways, modulating the processing of incoming visual stimuli by aligning textual semantics with fine-grained temporal features of facial dynamics. The second stream, a Latent Semantic Emotion Aggregator (LSEA), computationally models the knowledge integration process, akin to the mechanism described by the Conceptual Act Theory. It aggregates sensory inputs and synthesizes them with learned conceptual knowledge, reflecting the role of the hippocampus and default mode network in constructing a coherent emotional experience. By explicitly modeling these neuro-cognitive mechanisms, DuSE provides a more neurally plausible and robust framework for dynamic facial expression recognition (DFER). Extensive experiments on challenging in-the-wild benchmarks validate our cognition-centric approach, demonstrating that emulating the brain's strategies for emotion processing yields state-of-the-art performance and enhances model interpretability.

Figures & tables

Fig. 1: The priming effect and knowledge integration mechanisms in human emotional cognition inspire us to explore the progressive complementary relationship between prompts and knowledge in visual language sentiment modeling.

Fig. 2: Supervision from natural language can effectively alleviate the problem of difficulty in classifying expressions with inter-class similarities. Our method aims to bridge the gap between general image-based pre-training models and specialized recognition tasks based on dynamic expression sequences.

Fig. 3: Overall architecture of DuSE . (a) shows the overall methodological framework. (b) shows HTPC, which contributes to the cross-modal prompt streaming. (c) shows LSEA, which contributes to cross-domain knowledge streaming.

Method	Publication	DFEW		FERV39k
Method	Publication	UAR	WAR	UAR	WAR
C3D [ 22 ]	CVPR’15	42.74	53.54	22.68	31.69
P3D [ 32 ]	ICCV’17	43.97	54.47	23.20	33.39
I3D-RGB [ 33 ]	ICCV’17	43.40	54.27	30.17	38.78
3D ResNet18 [ 34 ]	CVPR’18	46.52	58.27	26.67	37.57
EC-STFL [ 20 ]	MM’20	45.35	56.51	-	-

TABLE I: Comparative results (%). Our proposed DuSE performs well on both datasets for 7-class classification. Best results are in bold and second-best results are underlined.

Method	Publication	Happy	Sad	Neutral	Angry	Surprise	Disgust	Fear	UAR	WAR
C3D [ 22 ]	CVPR’15	75.17	39.49	55.11	62.49	45.00	1.38	20.51	42.74	53.54
I3D-RGB [ 33 ]	CVPR’17	78.61	44.19	56.69	55.87	45.88	2.07	20.51	43.40	54.27
P3D [ 32 ]	ICCV’17	74.85	43.40	54.18	60.42	50.99	0.69	23.28	43.97	54.47
3D ResNet18 [ 34 ]	CVPR’18	76.32	50.21	64.18	62.85	47.52	0.00	24.56	46.52	58.27
EC-STFL [ 20 ]	MM’20	79.18	49.05	57.85	60.98	46.15	2.76	21.51	45.35	56.51
Former-DFER [ 35 ]	MM’21	84.05	62.57	67.52	70.03	56.43	3.45	31.78	53.69	65.70

TABLE II: Comparative results (%) across different methods on various emotion categories in DFEW. Best results are in bold and second-best results are underlined.

Fig. 4: Global and category-specific t-SNE visualization of DuSE on DFEW-fold5. The clustering results show that DuSE has a significant enhancement effect.

Fig. 5: Confusion matrices of the DuSE on DFEW and FERV39k.

HTPC	LSEA	DFEW		FERV39k
HTPC	LSEA	UAR	WAR	UAR	WAR
$\times$	$\times$	55.64	66.80	33.74	46.26
$\times$	✓	58.86	70.28	35.77	47.85
✓	$\times$	60.27	71.83	36.53	48.80
✓	✓	64.88	75.36	43.39	53.05

TABLE III: Comparative results (%) of the inter-module ablation experiments.

Pre-trained model	Strategy	DFEW		FERV39k
(parameters-frozen)	Strategy	UAR	WAR	UAR	WAR
CLIP ViT-B/32	$Shallow$	49.95	61.35	33.47	44.92
	$Normal$	53.22	64.70	35.02	46.87
	$Deep$	57.75	68.76	36.93	48.54
CLIP ViT-B/16	$Shallow$	55.27	65.97	35.14	46.13
	$Normal$	56.84	68.15	37.32	48.05

TABLE IV: Comparative results (%) of the intra-module ablation experiments of HTPC (cross-modal prompt streaming).

Hyperparameter	Value	DFEW		FERV39k
Hyperparameter	Value	UAR	WAR	UAR	WAR
$\mathcal{N}$	2	64.37	74.92	43.16	52.71
	4	64.88	75.36	43.39	53.05
	6	64.21	74.83	42.94	52.08
$\beta$	0.3	61.26	71.82	38.55	48.73
	0.5	62.01	73.56	40.98	50.09

TABLE V: Comparative results (%) of the hyperparameter ablation experiments of LSEA (cross-domain knowledge streaming).

Fig. 6: t-SNE visualization on the DFEW 5-fold dataset.

Fig. 7: Deployment modes and implementation results, which are also mentioned in the demo video.

Explore similar work

ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception
Apr 14, 2026Huanzhen Wang, Ziheng Zhou, Jiaqi Song +4Synthetic Data Augmentation Facial Expression Recognition
Fudan University · East China Normal University
Why Do Vision Language Models Struggle To Recognize Human Emotions?
Apr 16, 2026Madhav Agarwal, Sotirios A. Tsaftaris, Laura Sevilla-Lara +1VLM Evaluation Multimodal Emotion Recognition
The University of Edinburgh, UK
Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow
May 21, 2026Chengsheng Zhang, Chenghao Sun, Zhining Xie +1VLM Hallucination VLM Interpretability
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China · AIPD, Tencent

Apr 14, 2026J/K move · Enter open · S save

ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception

Huanzhen Wang, Ziheng Zhou, Jiaqi Song, Li He+3

Fudan University · East China Normal University

Synthetic Data Augmentation Facial Expression Recognition Video Diffusion Models Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Dynamic facial expression recognition in the wild remains challenging due to data scarcity and long-tail distributions, which hinder models from effectively learning the temporal dynamics of scarce emotions. To address these limitations, we propose ARGen, an Affect-Reinforced Generative Augmentation Framework that enables data-adaptive dynamic expression generation for robust emotion perception. ARGen operates in two stages: Affective Semantic Injection (ASI) and Adaptive Reinforcement Diffusion (ARD). The ASI stage establishes affective knowledge alignment through facial Action Units and employs a retrieval-augmented prompt generation strategy to synthesize consistent and fine-grained affective descriptions via large-scale visual-language models, thereby injecting interpretable emotional priors into the generation process. The ARD stage integrates text-conditioned image-to-video diffusion with reinforcement learning, introducing inter-frame conditional guidance and a multi-objective reward function to jointly optimize expression naturalness, facial integrity, and generative efficiency. Extensive experiments on both generation and recognition tasks verify that ARGen substantially enhances synthesis fidelity and improves recognition performance, establishing an interpretable and generalizable generative augmentation paradigm for vision-based affective computing.

Abstract

Explore similar work

ARGen: Affect-Reinforced Generative Augmentation towards Vision-based Dynamic Emotion Perception

Why Do Vision Language Models Struggle To Recognize Human Emotions?

Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow