Event-Level Emotion Recognition in the Wild Using Deep Facial Expression Analysis

Aleksandr Semerikov; Pakizar Shamoi

cs.CVSep 12, 2026

Event-Level Emotion Recognition in the Wild Using Deep Facial Expression Analysis

Authors: Aleksandr Semerikov, Pakizar Shamoi

Organizations: School of Information Technology and Engineering Kazakh-British Technical University Almaty, Kazakhstan

Abstract

Facial emotion recognition (FER) in real-world environments remains challenging due to unconstrained imaging conditions, including multiple faces, occlusions, pose variations, and complex lighting. Most existing studies focus on individual facial emotion classification and do not address the analysis of collective emotional states at the event level. This paper proposes an end-to-end pipeline for event-level emotion recognition from photographs. The approach detects faces in each image, classifies facial expressions using a deep convolutional neural network, and aggregates face-level emotion probabilities to estimate the overall emotional distribution of a public event. A comparative evaluation of several CNN architectures on the FER- 2013 and RAF-DB datasets demonstrates that transfer learning with EfficientNet-B2 trained on RAF-DB is more suitable for real-world RGB data. The proposed method is evaluated on a real-world event dataset containing 1658 images. Experimental results show stable emotion distributions across event subsets, confirming the effectiveness of event-level aggregation for emotion analysis in the wild.

Figures & tables

Fig. 1: Sample images from FER-2013 dataset.

Fig. 2: Sample images from the RAF-DB dataset.

Fig. 3: Examples of Cave Fall Fest Dataset images.

Fig. 4: Overview of the proposed event-level emotion recognition pipeline.

Dataset	Model	Input size	Params (M)	Test accuracy
FER-2013	Mini-Xception	$48\times 48\times 1$	1.54	0.656
	Custom CNN	$48\times 48\times 1$	1.28	0.632
	MobileNetV2 (TL)	$128\times 128\times 3$	2.27	0.247
	EfficientNet-B2 (TL)	$224\times 224\times 3$	7.78	0.247
RAF-DB	Mini-Xception	$96\times 96\times 3$	1.54	0.684
	MobileNetV2 (TL)	$128\times 128\times 3$	2.27	0.701

TABLE I: Comparison of CNN architectures on FER-2013 and RAF-DB datasets

Set	Surprised	Fear	Disgust	Happy	Sad	Angry	Neutral
1 – Set	0.180	0.007	0.032	0.085	0.167	0.014	0.514
2 – Set	0.203	0.003	0.021	0.020	0.162	0.007	0.584
3 – Set	0.211	0.006	0.036	0.071	0.158	0.015	0.503
4 – Set	0.175	0.006	0.023	0.057	0.153	0.017	0.570

TABLE II: Mean Emotion Probabilities Across the Climbing Sets

Fig. 5: Aggregated emotion distributions across four event sets.

Fig. 6: Examples of detected faces with predicted emotions and confidence scores.

Fig. 7: Violin plot of the emotion distribution of the faces in Cave Fall Fest dataset.

Fig. 8: Confidence distribution for happy class.

Explore similar work

Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras
Sep 7, 2026Jiaqi Chen, Qinfu Xu, Hao Zhuang +1Multimodal Emotion Recognition Emotion Recognition
Beijing Institute of Technology Beijing, China
Facial Expression Recognition in the Deep Learning Era: A Systematic Multi-Criteria Review of Methods, Models, Datasets, Performance, Challenges, and Future Research Directions
Jun 7, 2026Spyridon Georgiou, Aggelos Psiris, Spyridon Evangelatos +5Facial Expression Recognition
Department of Informatics and Telematics, Harokopio University of Athens, Athens, Greece · Research & Innovation Development Department, Netcompany-Intrasoft S.A., Luxembourg, Luxembourg · Department of Informatics, Faculty of Science, Democritus University of Thrace, Kavala, Greece +3
SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition
Aug 9, 2026Jiaye Song, Ruochen Zhang, Yuliang Wang +1Residual Learning Facial Expression Recognition
Tiangong University, No. 399 Binshuixi Road, Xiqing District, Tianjin 300387, China

Sep 7, 2026J/K move · Enter open · S save

Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras

Jiaqi Chen, Qinfu Xu, Hao Zhuang, Liyuan Pan

Beijing Institute of Technology Beijing, China

Multimodal Emotion Recognition Emotion Recognition Event-Based Vision Multimodal Sensor Fusion Privacy-Preserving ML

Open paper AskPDF arXiv

Emotion analysis is a fundamental task in computer vision, but its practical deployment remains constrained by the privacy risks inherent to conventional RGB cameras. Bio-inspired event cameras present a promising hardware-level solution because they capture asynchronous brightness changes, thereby reducing exposure of facial identity details while leveraging high dynamic range for robust perception under challenging illumination conditions. Despite these advantages, existing event-based methods struggle in complex real-world settings due to limited dataset scales, simple acquisition conditions, and reliance on single-modality visual cues. To address these, we establish a challenging tri-modal benchmark with event, audio, and text modalities and propose the Information-Guided Gated Fusion (IGF) framework, which first pre-trains an event encoder on the FAU subset of Emo-DVS to capture fine-grained facial dynamics, then employs adaptive modality gating to suppress modality-specific noise, and finally leverages mutual information maximization to align robust cross-modal representations. To alleviate data scarcity, we introduce Emo-DVS, the first large-scale event-based emotion analysis dataset, which couples dynamic illumination with the Facial Action Unit (FAU) subset and emotion subset. Extensive experiments demonstrate that IGF achieves state-of-the-art performance.

Abstract

Explore similar work

Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras

Facial Expression Recognition in the Deep Learning Era: A Systematic Multi-Criteria Review of Methods, Models, Datasets, Performance, Challenges, and Future Research Directions

SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition