Facial Expression Recognition Using Residual Masking Network

Luan Pham; The Huynh Vu; Tuan Anh Tran

cs.CVMar 6, 2026

Facial Expression Recognition Using Residual Masking Network

Authors: Luan Pham, The Huynh Vu, Tuan Anh Tran

Organizations: Research Department-Cinnamon AI, Viet Nam · Faculty of Computer Science & Engineering, Ho Chi Minh City-University of Technology (HCMUT), Viet Nam

Abstract

Automatic facial expression recognition (FER) has gained much attention due to its applications in human-computer interaction. Among the approaches to improve FER tasks, this paper focuses on deep architecture with the attention mechanism. We propose a novel Masking idea to boost the performance of CNN in facial expression task. It uses a segmentation network to refine feature maps, enabling the network to focus on relevant information to make correct decisions. In experiments, we combine the ubiquitous Deep Residual Network and Unet-like architecture to produce a Residual Masking Network. The proposed method holds state-of-the-art (SOTA) accuracy on the well-known FER2013 and private VEMO datasets. The source code is available at https://github.com/phamquiluan/ResidualMaskingNetwork.

Figures & tables

Fig. 1 : Example of landmark detections and features of Masking Block as follows: landmark detection, original image, feature map before the $3^{rd}$ Masking Block, feature map after the $3^{rd}$ Masking Block.

Layer name	Ouput size	Detail
Conv1	$64\times 112\times 112$	$7\times 7$ , stride 2
MaxPooling	$64\times 56\times 56$	$3\times 3$ , stride 2
Resmasking Block 1	$64\times 56\times 56$	RL 1, MB 1
Resmasking Block 2	$128\times 28\times 28$	RL 2, MB 2
Resmasking Block 3	$256\times 14\times 14$	RL 3, MB 3
Resmasking Block 4	$512\times 7\times 7$	RL 4, MB 4

TABLE I : The detailed Residual Masking Network configuration (RL: Residual Layer, MB: Masking Block).

Fig. 2 : The overview of Residual Masking Network.

Panel of Figure 4 — Fig. 3 : Example images of two datasets.

Panel of Figure 5 — Fig. 4 : The statistics of training, validation, and testing set.

Fig. 5 : The framework structure of the experiment setup.

Fig. 6 : Grad-CAM visualization [ 41 ] of some images in the testing sets.

Networks	Parameters x $10^{6}$	Accuracy (%)
VGG19 [ 43 ]	139.5	70.80
Eficientnet_b2b [ 44 ]	7.7	70.80
Googlenet [ 43 ]	5.6	71.97
Resnet34 [ 31 ]	21.2	72.41
ResAttNet56 [ 10 ]	29.0	72.63
Inception_v3 [ 30 ]	25.1	72.72

TABLE II : Performance evaluation of well-known classification networks and our method on FER2013

Networks	E (%)	WE (%)
Human Accuracy [ 27 ]	-	65 $\pm 5$
DNN_L_Reg [ 45 ]	-	66.4
Deep-Emotion [ 46 ]	-	70.02
DL-SVM [ 47 ]	-	71.16
CNN-SIFT [ 22 ]	-	73.4
Ensemble N&A [ 48 ]	73.31	-

TABLE III : Performance evaluation (Accuracy) of reported methods without ensemble (WE) and ensemble (E) on FER2013.

Networks	Accuracy (%)
ResAttNet56 [ 10 ]	60.82
Resnet18	63.94
Resnet34	64.84
Our ResMaskingNet	65.94

TABLE IV : Performance evaluation of three classification networks and our Residual Masking Network on the VEMO dataset.

Panel of Figure 11 — Fig. 7 : Quantitative results in form of confusion matrix on two testing sets

Fig. 8 : Examples of wrong emotion recognition on FER2013 and VEMO dataset (Ground truth / Predict)

Explore similar work

SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition
Aug 9, 2026Jiaye Song, Ruochen Zhang, Yuliang Wang +1Residual Learning Facial Expression Recognition
Tiangong University, No. 399 Binshuixi Road, Xiqing District, Tianjin 300387, China
A cross-modal network for facial expression recognition
May 6, 2026Chunwei Tian, Jingyuan Xie, Qi Zhang +3Facial Expression Recognition
School of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China · School of Software, Northwestern Polytechnical University, Xi’an 710072, China · School of Economics and Management, Harbin Institute of Technology at Weihai, Weihai, 264209, China +2
Facial Expression Recognition in the Deep Learning Era: A Systematic Multi-Criteria Review of Methods, Models, Datasets, Performance, Challenges, and Future Research Directions
Jun 7, 2026Spyridon Georgiou, Aggelos Psiris, Spyridon Evangelatos +5Facial Expression Recognition
Department of Informatics and Telematics, Harokopio University of Athens, Athens, Greece · Research & Innovation Development Department, Netcompany-Intrasoft S.A., Luxembourg, Luxembourg · Department of Informatics, Faculty of Science, Democritus University of Thrace, Kavala, Greece +3

Aug 9, 2026J/K move · Enter open · S save

SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition

Jiaye Song, Ruochen Zhang, Yuliang Wang, Jiaqi Wu

Tiangong University, No. 399 Binshuixi Road, Xiqing District, Tianjin 300387, China

Residual Learning Facial Expression Recognition

Open paper AskPDF arXiv

Fine-grained facial expression recognition (FER) hinges on capturing subtle muscular cues that distinguish adjacent emotions. Yet capturing these cues presents a dilemma. Detector-based methods depend on fragile landmark pipelines, whereas we find that directly transferring foundation models such as DINOv3 under conventional global readouts can cause local evidence dilution: early global aggregation washes out sparse muscular signals and leaves persistent confusion between categories such as fear/surprise and sad/neutral. To recover this evidence, we propose SRE-FER, a readout-level regional residual evidence learning framework. Its core module, RERA, adds zero-initialized residual logits that refine class boundaries while preserving the backbone's global prediction. Training-time action unit (AU) guidance steers regional features toward expression-relevant areas using Facial Action Coding System (FACS)-based anatomical priors, without requiring an external facial pipeline at inference. An optional Full setting further routes sample-specific non-redundant tokens. On three benchmarks, SRE-FER attains 92.76% on RAF-DB, 91.32% on FERPlus, and 67.78% on AffectNet-7, demonstrating highly competitive performance compared to existing FER methods.

Abstract

Explore similar work

SRE-FER: Regional residual evidence learning for mitigating local evidence dilution in fine-grained facial expression recognition

A cross-modal network for facial expression recognition

Facial Expression Recognition in the Deep Learning Era: A Systematic Multi-Criteria Review of Methods, Models, Datasets, Performance, Challenges, and Future Research Directions