EPIR: An Efficient Patch Tokenization, Integration and Representation Framework for Micro-expression Recognition

Junbo Wang; Liangyu Fu; Yuke Li; Yining Zhu; Xuecheng Wu; Kun Hu

cs.CVApr 9, 2026

EPIR: An Efficient Patch Tokenization, Integration and Representation Framework for Micro-expression Recognition

Authors: Junbo Wang, Liangyu Fu, Yuke Li, Yining Zhu, Xuecheng Wu, Kun Hu

Organizations: School of Software, Northwestern Polytechnical University, Xi’an, 710129, China · School of Computer Science, Northwestern Polytechnical University, Xi’an, 710129, China · School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China · School of Science, Edith Cowan University, WA, 6027, Australia

Abstract

Micro-expression recognition can obtain the real emotion of the individual at the current moment. Although deep learning-based methods, especially Transformer-based methods, have achieved impressive results, these methods have high computational complexity due to the large number of tokens in the multi-head self-attention. In addition, the existing micro-expression datasets are small-scale, which makes it difficult for Transformer-based models to learn effective micro-expression representations. Therefore, we propose a novel Efficient Patch tokenization, Integration and Representation framework (EPIR), which can balance high recognition performance and low computational complexity. Specifically, we first propose a dual norm shifted tokenization (DNSPT) module to learn the spatial relationship between neighboring pixels in the face region, which is implemented by a refined spatial transformation and dual norm projection. Then, we propose a token integration module to integrate partial tokens among multiple cascaded Transformer blocks, thereby reducing the number of tokens without information loss. Furthermore, we design a discriminative token extractor, which first improves the attention in the Transformer block to reduce the unnecessary focus of the attention calculation on self-tokens, and uses the dynamic token selection module (DTSM) to select key tokens, thereby capturing more discriminative micro-expression representations. We conduct extensive experiments on four popular public datasets (i.e., CASME II, SAMM, SMIC, and CAS(ME)3. The experimental results show that our method achieves significant performance gains over the state-of-the-art methods, such as 9.6% improvement on the CAS(ME) $^3$ dataset in terms of UF1 and 4.58% improvement on the SMIC dataset in terms of UAR metric.

Figures & tables

Figure 1: Comparison of (a) our EPIR and (b) the previous Transformer-based micro-expression recognition model. The blue square denotes a visual token.

Figure 2: The overall framework of our proposed EPIR. The left column shows the overall framework of EPIR, and the right column shows the detailed working process of its three main modules. $\mathcal{L}_{ce}$ and $\mathcal{L}_{con}$ denote the cross-entropy and contrastive loss function, respectively.

Method	Params	FLOPs	Full		SAMM		CASME II		SMIC
Method	Params	FLOPs	$\mathrm{UF}1$	UAR	$\mathrm{UF}1$	UAR	$\mathrm{UF}1$	$\mathrm{UAR}$	$\mathrm{UF}1$	$\mathrm{UAR}$
Traditional micro-expression recognition methods
LBP-TOP [ 36 ]	-	-	0.5882	0.5785	0.3954	0.4102	0.7026	0.7429	0.2000	0.5280
Bi-WOOF [ 23 ]	-	-	0.6296	0.6227	0.5211	0.5139	0.7805	0.8026	0.5727	0.5829
Deep learning-based micro-expression recognition methods
OFF-ApexNet [ 8 ]	-	-	0.7196	0.7069	0.5409	0.5392	0.8764	0.8681	0.6817	0.6695

Table 1: Comparative experiments with existing methods for micro-expression recognition under CDE protocol.

Figure 3: Confusion matrices for the proposed EPIR on the composite database (SAMM, SMIC, CASME II). The data in each cell denotes the accuracy for the particular class.

Figure 4: Confusion matrices for the proposed EPIR on the CAS(ME) 3 . The data in each cell denotes the accuracy for the particular class.

Method	Params	FLOPs	CAS(ME) 3
Method	Params	FLOPs	$\mathrm{UF}1$	UAR
STSTNet [ 22 ]	-	-	0.3795	0.3792
FeatRef [ 50 ]	-	-	0.3493	0.3413
$\mu$ -BERT [ 28 ]	333.4	414.5	0.5604	0.6125
HTNet [ 37 ]	438.51	214.6	0.5767	0.5415
Ours	7.03	84.03	0.6727	0.6345

Table 2: Comparative experiments with existing methods for micro-expression recognition on CAS(ME) 3 under SDE protocol.

Method	Params	FLOPs	SAMM		CASME II		SMIC
Method	Params	FLOPs	$\mathrm{UF}1$	UAR	$\mathrm{UF}1$	$\mathrm{UAR}$	$\mathrm{UF}1$	$\mathrm{UAR}$
Traditional micro-expression recognition methods
LBP-TOP [ 36 ]	-	-	0.3589	0.3968	0.3589	0.3968	0.3421	0.4338
MDMO [ 25 ]	-	-	-	-	0.4966	0.5169	0.5845	0.5897
Bi-WOOF [ 23 ]	-	-	-	-	0.6100	0.5885	0.6200	0.6220
Deep learning-based micro-expression recognition methods

Table 3: Comparative experiments for micro-expression recognition under SDE protocol.

Method	Full		SAMM		CASME II		SMIC
Method	$\mathrm{UF}1$	UAR	$\mathrm{UF}1$	UAR	$\mathrm{UF}1$	$\mathrm{UAR}$	$\mathrm{UF}1$	$\mathrm{UAR}$
Baseline	0.7325	0.7214	0.6318	0.6307	0.8511	0.8532	0.6532	0.6767
$+$ DTSM	0.7736	0.7838	0.7351	0.7253	0.8741	0.8703	0.7147	0.7052
$+$ ITALS	0.8021	0.8165	0.7962	0.7815	0.8753	0.8712	0.7501	0.7326
$+$ SPT	0.8513	0.8627	0.8135	0.7831	0.9797	0.9714	0.8031	0.8122
$+$ DNSPT	0.8759	0.8747	0.8288	0.7941	0.9808	0.9791	0.8167	0.8215

Table 4: Ablation experiments for micro-expression recognition.

Method	CAS(ME) 3
Method	$\mathrm{UF}1$	UAR
Baseline	0.3647	0.3129
$+$ DTSM	0.4497	0.4792
$+$ ITALS	0.5318	0.5132
$+$ SPT	0.6233	0.5613
$+$ DNSPT	0.6604	0.6169

Table 5: Ablation experiments for micro-expression recognition on CAS(ME) 3 dataset.

Figure 5: Ablation experiments on the number of Transformer blocks, the two subfigures on the left are the UAR and UF1 metrics under the CDE protocol, and the two subfigures on the right are the UAR and UF1 metrics on the CAS(ME) 3 dataset. The abscissa is the number n of Transformer blocks in all cases.

Figure 6: Actual effect of token integration module on micro-expression samples. We take two micro-expression samples (first and third rows) and their corresponding optical flow feature maps (second and fourth rows), from left to right, with integration rates of 0%, 30%, 60%, and 80%, respectively.

Micro-Expression Recognition

See all 12 papers

Jun 26, 2026

STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

Micro-expression recognition is challenging due to subtle and short-lived facial muscle movements. Existing methods rely heavily on apex-onset frames, overlook fine-grained inter-frame dynamics, and separately model spatial and temporal information, limiting generalization across datasets. To address these challenges, we propose STAG, a dynamic ROI-AU-coupled spatial-temporal network that jointly models motion flow and adaptive facial connectivity. The framework extracts optical flow from discriminative frames using magnitude-based selection and temporal attention. A dual-branch architecture combines an enhanced graph attention network for structured spatial reasoning with a transformer encoder for temporal modeling. A bidirectional cross-attention module enables mutual refinement of spatial and temporal features, while AU-guided dynamic connectivity adapts facial region interactions according to muscle activation patterns. The transformer captures subtle temporal dynamics beyond apex-based approaches, improving semantic consistency and interpretability for explainable micro-expression recognition. The fused representation is optimized using focal loss and evaluated on CASME II, 4DME, DFME, NaME, SAMM, and SMIC-HS. Extensive experiments demonstrate improved robustness, generalization, interpretability, and computational efficiency, confirming the effectiveness of adaptive relational reasoning, AU-guided dynamic connectivity, and deep spatial-temporal feature fusion for accurate cross-dataset micro-expression recognition.

Nandani Sharma, Varun Sharma, Dinesh Singh

Vision Intelligence and Machine Learning (VIML) Group, School of Computing and Electrical Engineering, Indian Institute of Technology Mandi, Mandi, 175005, Himachal Pradesh, India · Indian Institute of Information Technology Bhagalpur, Sabour Rd, Bhagalpur, 813210, Bihar, India

Facial Expression Recognition Micro-Expression Recognition Spatiotemporal GNNs

Ask PDF

Feb 27, 2026

Micro-expression Recognition Based on Dual-branch Feature Extraction and Fusion

Micro-expressions, characterized by transience and subtlety, pose challenges to existing optical flow-based recognition methods. To address this, this paper proposes a dual-branch micro-expression feature extraction network integrated with parallel attention. Key contributions include: 1) a residual network designed to alleviate gradient anishing and network degradation; 2) an Inception network constructed to enhance model representation and suppress interference from irrelevant regions; 3) an adaptive feature fusion module developed to integrate dual-branch features. Experiments on the CASME II dataset demonstrate that the proposed method achieves 74.67% accuracy, outperforming LBP-TOP (by 11.26%), MSMMT (by 3.36%), and other comparative methods.

Mingjie Zhang, Bo Li, Wanting Liu +5

School of Petroleum China University of Petroleum (Beijing) Karamay Campus Karamay, China · Artificial Intelligence Research Institute China Tower Corporation Limited Beijing, China · Low Carbon Business Department South Electric Power Science and Technology Development Co., Ltd. Guangzhou, China +1

Micro-Expression Recognition Facial Expression Recognition Feature Fusion

Ask PDF

Apr 20, 2026

MEDN: Motion-Emotion Feature Decoupling Network for Micro-Expression Recognition

Unlike macro-expression, micro-expression does not follow a strictly consistent mapping rule between emotions and Action Units (AUs). As a result, some micro-expressions share identical AUs yet represent completely opposite emotional categories, making them highly visually similar. Existing microexpression recognition (MER) methods mostly rely on explicit facial motion cues (e.g., optical flow, frame differences, AU features) while ignoring implicit emotion information. To tackle this issue, this paper presents a Motion Emotion Feature Decoupling Network (MEDN) for MER. We design a dual-branch framework to separately extract motion and emotion features. In the motion branch, an AU-detection task restricts features to the explicit motion domain, and orthogonal loss is adopted to reduce motion emotion feature coupling. For implicit emotion modeling, we propose a Sparse Emotion Vision Transformer (SEVit) that sparsifies spatial tokens to highlight local temporal variations with multi-scale sparsity rates. A Collaborative Fusion Module (CoFM) is further developed to fuse disentangled motion and emotion features adaptively. Extensive experiments on three benchmark datasets validate that MEDN effectively decouples motion and emotion features and achieves superior recognition performance, offering a new perspective for enhancing recognition accuracy and generalization.

Chenxing Hu, Kun Xie, Qiguang Miao +3

School of Computer Science and Technology, Xidian University, Xi’an, Shaanxi 710071, China · Xi’an Key Laboratory of Big Data and Intelligent Vision, Xi’an, Shaanxi 710071, China · Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University, Xi’an 710071, China +1

Disentangled Representation Learning Vision Transformer Facial Action Unit Detection Facial Expression Recognition Micro-Expression Recognition

Ask PDF

Abstract

Micro-Expression Recognition

Explore similar work

STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

Micro-expression Recognition Based on Dual-branch Feature Extraction and Fusion

MEDN: Motion-Emotion Feature Decoupling Network for Micro-Expression Recognition