Evaluating and Correcting Human Annotation Bias in Dynamic Micro-Expression Recognition

Feng Liu; Bingyu Nan; Xuezhong Qian; Xiaolan Fu

cs.CVMar 5, 2026

Evaluating and Correcting Human Annotation Bias in Dynamic Micro-Expression Recognition

Authors: Feng Liu, Bingyu Nan, Xuezhong Qian, Xiaolan Fu

Organizations: School of Psychology, Shanghai Jiao Tong University, Shanghai 200030, China · School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China

Abstract

Existing manual labeling of micro-expressions is subject to errors in accuracy, especially in cross-cultural scenarios where deviation in labeling of key frames is more prominent. To address this issue, this paper presents a novel Global Anti-Monotonic Differential Selection Strategy (GAMDSS) architecture for enhancing the effectiveness of spatio-temporal modeling of micro-expressions through keyframe re-selection. Specifically, the method identifies Onset and Apex frames, which are characterized by significant micro-expression variation, from complete micro-expression action sequences via a dynamic frame reselection mechanism. It then uses these to determine Offset frames and construct a rich spatio-temporal dynamic representation. A two-branch structure with shared parameters is then used to efficiently extract spatio-temporal features. Extensive experiments are conducted on seven widely recognized micro-expression datasets. The results demonstrate that GAMDSS effectively reduces subjective errors caused by human factors in multicultural datasets such as SAMM and 4DME. Furthermore, quantitative analyses confirm that offset-frame annotations in multicultural datasets are more uncertain, providing theoretical justification for standardizing micro-expression annotations. These findings directly support our argument for reconsidering the validity and generalizability of dataset annotation paradigms. Notably, this design can be integrated into existing models without increasing the number of parameters, offering a new approach to enhancing micro-expression recognition performance. The source code is available on GitHub[https://github.com/Cross-Innovation-Lab/GAMDSS].

Figures & tables

Fig. 1: This paper attempts to address the distortion of ground truth labeling due to human subjective errors in the annotation of micro-expression datasets. (a) Traditional manual annotation of key frames involves three steps: global observation, local observation, and frame-by-frame observation. However, this process requires high expertise from annotators, and the frame-by-frame observation stage may introduce subjective errors. (b) The automatic annotation method locates the Apex by calculating inter-frame changes through a sliding window, without requiring manual frame-by-frame comparison. However, this method typically relies on an extra training process and introduces extra model parameters. (c) The proposed GAMDSS method can re-select key frames based on manual annotations, effectively avoiding human subjective errors. Importantly, it is plug-and-play and does not introduce additional parameters.

Fig. 2: Under the three-classification conditions, the individual sample difference curves obtained after calculating differences across three datasets are shown in the figure. Specifically, the L2 norm of pixel values between frames is computed frame-by-frame to quantify motion intensity, with its apex serving as a key objective metric for assessing changes in expression intensity. The original manually annotated Apex frames in the dataset are marked with red dashed lines, while the maximum difference values derived from the difference calculations are highlighted with dashed boxes.

Fig. 3: An overview of the proposed GAMDSS architecture is provided below. (a) The GAMDSS pipeline consists of the following steps: First, Dynamic Frame Reselection Mechanism reselects the three frames with the richest action changes based on different datasets. Second, a backbone model and feature processing method are selected. Next, spatio-temporal features are extracted at different stages using spatio-temporal units with two shared parameters. Where the temporal stream integrates the RMT module, which efficiently models long-term temporal dependencies through a retention mechanism based on Manhattan distance decay. Finally, the spatio-temporal features are integrated, and an auxiliary loss function is introduced to inject additional knowledge, thereby enabling the modeling of the complete evolution process of micro-expressions. (b) The designed method for extracting spatio-temporal features and their fusion approach, where Swish activation layers are employed to enhance feature nonlinearity and improve optimization stability.

Fig. 4: Conceptual design of GAMDSS. First, a set of relevant frames is determined based on manually annotated key frames. Second, frame pairs with the greatest action changes are reselected from the frame set through difference calculation, and these are used as the reselected onset frames and apex frames. Finally, the reselected offset is determined based on the reselected apex frames.

Panel of Figure 5 — Algorithm 1 Pseudo code for Dynamic frame reselection mechanism.

Dataset	Year	Subs	Gender (M/F)	Age	Samples	FPS	Emotions	Cultural composition
CASME [ 38 ]	2013	35	22/13	Mean=22.03	195	60	contempt disgust fear happiness repression sadness surprise tense	As: 35
CASME II [ 39 ]	2014	35	22/13	Mean=22.03	247	200	disgust happiness repression surprise others	As: 35
SAMM [ 40 ]	2016	32	16/16	Mean=33.24 (Range:19-57)	159	200	anger contempt disgust fear happiness sadness surprise other	E: 18 As: 8 ME: 2 Af: 3 Mix: 1
CAS(ME) 2 [ 41 ]	2017	22	9/13	Mean=22.59 (Range:19-26)	57	30	anger disgust happiness	As: 22
MMEW [ 42 ]	2021	36	/	Mean=22.35	300	90	anger disgust fear happiness sadness surprise others	As: 36
4DME [ 43 ]	2022	56	38/27	Mean=27.80 (Range:22-57)	267	30	negative positive surprise repression others	As: 37 E: 28

TABLE I: Statistical information on the mainstream micro-expression dataset.

Model	CASME II				SAMM				#param.	FLOPs
	5-class (% ↑)		3-class (% ↑)		5-class (% ↑)		3-class (% ↑)		#param.	FLOPs
	ACC.	UF1	ACC.	UF1	ACC.	UF1	ACC.	UF1
Swin-T [ 45 ] ’21	67.21	62.47	84.42	79.87	65.67	55.56	77.09	60.52	28M	4.5G
MERSiamC3D [ 46 ] ’21	81.89	83.00	87.63	88.18	68.75	64.00	72.80	74.75	-	-
AU-GCN [ 47 ] ’21	74.27	70.47	87.10	87.98	74.26	70.45	78.90	77.51	-	-
ConvNeXt-T [ 48 ] ’22	84.62	82.11	91.56	89.67	73.1	60.43	82.44	66.86	29M	4.5G

TABLE II: Performance comparison on CASME II and SAMM, ACC. stands for accuracy, and UF1 stands for unweighted F1 score. The best results are highlighted in bold, and the second-best results are underlined.

Model	7-class (% ↑)		4-class (% ↑)
Model	UF1	UAR	UF1	UAR
AlexNet [ 44 ] ’22	17.59	18.01	29.15	29.10
$\mu$ -BERT [ 54 ] ’23	32.64	32.54	47.18	49.13
SFAMNet [ 62 ] ’24	23.65	23.73	44.62	47.97
ATM-GCN [ 55 ] ’24	43.08	42.83	54.23	53.49
PC-GCN [ 63 ] ’25	35.64	41.59	47.64	53.66

TABLE III: Performance comparison on CAS(ME) 3

DataSet	Model		7 or 5-class (% ↑)		4 or 3-class (% ↑)
DataSet	D	S	ACC./UAR	UF1	ACC./UAR	UF1
CASME II	$\times$	$\times$	82.19	79.80	90.90	89.69
	$\checkmark$	$\times$	85.02(+2.83)	83.08(+3.28)	93.50(+2.60)	91.81(+2.12)
	$\times$	$\checkmark$	85.82(+3.63)	83.73(+3.93)	93.50(+2.60)	92.25(+2.56)
	$\checkmark$	$\checkmark$	87.04(+4.85)	85.48(+5.68)	94.19(+3.29)	93.62(+3.93)
SAMM	$\times$	$\times$	78.35	72.56	85.49	76.76

TABLE IV: Ablation study on CASME II and SAMM.

$\lambda_{r}$		SAMM (% ↑)		CASME II(% ↑)		4DME (% ↑)		MMEW (% ↑)
$\lambda_{rise}$	$\lambda_{fall}$	ACC.	UF1	ACC.	UF1	ACC.	UF1	ACC.	UF1
0.00	0.00	79.85	75.81	85.82	83.73	85.09	68.29	79.91	64.41
0.05	0.05	81.34	78.29	87.04	85.48	85.09	68.29	80.34	68.20
0.10	0.10	82.84	81.47	85.02	84.44	86.84	74.98	79.48	66.14
0.15	0.15	82.09	77.25	85.82	84.66	85.53	77.61	79.48	62.84
0.20	0.20	81.34	77.65	85.43	82.96	82.89	69.56	79.05	61.36

TABLE V: Strategies for range of frame reselection.

Model	SAMM				Train Time	CASME II				Train Time
	5-class (% ↑)		3-class (% ↑)			5-class (% ↑)		3-class (% ↑)
	ACC.	UF1	ACC.	UF1		ACC	UF1	ACC.	UF1
ResNet [ 64 ]	75.37	66.32	83.96	74.30	1 h 10 min 38.97 s	81.37	78.12	89.61	86.94	30 min 7.36 s
ResNet (+3DF-N [ 65 ] )	77.61	67.51	85.49	79.29	2 h 12 min 37.38 s	80.97	77.82	88.96	86.03	1 h 12 min 55.08 s
ResNet (+GAMDSS)	77.61	69.95	86.25	78.50	1 h 51 min 54.53 s	81.78	79.23	90.90	87.51	54 min 13.24 s
ConvNeXt [ 48 ]	73.13	60.43	82.44	66.86	1 h 38 min 43.76 s	84.62	82.11	91.56	89.67	46 min 16.48 s

TABLE VI: Backbone study on SAMM and CASME II.

Model	Feature	SAMM				Training Time	CASME II				Training Time
		5-class (% ↑)		3-class (% ↑)			5-class (% ↑)		3-class (% ↑)
		ACC.	UF1	ACC.	UF1		ACC	UF1	ACC.	UF1
ResNet [ 64 ]	flow	76.12	65.47	81.82	73.57	1 h 44 min 10.57 s	78.95	74.71	87.66	84.27	52 min 31.96 s
ResNet [ 64 ]	difference	75.37	66.32	83.96	74.30	1 h 8 min 21.87 s	81.37	78.12	89.61	86.94	30 min 7.36 s
ConvNeXt [ 48 ]	flow	62.69	52.33	77.86	61.54	2 h 18 min 13.26 s	63.16	53.94	72.73	60.62	1 h 16 min 43.40 s
ConvNeXt [ 48 ]	difference	73.13	60.43	82.44	66.86	1 h 38 min 43.76 s	84.62	82.11	91.56	89.67	46 min 16.48 s

TABLE VII: Feature processing study on the SAMM and CASME II.

Fig. 5: The GAMDSS selection sample is represented visually as follows: pink indicates the original annotation information, blue indicates the GAMDSS re-selected annotation information, and yellow indicates the remaining frames.

Fig. 6: Visualization of feature distributions for the benchmark model and GAMDSS on three datasets. The dashed circles highlight regions where class clusters show clearer separation under GAMDSS compared with the baseline model.

Fig. 7: Quantitative evaluation of keyframe annotation deviations across datasets. (a) Bar plots show relative deviations ( $D\%$ ) computed via Eq. 15 . (b) Line plots show the average absolute deviation calculated based on Eq. 17 , which represents the mean of the absolute deviations for the three key frames at onset, apex, and offset.

Fig. 8: Average difference distance between original manual keyframe annotations and GAMDSS re-annotations, in ms.

Micro-Expression Recognition

See all 12 papers

Jun 26, 2026

STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

Micro-expression recognition is challenging due to subtle and short-lived facial muscle movements. Existing methods rely heavily on apex-onset frames, overlook fine-grained inter-frame dynamics, and separately model spatial and temporal information, limiting generalization across datasets. To address these challenges, we propose STAG, a dynamic ROI-AU-coupled spatial-temporal network that jointly models motion flow and adaptive facial connectivity. The framework extracts optical flow from discriminative frames using magnitude-based selection and temporal attention. A dual-branch architecture combines an enhanced graph attention network for structured spatial reasoning with a transformer encoder for temporal modeling. A bidirectional cross-attention module enables mutual refinement of spatial and temporal features, while AU-guided dynamic connectivity adapts facial region interactions according to muscle activation patterns. The transformer captures subtle temporal dynamics beyond apex-based approaches, improving semantic consistency and interpretability for explainable micro-expression recognition. The fused representation is optimized using focal loss and evaluated on CASME II, 4DME, DFME, NaME, SAMM, and SMIC-HS. Extensive experiments demonstrate improved robustness, generalization, interpretability, and computational efficiency, confirming the effectiveness of adaptive relational reasoning, AU-guided dynamic connectivity, and deep spatial-temporal feature fusion for accurate cross-dataset micro-expression recognition.

Nandani Sharma, Varun Sharma, Dinesh Singh

Vision Intelligence and Machine Learning (VIML) Group, School of Computing and Electrical Engineering, Indian Institute of Technology Mandi, Mandi, 175005, Himachal Pradesh, India · Indian Institute of Information Technology Bhagalpur, Sabour Rd, Bhagalpur, 813210, Bihar, India

Facial Expression Recognition Micro-Expression Recognition Spatiotemporal GNNs

Ask PDF

Feb 27, 2026

Micro-expression Recognition Based on Dual-branch Feature Extraction and Fusion

Micro-expressions, characterized by transience and subtlety, pose challenges to existing optical flow-based recognition methods. To address this, this paper proposes a dual-branch micro-expression feature extraction network integrated with parallel attention. Key contributions include: 1) a residual network designed to alleviate gradient anishing and network degradation; 2) an Inception network constructed to enhance model representation and suppress interference from irrelevant regions; 3) an adaptive feature fusion module developed to integrate dual-branch features. Experiments on the CASME II dataset demonstrate that the proposed method achieves 74.67% accuracy, outperforming LBP-TOP (by 11.26%), MSMMT (by 3.36%), and other comparative methods.

Mingjie Zhang, Bo Li, Wanting Liu +5

School of Petroleum China University of Petroleum (Beijing) Karamay Campus Karamay, China · Artificial Intelligence Research Institute China Tower Corporation Limited Beijing, China · Low Carbon Business Department South Electric Power Science and Technology Development Co., Ltd. Guangzhou, China +1

Micro-Expression Recognition Facial Expression Recognition Feature Fusion

Ask PDF

Apr 20, 2026

MEDN: Motion-Emotion Feature Decoupling Network for Micro-Expression Recognition

Unlike macro-expression, micro-expression does not follow a strictly consistent mapping rule between emotions and Action Units (AUs). As a result, some micro-expressions share identical AUs yet represent completely opposite emotional categories, making them highly visually similar. Existing microexpression recognition (MER) methods mostly rely on explicit facial motion cues (e.g., optical flow, frame differences, AU features) while ignoring implicit emotion information. To tackle this issue, this paper presents a Motion Emotion Feature Decoupling Network (MEDN) for MER. We design a dual-branch framework to separately extract motion and emotion features. In the motion branch, an AU-detection task restricts features to the explicit motion domain, and orthogonal loss is adopted to reduce motion emotion feature coupling. For implicit emotion modeling, we propose a Sparse Emotion Vision Transformer (SEVit) that sparsifies spatial tokens to highlight local temporal variations with multi-scale sparsity rates. A Collaborative Fusion Module (CoFM) is further developed to fuse disentangled motion and emotion features adaptively. Extensive experiments on three benchmark datasets validate that MEDN effectively decouples motion and emotion features and achieves superior recognition performance, offering a new perspective for enhancing recognition accuracy and generalization.

Chenxing Hu, Kun Xie, Qiguang Miao +3

School of Computer Science and Technology, Xidian University, Xi’an, Shaanxi 710071, China · Xi’an Key Laboratory of Big Data and Intelligent Vision, Xi’an, Shaanxi 710071, China · Key Laboratory of Collaborative Intelligence Systems, Ministry of Education, Xidian University, Xi’an 710071, China +1

Disentangled Representation Learning Vision Transformer Facial Action Unit Detection Facial Expression Recognition Micro-Expression Recognition

Ask PDF

Abstract

Micro-Expression Recognition

Explore similar work

STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

Micro-expression Recognition Based on Dual-branch Feature Extraction and Fusion

MEDN: Motion-Emotion Feature Decoupling Network for Micro-Expression Recognition