GM-GReFEL: Geometry-Guided Spatiotemporal State-Space Learning for Robust Dynamic Facial Expression Recognition

Yudhistira Arditya Pratama; Yi-Zeng Hsieh

cs.CVOct 22, 2026

GM-GReFEL: Geometry-Guided Spatiotemporal State-Space Learning for Robust Dynamic Facial Expression Recognition

Authors: Yudhistira Arditya Pratama, Yi-Zeng Hsieh

Organizations: Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei 106335, Taiwan

Abstract

Recognizing facial expressions in unconstrained video suffers from two coupled problems: self-attention grows quadratically in cost with clip length, and crowdsourced labels are noisy. We present GM-GReFEL, a spatiotemporal state-space network with a geometry-aware decision stage. Its Modulated GroupMamba backbone replaces a spatiotemporal Vision Transformer. Six channel groups are scanned separately, four over the frame plane and two over time, so the cost grows linearly with sequence length ( $\mathcal{O}(N)$ ); after each layer, a Channel Affinity Modulation gate mixes the groups again. On top of the backbone, a GReFEL reliability module turns the normalized entropy of each prediction into a mixing weight between the classifier output and a vote from trainable geometric anchors, so that facial geometry counts for more when the network is unsure. The model is initialized by static-to-dynamic weight transfer and trained with a Triple Loss. In five-fold cross-validation, the best fold scores 66.57%/76.46% UAR/WAR on DFEW, 0.3 points or less below the strongest compared method, and 45.66%/62.27% on MAFW, ahead of all compared methods; on FERV39k the mean over five splits, 42.30%/52.66%, is competitive. The five-fold means are 64.07%/74.54% on DFEW, on par with recent large pre-trained models, and 38.90%/54.37% on MAFW, below the best of them. A paired $t$ -test on DFEW indicates a modest benefit of temporal modeling (+1.61 UAR points, one-sided $p=0.031$ ). Attention stays on plausible facial regions, and latency is close to a ViT-B/16 baseline (47.4 vs.\ 46.8 ms per clip). Code: https://github.com/ardiytama/GroupMamba-DFER.

Figures & tables

Fig. 1: Structure of the GM-GReFEL pipeline. Top row: the four-stage forward pass, (a) 3D spatiotemporal tubelet tokenization with positional encoding, (b) sequence modeling via the Modulated GroupMamba engine with Channel Affinity Modulation (CAM), (c) geometry modeling via the Temporal Geometry-Aware module that combines the primary MLP’s confidence score with the geometric anchor bank, and (d) Triple Loss optimization, together with example output visualizations (facial landmarks, Grad-CAM attention). Bottom row: internal detail of each stage. Bottom left: the Modulated GroupMamba layer, in which the Spatial Group and Temporal Group each pass through a stack of VSSS blocks before Channel Affinity and Channel Modulation gating produce the $B\times 1\times C$ layer output $Z^{(out)}$ . Bottom center: the Geometry-Aware Reliability mechanism, showing the Statistical Track (Process, Softmax, Normalized Entropy) computed in parallel with the Geometric Track (a similarity lookup against the geometric anchor database via Softmax), combined before Loss Optimization. Bottom right: the six VSSS scanning directions — four boustrophedon spatial-group sweeps (Direction 1–4) and two temporal-group directions (Forward, Backward).

Fig. 2: VSSS block: a Mamba-style selective scan core and a subsequent feed-forward layer.

Configuration	WAR (%)	UAR (%)
GM-GReFEL (full)	73.46 $\pm$ 1.21	64.69 $\pm$ 1.68
w/o Temporal Modeling	73.59 $\pm$ 1.28	63.08 $\pm$ 1.70
w/o Trainable Anchors (frozen)	74.69 $\pm$ 1.19	64.95 $\pm$ 2.20
GroupMamba-SSM depth $L=1$	74.58 $\pm$ 1.49	64.94 $\pm$ 1.53
w/ Single-Stage Training	74.80 $\pm$ 1.37	64.40 $\pm$ 2.00
w/o Jittered Sampling	74.73 $\pm$ 1.52	64.68 $\pm$ 1.90

TABLE I: Component Ablation on DFEW, Mean

\pm

SD Over Five Folds (Population SD,

n=5

). Each Fold Is Taken at Its Epoch of Highest UAR, With WAR at That Epoch. The Full Model Uses

L=3

Hyperparameter	Values tested	Best	UAR / WAR (%)
Dropout $p$	0, 0.25, 0.5, 0.75	0.75	64.10 / 74.93
Layers $L$	1, 3, 5	1	64.19 / 75.69
$\lambda_{2}$ (center)	0, 0.1, 1, 10	10	64.02 / 75.95
$\lambda_{1}$ (orth.)	0, 0.1, 1, 10	10	64.08 / 75.69
Temperature $\tau$	0.01, 0.05, 0.1, 0.5	0.10	65.20 / 75.22

TABLE II: Hyperparameter Sensitivity Summary (DFEW Fold 1, Accelerated Schedule). Differences Below About Two Points Are Within Run-to-Run Variation. Full Sweeps in the Supplementary Material.

Dataset	Mean UAR	Std UAR	Mean WAR	Std WAR
DFEW	64.07%	$\pm$ 1.80	74.54%	$\pm$ 1.54
FERV39k	42.30%	$\pm$ 0.84	52.66%	$\pm$ 0.69
MAFW	38.90%	$\pm$ 5.65	54.37%	$\pm$ 6.35

TABLE III: Cross-Fold Mean and Standard Deviation (Sample SD), DFEW/FERV39k/MAFW

Method	H	S	N	A	Sp	D	F	UAR	WAR
C3D [ 18 ]	75.1	39.4	55.1	62.4	45.0	1.3	20.5	42.7	53.5
P3D [ 19 ]	74.8	43.4	54.1	60.4	50.9	0.6	23.2	43.9	54.4
I3D [ 20 ]	78.6	44.1	56.6	55.8	45.8	2.0	20.5	43.4	54.2
R(2+1)D18 [ 21 ]	79.6	39.0	57.6	50.3	48.2	3.4	21.0	42.7	53.2
3D ResNet18 [ 22 ]	73.1	48.2	50.5	64.7	50.1	0.0	26.3	44.7	54.9
ResNet18+LSTM [ 2 ]	78.0	40.6	53.7	56.8	45.0	4.1	21.6	42.8	53.0

TABLE IV: Recall (%) of GM-GReFEL and Published Methods on DFEW. H = Happy, S = Sad, N = Neutral, A = Angry, Sp = Surprise, D = Disgust, F = Fear. GM-GReFEL is shown for its best fold (Fold 5) and for the mean over five folds, both from one evaluation of the saved best checkpoints. Bold marks the best value in each column.

Fig. 3: Normalized confusion matrix for DFEW, Fold 5 (repeated evaluation of the saved checkpoint; see text).

Method	WAR	UAR
C3D [ 18 ]	31.69	22.68
P3D [ 19 ]	33.39	23.20
I3D [ 20 ]	38.78	30.17
R(2+1)D18 [ 21 ]	41.28	31.55
3D ResNet18 [ 22 ]	37.57	26.67
R18+LSTM [ 1 ]	42.95	30.92

TABLE V: UAR and WAR (%) of GM-GReFEL and Published Methods on FERV39k

Method	WAR	UAR
Former-DFER [ 23 ]	43.27	31.16
T-MEP [ 31 ]	52.85	39.37
DFER-CLIP [ 32 ]	52.55	39.89
EmoCLIP [ 33 ]	41.46	34.24
A 3 lign-DFER [ 34 ]	53.24	42.07
SVFAP [ 35 ]	54.28	41.19

TABLE VI: UAR and WAR (%) of GM-GReFEL and Published Methods on MAFW

Method (Sampling)/Backbone	DFEW		FERV39k		MAFW
Method (Sampling)/Backbone	UAR	WAR	UAR	WAR	UAR	WAR
EC-STFL (TI)/C3D+P3D [ 2 ]	45.3	56.5	–	–	–	–
Former-DFER (DS)/Trans. [ 23 ]	53.6	65.7	37.2	46.8	31.1	43.2
CEFLNet (CLIP)/ResNet-18 [ 40 ]	51.1	65.3	–	–	–	–
NR-DFERNet (DS)/CNN-Trans. [ 26 ]	54.2	68.1	33.9	45.9	–	–
STT (DS)/ResNet-18 [ 24 ]	54.5	66.6	37.7	48.1	–	–

TABLE VII: UAR and WAR (%) on DFEW, FERV39k, and MAFW, Grouped by Method, Sampling, and Backbone. DS = Dynamic Sampling; TI = Time Interpolation; GroupMamba-T = the tiny-configuration GroupMamba backbone used throughout this paper (Section III-D ). For FERV39k, both GM-GReFEL rows give the five-split mean. Bold marks the best value in each column.

Fig. 4: Mean training and validation loss trajectory on DFEW, averaged across all five cross-validation folds with shaded variance bands.

Fig. 5: Progression of UAR and WAR on DFEW across training, averaged across five folds.

Fig. 6: Grad-CAM attention across a DFEW clip labeled “Angry,” showing attention shifting from the ocular region toward the mouth and jawline as the expression reaches its apex.

Fig. 7: Comparison of attention maps for a “Surprise” sample: full GroupMamba-SSM model (localized on mouth and eyes) vs. a mean-pooling baseline (diffuse, drifting into background).

Fig. 8: $t$ -SNE projections of the learned 768-dimensional latent feature space for DFEW, FERV39k, and MAFW.

Metric	GM-GReFEL	ViT-B/16	$\Delta$
Total Params	155.99 M	147.69 M	+8.30 M (+5.6%)
GFLOPs	539.62	548.75	$-9.13$
FPS (clips/s)	21.1	21.4	$-0.3$ ( $-1.4\%$ )
Latency (ms)	47.4	46.8	+0.6

TABLE VIII: Computational Complexity and Inference Speed Comparison

Method	Backbone	Params (M)	GFLOPs
M3DFEL [ 28 ]	ResNet-18-3D	37.5	58.0
SVFAP-B [ 35 ]	ViT-B/16	78.0	44.0
MAE-DFER [ 37 ]	ViT-B/16	85.0	50.0
VideoMAE [ 17 ]	ViT-B/16	86.0	82.0
S4D [ 39 ]	ViT-B/16 (MoAE)	90.0	–
A 3 lign-DFER [ 34 ]	CLIP-ViT-L/14	–	–

TABLE IX: Compute Comparison With Other DFER Methods. † : Measured Locally (224

\times

224,

T{=}16

, thop , RTX 3090). Other Rows Are Taken From Each Method’s Publication and May Use Different Settings; – Means Not Reported.

Yudhistira Arditya Pratama received the B.S. degree in Mechatronics Engineering from Politeknik Elektronika Negeri Surabaya (PENS), Surabaya, Indonesia, and the M.S. degree in Electrical Engineering from National Taiwan University of Science and Technology (NTUST), Taipei, Taiwan. He is currently pursuing the Ph.D. degree in Electrical Engineering at NTUST. His research interests include computer vision, dynamic facial expression recognition, spatiotemporal state-space modeling, and underwater image processing.

Table 18

Yi-Zeng Hsieh (Senior Member, IEEE) received the B.S., M.S., and Ph.D. degrees in computer science and information engineering from National Central University, Taoyuan City, Taiwan, in 2004, 2006, and 2012, respectively. He is currently a Professor with the Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan. He has authored or coauthored more than 100 journal and refereed conference papers. His research interests include deep learning, machine learning, neural networks, pattern recognition, image processing, computer vision, and swarm intelligence. He is a Senior Member of both the IEEE Consumer Technology Society and the IEEE Signal Processing Society, and was recognized as an Outstanding Young Scholar by the IEEE Consumer Technology Society in 2023.

Table 19

Facial Expression Recognition

Facial Expression Recognition studies how computers infer emotion and facial actions from images and video.

See all 79 papers

Jun 29, 2026

Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

Understanding facial expressions in videos requires modeling subtle and localized facial dynamics under unconstrained conditions. Although recent Vision Transformer (ViT)-based video models have shown strong performance through large-scale self-supervised pretraining, their attention mechanisms often emphasize dominant global motions and coarse temporal dynamics, limiting sensitivity to fine-grained facial variations. To address this limitation, we propose MiRA (Marginal-induced Attention Redistribution), a plug-in frame-marginal attention redistribution framework for ViT backbones that enhances spatio-temporal selectivity toward subtle facial dynamics without introducing additional trainable parameters. MiRA derives frame-level confidence and intra-frame concentration statistics from self-attention maps to estimate frame-wise marginal importance and redistribute attention toward spatiotemporally localized facial cues. We first introduce a principled exact mode based on post-softmax attention redistribution. To further improve efficiency, we propose flashLite mode, a lightweight pre-softmax approximation that integrates frame-marginal redistribution into FlashAttention kernels while preserving the effectiveness of the exact formulation. Experimental results on challenging Facial Expression Recognition (FER) benchmarks demonstrate consistent improvements over strong ViT baselines.

Seongro Yoon, Donghyeon Cho, Jinsun Park +1

Inria, Université Côte d’Azur, France · Hanyang University, South Korea · Pusan National University, South Korea

Vision Transformer Facial Expression Recognition Attention Mechanisms Temporal Attention

Ask PDF

Jun 26, 2026

STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

Micro-expression recognition is challenging due to subtle and short-lived facial muscle movements. Existing methods rely heavily on apex-onset frames, overlook fine-grained inter-frame dynamics, and separately model spatial and temporal information, limiting generalization across datasets. To address these challenges, we propose STAG, a dynamic ROI-AU-coupled spatial-temporal network that jointly models motion flow and adaptive facial connectivity. The framework extracts optical flow from discriminative frames using magnitude-based selection and temporal attention. A dual-branch architecture combines an enhanced graph attention network for structured spatial reasoning with a transformer encoder for temporal modeling. A bidirectional cross-attention module enables mutual refinement of spatial and temporal features, while AU-guided dynamic connectivity adapts facial region interactions according to muscle activation patterns. The transformer captures subtle temporal dynamics beyond apex-based approaches, improving semantic consistency and interpretability for explainable micro-expression recognition. The fused representation is optimized using focal loss and evaluated on CASME II, 4DME, DFME, NaME, SAMM, and SMIC-HS. Extensive experiments demonstrate improved robustness, generalization, interpretability, and computational efficiency, confirming the effectiveness of adaptive relational reasoning, AU-guided dynamic connectivity, and deep spatial-temporal feature fusion for accurate cross-dataset micro-expression recognition.

Nandani Sharma, Varun Sharma, Dinesh Singh

Vision Intelligence and Machine Learning (VIML) Group, School of Computing and Electrical Engineering, Indian Institute of Technology Mandi, Mandi, 175005, Himachal Pradesh, India · Indian Institute of Information Technology Bhagalpur, Sabour Rd, Bhagalpur, 813210, Bihar, India

Facial Expression Recognition Micro-Expression Recognition Spatiotemporal GNNs

Ask PDF

Apr 10, 2025

End-to-End Facial Expression Detection in Long Videos

Facial expression detection requires spotting when expressions occur and recognizing which emotional category they belong to. Despite their close relationships, existing approaches typically address these tasks separately, limiting performance and robustness in real-world settings. In this work, we propose FEDN, a Facial Expression Detection Network, which unifies spotting and recognition into a single detection task performed fully end-to-end. FEDN introduces two temporal attention modules, segment-level attention to capture fine-grained local dynamics and sliding window attention to capture the broader temporal context. Their output is combined in a multi-scale temporal feature pyramid, which enables spotting of expressions with varying duration. This unified framework enables joint optimization and shared representation learning across tasks. FEDN outperforms strong baselines in both spotting and detection on three public benchmarks, demonstrating the effectiveness of unifying spotting and recognition across multiple temporal scales. Additionally, we uncover a previously unreported discrepancy between expert-annotated and self-reported emotion labels, highlighting a key challenge in expression benchmarking and motivating the development of more nuanced annotation protocols.

Yini Fang, Alec F. Diallo, Frederic Jumelle +1

Hong Kong University of Science and Technology · University of Edinburgh · Ydentity Organization

Facial Expression Recognition

Ask PDF

Abstract

Facial Expression Recognition

Explore similar work

Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

End-to-End Facial Expression Detection in Long Videos