GM-GReFEL: Geometry-Guided Spatiotemporal State-Space Learning for Robust Dynamic Facial Expression Recognition

Yudhistira Arditya Pratama; Yi-Zeng Hsieh

cs.CVOct 22, 2026

GM-GReFEL: Geometry-Guided Spatiotemporal State-Space Learning for Robust Dynamic Facial Expression Recognition

Authors: Yudhistira Arditya Pratama, Yi-Zeng Hsieh

Organizations: Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei 106335, Taiwan

Abstract

Recognizing facial expressions in unconstrained video suffers from two coupled problems: self-attention grows quadratically in cost with clip length, and crowdsourced labels are noisy. We present GM-GReFEL, a spatiotemporal state-space network with a geometry-aware decision stage. Its Modulated GroupMamba backbone replaces a spatiotemporal Vision Transformer. Six channel groups are scanned separately, four over the frame plane and two over time, so the cost grows linearly with sequence length ( $\mathcal{O}(N)$ ); after each layer, a Channel Affinity Modulation gate mixes the groups again. On top of the backbone, a GReFEL reliability module turns the normalized entropy of each prediction into a mixing weight between the classifier output and a vote from trainable geometric anchors, so that facial geometry counts for more when the network is unsure. The model is initialized by static-to-dynamic weight transfer and trained with a Triple Loss. In five-fold cross-validation, the best fold scores 66.57%/76.46% UAR/WAR on DFEW, 0.3 points or less below the strongest compared method, and 45.66%/62.27% on MAFW, ahead of all compared methods; on FERV39k the mean over five splits, 42.30%/52.66%, is competitive. The five-fold means are 64.07%/74.54% on DFEW, on par with recent large pre-trained models, and 38.90%/54.37% on MAFW, below the best of them. A paired $t$ -test on DFEW indicates a modest benefit of temporal modeling (+1.61 UAR points, one-sided $p=0.031$ ). Attention stays on plausible facial regions, and latency is close to a ViT-B/16 baseline (47.4 vs.\ 46.8 ms per clip). Code: https://github.com/ardiytama/GroupMamba-DFER.

Figures & tables

Fig. 1: Structure of the GM-GReFEL pipeline. Top row: the four-stage forward pass, (a) 3D spatiotemporal tubelet tokenization with positional encoding, (b) sequence modeling via the Modulated GroupMamba engine with Channel Affinity Modulation (CAM), (c) geometry modeling via the Temporal Geometry-Aware module that combines the primary MLP’s confidence score with the geometric anchor bank, and (d) Triple Loss optimization, together with example output visualizations (facial landmarks, Grad-CAM attention). Bottom row: internal detail of each stage. Bottom left: the Modulated GroupMamba layer, in which the Spatial Group and Temporal Group each pass through a stack of VSSS blocks before Channel Affinity and Channel Modulation gating produce the $B\times 1\times C$ layer output $Z^{(out)}$ . Bottom center: the Geometry-Aware Reliability mechanism, showing the Statistical Track (Process, Softmax, Normalized Entropy) computed in parallel with the Geometric Track (a similarity lookup against the geometric anchor database via Softmax), combined before Loss Optimization. Bottom right: the six VSSS scanning directions — four boustrophedon spatial-group sweeps (Direction 1–4) and two temporal-group directions (Forward, Backward).

Fig. 2: VSSS block: a Mamba-style selective scan core and a subsequent feed-forward layer.

Configuration	WAR (%)	UAR (%)
GM-GReFEL (full)	73.46 $\pm$ 1.21	64.69 $\pm$ 1.68
w/o Temporal Modeling	73.59 $\pm$ 1.28	63.08 $\pm$ 1.70
w/o Trainable Anchors (frozen)	74.69 $\pm$ 1.19	64.95 $\pm$ 2.20
GroupMamba-SSM depth $L=1$	74.58 $\pm$ 1.49	64.94 $\pm$ 1.53
w/ Single-Stage Training	74.80 $\pm$ 1.37	64.40 $\pm$ 2.00
w/o Jittered Sampling	74.73 $\pm$ 1.52	64.68 $\pm$ 1.90

TABLE I: Component Ablation on DFEW, Mean

\pm

SD Over Five Folds (Population SD,

n=5

). Each Fold Is Taken at Its Epoch of Highest UAR, With WAR at That Epoch. The Full Model Uses

L=3

Hyperparameter	Values tested	Best	UAR / WAR (%)
Dropout $p$	0, 0.25, 0.5, 0.75	0.75	64.10 / 74.93
Layers $L$	1, 3, 5	1	64.19 / 75.69
$\lambda_{2}$ (center)	0, 0.1, 1, 10	10	64.02 / 75.95
$\lambda_{1}$ (orth.)	0, 0.1, 1, 10	10	64.08 / 75.69
Temperature $\tau$	0.01, 0.05, 0.1, 0.5	0.10	65.20 / 75.22

TABLE II: Hyperparameter Sensitivity Summary (DFEW Fold 1, Accelerated Schedule). Differences Below About Two Points Are Within Run-to-Run Variation. Full Sweeps in the Supplementary Material.

Dataset	Mean UAR	Std UAR	Mean WAR	Std WAR
DFEW	64.07%	$\pm$ 1.80	74.54%	$\pm$ 1.54
FERV39k	42.30%	$\pm$ 0.84	52.66%	$\pm$ 0.69
MAFW	38.90%	$\pm$ 5.65	54.37%	$\pm$ 6.35

TABLE III: Cross-Fold Mean and Standard Deviation (Sample SD), DFEW/FERV39k/MAFW

Method	H	S	N	A	Sp	D	F	UAR	WAR
C3D [ 18 ]	75.1	39.4	55.1	62.4	45.0	1.3	20.5	42.7	53.5
P3D [ 19 ]	74.8	43.4	54.1	60.4	50.9	0.6	23.2	43.9	54.4
I3D [ 20 ]	78.6	44.1	56.6	55.8	45.8	2.0	20.5	43.4	54.2
R(2+1)D18 [ 21 ]	79.6	39.0	57.6	50.3	48.2	3.4	21.0	42.7	53.2
3D ResNet18 [ 22 ]	73.1	48.2	50.5	64.7	50.1	0.0	26.3	44.7	54.9
ResNet18+LSTM [ 2 ]	78.0	40.6	53.7	56.8	45.0	4.1	21.6	42.8	53.0

TABLE IV: Recall (%) of GM-GReFEL and Published Methods on DFEW. H = Happy, S = Sad, N = Neutral, A = Angry, Sp = Surprise, D = Disgust, F = Fear. GM-GReFEL is shown for its best fold (Fold 5) and for the mean over five folds, both from one evaluation of the saved best checkpoints. Bold marks the best value in each column.

Fig. 3: Normalized confusion matrix for DFEW, Fold 5 (repeated evaluation of the saved checkpoint; see text).

Method	WAR	UAR
C3D [ 18 ]	31.69	22.68
P3D [ 19 ]	33.39	23.20
I3D [ 20 ]	38.78	30.17
R(2+1)D18 [ 21 ]	41.28	31.55
3D ResNet18 [ 22 ]	37.57	26.67
R18+LSTM [ 1 ]	42.95	30.92

TABLE V: UAR and WAR (%) of GM-GReFEL and Published Methods on FERV39k

Method	WAR	UAR
Former-DFER [ 23 ]	43.27	31.16
T-MEP [ 31 ]	52.85	39.37
DFER-CLIP [ 32 ]	52.55	39.89
EmoCLIP [ 33 ]	41.46	34.24
A 3 lign-DFER [ 34 ]	53.24	42.07
SVFAP [ 35 ]	54.28	41.19

TABLE VI: UAR and WAR (%) of GM-GReFEL and Published Methods on MAFW

Method (Sampling)/Backbone	DFEW		FERV39k		MAFW
Method (Sampling)/Backbone	UAR	WAR	UAR	WAR	UAR	WAR
EC-STFL (TI)/C3D+P3D [ 2 ]	45.3	56.5	–	–	–	–
Former-DFER (DS)/Trans. [ 23 ]	53.6	65.7	37.2	46.8	31.1	43.2
CEFLNet (CLIP)/ResNet-18 [ 40 ]	51.1	65.3	–	–	–	–
NR-DFERNet (DS)/CNN-Trans. [ 26 ]	54.2	68.1	33.9	45.9	–	–
STT (DS)/ResNet-18 [ 24 ]	54.5	66.6	37.7	48.1	–	–

TABLE VII: UAR and WAR (%) on DFEW, FERV39k, and MAFW, Grouped by Method, Sampling, and Backbone. DS = Dynamic Sampling; TI = Time Interpolation; GroupMamba-T = the tiny-configuration GroupMamba backbone used throughout this paper (Section III-D ). For FERV39k, both GM-GReFEL rows give the five-split mean. Bold marks the best value in each column.

Fig. 4: Mean training and validation loss trajectory on DFEW, averaged across all five cross-validation folds with shaded variance bands.

Fig. 5: Progression of UAR and WAR on DFEW across training, averaged across five folds.

Fig. 6: Grad-CAM attention across a DFEW clip labeled “Angry,” showing attention shifting from the ocular region toward the mouth and jawline as the expression reaches its apex.

Fig. 7: Comparison of attention maps for a “Surprise” sample: full GroupMamba-SSM model (localized on mouth and eyes) vs. a mean-pooling baseline (diffuse, drifting into background).

Fig. 8: $t$ -SNE projections of the learned 768-dimensional latent feature space for DFEW, FERV39k, and MAFW.

Metric	GM-GReFEL	ViT-B/16	$\Delta$
Total Params	155.99 M	147.69 M	+8.30 M (+5.6%)
GFLOPs	539.62	548.75	$-9.13$
FPS (clips/s)	21.1	21.4	$-0.3$ ( $-1.4\%$ )
Latency (ms)	47.4	46.8	+0.6

TABLE VIII: Computational Complexity and Inference Speed Comparison

Method	Backbone	Params (M)	GFLOPs
M3DFEL [ 28 ]	ResNet-18-3D	37.5	58.0
SVFAP-B [ 35 ]	ViT-B/16	78.0	44.0
MAE-DFER [ 37 ]	ViT-B/16	85.0	50.0
VideoMAE [ 17 ]	ViT-B/16	86.0	82.0
S4D [ 39 ]	ViT-B/16 (MoAE)	90.0	–
A 3 lign-DFER [ 34 ]	CLIP-ViT-L/14	–	–

TABLE IX: Compute Comparison With Other DFER Methods. † : Measured Locally (224

\times

224,

T{=}16

, thop , RTX 3090). Other Rows Are Taken From Each Method’s Publication and May Use Different Settings; – Means Not Reported.

Yudhistira Arditya Pratama received the B.S. degree in Mechatronics Engineering from Politeknik Elektronika Negeri Surabaya (PENS), Surabaya, Indonesia, and the M.S. degree in Electrical Engineering from National Taiwan University of Science and Technology (NTUST), Taipei, Taiwan. He is currently pursuing the Ph.D. degree in Electrical Engineering at NTUST. His research interests include computer vision, dynamic facial expression recognition, spatiotemporal state-space modeling, and underwater image processing.

Table 18

Yi-Zeng Hsieh (Senior Member, IEEE) received the B.S., M.S., and Ph.D. degrees in computer science and information engineering from National Central University, Taoyuan City, Taiwan, in 2004, 2006, and 2012, respectively. He is currently a Professor with the Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan. He has authored or coauthored more than 100 journal and refereed conference papers. His research interests include deep learning, machine learning, neural networks, pattern recognition, image processing, computer vision, and swarm intelligence. He is a Senior Member of both the IEEE Consumer Technology Society and the IEEE Signal Processing Society, and was recognized as an Outstanding Young Scholar by the IEEE Consumer Technology Society in 2023.

Table 19

Facial Expression Recognition

Facial Expression Recognition studies how computers infer emotion and facial actions from images and video.

See all 79 papers

Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding
Jun 29, 2026Seongro Yoon, Donghyeon Cho, Jinsun Park +1Vision Transformer Facial Expression Recognition
Inria, Université Côte d’Azur, France · Hanyang University, South Korea · Pusan National University, South Korea
STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition
Jun 26, 2026Nandani Sharma, Varun Sharma, Dinesh SinghFacial Expression Recognition Micro-Expression Recognition
Vision Intelligence and Machine Learning (VIML) Group, School of Computing and Electrical Engineering, Indian Institute of Technology Mandi, Mandi, 175005, Himachal Pradesh, India · Indian Institute of Information Technology Bhagalpur, Sabour Rd, Bhagalpur, 813210, Bihar, India
End-to-End Facial Expression Detection in Long Videos
Apr 10, 2025Yini Fang, Alec F. Diallo, Frederic Jumelle +1Facial Expression Recognition
Hong Kong University of Science and Technology · University of Edinburgh · Ydentity Organization

Jun 29, 2026J/K move · Enter open · S save

Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

Seongro Yoon, Donghyeon Cho, Jinsun Park, François Brémond

Inria, Université Côte d’Azur, France · Hanyang University, South Korea · Pusan National University, South Korea

Vision Transformer Facial Expression Recognition Attention Mechanisms Temporal Attention

Open paper AskPDF arXiv

Understanding facial expressions in videos requires modeling subtle and localized facial dynamics under unconstrained conditions. Although recent Vision Transformer (ViT)-based video models have shown strong performance through large-scale self-supervised pretraining, their attention mechanisms often emphasize dominant global motions and coarse temporal dynamics, limiting sensitivity to fine-grained facial variations. To address this limitation, we propose MiRA (Marginal-induced Attention Redistribution), a plug-in frame-marginal attention redistribution framework for ViT backbones that enhances spatio-temporal selectivity toward subtle facial dynamics without introducing additional trainable parameters. MiRA derives frame-level confidence and intra-frame concentration statistics from self-attention maps to estimate frame-wise marginal importance and redistribute attention toward spatiotemporally localized facial cues. We first introduce a principled exact mode based on post-softmax attention redistribution. To further improve efficiency, we propose flashLite mode, a lightweight pre-softmax approximation that integrates frame-marginal redistribution into FlashAttention kernels while preserving the effectiveness of the exact formulation. Experimental results on challenging Facial Expression Recognition (FER) benchmarks demonstrate consistent improvements over strong ViT baselines.

Abstract

Facial Expression Recognition

Explore similar work

Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding

STAG: Spatio-temporal Evolving Structural Representation of Action Units for Micro-expression Recognition

End-to-End Facial Expression Detection in Long Videos