Variable interaction distances continuously degrade facial resolution in edge-based affective interaction, making low-resolution facial expression recognition (FER) difficult under limited computation. Full-face image or feature restoration incurs substantial overhead, whereas lightweight recognition pipelines may miss local cues that remain discriminative after degradation. We propose STAR-FER, a scale-and-task-adaptive selective restoration framework that balances recognition performance and resource constraints. A lightweight image-scale conditioned Gate jointly predicts a pixel-wise spatial ranking and a continuous restoration budget, then applies exact-support projection to obtain an input-dependent free-form mask. The mask-conditioned restorer retains complete facial context while concentrating sparse restoration and residual insertion on the selected support. Across seven degradation scales from x2 to x8 on the complete RAF-DB test set, STAR-FER achieves 80.92% mean accuracy and 71.64% Macro-F1, improving the low-resolution ResNet-18 baseline by 2.25 and 2.71 percentage points, respectively. It surpasses SR-FER by 1.04 accuracy points while requiring 8.84 GFLOPs, a 54.29% reduction, with 28.01% mean mask coverage, and maintains effective recognition at unseen non-integer scales. These results show that spatial-ranking and continuous-budget driven selective restoration reduces spatial modification and computation while maintaining low-resolution FER performance, offering an efficient path to resource-constrained affective interaction.
Figures & tables
Fig. 1: Selective restoration and region decisions in STAR-FER: (a) full-face and selective restoration, (b) Gate decision features and predicted supports for two parent images at the same scale, and (c) predicted supports for one parent image under different degradation scales.
Fig. 2: Overview of STAR-FER. The lightweight continuous-scale Gate extracts visual information from a frozen SMSR prefix and jointly predicts a spatial ranking field and an image–scale conditioned restoration budget. Exact Top- k projection converts them into a free-form mask. Mask-conditioned SMSR performs sparse restoration on the selected support, and the locally blended image is passed to a ResNet-18 classifier. Network features and sparse activations are schematic. The reported 28.01% and 8.84 GFLOPs denote mean coverage across ×2 – ×8 and forward computation under the protocol used in this work, respectively.
Fig. 3: Scale-conditioned Gate. The frozen prefix and source spatial branch provide base features and a source ranking field. The lightweight visual–scale encoder predicts a spatial correction, the continuous budget head estimates restoration area, and exact Top- k generates the final mask.
Fig. 4: Recognition and base support across seven RAF-DB scales. (a) Accuracy; (b) Macro-F1; (c) Mask coverage.
Fig. 5: t-SNE visualization of RAF-DB features at ×8 .
Method
Accuracy (%)
Macro-F1 (%)
ResNet-18 (LR)
78.66
68.94
STAR-FER
80.92
71.64
SR-FER
79.88
70.76
FSR-FER
77.01
68.09
TABLE I: Mean recognition results across RAF-DB scales.
STAR-FER
ResNet-18 (LR)
Scale
Acc (%)
F1 (%)
Coverage (%)
Acc (%)
F1 (%)
×2
85.71
75.67
17.23
85.68
75.72
×3
85.17
74.92
23.20
85.14
74.89
×4
84.34
74.45
28.87
84.02
74.02
×5
83.07
72.73
33.64
82.53
72.65
×6
82.68
72.25
34.77
80.80
68.62
TABLE II: Recognition performance and Gate coverage on FERPlus. F1 denotes Macro-F1.
Fig. 6: Illustration of fixed eye-and-mouth support and adaptive selection. (a) Parent face. (b) Adaptive mask for a ×4 input with its corresponding scale condition. (c) Fixed eye-and-mouth mask on the same ×4 input. (d) Adaptive mask with the input kept at ×4 and the scale code fixed to ×2 .
Input scale
Accuracy (%)
Coverage (%)
Normal
Fixed ×2
Normal
Fixed ×2
×2
84.53
84.53
18.89
18.89
×3
83.04
83.94
23.76
18.93
×4
81.13
80.87
27.27
19.09
×5
80.70
80.61
29.70
19.33
×6
80.08
79.53
31.14
19.34
TABLE III: Normal versus constant ×2 scale encoding. The actual input degradation is unchanged.
Scale
STAR-FER Acc (%)
Fixed Acc (%)
Δ Acc (pp)
×2
84.53
84.63
−0.10
×3
83.04
83.84
−0.80
×4
81.13
81.02
+0.11
×5
80.70
80.44
+0.26
×6
80.08
79.37
+0.71
×7
78.88
77.77
+1.11
TABLE IV: RAF-DB accuracy of the adaptive Gate and fixed eye-and-mouth support across seven scales. Δ Acc denotes STAR-FER minus fixed accuracy.
Execution path
Cost (GFLOPs)
Reduction (%)
Dense masked
9.5646
—
True SMSR sparse
8.8380
7.60
TABLE V: Execution-path comparison under identical weights and masks. Computation is averaged over seven scales for the complete pipeline.
Method
Parameters (M)
Coverage (%)
Cost (GFLOPs)
STAR-FER
12.885
28.01
8.84
SR-FER
14.21
100.00
19.34
FSR-FER
35.42
100.00
19.41
TABLE VI: Model parameters, mean restoration coverage, and computational cost.
Scale
Acc (%)
F1 (%)
Coverage (%)
×4.55
80.35
71.50
28.60
×6.25
79.47
70.27
31.57
×7.14
78.10
69.31
32.52
TABLE VII: RAF-DB recognition and Gate coverage at unseen non-integer scales.
Fig. 7: Restoration masks predicted by the current continuous-scale Gate for the same parent image across degradation scales. Orange, green, and blue denote the three largest connected components in descending area; values below each image give union-mask coverage.
Fig. 8: Selective restoration and scale response in the PC deployment experiment. (a) Input with the predicted mask. (b) Locally fused result X and channel-mean ∣X−L∣ to its right. (c) Effective degradation scale s . (d) Mask coverage. Curves show valid observations from 5 to 33 s; dots mark 22.97 s, with s=3.33 and 24.02% coverage.
Fig. 9: Overall PC deployment statistics. (a) Prediction switching rates over 913 adjacent valid-frame pairs. (b) Mean method latency over 914 detected faces.
Method
Mean (ms)
P95 (ms)
Peak CUDA memory (MB)
ResNet-18 (LR)
15.37
23.58
92.40
SR-FER
132.06
141.51
727.54
FSR-FER
146.21
149.54
202.75
STAR-FER (Dense)
85.46
88.93
115.29
STAR-FER (Sparse)
122.87
127.53
115.26
TABLE IX: Single-frame inference on Jetson Orin Nano Super with batch size 1. Memory denotes peak CUDA tensor allocation measured by PyTorch.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Stage
Updated components
Main supervision
G/F calibration I
Continuous restorer parameters
Eq. ( 12 )
G/F calibration II
Restorer and FER layer3/4/FC
Eq. ( 12 )
Spatial Gate learning
Lightweight visual, spatial, and auxiliary branches
Fine-grained facial expression recognition (FER) hinges on capturing subtle muscular cues that distinguish adjacent emotions. Yet capturing these cues presents a dilemma. Detector-based methods depend on fragile landmark pipelines, whereas we find that directly transferring foundation models such as DINOv3 under conventional global readouts can cause local evidence dilution: early global aggregation washes out sparse muscular signals and leaves persistent confusion between categories such as fear/surprise and sad/neutral. To recover this evidence, we propose SRE-FER, a readout-level regional residual evidence learning framework. Its core module, RERA, adds zero-initialized residual logits that refine class boundaries while preserving the backbone's global prediction. Training-time action unit (AU) guidance steers regional features toward expression-relevant areas using Facial Action Coding System (FACS)-based anatomical priors, without requiring an external facial pipeline at inference. An optional Full setting further routes sample-specific non-redundant tokens. On three benchmarks, SRE-FER attains 92.76% on RAF-DB, 91.32% on FERPlus, and 67.78% on AffectNet-7, demonstrating highly competitive performance compared to existing FER methods.
Automatic facial expression recognition (FER) has gained much attention due to its applications in human-computer interaction. Among the approaches to improve FER tasks, this paper focuses on deep architecture with the attention mechanism. We propose a novel Masking idea to boost the performance of CNN in facial expression task. It uses a segmentation network to refine feature maps, enabling the network to focus on relevant information to make correct decisions. In experiments, we combine the ubiquitous Deep Residual Network and Unet-like architecture to produce a Residual Masking Network. The proposed method holds state-of-the-art (SOTA) accuracy on the well-known FER2013 and private VEMO datasets. The source code is available at https://github.com/phamquiluan/ResidualMaskingNetwork.
Luan Pham, The Huynh Vu, Tuan Anh Tran
Research Department-Cinnamon AI, Viet Nam · Faculty of Computer Science & Engineering, Ho Chi Minh City-University of Technology (HCMUT), Viet Nam
Facial Emotion Recognition (FER) is an important task that has significant implications across various fields such as biometrics, health, and human-computer interaction. Current Vision Transformer-based approaches display quadratic complexity O(N2), with N being the input sequence length, making them cumbersome to deploy at the edge. In this paper, we hypothesize that the FER task does not necessarily require all facial information to correctly interpret emotional states, as specific regions such as the eyes, the mouth, and parts of the cheeks carry discriminative information that can be sufficient to recognize emotions. Based on this, we propose Sparse Attention to Emotion (SAE), a model that discards image tokens that have no added value to the emotional context, while preserving good accuracy and achieving a significant gain in computational cost. Surprisingly, even after suppressing 90% of the image tokens, our model achieves competitive accuracy to state of the art methods at much lower cost, providing a lightweight Facial Emotion Recognition approach. Experimental results demonstrate that SAE achieves new state of the art results on the RAF-DB dataset while reducing the computational complexity by up to 90%.
Aya Manel Zitouni, Aicha Zenakhri, Karim Haroun +1