Variable interaction distances continuously degrade facial resolution in edge-based affective interaction, making low-resolution facial expression recognition (FER) difficult under limited computation. Full-face image or feature restoration incurs substantial overhead, whereas lightweight recognition pipelines may miss local cues that remain discriminative after degradation. We propose STAR-FER, a scale-and-task-adaptive selective restoration framework that balances recognition performance and resource constraints. A lightweight image-scale conditioned Gate jointly predicts a pixel-wise spatial ranking and a continuous restoration budget, then applies exact-support projection to obtain an input-dependent free-form mask. The mask-conditioned restorer retains complete facial context while concentrating sparse restoration and residual insertion on the selected support. Across seven degradation scales from x2 to x8 on the complete RAF-DB test set, STAR-FER achieves 80.92% mean accuracy and 71.64% Macro-F1, improving the low-resolution ResNet-18 baseline by 2.25 and 2.71 percentage points, respectively. It surpasses SR-FER by 1.04 accuracy points while requiring 8.84 GFLOPs, a 54.29% reduction, with 28.01% mean mask coverage, and maintains effective recognition at unseen non-integer scales. These results show that spatial-ranking and continuous-budget driven selective restoration reduces spatial modification and computation while maintaining low-resolution FER performance, offering an efficient path to resource-constrained affective interaction.
Figures & tables
Fig. 1: Selective restoration and region decisions in STAR-FER: (a) full-face and selective restoration, (b) Gate decision features and predicted supports for two parent images at the same scale, and (c) predicted supports for one parent image under different degradation scales.
Fig. 2: Overview of STAR-FER. The lightweight continuous-scale Gate extracts visual information from a frozen SMSR prefix and jointly predicts a spatial ranking field and an image–scale conditioned restoration budget. Exact Top- k projection converts them into a free-form mask. Mask-conditioned SMSR performs sparse restoration on the selected support, and the locally blended image is passed to a ResNet-18 classifier. Network features and sparse activations are schematic. The reported 28.01% and 8.84 GFLOPs denote mean coverage across ×2 – ×8 and forward computation under the protocol used in this work, respectively.
Fig. 3: Scale-conditioned Gate. The frozen prefix and source spatial branch provide base features and a source ranking field. The lightweight visual–scale encoder predicts a spatial correction, the continuous budget head estimates restoration area, and exact Top- k generates the final mask.
Fig. 4: Recognition and base support across seven RAF-DB scales. (a) Accuracy; (b) Macro-F1; (c) Mask coverage.
Fig. 5: t-SNE visualization of RAF-DB features at ×8 .
Method
Accuracy (%)
Macro-F1 (%)
ResNet-18 (LR)
78.66
68.94
STAR-FER
80.92
71.64
SR-FER
79.88
70.76
FSR-FER
77.01
68.09
TABLE I: Mean recognition results across RAF-DB scales.
STAR-FER
ResNet-18 (LR)
Scale
Acc (%)
F1 (%)
Coverage (%)
Acc (%)
F1 (%)
×2
85.71
75.67
17.23
85.68
75.72
×3
85.17
74.92
23.20
85.14
74.89
×4
84.34
74.45
28.87
84.02
74.02
×5
83.07
72.73
33.64
82.53
72.65
×6
82.68
72.25
34.77
80.80
68.62
TABLE II: Recognition performance and Gate coverage on FERPlus. F1 denotes Macro-F1.
Fig. 6: Illustration of fixed eye-and-mouth support and adaptive selection. (a) Parent face. (b) Adaptive mask for a ×4 input with its corresponding scale condition. (c) Fixed eye-and-mouth mask on the same ×4 input. (d) Adaptive mask with the input kept at ×4 and the scale code fixed to ×2 .
Input scale
Accuracy (%)
Coverage (%)
Normal
Fixed ×2
Normal
Fixed ×2
×2
84.53
84.53
18.89
18.89
×3
83.04
83.94
23.76
18.93
×4
81.13
80.87
27.27
19.09
×5
80.70
80.61
29.70
19.33
×6
80.08
79.53
31.14
19.34
TABLE III: Normal versus constant ×2 scale encoding. The actual input degradation is unchanged.
Scale
STAR-FER Acc (%)
Fixed Acc (%)
Δ Acc (pp)
×2
84.53
84.63
−0.10
×3
83.04
83.84
−0.80
×4
81.13
81.02
+0.11
×5
80.70
80.44
+0.26
×6
80.08
79.37
+0.71
×7
78.88
77.77
+1.11
TABLE IV: RAF-DB accuracy of the adaptive Gate and fixed eye-and-mouth support across seven scales. Δ Acc denotes STAR-FER minus fixed accuracy.
Execution path
Cost (GFLOPs)
Reduction (%)
Dense masked
9.5646
—
True SMSR sparse
8.8380
7.60
TABLE V: Execution-path comparison under identical weights and masks. Computation is averaged over seven scales for the complete pipeline.
Method
Parameters (M)
Coverage (%)
Cost (GFLOPs)
STAR-FER
12.885
28.01
8.84
SR-FER
14.21
100.00
19.34
FSR-FER
35.42
100.00
19.41
TABLE VI: Model parameters, mean restoration coverage, and computational cost.
Scale
Acc (%)
F1 (%)
Coverage (%)
×4.55
80.35
71.50
28.60
×6.25
79.47
70.27
31.57
×7.14
78.10
69.31
32.52
TABLE VII: RAF-DB recognition and Gate coverage at unseen non-integer scales.
Fig. 7: Restoration masks predicted by the current continuous-scale Gate for the same parent image across degradation scales. Orange, green, and blue denote the three largest connected components in descending area; values below each image give union-mask coverage.
Fig. 8: Selective restoration and scale response in the PC deployment experiment. (a) Input with the predicted mask. (b) Locally fused result X and channel-mean ∣X−L∣ to its right. (c) Effective degradation scale s . (d) Mask coverage. Curves show valid observations from 5 to 33 s; dots mark 22.97 s, with s=3.33 and 24.02% coverage.
Fig. 9: Overall PC deployment statistics. (a) Prediction switching rates over 913 adjacent valid-frame pairs. (b) Mean method latency over 914 detected faces.
Method
Mean (ms)
P95 (ms)
Peak CUDA memory (MB)
ResNet-18 (LR)
15.37
23.58
92.40
SR-FER
132.06
141.51
727.54
FSR-FER
146.21
149.54
202.75
STAR-FER (Dense)
85.46
88.93
115.29
STAR-FER (Sparse)
122.87
127.53
115.26
TABLE IX: Single-frame inference on Jetson Orin Nano Super with batch size 1. Memory denotes peak CUDA tensor allocation measured by PyTorch.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Stage
Updated components
Main supervision
G/F calibration I
Continuous restorer parameters
Eq. ( 12 )
G/F calibration II
Restorer and FER layer3/4/FC
Eq. ( 12 )
Spatial Gate learning
Lightweight visual, spatial, and auxiliary branches