STAG-Net: Structure-Aware Attention over Landmark Groups for Lightweight Head Pose Estimation
Organizations: Faculty of Engineering, Sri Lanka Institute of Information Technology, Sri Lanka · Faculty of Engineering, University of Sri Jayewardenepura, Sri Lanka · University of Colombo School of Computing, Colombo 07, Sri Lanka · Faculty of Computing, Sri Lanka Institute of Information Technology, Sri Lanka
Abstract
Head pose estimation is important for human computer interaction and driver monitoring, but many image based methods remain too costly for resource constrained deployment. We propose STAG-Net, a structure-aware attention over landmark groups for lightweight head Pose estimation that estimates head pose from facial landmarks using movement aware selection, spatial clustering, compact DeepSet encoders, self attention, and a global geometry descriptor. We develop variants using FaceMesh and Face Alignment Network landmarks and evaluate them under cross dataset and subject independent protocols on 300W-LP, AFLW2000, and BIWI. STAG-Net achieves the lowest mean absolute error among all compared landmark based methods in every evaluated setting. On BIWI frames with valid detections, the FAN variant also records the lowest overall error among all compared methods, reaching under cross dataset evaluation and under subject independent evaluation. The regressors use only 0.012 to 0.043 million parameters and 24.1 to 83.6 thousand FLOPs, about three orders of magnitude fewer FLOPs than FSA-Net. Despite including landmark extraction and regression, the FaceMesh pipeline reaches 322.9 frames per second on a Tesla T4 GPU, nearly twice the 164.5 frames per second of the fastest measured image based baseline, and 111.0 frames per second on CPU. After normalization by dataset size, STAG-Net processes training samples seven to eight times faster than FSA-Net and more than one hundred times faster than HopeNet and TokenHPE, while complete training finishes within 28 minutes. These results show that STAG-Net combines leading landmark based accuracy with substantially lower training and inference cost.
Figures & tables
| Dataset | Yaw ( ∘ ) | Pitch ( ∘ ) | Roll ( ∘ ) |
|---|---|---|---|
| BIWI | to | to | to |
| 300W-LP | to | to | to |
| AFLW2000 | to | to | to |
| Parameter | Value |
|---|---|
| Framework | PyTorch |
| Optimizer | Adam |
| Loss function | Mean absolute error (MAE) |
| Batch size | 128 |
| Training epochs | 150 |
| Initial learning rate |
| Method | Params (M) | BIWI | AFLW2000 | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Yaw | Pitch | Roll | MAE | Yaw | Pitch | Roll | MAE | ||
| RGB image based methods | |||||||||
| KEPLER (2017) [ 47 ] | N/R | 8.80 | 17.30 | 16.20 | 13.90 | N/R | N/R | N/R | N/R |
| 3DDFA (2019) [ 48 ] | N/R | 36.20 | 12.30 | 8.78 | 19.10 | 5.40 | 8.53 | 8.25 | 7.39 |
| TokenHPE (2023) [ 19 ] | 86.42 | 4.51 | 3.95 | 2.71 | 3.72 | 5.54 | 3.36 | 4.08 | 4.66 |
| 6DRepNet (2022) [ 17 ] | 39.32 | 3.23 | 5.32 | 2.78 | 3.78 | 3.27 | 4.58 | 2.98 | 3.61 |
| Method | Params (M) | BIWI | |||
|---|---|---|---|---|---|
| Yaw | Pitch | Roll | MAE | ||
| RGB image based methods | |||||
| Martin (2014) [ 55 ] | N/R | 3.60 | 2.50 | 2.60 | 2.90 |
| VGG16-RNN (2017) [ 56 ] | 138.00 | 3.14 | 3.48 | 2.60 | 3.07 |
| VGG16(2017) [ 56 ] | 138.00 | 3.91 | 4.03 | 3.03 | 3.66 |
| TokenHPE (2023) [ 19 ] | 86.42 | 3.01 | 2.28 | 2.01 | 2.49 |
| Dataset | Angle | Full range | FaceMesh | FAN |
|---|---|---|---|---|
| BIWI | Yaw | |||
| Pitch | ||||
| Roll | ||||
| 300W-LP | Yaw | |||
| Pitch | ||||
| Roll |
| Neighbors | Clusters | Yaw | Pitch | Roll | MAE |
|---|---|---|---|---|---|
| 2 | 5 | 2.517 | 3.218 | 2.367 | 2.701 |
| 4 | 5 | 2.440 | 3.235 | 2.296 | 2.657 |
| 6 | 5 | 2.373 | 3.091 | 2.239 | 2.568 |
| Retained | Yaw | Pitch | Roll | MAE |
|---|---|---|---|---|
| 25% | 2.408 | 3.162 | 2.184 | 2.584 |
| 20% | 2.334 | 3.242 | 2.276 | 2.617 |
| 15% | 2.373 | 3.091 | 2.239 | 2.568 |
| 10% | 2.428 | 3.180 | 2.329 | 2.646 |
| 5% | 2.572 | 2.930 | 2.421 | 2.641 |
| Setting | Yaw | Pitch | Roll | MAE | |
|---|---|---|---|---|---|
| Dropout | 0 | 2.60 | 3.05 | 2.31 | 2.65 |
| 0.01 † | 2.37 | 3.09 | 2.24 | 2.57 | |
| 0.05 | 2.49 | 3.26 | 2.42 | 2.72 | |
| Heads | 2 † | 2.37 | 3.09 | 2.24 | 2.57 |
| 4 | 2.44 | 3.26 | 2.29 | 2.67 | |
| 6 | 2.40 | 3.26 | 2.34 | 2.66 |
| Scheduler | Yaw | Pitch | Roll | MAE |
|---|---|---|---|---|
| CosineAnnealingWarmRestarts | 2.373 | 3.091 | 2.239 | 2.568 |
| CosineAnnealingLR | 2.456 | 3.079 | 2.192 | 2.576 |
| MultiStepLR | 3.576 | 3.538 | 2.351 | 3.155 |
| Constant learning rate | 2.365 | 3.100 | 2.307 | 2.591 |
| Method | Params (M) | FLOPs |
|---|---|---|
| TokenHPE | 86.42 | G |
| 6DRepNet | 39.32 | G |
| HopeNet | 23.90 | G |
| FSA-Net | 1.20 | M |
| STAG-Net (FaceMesh), Protocol I | 0.043 | K |
| STAG-Net (FAN), Protocol I | 0.034 | K |
| Method | CPU (FPS) | GPU (FPS) |
|---|---|---|
| HopeNet | 4.4 | 147.3 |
| 6DRepNet | 6.9 | 62.0 |
| FSA-Net | 114.0 | 164.5 |
| TokenHPE | 6.6 | 69.9 |
| STAG-Net (FaceMesh) | 111.0 | 322.9 |
| STAG-Net (FAN) | 1.6 | 17.7 |
| Method | CPU (s) | GPU (s) | GPU (ms/sample) |
|---|---|---|---|
| HopeNet | 8051.9 | 127.5 | 11.6 |
| 6DRepNet | 5232.2 | 253.2 | 23.0 |
| TokenHPE | 6365.5 | 410.3 | 37.4 |
| FSA-Net | 203.1 | 6.7 | 0.61 |
| STAG-Net (FaceMesh) | 25.9 | 9.5 | 0.077 |
| STAG-Net (FAN) | 28.0 | 10.9 | 0.089 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Protocol I | Protocol II | |||
| Configuration | FaceMesh | FAN | FaceMesh | FAN |
| Encoder width, | 64 | 64 | 32 | 64 |
| Token dimension, | 32 | 32 | 16 | 32 |
| Attention heads, | 4 | 4 | 2 | 2 |
| Clusters, | 7 | 5 | 5 | 5 |
| Trunk output, | 36 | 36 | 32 | 36 |
Object Pose Estimation
Object pose estimation recovers an object's 3D position and orientation from images, depth, point clouds, or contact sensing.
See all 199 papers