Gaze gestures can provide hands free input on mobile devices, but practical use requires (i) gestures users can learn and recall and (ii) recognition models that are efficient enough for on-device deployment. We present an end-to-end pipeline using commodity ARKit head/eye transforms and a scaffolded guidance-to-recall protocol grounded in learning theory. In a pilot feasibility study (N=4 participants; 240 trials; controlled single-session setting), we benchmark a compact time-series model (TinyHAR) against deeper baselines (DeepConvLSTM, SA-HAR) on 5-way gesture recognition and 4-way user identification. TinyHAR achieves strong performance in this pilot benchmark (Macro F1 = 0.960 for gesture recognition; Macro F1 = 0.997 for user identification) while using only 46k parameters. A modality analysis further indicates that head pose dynamics are highly informative for mobile gaze gestures, highlighting embodied head--eye coordination as a key design consideration. Although the small sample size and controlled setting limit generalizability, these results indicate a potential direction for further investigation into on-device gaze gesture recognition.
Figures & tables
Figure 1. TinyGaze overview. TinyGaze logs ARKit head and eye pose time series and represents each gaze gesture as a spatio-temporal trajectory (from t0 to tn ). The same sequential signal supports two tasks: (i) gesture control and (ii) continuous-style authentication via user identification. An overview diagram of the TinyGaze system. The figure shows two stages: data capture and dual-purpose use. In the data capture stage, ARKit head and eye movement streams are recorded and represented as a time-ordered trajectory from t0 to tn on a smartphone screen. In the dual-purpose use stage, the same sequential signal is used for two downstream tasks: gesture-based interaction control and user identification for continuous-style authentication.
Figure 2. Gesture set. Five gaze-gesture patterns with increasing directional complexity: Vertical, Horizontal, L-shape ( 0∘ ), L-shape ( 270∘ ), and Z-shape ( 0∘ ). These templates are simple to learn yet sufficiently distinct to evaluate recognition under noisy mobile sensing. Illustration of the five gaze gestures used in the study, arranged as simple line and shape templates with increasing complexity: a vertical line, a horizontal line, an L-shape at 0 degrees, an L-shape at 270 degrees, and a Z-shape at 0 degrees. The set spans straight strokes, single-turn gestures, and a multi-segment gesture to test recognition under noisy mobile sensing.
Figure 3. Scaffolded study protocol. (a) Gesture configuration UI for adjusting parameters (e.g., size, speed, repetitions). (b) Four-stage guidance-to-recall sequence: Follow (moving guide), Fixed (static outline), IRecall (start-point cue), and Recall (no visual guidance), designed to transition from low-variance tracing to memory-based production. Composite figure showing the TinyGaze study interface and scaffolded protocol. Panel (a) shows the gesture configuration screen, where parameters such as pattern size, speed, and repetitions can be adjusted. Panel (b) shows four study stages that progressively reduce visual guidance: Follow with a moving guide dot, Fixed with a static gesture outline, IRecall with only a start-point cue, and Recall with no visual guidance. The sequence illustrates a transition from guided tracing to memory-based gesture production.
Model
Macro F1
Params
Exec. Time (ms)
TinyHAR ( Zhou et al., 2022 )
0.960
46,094
40.0
DeepConvLSTM ( Wan et al., 2019 )
0.762
1,050,208
39.4
SA-HAR ( Mahmud et al., 2020 )
0.853
440,412
77.3
Table 1. Gesture recognition performance on P0123-I. A four-column table comparing gesture recognition performance across three models on dataset P0123-I. Columns are Model, Macro F1, Params, and Execution Time in milliseconds. TinyHAR achieves the highest Macro F1 of 0.960 with 46,094 parameters and 40.0 ms execution time. DeepConvLSTM achieves 0.762 Macro F1 with 1,050,208 parameters and 39.4 ms. SA-HAR achieves 0.853 Macro F1 with 440,412 parameters and 77.3 ms. The table shows that TinyHAR has the best recognition performance with far fewer parameters than the deeper baselines.
Model
Macro F1
Exec. Time (ms)
TinyHAR ( Zhou et al., 2022 )
0.997
190.1
DeepConvLSTM ( Wan et al., 2019 )
1.000
329.2
SA-HAR ( Mahmud et al., 2020 )
0.961
513.0
Table 2. User identification on P0123-I. TinyHAR achieves near-optimal performance while using far fewer parameters and reduced compute relative to deep baselines. A three-column table summarizing user identification performance on dataset P0123-I for three models. Columns are Model, Macro F1, and Execution Time in milliseconds. TinyHAR achieves a Macro F1 of 0.997 with 190.1 ms execution time. DeepConvLSTM achieves 1.000 Macro F1 with 329.2 ms, and SA-HAR achieves 0.961 Macro F1 with 513.0 ms. The table indicates near-ceiling performance for TinyHAR in this pilot setting while using less compute time than the deeper baselines.
Gesture type
Accuracy
Weighted F1
Macro F1
Auth (all)
0.997
0.997
0.997
Vertical
1.000
1.000
1.000
Horizontal
1.000
1.000
1.000
L 0
1.000
1.000
1.000
L 270
0.938
0.937
0.937
Z 0
1.000
1.000
1.000
Table 3. User identification across gesture subsets and full set. A four-column table reporting TinyHAR user identification performance across gesture subsets and the full authentication set. Columns are Gesture type, Accuracy, Weighted F1, and Macro F1. The full set labelled Auth (all) has 0.997 for all three metrics. Vertical, Horizontal, L 0, and Z 0 each achieve perfect scores of 1.000. The L 270 subset shows lower performance, with Accuracy 0.938, Weighted F1 0.937, and Macro F1 0.937. The table shows that user identification is good for most gesture types, with L 270 being the most challenging subset.
Input
P0
P1
P2
P3
Avg
Eye_Head
0.796
0.683
0.928
0.889
0.824
Eyes
0.834
0.469
0.659
0.659
0.655
Left_Eye
0.699
0.301
0.699
0.831
0.633
Right_Eye
0.671
0.308
0.784
0.834
0.649
Head
0.960
0.506
0.879
0.919
0.816
Avg
0.792
0.453
0.790
0.826
0.715
Table 4. Gesture recognition modality ablation using TinyHAR (Macro F1). A six-column table showing TinyHAR gesture recognition Macro F1 scores for different sensor input modalities across participants P0, P1, P2, and P3, plus an average column. Rows correspond to Eye_Head, Eyes, Left_Eye, Right_Eye, and Head inputs, followed by an overall average row. Eye_Head has the highest average performance at 0.824, followed by Head at 0.816. Eyes, Left_Eye, and Right_Eye achieve lower averages of 0.655, 0.633, and 0.649, respectively. Participant-level scores vary substantially, with P1 lower than the others across most modalities. The table indicates that head pose is highly informative and that fusing head and eye signals yields the strongest average performance.
Gesture recognition is a cornerstone of Human-Computer Interaction (HCI) for smart eyewear, enabling natural and device-free control in augmented reality environments. Traditional vision-based approaches face significant challenges regarding power consumption, computational latency, and user privacy. This paper proposes a lightweight, privacy-preserving gesture recognition system based on the fusion of low-resolution Time-of-Flight (ToF) and Infrared (IR) thermal sensors. We used an 8 times 8 multizone ToF sensor (VL53L8CH) and an 8 times 8 IR array (AMG8833) to capture complementary depth and thermal cues. A compact Convolutional Neural Network (CNN) with a specialized grouped-convolution architecture is designed to fuse these modalities efficiently on a microcontroller (MCU). Experimental results on a custom dataset of 7 static gestures, validated via k-fold cross-validation, demonstrate that the proposed fusion strategy significantly outperforms single-sensor baselines with an accuracy of 92.3% and a macro F1-score of 0.93. Finally, on-device benchmarks on STM32F4 and STM32H7 MCUs confirm the system's suitability for resource-constrained wearables, requiring only 6,343 parameters and achieving millisecond-level inference latency with a total system power of 50 mW.
Pietro Bartoli, Christian Veronesi, Tommaso Bondini +2
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB) Politecnico di Milano, Milan, Italy
Realizing on-device ML-based gesture detection under tight real-time performance, energy and memory constraints is challenging, especially when considering mobile devices with varying battery-power levels. Existing EdgeAI deployments typically rely on a single fixed detector, limiting optimization opportunities. We present Scale-Gest, a novel run-time adaptive gesture detection framework that expands the detector space into a dense family of tiny-YOLO architectures. We introduce multiple novel device-calibrated ACE (Accuracy-Complexity-Energy) profiles by analyzing different model-resolution-stride operating points. A lightweight run-time controller selects an appropriate ACE mode under user-defined and battery constraints, while a motion-aware hand-gesture-tracking ROI gate crops the input for reduced complexity detection. To evaluate performance of our system in real-world car driving scenarios, we introduce a temporally-annotated Driver Simulated Gesture (DSG-18) dataset. Scale-Gest maintains event-level F1 while significantly reducing energy and latency compared to single-detector approaches. On a battery-powered laptop running gesture streams, our ACE controller reduces per-frame energy by 4x (from 6.9 mJ to 1.6 mJ) while maintaining high gesture-detection performance (event-level F1 = 0.8-0.9) and low mean latency (6 ms).
Abdul Basit, Saim Rehman, Muhammad Shafique
New York University (NYU) Abu Dhabi, Abu Dhabi, UAE
Mobile gaze tracking on handheld devices commonly relies on appearance-based 2D gaze estimation from the device's front-facing RGB camera. Although this approach makes eye tracking more accessible by avoiding dedicated infrared hardware, maintaining accuracy remains difficult because the spatial relationship among the user's face, the camera, and the screen changes continuously during handheld use. This paper examines how motion-related geometry change affects a representative front-camera RGB 2D gaze-estimation pipeline. We report two complementary within-subject user studies (n=10) that capture handheld behaviour, calibration data, and gaze-estimation error during everyday-style mobile tasks across conditions ranging from lying and sitting to walking and maze navigation. Rather than treating motion as a single binary factor, we analyse within-task versus task-switch behaviour, calibration-test mismatch, and condition-specific motion signatures. Behavioural results show that head-to-camera distance is comparatively stable during sustained within-task use, while task switches introduce much larger geometry fluctuations, with head-distance standard deviations of approximately 7.42--10.10,cm across conditions. In the gaze-evaluation study, one-off calibration reduced average RMSE from 4.49,cm to 3.44,cm; under this regime, the global effect of motion condition on RMSE was not statistically reliable, whereas clearer patterns appeared in task-specific and motion-specific calibration analyses, where better calibration-test match reduced error.
Yaxiong Lei, Yuheng Wang, Fergus Buchanan +5
University of St Andrews, St Andrews, Fife, UK, KY16 9SX · University of Tokyo, Tokyo, Japan · King’s College London, London, UK +1