Gaze gestures can provide hands free input on mobile devices, but practical use requires (i) gestures users can learn and recall and (ii) recognition models that are efficient enough for on-device deployment. We present an end-to-end pipeline using commodity ARKit head/eye transforms and a scaffolded guidance-to-recall protocol grounded in learning theory. In a pilot feasibility study (N=4 participants; 240 trials; controlled single-session setting), we benchmark a compact time-series model (TinyHAR) against deeper baselines (DeepConvLSTM, SA-HAR) on 5-way gesture recognition and 4-way user identification. TinyHAR achieves strong performance in this pilot benchmark (Macro F1 = 0.960 for gesture recognition; Macro F1 = 0.997 for user identification) while using only 46k parameters. A modality analysis further indicates that head pose dynamics are highly informative for mobile gaze gestures, highlighting embodied head--eye coordination as a key design consideration. Although the small sample size and controlled setting limit generalizability, these results indicate a potential direction for further investigation into on-device gaze gesture recognition.
Figures & tables
Figure 1. TinyGaze overview. TinyGaze logs ARKit head and eye pose time series and represents each gaze gesture as a spatio-temporal trajectory (from t0 to tn ). The same sequential signal supports two tasks: (i) gesture control and (ii) continuous-style authentication via user identification. An overview diagram of the TinyGaze system. The figure shows two stages: data capture and dual-purpose use. In the data capture stage, ARKit head and eye movement streams are recorded and represented as a time-ordered trajectory from t0 to tn on a smartphone screen. In the dual-purpose use stage, the same sequential signal is used for two downstream tasks: gesture-based interaction control and user identification for continuous-style authentication.
Figure 2. Gesture set. Five gaze-gesture patterns with increasing directional complexity: Vertical, Horizontal, L-shape ( 0∘ ), L-shape ( 270∘ ), and Z-shape ( 0∘ ). These templates are simple to learn yet sufficiently distinct to evaluate recognition under noisy mobile sensing. Illustration of the five gaze gestures used in the study, arranged as simple line and shape templates with increasing complexity: a vertical line, a horizontal line, an L-shape at 0 degrees, an L-shape at 270 degrees, and a Z-shape at 0 degrees. The set spans straight strokes, single-turn gestures, and a multi-segment gesture to test recognition under noisy mobile sensing.
Figure 3. Scaffolded study protocol. (a) Gesture configuration UI for adjusting parameters (e.g., size, speed, repetitions). (b) Four-stage guidance-to-recall sequence: Follow (moving guide), Fixed (static outline), IRecall (start-point cue), and Recall (no visual guidance), designed to transition from low-variance tracing to memory-based production. Composite figure showing the TinyGaze study interface and scaffolded protocol. Panel (a) shows the gesture configuration screen, where parameters such as pattern size, speed, and repetitions can be adjusted. Panel (b) shows four study stages that progressively reduce visual guidance: Follow with a moving guide dot, Fixed with a static gesture outline, IRecall with only a start-point cue, and Recall with no visual guidance. The sequence illustrates a transition from guided tracing to memory-based gesture production.
Model
Macro F1
Params
Exec. Time (ms)
TinyHAR ( Zhou et al., 2022 )
0.960
46,094
40.0
DeepConvLSTM ( Wan et al., 2019 )
0.762
1,050,208
39.4
SA-HAR ( Mahmud et al., 2020 )
0.853
440,412
77.3
Table 1. Gesture recognition performance on P0123-I. A four-column table comparing gesture recognition performance across three models on dataset P0123-I. Columns are Model, Macro F1, Params, and Execution Time in milliseconds. TinyHAR achieves the highest Macro F1 of 0.960 with 46,094 parameters and 40.0 ms execution time. DeepConvLSTM achieves 0.762 Macro F1 with 1,050,208 parameters and 39.4 ms. SA-HAR achieves 0.853 Macro F1 with 440,412 parameters and 77.3 ms. The table shows that TinyHAR has the best recognition performance with far fewer parameters than the deeper baselines.
Model
Macro F1
Exec. Time (ms)
TinyHAR ( Zhou et al., 2022 )
0.997
190.1
DeepConvLSTM ( Wan et al., 2019 )
1.000
329.2
SA-HAR ( Mahmud et al., 2020 )
0.961
513.0
Table 2. User identification on P0123-I. TinyHAR achieves near-optimal performance while using far fewer parameters and reduced compute relative to deep baselines. A three-column table summarizing user identification performance on dataset P0123-I for three models. Columns are Model, Macro F1, and Execution Time in milliseconds. TinyHAR achieves a Macro F1 of 0.997 with 190.1 ms execution time. DeepConvLSTM achieves 1.000 Macro F1 with 329.2 ms, and SA-HAR achieves 0.961 Macro F1 with 513.0 ms. The table indicates near-ceiling performance for TinyHAR in this pilot setting while using less compute time than the deeper baselines.
Gesture type
Accuracy
Weighted F1
Macro F1
Auth (all)
0.997
0.997
0.997
Vertical
1.000
1.000
1.000
Horizontal
1.000
1.000
1.000
L 0
1.000
1.000
1.000
L 270
0.938
0.937
0.937
Z 0
1.000
1.000
1.000
Table 3. User identification across gesture subsets and full set. A four-column table reporting TinyHAR user identification performance across gesture subsets and the full authentication set. Columns are Gesture type, Accuracy, Weighted F1, and Macro F1. The full set labelled Auth (all) has 0.997 for all three metrics. Vertical, Horizontal, L 0, and Z 0 each achieve perfect scores of 1.000. The L 270 subset shows lower performance, with Accuracy 0.938, Weighted F1 0.937, and Macro F1 0.937. The table shows that user identification is good for most gesture types, with L 270 being the most challenging subset.
Input
P0
P1
P2
P3
Avg
Eye_Head
0.796
0.683
0.928
0.889
0.824
Eyes
0.834
0.469
0.659
0.659
0.655
Left_Eye
0.699
0.301
0.699
0.831
0.633
Right_Eye
0.671
0.308
0.784
0.834
0.649
Head
0.960
0.506
0.879
0.919
0.816
Avg
0.792
0.453
0.790
0.826
0.715
Table 4. Gesture recognition modality ablation using TinyHAR (Macro F1). A six-column table showing TinyHAR gesture recognition Macro F1 scores for different sensor input modalities across participants P0, P1, P2, and P3, plus an average column. Rows correspond to Eye_Head, Eyes, Left_Eye, Right_Eye, and Head inputs, followed by an overall average row. Eye_Head has the highest average performance at 0.824, followed by Head at 0.816. Eyes, Left_Eye, and Right_Eye achieve lower averages of 0.655, 0.633, and 0.649, respectively. Participant-level scores vary substantially, with P1 lower than the others across most modalities. The table indicates that head pose is highly informative and that fusing head and eye signals yields the strongest average performance.