Quantifying the Impact of Motion on 2D Gaze Estimation in Real-World Mobile Interactions
Authors: Yaxiong Lei, Yuheng Wang, Fergus Buchanan, Mingyue Zhao, Yusuke Sugano, Shijing He, Mohamed Khamis, Juan Ye
Organizations: University of St Andrews, St Andrews, Fife, UK, KY16 9SX · University of Tokyo, Tokyo, Japan · King’s College London, London, UK · University of Glasgow, Glasgow, UK
Mobile gaze tracking on handheld devices commonly relies on appearance-based 2D gaze estimation from the device's front-facing RGB camera. Although this approach makes eye tracking more accessible by avoiding dedicated infrared hardware, maintaining accuracy remains difficult because the spatial relationship among the user's face, the camera, and the screen changes continuously during handheld use. This paper examines how motion-related geometry change affects a representative front-camera RGB 2D gaze-estimation pipeline. We report two complementary within-subject user studies (n=10) that capture handheld behaviour, calibration data, and gaze-estimation error during everyday-style mobile tasks across conditions ranging from lying and sitting to walking and maze navigation. Rather than treating motion as a single binary factor, we analyse within-task versus task-switch behaviour, calibration-test mismatch, and condition-specific motion signatures. Behavioural results show that head-to-camera distance is comparatively stable during sustained within-task use, while task switches introduce much larger geometry fluctuations, with head-distance standard deviations of approximately 7.42--10.10,cm across conditions. In the gaze-evaluation study, one-off calibration reduced average RMSE from 4.49,cm to 3.44,cm; under this regime, the global effect of motion condition on RMSE was not statistically reliable, whereas clearer patterns appeared in task-specific and motion-specific calibration analyses, where better calibration-test match reduced error.
Figures & tables
Figure 1: Overview of the experimental mobile eye-tracking system used to synchronise front-camera RGB input, calibration and evaluation events, and motion-sensing data.
Figure 2: Head-to-camera distance estimation from the relationship between physical interpupillary distance and image-frame interpupillary scale.
Sensor Events
Frequency
Dimension
Description
UserAccelerometer
50Hz
X,Y,Z
Acceleration (m/s 2 ) without the effects of gravity.
Accelerometer
50Hz
X,Y,Z
Acceleration (m/s 2 ) with the effects of gravity.
Gyroscope
50Hz
X,Y,Z
Device rotation.
Magnetometer
50Hz
X,Y,Z
Ambient magnetic field surrounding the device.
Head distance
30Hz
distance
Estimated distance between the eye region and the front camera.
Head pose
30Hz
Pitch, Yaw
Head orientation relative to the camera.
Table 1: Sensor data collected or estimated by the Motion Sensing Module.
Figure 3: Schematic overview of the user study design. User Study 1 characterises handheld geometry and motion patterns during everyday-style tasks. User Study 2 evaluates 2D gaze-estimation performance under task-elicited semi-controlled handheld conditions.
Task
Cue Phrase
Document editing
“Create a new document using your preferred document editing app and write today’s diary entry.”
Social chatting
“Open your preferred social media app, such as WhatsApp, and respond to any unread messages.”
Social media browsing
“Open a social media app like Instagram to browse content or like posts.”
Video calling
“Use your preferred video calling app to either simulate a video call or engage in a brief call with a friend.”
News reading
“Open your preferred news app to read some articles.”
Table 2: Cue phrases for everyday-style task prompts.
Table 3: Distribution of data samples across motion conditions and sensor types in User Study 1.
Table 4: Distribution of data samples across motion conditions, sensor types, and experimental phases in User Study 2.
Table 5: Summary of extracted features used in the motion analysis and regression-based factor identification. The full feature set contains 552 dimensions aggregated across raw channels and feature families.
Figure 4: Box plots of sensor data under each motion condition: Ly for lying, St for sitting, Ws for walking slowly and Wm for walking in maze.
Figure 5: t-SNE plots of sensor features under each motion condition
Figure 6: Comparison of the number of motion clusters under different motion conditions
Figure 7: GMM clustering mapped to time-series sensor data on Participant 1 under static conditions
Figure 8: GMM clustering mapped to time-series sensor data on Participant 1 under dynamic conditions
Table 6: Comparison of mean and standard deviation of motion data for in-task and switch-task periods under static conditions.
Table 7: Comparison of mean and standard deviation of motion data for in-task and switch-task periods under dynamic conditions.
Figure 9: Box plots of motion data under each motion condition: Ly for lying, St for sitting, Sd for standing, Wk for walking and Wm for walking in maze.
Figure 10: Cross-comparison t-SNE plots for User Study 1 and 2
Figure 11: Overview of participant-specific calibration and test regimes at different behavioural granularities.
Table 8: Comparison of RMSE for the base model and one-off calibration. Values are mean (standard deviation) in cm.
Figure 12: Held-out RMSE as a function of elapsed time within the test sequence (four equal temporal bins after one-off calibration), per motion condition. Error bars indicate ±1 standard error across participants. Within the present dataset there is no monotonic post-calibration drift across the four bins.
Figure 13: Within-task and between-task RMSE under task-specific calibration.
Figure 14: t-SNE visualisation of calibration and test motion-feature distributions under each motion condition.
Figure 15: Within-motion and between-motion RMSE under motion-specific calibration.
Table 9: Train-test RMSE matrix for the motion-specific calibration setting. Values are mean (standard deviation) in cm.
Table 10: Impact factors identified between and within motion conditions
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
R2
MAE (cm)
Top factor groups
Ordinary Least Squares
0.32±0.29
0.66±0.20
distance, orientation, head movement
Ridge
0.32±0.29
0.66±0.20
distance, orientation, head movement
Elastic Net
−0.07±0.15
0.85±0.13
orientation, head movement
Random Forest
0.16±0.29
0.74±0.12
distance, orientation, head movement
Lasso
−0.09±0.21
0.87±0.16
head movement, orientation
Appendix
Table 11: Supplementary comparison of regression models used for factor identification, evaluated by 5-fold cross-validation on participant-by-motion-condition aggregated features ( n=50 cells, 477 standardised features). OLS, Ridge, and Random Forest recover broadly similar geometry-related groups; Lasso is retained for the main analysis because it yields the sparsest coefficient profile.
Motion condition
Head distance
Head movement
Device orientation
Lying
≈ 0%
≈ 0%
≈ 0%
Sitting
≈ 0%
≈ 15%
≈ 20%
Standing
≈ 33%
≈ 18%
≈ 43%
Walking
≈ 0%
≈ 100%
≈ 0%
Walking in maze
≈ 0%
≈ 100%
≈ 0%
Appendix
Table 12: Per-condition decomposition of Lasso coefficient mass into three broad factor groups (head-to-camera distance, head movement, device-orientation change). Percentages represent the share of ∑∣β∣ within that condition. Methodological notes (rounding, residual “other” group, and the degenerate behaviour of the lying and walking conditions under the small per-condition sample size) are given in B .
Practical webcam gaze tracking is constrained not only by error, but also by calibration burden, robustness to head motion and session drift, runtime footprint, and browser use. We therefore target a deployment-oriented operating point rather than the image large-backbone regime. We cast landmark-based point-of-regard estimation as session-wise adaptation: a shared geometric encoder produces embeddings that can be aligned to a new session from a small calibration set. We present Equivariant Meta-Calibrated Gaze (EMC-Gaze), a lightweight landmark-only method combining an E(3)-equivariant landmark-graph encoder, local eye geometry, binocular emphasis, auxiliary 3D gaze-direction supervision, and a closed-form ridge calibrator differentiated through episodic meta-training. To reduce pose leakage, we use a two-view canonicalization consistency loss. The deployed predictor uses only facial landmarks and fits a per-session ridge head from brief calibration. In a fixation-style interactive evaluation over 33 sessions at 100 cm, EMC-Gaze achieves 5.79 +/- 1.81 deg RMSE after 9-point calibration versus 6.68 +/- 2.34 deg for Elastic Net; the gain is larger on still-head queries (2.92 +/- 0.75 deg vs. 4.45 +/- 0.30 deg). Across three subject holdouts of 10 subjects each, EMC-Gaze retains an advantage (5.66 +/- 0.19 deg vs. 6.49 +/- 0.33 deg). On MPIIFaceGaze with short per-session calibration, the eye-focused model reaches 8.82 +/- 1.21 deg at 16-shot calibration, ties Elastic Net at 1-shot, and outperforms it from 3-shot onward. The exported eye-focused encoder has 944,423 parameters, is 4.76 MB in ONNX, and supports calibrated browser prediction in 12.58/12.58/12.90 ms per sample (mean/median/p90) in Chromium 145 with ONNX Runtime Web. These results position EMC-Gaze as a calibration-friendly operating point rather than a universal state-of-the-art claim against heavier appearance-based systems.
Chenkai Zhang
Independent Researcher, Wenzhou, Zhejiang 325000, China
Appearance-based gaze estimation (AGE) has achieved remarkable performance in constrained settings, yet we reveal a significant generalization gap where existing AGE models often fail in practical, unconstrained scenarios, particularly those involving facial wearables and poor lighting conditions. We attribute this failure to two core factors: limited image diversity and inconsistent label fidelity across different datasets, especially along the pitch axis. To address these, we propose a robust AGE framework that enhances generalization without requiring additional human-annotated data. First, we expand the image manifold via an ensemble of augmentation techniques, including synthesis of eyeglasses, masks, and varied lighting. Second, to mitigate the impact of anisotropic inter-dataset label deviation, we reformulate gaze regression as a multi-task learning problem, incorporating multi-view supervised contrastive (SupCon) learning, discretized label classification, and eye-region segmentation as auxiliary objectives. To rigorously validate our approach, we curate new benchmark datasets designed to evaluate gaze robustness under challenging conditions, a dimension largely overlooked by existing evaluation protocols. Our MobileNet-based lightweight model achieves generalization performance competitive with the state-of-the-art (SOTA) UniGaze-H, while utilizing less than 1% of its parameters, enabling high-fidelity, real-time gaze tracking on mobile devices.
Zhenhao Li, Zheng Liu, Seunghyun Lee +2
Huawei Technologies Canada · University of Toronto
We present a new and accurate approach for gaze estimation on consumer computing devices. We take advantage of continued strides in the quality of user-facing cameras found in e.g., smartphones, laptops, and desktops - 4K or greater in high-end devices - such that it is now possible to capture the 2D reflection of a device's screen in the user's eyes. This alone is insufficient for accurate gaze tracking due to the near-infinite variety of screen content. Crucially, however, the device knows what is being displayed on its own screen - in this work, we show this information allows for robust segmentation of the reflection, the location and size of which encodes the user's screen-relative gaze target. We explore several strategies to leverage this useful signal, quantifying performance in a user study. Our best performing model reduces mean tracking error by ~8% compared to a baseline appearance-based model. A supplemental study reveals an additional 10-20% improvement if the gaze-tracking camera is located at the bottom of the device.
Taejun Kim, Vimal Mollyn, Riku Arakawa +1
Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA