We present a new and accurate approach for gaze estimation on consumer computing devices. We take advantage of continued strides in the quality of user-facing cameras found in e.g., smartphones, laptops, and desktops - 4K or greater in high-end devices - such that it is now possible to capture the 2D reflection of a device's screen in the user's eyes. This alone is insufficient for accurate gaze tracking due to the near-infinite variety of screen content. Crucially, however, the device knows what is being displayed on its own screen - in this work, we show this information allows for robust segmentation of the reflection, the location and size of which encodes the user's screen-relative gaze target. We explore several strategies to leverage this useful signal, quantifying performance in a user study. Our best performing model reduces mean tracking error by ~8% compared to a baseline appearance-based model. A supplemental study reveals an additional 10-20% improvement if the gaze-tracking camera is located at the bottom of the device.
Figures & tables
Figure 1. HiFiGaze leverages two pieces of information: 1) crop of the user’s eyes (as captured by increasingly common high-quality, 4K "selfie" cameras), and 2) the contents of the screen (which the device already knows). We quantify how combining these two information streams improves eye tracking accuracy over a conventional appearance-based approach. This figure shows the overall HiFiGaze pipeline. A user looks at a smartphone, where both the front-facing camera image of the face and the screen content are captured. The eye region is cropped and analyzed together with screen information to estimate the gaze point on the display.
Figure 2. An example smartphone screen drawn from the WebUI dataset ( Wu et al., 2023 ) . A naive screen glint approach will detect the light element, but incorrectly estimate the reflection center. With knowledge of the screen content, HiFiGaze finds the correct screen reflection center, necessary for accurate gaze estimation. Right: additional examples where a naive screen glint center approach would fail. This figure illustrates why naïvely using the brightest reflection in the eye can fail. Bright elements in the screen content can shift the detected glint away from the true screen center. Using screen content knowledge enables more accurate identification of the correct reflection corresponding to the display.
Figure 3. Iris center estimation. The purple dot shows the initial center estimate from MediaPipe ( Lugaresi et al., 2019 ) , which can be noisy. To correct this estimate, we extract the iris contour (shown in green) using the GrabCut algorithm ( Rother et al., 2004 ) , then apply RANSAC ( Fischler and Bolles, 1981 ) to reject outliers before circle fitting. The blue dots show the final points used for fitting, and the yellow circle with its center dot shows the fitted result. This figure depicts the iris center refinement process. An initial iris center estimate is obtained, followed by iris segmentation and contour fitting. The final fitted circle produces a more accurate iris center used for gaze estimation.
Figure 4. HiFiGaze architecture overview. Our model can draw on different sources of information, much of which leverages knowledge of the screen content. Our user study systematically compares the information power of these different inputs. This figure presents the HiFiGaze model architecture. Multiple inputs, including eye crops, eye bounds, screen thumbnails, and reflection-based features, are encoded by separate modules. These features are fused to predict the final two-dimensional gaze location.
Figure 5. Example screens drawn from the WebUI ( Wu et al., 2023 ) dataset, which we used in our data collection. Approximately 9% of the dataset contains predominantly dark screens, which are more challenging for our approach.
Figure 6. Illustration of the gaze target trajectories used in our data collection software, running on an iPhone 14 Pro Max. Orange and blue lines show the extent and spacing of our horizontal and vertical paths. The on-screen gaze target is shown at the same scale. This figure illustrates the data collection procedure. A participant looks at a smartphone while a gaze target moves across the screen in predefined horizontal and vertical paths. The trajectories ensure dense coverage of the display area during calibration-free data collection.
Figure 7. Calibration-free gaze estimation accuracy across eight encoding approaches (note all models are given Eye Bounds data as input). Gaze error reported in blue; within-participant normalized gaze error plotted in orange. Lower values are better. Reflection Vector * is evaluated only on a subset of gaze instances (153K of 171K; 89%). Error bars are ±1 standard deviation. This figure compares gaze estimation accuracy across different input configurations. Results are shown both as absolute error and normalized within-participant performance. Methods that incorporate screen-content-derived features achieve lower error than eye-only baselines.
Figure 8. Spatial heatmaps of gaze error across the screen. This figure visualizes spatial gaze error distributions over the phone screen. Heatmaps show higher error near the lower screen region for baseline methods. Incorporating screen knowledge reduces error, particularly in these challenging areas.
Figure 9. Mean gaze error broken out by method and instances with bright and dark screen content. Note that the Eye Crops condition is equivalent to prior appearance-based methods. See bright and dark screen examples in Figure 5 . This figure compares gaze estimation performance under bright and dark screen conditions. Errors are generally higher for dark screens. However, models using screen knowledge consistently outperform the baseline in both cases.
Figure 10. Calibration-free gaze estimation error for top and bottom camera location. Error bars are ±1 standard deviation. This figure compares top-camera and bottom-camera smartphone configurations. Bottom-camera placement results in lower gaze estimation error across methods. The improvement highlights the impact of camera geometry on appearance-based gaze tracking.
Figure 11. Various everyday computing devices where the HiFiGaze eye tracking method is applicable. A four-column comparison showing eye reflections from different display types. The top row presents close-up eye crops with checkerboard patterns reflected in the pupil. The bottom row shows the corresponding reference photos of the displays: an Apple Watch Ultra smartwatch at 30 cm, an iPad Mini tablet at 30 cm, a 13-inch MacBook Air laptop at 40 cm, and a 50-inch TV at 100 cm. The checkerboard reflection grows smaller and less distinct as viewing distance and display size increase.
Mobile gaze tracking on handheld devices commonly relies on appearance-based 2D gaze estimation from the device's front-facing RGB camera. Although this approach makes eye tracking more accessible by avoiding dedicated infrared hardware, maintaining accuracy remains difficult because the spatial relationship among the user's face, the camera, and the screen changes continuously during handheld use. This paper examines how motion-related geometry change affects a representative front-camera RGB 2D gaze-estimation pipeline. We report two complementary within-subject user studies (n=10) that capture handheld behaviour, calibration data, and gaze-estimation error during everyday-style mobile tasks across conditions ranging from lying and sitting to walking and maze navigation. Rather than treating motion as a single binary factor, we analyse within-task versus task-switch behaviour, calibration-test mismatch, and condition-specific motion signatures. Behavioural results show that head-to-camera distance is comparatively stable during sustained within-task use, while task switches introduce much larger geometry fluctuations, with head-distance standard deviations of approximately 7.42--10.10,cm across conditions. In the gaze-evaluation study, one-off calibration reduced average RMSE from 4.49,cm to 3.44,cm; under this regime, the global effect of motion condition on RMSE was not statistically reliable, whereas clearer patterns appeared in task-specific and motion-specific calibration analyses, where better calibration-test match reduced error.
Yaxiong Lei, Yuheng Wang, Fergus Buchanan +5
University of St Andrews, St Andrews, Fife, UK, KY16 9SX · University of Tokyo, Tokyo, Japan · King’s College London, London, UK +1
According to commonly consented theories, the minimum hardware requirement for gaze tracker is one camera and two light sources to realize gaze estimation with free head movements. However, in some scenarios such as eye tracking on mobile devices, it is preferable to use less components, especially light sources. We propose a gaze estimation method with one camera and one light source. A "virtual light source" is introduced, which is geometrically placed symmetrically to the real light source with respect to the camera, and generates a "virtual glint" in the acquired image. We estimate the "virtual glint" by exploiting the relationship between the distance between two pupils and two glints in the captured image, and estimate the gaze with polynomial regression assuming two light sources are available. A new normalization factor for regression method is verified, which turns out to be practical for one-glint system. The performance is proved to be acceptable, while degradation is noticed compared to system with two actual light sources.
Gaze estimation under natural head-eye motion underpins applications from driver monitoring to human-computer interaction. Single-frame methods predict each frame independently, so consecutive outputs fluctuate as jitter. Multi-frame methods reduce this, but they learn motion implicitly inside appearance features, so the gaze trajectory is never an explicit variable. We propose EyeTAG (Eye Trajectory-Aware Gaze Estimation), a causal multi-frame framework built around an explicit first-order gaze prior: at each step it differentiates its own recent predictions and feeds the resulting trajectory back as a compact kinematic token. Because differencing is translation-invariant in gaze space, this token carries subject-invariant motion rather than personal gaze offsets. Face and eye streams supply visual evidence, fused by cross-attention and a causal Transformer decoder. EyeTAG reduces the mean angular error by about 1.0∘ on Gaze360 and performs on par with the strongest baseline on EVE (2.56∘ vs. 2.58∘). Within-model ablations, which keep the encoder and the rest of the architecture fixed and vary only the gaze history, show that the differential formulation, rather than temporal context alone, removes the systematic saccade bias that persists even with an absolute gaze-history prior. Our code is available at https://github.com/peter8366/EyeTAG.
Jungmin Lee, Niamat Ullah, Yoseob Han
Department of Information and Telecommunication Engineering Soongsil University Seoul, Republic of Korea · Department of Electronic Engineering Soongsil University Seoul, Republic of Korea