cs.HCMar 20, 2026
SaveHiFiGaze: Improving Eye Tracking Accuracy Using Screen Content Knowledge
Organizations: Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
Abstract
We present a new and accurate approach for gaze estimation on consumer computing devices. We take advantage of continued strides in the quality of user-facing cameras found in e.g., smartphones, laptops, and desktops - 4K or greater in high-end devices - such that it is now possible to capture the 2D reflection of a device's screen in the user's eyes. This alone is insufficient for accurate gaze tracking due to the near-infinite variety of screen content. Crucially, however, the device knows what is being displayed on its own screen - in this work, we show this information allows for robust segmentation of the reflection, the location and size of which encodes the user's screen-relative gaze target. We explore several strategies to leverage this useful signal, quantifying performance in a user study. Our best performing model reduces mean tracking error by ~8% compared to a baseline appearance-based model. A supplemental study reveals an additional 10-20% improvement if the gaze-tracking camera is located at the bottom of the device.
Figures & tables
Figure 1. HiFiGaze leverages two pieces of information: 1) crop of the user’s eyes (as captured by increasingly common high-quality, 4K "selfie" cameras), and 2) the contents of the screen (which the device already knows). We quantify how combining these two information streams improves eye tracking accuracy over a conventional appearance-based approach. This figure shows the overall HiFiGaze pipeline. A user looks at a smartphone, where both the front-facing camera image of the face and the screen content are captured. The eye region is cropped and analyzed together with screen information to estimate the gaze point on the display.
Figure 2. An example smartphone screen drawn from the WebUI dataset ( Wu et al., 2023 ) . A naive screen glint approach will detect the light element, but incorrectly estimate the reflection center. With knowledge of the screen content, HiFiGaze finds the correct screen reflection center, necessary for accurate gaze estimation. Right: additional examples where a naive screen glint center approach would fail. This figure illustrates why naïvely using the brightest reflection in the eye can fail. Bright elements in the screen content can shift the detected glint away from the true screen center. Using screen content knowledge enables more accurate identification of the correct reflection corresponding to the display.
Figure 3. Iris center estimation. The purple dot shows the initial center estimate from MediaPipe ( Lugaresi et al., 2019 ) , which can be noisy. To correct this estimate, we extract the iris contour (shown in green) using the GrabCut algorithm ( Rother et al., 2004 ) , then apply RANSAC ( Fischler and Bolles, 1981 ) to reject outliers before circle fitting. The blue dots show the final points used for fitting, and the yellow circle with its center dot shows the fitted result. This figure depicts the iris center refinement process. An initial iris center estimate is obtained, followed by iris segmentation and contour fitting. The final fitted circle produces a more accurate iris center used for gaze estimation.
Figure 4. HiFiGaze architecture overview. Our model can draw on different sources of information, much of which leverages knowledge of the screen content. Our user study systematically compares the information power of these different inputs. This figure presents the HiFiGaze model architecture. Multiple inputs, including eye crops, eye bounds, screen thumbnails, and reflection-based features, are encoded by separate modules. These features are fused to predict the final two-dimensional gaze location.
Figure 5. Example screens drawn from the WebUI ( Wu et al., 2023 ) dataset, which we used in our data collection. Approximately 9% of the dataset contains predominantly dark screens, which are more challenging for our approach.
Figure 6. Illustration of the gaze target trajectories used in our data collection software, running on an iPhone 14 Pro Max. Orange and blue lines show the extent and spacing of our horizontal and vertical paths. The on-screen gaze target is shown at the same scale. This figure illustrates the data collection procedure. A participant looks at a smartphone while a gaze target moves across the screen in predefined horizontal and vertical paths. The trajectories ensure dense coverage of the display area during calibration-free data collection.
Figure 7. Calibration-free gaze estimation accuracy across eight encoding approaches (note all models are given Eye Bounds data as input). Gaze error reported in blue; within-participant normalized gaze error plotted in orange. Lower values are better. Reflection Vector * is evaluated only on a subset of gaze instances (153K of 171K; 89%). Error bars are standard deviation. This figure compares gaze estimation accuracy across different input configurations. Results are shown both as absolute error and normalized within-participant performance. Methods that incorporate screen-content-derived features achieve lower error than eye-only baselines.
Figure 8. Spatial heatmaps of gaze error across the screen. This figure visualizes spatial gaze error distributions over the phone screen. Heatmaps show higher error near the lower screen region for baseline methods. Incorporating screen knowledge reduces error, particularly in these challenging areas.
Figure 9. Mean gaze error broken out by method and instances with bright and dark screen content. Note that the Eye Crops condition is equivalent to prior appearance-based methods. See bright and dark screen examples in Figure 5 . This figure compares gaze estimation performance under bright and dark screen conditions. Errors are generally higher for dark screens. However, models using screen knowledge consistently outperform the baseline in both cases.
Figure 10. Calibration-free gaze estimation error for top and bottom camera location. Error bars are standard deviation. This figure compares top-camera and bottom-camera smartphone configurations. Bottom-camera placement results in lower gaze estimation error across methods. The improvement highlights the impact of camera geometry on appearance-based gaze tracking.
Figure 11. Various everyday computing devices where the HiFiGaze eye tracking method is applicable. A four-column comparison showing eye reflections from different display types. The top row presents close-up eye crops with checkerboard patterns reflected in the pupil. The bottom row shows the corresponding reference photos of the displays: an Apple Watch Ultra smartwatch at 30 cm, an iPad Mini tablet at 30 cm, a 13-inch MacBook Air laptop at 40 cm, and a 50-inch TV at 100 cm. The checkerboard reflection grows smaller and less distinct as viewing distance and display size increase.
More in this topic
Gaze Estimation
Gaze estimation predicts where a person is looking from eye, face, head-pose, and scene cues.
See all 66 papers