Gaze estimation is instrumental in modern virtual reality (VR) systems. Despite significant progress in remote-camera gaze estimation, VR gaze research remains constrained by data scarcity, particularly the lack of large-scale, accurately labeled datasets captured with the off-axis camera configurations typical of modern headsets. Gaze annotation is difficult since fixation on intended targets cannot be guaranteed. To address these challenges, we introduce VRGaze, the first large-scale off-axis gaze estimation dataset for VR, comprising 2.1 million near-eye infrared images collected from 68 participants. We further propose GazeShift, an attention-guided unsupervised framework for learning gaze representations without labeled data. Unlike prior redirection-based methods that rely on multi-view or 3D geometry, GazeShift is tailored to near-eye imagery, achieving effective gaze-appearance disentanglement in a compact, real-time model. GazeShift embeddings can be optionally adapted to individual users via lightweight few-shot calibration, achieving a 1.84° mean error on VRGaze. On the remote-camera MPIIGaze dataset, the model achieves a 7.15° person-agnostic error, doing so with 10x fewer parameters and 35x fewer FLOPs than baseline methods. Deployed natively on a VR headset GPU, inference takes only 5 ms. Combined with demonstrated robustness to illumination changes, these results highlight GazeShift as a label-efficient, real-time solution for VR gaze tracking. Project code and the VRGaze dataset are released at https://github.com/gazeshift3/gazeshift
Figures & tables
Figure 1: GazeShift architecture. The model is trained to reconstruct the appearance source image to match the gaze target, conditioned on the target’s gaze embedding. A lightweight gaze encoder extracts the embedding, while a separate appearance encoder preserves spatial structure. Appearance tokens undergo self-attention and cross-attention (conditioned on the gaze embedding) before being decoded. Attention maps also guide a gaze-aware loss focused on gaze-relevant regions. At inference, only the gaze encoder and a lightweight calibration module are used to predict gaze.
Dataset
Year
Gaze Ann.
Participants
Images
(PoR)
VR off-axis /
Total
VR off-axis /
Total
NVGaze [ 15 ]
2018
Yes
14 /
49
260K /
2.5M
OpenEDS2020 [ 23 ]
2020
Yes
0 /
80
0 /
550K
TEyeD [ 7 ]
2021
No
14 /
132
260K /
20M
VRGaze (ours)
2025
Yes
68 /
68
2.1M /
2.1M
Table 1: Overview of VR eye-tracking datasets.
Gender
Ethnicity
Age (years)
Train / Val
(F / M)
(Asian / Cauc)
18–30 /
31–40 /
41–50 /
51+
Count
14 / 54
29 / 39
22 /
30 /
13 /
3
61 / 7
Table 2: VRGaze dataset statistics.
Figure 2: Sample images from our off-axis VRGaze dataset (left), example images from on-axis OpenEDS2020 dataset (center), 2D gaze angle distribution of VRGaze data (right).
Supervision
Method
Calibration Type
Avg. Error [ ∘ ]
Supervised
Appearance Based
Per-person
1.54
Feature Based
3.2
Unsupervised
VAE
5.30
Cross-Encoder
2.15
GazeShift
1.84
Cross-Encoder
Person-agnostic (K=100)
2.83
Table 3: Performance on VRGaze.
Model
Calibration Type
Avg. Error [°]
Cross-Encoder
Per-person
3.69
GazeShift
Per-person
3.43
Cross-Encoder
Person-agnostic
5.20
GazeShift
Person-agnostic
4.20
Table 4: Performance on OpenEDS2020.
#
Separate Encoders
Attention- Based Redirection
Gaze- Focused Loss
Avg. Error [°]
1
×
×
×
2.15
2
✓
×
×
2.10
3
✓
✓
×
2.07
4
✓
✓
✓
1.84
Table 5: GazeShift ablation study on VRGaze.
Figure 3: Disentanglement analysis of learned embeddings. Left: appearance perturbations under fixed gaze. Right: gaze perturbations under fixed appearance. Gaze embeddings vary primarily with gaze direction, while appearance embeddings remain comparatively stable, indicating effective disentanglement.
γ
0.5
1.0
2.0
4.0
Avg. Error [ ∘ ]
2.03
1.84
2.19
2.41
Table 6: Average error under different γ settings.
Figure 4: Appearance source frames and their associated self-attention maps. The model learns to focus on regions that exhibit the greatest differences between the source and target images—primarily the gaze-relevant areas around the iris.
Supervision
Method
Avg. Error [ ∘ ]
Params
FLOPS
Supervised
Resnet-18
8.35
11M
75M
Unsupervised
Cross-Encoder
8.32
11M
75M
GazeShift (MobileNetV2)
8.00
1M
2M
GazeShift (ResNet-18)
7.56
11M
75M
Table 7: Calibration results on MPIIGaze using embeddings trained on Columbia Gaze.
Model
Avg. Error [ ∘ ]
Params
ResNet-18 (ImageNet)
10.60
11M
Auto-encoder
9.50
11M
Auto-encoder (EFC)
9.20
11M
SimCLR
10.00
25M
BYOL
11.10
25M
Cross-Encoder
7.20
11M
Table 8: Unsupervised fine-tuning results on MPIIGaze.
Figure 5: Latent space interpolation and gaze redirection. We interpolate between two target gaze embeddings ( gt1→gt2 , top green boxes) while keeping the appearance embeddings ( As1,As2,As3 , left blue boxes) fixed for each row. The smooth eye movement between the gaze targets, combined with the preservation of the source images’ unique appearance, visually confirms effective disentanglement and a continuous latent manifold.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1 : Supervised model architecture.
Figure S2 : Single person data visualization.
Figure S3 : Gaze direction visualization.
Figure S4 : Pupils dilation illustration.
Figure S5 : Pupil center and glints from manual annotations (left) and model predictions (right).
Figure S6 : Gaze embeddings across directions. Each quadrant shows two eye images with similar gaze directions, along with their corresponding 12D gaze embeddings. Within a quadrant, the embeddings align closely, while embeddings across quadrants differ, reflecting distinct gaze directions. Notably, in the top-left quadrant, the gaze embeddings remain similar despite clear differences in eye appearance.
Figure S7 : Latent space visualization. A t-SNE projection of the representations extracted by GazeShift. Each point corresponds to an eye image from the Columbia Gaze dataset and is colored according to the subject’s identity, illustrating the clustering of appearance features.
Figure S8 : VRGaze hardware setup. The internal view of the custom virtual reality headset used for data collection. The ring of 850 nm near-infrared LEDs (visible as purple lights) provides active illumination for robust corneal glint formation, while the restrictive form factor necessitates the off-axis camera placement.
Eye tracking (ET) plays a critical role in augmented and virtual reality applications. However, rapidly deploying high-accuracy, on-device gaze estimation for new products remains challenging because hardware configurations (e.g., camera placement, camera pose, and illumination) often change across device generations. Visual foundation models (VFMs) excel on natural-image benchmarks and offer a promising path to rapid training and deployment; yet, we find that off-the-shelf VFMs still struggle to reach high accuracy on specialized near-eye infrared images. To close this gap, we introduce DistillGaze, a framework that distills a VFM using labeled synthetic data and unlabeled real data for rapid, high-accuracy on-device gaze estimation. DistillGaze proceeds in two stages. First, we adapt a VFM into a domain-specialized teacher using synthetic gaze labels and unlabeled real images. Synthetic data provide scalable, high-quality gaze supervision, while unlabeled real data bridges the synthetic-to-real domain gap. Second, we train an on-device student from both teacher guidance and self-training. Evaluated on a large-scale crowd-sourced dataset spanning more than 2,000 participants, DistillGaze reduces median gaze error by 58.6% relative to synthetic-only baselines while maintaining a lightweight 256K-parameter model suitable for real-time on-device deployment. More broadly, DistillGaze offers an efficient path to training and deploying ET models that adapt to hardware changes, and a recipe for combining synthetic supervision with unlabeled real data in on-device regression tasks.
Eye tracking (ET) is a foundational technology for advanced AR/VR applications. However, training ET models for every new ET device is challenging: real data collection is costly and time-consuming, while existing synthetic data generation methods lack realism. To remove the need for additional data collection while maintaining data quality, we introduce a data-driven 3D prior that models the distribution of human eyes across diverse identities, gaze directions, and light settings. This model, which we coin GazePrior, then enables sparse-input 3D reconstruction of annotated data collected with previous ET devices, which can in turn be rendered from the cameras of any target ET device. Our approach synthesizes data with the realism, diversity and ground-truth accuracy of real data collection without its prohibitive costs. Our experiments demonstrate that ET models trained with our synthesized data outperform previous zero-shot methods, achieving higher accuracy and robustness.
Corentin Dumery, David Colmenares, Alexander Fix +3
Meta Reality Labs, Redmond, USA · EPFL, Lausanne, Switzerland · Work done during an internship at Meta
We present PicoEyes, a unified gaze estimation framework that directly predicts all key attributes of gaze, including 3D eye parameters, eye-region segmentation, optical axis, visual axis, and depth maps, from either monocular or binocular inputs. The framework simultaneously addresses calibration, gaze forecasting, and varying device postures, while also supporting 3D eye reconstruction via joint estimation of eye parameters and depth maps in an end-to-end manner. In addition, we introduce a large-scale multi-view near-eye dataset containing comprehensive 2D and 3D annotations under diverse conditions, including train, test, rewear-test, and calibration sessions. Extensive experiments demonstrate that PicoEyes achieves state-ofthe-art performance, consistently outperforming both academic and industrial gaze tracking methods across nocalibration, calibration, rewear-after-calibration, and forecasting settings. This work establishes a practical, end-toend paradigm for robust and generalizable gaze estimation in mixed reality (MR) applications.