Synthetic eye data are critical for rapid development of eye tracking (ET) in AR/VR applications. However, existing synthetic eye generators are largely image-centric and lack a high-quality underlying 3D anatomy of the eye, which limits their control mainly to the gaze and provides no anatomical labels. To address this problem, we propose EyeDiT, an anatomy-grounded, multi-modal 3D eye generation and simulation model trained with optical coherence tomography (OCT), labeled anatomy point clouds and near-eye images. Our pipeline first trains a Multi-Modal Anatomy Variational Autoencoder (VAE), which encodes an anatomy-aware latent representation from all input modalities. Then, our Eye Latent DiT is trained to generate latent samples conditioned on the optical axis, the pupil position and radius, and the subject identity. Together with Gaussian decoders for novel-view simulation, our model enables control over the pupil, subject identity and viewpoint for data synthesis. We show that our model outperforms previous eye image synthesis models in novel-view synthesis (24.9 against 21.8 dB PSNR from a single view), and that a public ET neural network trained on our synthetic data shows a smaller sim-to-real gap than when trained on data from other generators (3.5∘ vs. 8.4∘ gaze error), approaches a network trained on real data, and further improves ET performance when combined with real data (2.3∘ vs. 2.6∘). More information is available on the project page: https://lyhan12.github.io/eyedit
Figures & tables
Figure 1: We present EyeDiT, which demonstrates controlled simulation (right) of gaze, pupil radius, subject identity and novel views for synthetic near-eye images. The examples show simulated gaze and pupil radius sweeps, identities from the learned identity embedding, and novel views rendered from decoded 3D Gaussians [ 19 ] . Our core novelty is anatomy-grounded eye data generation (middle), in which images, anatomy-labeled (iris and cornea) point clouds and OCT B-scans are learned and generated jointly. With this anatomical grounding, our method accurately controls both gaze and pupil radius, whereas previous learned generators mainly control the gaze [ 28 , 5 , 54 , 25 ] . The other key capability (left) is one-shot reconstruction and simulation of a new eye, together with simulation of the trained eyes.
Figure 2: Overview of EyeDiT. (Stage 1) Our multi-modal anatomy VAE learns a shared latent space from OCT volumes, near-eye images and anatomy point clouds. (Stage 2) Our eye latent DiT is then trained on this latent space to simulate various gaze, pupil, and identity conditions. Our pipeline generates multi-modal outputs for a learned identity or for a new subject given by a context image.
Figure 3: Inputs (GT) and reconstructions (Ours) of our multi-modal anatomy VAE on OCT-Cam-2V (top two rows) and OCT-Cam-4V (bottom two rows), showing that our VAE preserves the main structures and details of the given inputs across all modalities. The columns show two views (nasal, temporal), two OCT B-scan positions, the projected OCT volume (en-face) and the PCD, each as a GT and Ours pair. Each row represents a subject either seen or unseen during training. Note that our VAE also reconstructs the unseen subjects faithfully.
Figure 4: Generation results when changing the gaze, the pupil radius and the identity embedding, respectively (rows), in five steps (columns). Only the changed factor varies along each row, and the generated pupil follows the commanded values. The green lines and circles mark the commanded optical axis and pupil radius. Top-left: commanded value. Bottom-right: readout value.
Table 4: Novel-view rendering on OCT-Cam-2V, OCT-Cam-4V and LightDome [ 29 ] . The top three methods are baselines for eye modeling, while the others are general 3D reconstruction and generation methods. Our model, with a prior learned from multi-view eye data, shows the best novel-view synthesis performance. × : the method needs at least two context views. –: ControllableGaze is not available on LightDome, which has no mesh or OCT data.
Figure 6: ET training protocol. An Aria ET model is trained on synthetic data from different sources, or on real data, and tested on real data to measure the sim-to-real gap.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Raw data and anatomy annotation. (a) A raw OCT B-scan and the near-eye camera images. (b) The point cloud lifted from the OCT volume with its five labels. (c) The pupil annotation on the point cloud. (d) The optical axes.
Figure 8: One example of OCT-Cam-2V, OCT-Cam-4V and LightDome [ 29 ] : camera views, and a central OCT B-scan and the anatomy PCD where available.
Figure 9: Subject embedding and label distributions of OCT-Cam-2V.
Figure 10: Subject embedding and label distributions of OCT-Cam-4V.
Encoder
Layers
Output
OCT
3 conv blocks, warping, strided conv
G=960 tokens
PCD
MLP, attention pooling
4K=32 tokens
Image
linear patch + Plücker ray embedding
T=256 tokens/view
Fusion
2 cross-attn, 2 self-attn
G×d=960×256
Appendix
Table 9: Encoders and decoders of the VAE.
Figure 11: Multi-modal reconstruction on OCT-Cam-2V.
Figure 12: Multi-modal reconstruction on OCT-Cam-4V.
Figure 13: Generated samples beside GT on OCT-Cam-2V.
Figure 14: Generated samples beside GT on OCT-Cam-4V.
Command
Cross-Modal Consistency
Error ↓
PCD
OCT
Image
OCT–PCD
Image–PCD
OCT–Image
Pupil position (mm)
0.086
0.159
0.473
0.146
0.467
0.510
Pupil radius (mm)
0.091
0.112
0.160
0.091
0.124
0.149
Appendix
Table 14: Pupil consistency between the generated modalities on OCT-Cam-2V. Command: error against the commanded pupil. Cross-Modal Consistency: error between two generated modalities.
Figure 15: Cross-modal consistency, one sample per dataset.
Figure 16: Field-of-view masks of the OCT-Cam-2V (left) and OCT-Cam-4V (right) cameras.
1 context view
3 context views
Method
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
GazePrior [ 5 ]
16.08
0.485
0.576
17.27
0.493
0.554
Lin et al. [ 29 ]
11.73
0.404
0.723
14.22
0.405
0.588
Ours
18.08
0.502
0.358
19.30
0.520
0.324
Appendix
Table 18: Novel-view rendering on LightDome [ 29 ] test-unseen subjects, target camera 9.
Figure 17: Novel-view rendering on OCT-Cam-2V test-seen samples.
Figure 18: Novel-view rendering on OCT-Cam-4V test-unseen samples from three context views (left of the gray rule).
Figure 19: Novel-view rendering from one context image on OCT-Cam-2V, OCT-Cam-4V and LightDome [ 29 ] .
Figure 20: View sparsification on LightDome [ 29 ] , target camera 9.
Figure 21: Training images of each sim-to-real source for the same condition (nasal camera). Red cross: the commanded pupil center projected with the rig calibration. GazePrior renders blurry images in which the pupil is barely visible, and UnityEyes 2 differs strongly from the IR images in appearance.
Figure 22: Eye-tracking estimates on real OCT-Cam-2V test frames of two subjects, by training source. Green: label. Each panel shows the estimate and its optical-axis error.
Figure 24: Novel-view rendering on test-unseen OCT-Cam-4V subjects from two context views, with MVSplat and DepthSplat trained on OCT-Cam-4V and with their released checkpoints. Regions hidden by the device housing are masked out.
Figure 25: Novel-view rendering on test-unseen OCT-Cam-4V subjects from three context views, in the format of Fig. 5 , with MapAnything+GS, MVSplat and DepthSplat trained on OCT-Cam-4V (Tab. 22 ). Three context views (left of the dashed line) are used as inputs to render the target view. Regions hidden by the device housing are masked out.
Eye tracking (ET) is a foundational technology for advanced AR/VR applications. However, training ET models for every new ET device is challenging: real data collection is costly and time-consuming, while existing synthetic data generation methods lack realism. To remove the need for additional data collection while maintaining data quality, we introduce a data-driven 3D prior that models the distribution of human eyes across diverse identities, gaze directions, and light settings. This model, which we coin GazePrior, then enables sparse-input 3D reconstruction of annotated data collected with previous ET devices, which can in turn be rendered from the cameras of any target ET device. Our approach synthesizes data with the realism, diversity and ground-truth accuracy of real data collection without its prohibitive costs. Our experiments demonstrate that ET models trained with our synthesized data outperform previous zero-shot methods, achieving higher accuracy and robustness.
Corentin Dumery, David Colmenares, Alexander Fix +3
Meta Reality Labs, Redmond, USA · EPFL, Lausanne, Switzerland · Work done during an internship at Meta
Diffusion models are increasingly used to generate synthetic training data, but precise label control remains difficult when the conditioning signal is low-dimensional and coarse. Text-conditioned images are judged by broad prompt consistency, whereas supervised training requires precise correspondence between each image and its numerical label. This is challenging in eye tracking, where a 4D binocular gaze is expressed through subtle, spatially localized pupil and iris geometry. We introduce GazeDiT, a diffusion model that generates images for a requested 4D gaze through an internally constructed spatial condition that grounds the global gaze label in this local geometry. During training, a frozen SegFormer extracts pupil/iris geometry from diverse real images, allowing the model to learn realistic appearance conditioned on that geometry. At inference, a physical eye renderer samples gaze-consistent geometries by varying anatomy and camera state, enabling diverse synthesis without a source image. GazeDiT achieves substantially lower tail gaze-label error than other diffusion baselines, approaching the error of the same frozen gaze estimator on real images. Its generated data also improves the downstream eye tracker, reducing gaze error on difficult cases from 3.05° to 2.80° in the smallest cohort.
Large vision-language models have achieved remarkable capabilities by training on massive internet-scale data, yet a fundamental asymmetry persists: while LLMs can leverage self-supervised pretraining on abundant text and image data, the same is not true for many behavioral modalities. Video-based behavioral data -- gestures, eye movements, social signals -- remains scarce, expensive to annotate, and privacy-sensitive. A promising alternative is simulation: replace real data collection with controlled synthetic generation to produce automatically labeled data at scale. We introduce infrastructure for this paradigm applied to eye movement, a behavioral signal with applications across vision-language modeling, virtual reality, robotics, accessibility systems, and cognitive science. We present a pipeline for generating synthetic labeled eye movement video by extracting real human iris trajectories from reference videos and replaying them on a 3D eye movement simulator via headless browser automation. Applying this to the task of script-reading detection during video interviews, we release final_dataset_v1: 144 sessions (72 reading, 72 conversation) totaling 12 hours of synthetic eye movement video at 25fps. Evaluation shows that generated trajectories preserve the temporal dynamics of the source data (KS D < 0.14 across all metrics). A matched frame-by-frame comparison reveals that the 3D simulator exhibits bounded sensitivity at reading-scale movements, attributable to the absence of coupled head movement -- a finding that informs future simulator design. The pipeline, dataset, and evaluation tools are released to support downstream behavioral classifier development at the intersection of behavioral modeling and vision-language systems.