DiTGaze: Geometry-Conditioned Diffusion Augmentation for Few-Shot Gaze Estimation

Abiram Panchalingam; Indu P. Bodala; Stuart E. Middleton

cs.CVNov 14, 2025

DiTGaze: Geometry-Conditioned Diffusion Augmentation for Few-Shot Gaze Estimation

Authors: Abiram Panchalingam, Indu P. Bodala, Stuart E. Middleton

Organizations: School of Electronics and Computer Science, University of Southampton

Abstract

Accurate person-specific gaze estimation benefits from labeled user data, but collecting images that densely cover gaze directions is burdensome. We present DiTGaze, a calibration-efficient augmentation system that follows a two-stream 3D Gaussian Splatting backbone and reformulates its convolutional rendering stage as a geometry-conditioned latent Diffusion Transformer (DiT). A learnable projection and frozen VAE bridge rasterized Gaussian features to latent tokens, while timestep and identity/expression conditioning are integrated through Adaptive Layer Normalization. Intermediate yaw-pitch sampling densifies gaze supervision, and an orthogonality constraint reduces interference between gaze, pose, and expression. On ETH-XGaze, DiTGaze reduces generator-level gaze error by 4.1 percent relative to GazeGaussian. In our runs of the expanded downstream protocol, it ranks first at every calibration budget from one to nine real images; the repeated primary comparison at N = 9 reduces subject-level estimator error by 0.73 degrees and improves 14 of 15 identities (p < 0.001). These results connect the proposed generator changes to statistically supported few-shot personalization gains.

Figures & tables

Panel of Figure 1 — Figure 1: Primary $N=9$ application protocol. The nine labeled calibration images both adapt the generator and remain in each augmented fine-tuning set (9 real + 191 synthetic); real-only uses the nine real images. Evaluation uses 185 held-out images from the same unseen subject.

Figure 2: DiTGaze generator. Face and eye Gaussians form a two-stream 3DGS backbone. Pose and expression drive the Face Deform Field to produce offsets $\Delta x$ , while sampled target gaze drives the Eye Rotation Field to produce rotations $\Delta q$ . Orthogonality regularizes the deformation embeddings, and the transformed Gaussians are rasterized into feature maps for the latent DiT renderer, which uses AdaLN conditioning to synthesize the gaze-redirected image.

Method	Gaze $\downarrow$	Pose $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	ID $\uparrow$
STED	16.217	13.153	0.726	17.530	0.300	115.020	24.347
GazeNeRF	6.944	3.470	0.733	15.453	0.291	81.816	45.207
GazeGaussian	6.622	2.128	0.823	18.734	0.216	41.972	67.749
DiTGaze	6.353	2.349	0.850	20.512	0.187	38.319	71.724

Table 1: Within-dataset redirection on the ETH-XGaze person-specific test set. All methods use the same training split and source–target pairs. Best results are bold.

Dataset	Method	Gaze $\downarrow$	Pose $\downarrow$	LPIPS $\downarrow$	ID $\uparrow$
ColumbiaGaze	STED	17.887	14.693	0.413	6.384
	GazeNeRF	9.464	3.811	0.352	23.157
	GazeGaussian	7.415	3.332	0.273	59.788
	DiTGaze	7.265	3.611	0.231	64.963
MPIIFaceGaze	STED	14.796	11.893	0.288	10.677
	GazeNeRF	14.933	7.118	0.272	30.981

Table 2: Cross-dataset redirection after training on ETH-XGaze. We report angular gaze/pose error ( ∘ ), LPIPS, and identity similarity. Best results are bold.

Variant	Gaze $\downarrow$	Pose $\downarrow$	LPIPS $\downarrow$	FID $\downarrow$
w/o DiT renderer	6.485	2.328	0.214	41.878
w/o orthogonality	6.396	2.572	0.191	39.337
w/o gaze sampling	6.714	2.344	0.190	38.582
Full DiTGaze	6.353	2.349	0.187	38.319

Table 3: Component ablation on ETH-XGaze. Each row removes one component from the full system.

Figure 3: Within-dataset qualitative comparison on ETH-XGaze. Columns show the input, prior 3D redirectors, DiTGaze, and target.

$N$ real	Real	STED	GazeNeRF	GazeGaussian	DiTGaze
1	9.1	8.8	6.9	6.1	5.7
2	9.0	8.5	6.5	6.0	5.3
3	8.8	9.0	6.4	6.0	5.4
4	8.0	8.9	6.0	5.9	5.1
5	6.8	8.0	6.4	5.8	5.2
6	6.9	8.2	6.3	5.7	5.1

Table 4: Rerun of the ETH-XGaze few-shot curve. Mean gaze error ( ∘ ) on held-out real images; augmented sets contain

B=200

samples. Lower is better, and the best result at each calibration budget is bold.

Method/statistic	Mean $\pm$ SD ( ∘ )	95% CI ( ∘ )
GazeGaussian	$5.63\pm 1.39$	$[4.96,\ 6.33]$
DiTGaze	$\mathbf{4.90\pm 1.29}$	$\mathbf{[4.29,\ 5.57]}$
Paired $\Delta$	$\mathbf{0.73\pm 0.49}$	$\mathbf{[0.49,\ 0.97]}$
Wilcoxon $p$ / effect size	$p<0.001$ / $r_{\mathrm{rb}}=0.967$

Table 5: Primary paired comparison at

N=9

. Results aggregate five draws and three seeds per subject;

\Delta

is GazeGaussian minus DiTGaze, so positive values favor ours.

Generator variant	Error ( ∘ ) $\downarrow$	$\Delta$ vs. full
w/o DiT renderer	5.220	+0.346
w/o orthogonality	4.995	+0.120
w/o gaze sampling	5.555	+0.680
Full DiTGaze	4.875	0.000

Table 6: Downstream component ablation at ETH-XGaze

N=9

S=191

, with fixed real samples, targets, estimator, and fine-tuning schedule.

$N$	Real only	GazeGaussian	DiTGaze
5	6.883	5.855	5.253
9	6.704	5.439	4.906
15	6.107	5.273	4.701

Table 7: Calibration and synthetic-data sensitivity on ETH-XGaze. Panel (a) varies the real calibration budget with

B=200

augmented samples. Panel (b) independently fixes

N=9

while varying synthetic count, so total size changes. Each panel is an internally paired controlled sweep.

Fine-tuning data	ETH $\rightarrow$ ETH	ETH $\rightarrow$ MPII
Real only	6.7	11.8
Real + STED	7.9	12.1
Real + GazeNeRF	5.9	10.4
Real + GazeGaussian	5.3	9.9
Real + DiTGaze	4.8	9.5

Table 8: Downstream domain evaluation at

N=9

. Mean angular error on held-out real images. ETH

\rightarrow

ETH is the in-domain reference; ETH

\rightarrow

MPII transfers the ETH-XGaze-pretrained estimator and generator before target-subject generator adaptation and estimator fine-tuning. MPII denotes MPIIFaceGaze.

Gaze Estimation

Gaze estimation predicts where a person is looking from eye, face, head-pose, and scene cues.

See all 69 papers

GazeDiT: Gaze-Accurate Diffusion Image Generation for Eye Tracking via Spatial Conditioning
Sep 15, 2026Dongze Wu, David Colmenares, Fengting Yang +4Conditional Image Generation Diffusion-Based Image Generation
Meta · Georgia Institute of Technology
PAGE: Towards Practical Human-level Gaze Target Estimation
Jul 6, 2026Zhoutong Ye, Chengwen Zhang, Zhaibin Cui +10Gaze Estimation
1Tsinghua University · 2Jinan University
GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer
May 1, 2026Xinyuan Zhao, Yihang Wu, Ahmad Chaddad +2Domain-Adversarial Learning Gaze Estimation
School of Artificial Intelligence, Guilin University of Electronic Technology, Guilin, China · The Laboratory for Imagery, Vision and Artificial Intelligence, École de Technologie Supérieure, Montreal, Canada · Information Technology Department, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia +2

Sep 15, 2026J/K move · Enter open · S save

GazeDiT: Gaze-Accurate Diffusion Image Generation for Eye Tracking via Spatial Conditioning

Dongze Wu, David Colmenares, Fengting Yang, Jogendra Nath Kundu+3

Meta · Georgia Institute of Technology

Conditional Image Generation Diffusion-Based Image Generation Conditional Diffusion Models Gaze Estimation Diffusion Transformer

Open paper AskPDF arXiv

Diffusion models are increasingly used to generate synthetic training data, but precise label control remains difficult when the conditioning signal is low-dimensional and coarse. Text-conditioned images are judged by broad prompt consistency, whereas supervised training requires precise correspondence between each image and its numerical label. This is challenging in eye tracking, where a 4D binocular gaze is expressed through subtle, spatially localized pupil and iris geometry. We introduce GazeDiT, a diffusion model that generates images for a requested 4D gaze through an internally constructed spatial condition that grounds the global gaze label in this local geometry. During training, a frozen SegFormer extracts pupil/iris geometry from diverse real images, allowing the model to learn realistic appearance conditioned on that geometry. At inference, a physical eye renderer samples gaze-consistent geometries by varying anatomy and camera state, enabling diverse synthesis without a source image. GazeDiT achieves substantially lower tail gaze-label error than other diffusion baselines, approaching the error of the same frozen gaze estimator on real images. Its generated data also improves the downstream eye tracker, reducing gaze error on difficult cases from 3.05° to 2.80° in the smallest cohort.

Abstract

Gaze Estimation

Explore similar work

GazeDiT: Gaze-Accurate Diffusion Image Generation for Eye Tracking via Spatial Conditioning

PAGE: Towards Practical Human-level Gaze Target Estimation

GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer