Head-mounted devices integrated with eye tracking promise a solution for natural human-computer interaction. However, they typically require per-user calibration for optimal performance due to inter-person variability. A differential personalization approach using Siamese architectures learns relative gaze displacements and reconstructs absolute gaze from a small set of calibration frames. In this paper, we benchmark Siamese personalization on polarization-enabled eye tracking. For benchmarking, we use a 338-subject dataset captured with a polarization-sensitive camera and 850 nm illumination. We achieve performance comparable to linear calibration with 10-fold fewer samples. Using polarization inputs for Siamese personalization reduces gaze error by up to 12% compared to near-infrared (NIR)-based inputs. Combining Siamese personalization with linear calibration yields further improvements of up to 13% over a linearly calibrated baseline. These results establish Siamese personalization as a practical approach enabling accurate eye tracking.
Figures & tables
Figure 1: Intensity
Figure 5: The polarization-based dataset is split into training and testing subjects. Both training and testing employ a Siamese network architecture. During training, the network learns to predict gaze displacement between pairs of images from the same eye of the same subject. During testing, the predicted gaze for an input image is computed by estimating gaze displacements between the input and each image in the subject’s calibration set, then aggregating the calibration gaze labels based on the predicted displacements. While our approach processes binocular images as input to each Siamese branch, we illustrate with single-eye images for clarity.
Linear
Gaze Error ( ∘ ) ↓
Data
Model
Calibration
P50
P75
P95
Polarization
Siamese
✓
0.91
1.51
2.88
×
1.08
1.65
2.98
Baseline
✓
1.05
1.69
3.15
×
2.36
3.18
4.78
Intensity
Siamese
✓
1.03
1.72
3.05
Table 1: Test gaze error percentiles in degrees (P50, P75, P95) comparing Siamese and Baseline models on Polarization (Intensity-DoLP-AoLP) and Intensity data. Siamese personalization outperforms linearly calibrated Baseline, with the combination of Siamese personalization and linear calibration achieving the best overall performance. The proposed Siamese model with Polarization data achieves lower gaze errors than the conventional linearly calibrated Baseline model with Intensity data.
Calibration
Gaze Error ( ∘ ) ↓
Data
Model
Images
P50
P75
P95
Polarization
Siamese
3
1.16
1.76
3.10
5
1.12
1.71
3.04
7
1.11
1.68
3.01
9
1.08
1.65
2.98
Baseline
∼ 100
1.05
1.69
3.15
Table 2: Effect of calibration/anchor image count on P50, P75, and P95 percentile errors in the Siamese pipeline. Performance improves with additional calibration images, with 9 images achieving competitive results compared to linear calibration that utilizes the entire calibration sequence. With Polarization data, both the Siamese personalization and linearly calibrated Baseline models outperform the conventional Intensity-based linearly calibrated Baseline.
Figure 6: P50 (50th percentile/median) of the error distribution
Figure 9: Random Sampling
Gaze Error
Data
Sampling Strategy
P50
P75
P95
Polarization
Random Sampling
1.08
1.65
2.98
Calibration Sampling
1.34
2.05
3.56
Table 3: Performance comparison of random sampling vs. calibration sampling (with 3 Images/Subject) for Siamese model training on Polarization data. Random sampling shows lower gaze errors than pre-selected fixed calibration sets.
Linear
Gaze Error ( ∘ ) ↓
Data
Calibration
P50
P75
P95
Polarization
✓
1.05
1.69
3.15
×
2.36
3.18
4.78
Intensity
✓
1.24
2.02
3.56
×
2.77
3.67
5.32
Single-channel Intensity
✓
1.26
2.04
3.57
Table 4: P50, P75, and P95 percentile errors for different input modalities. Triplicating the Intensity channels reduces error compared to single-channel Intensity input, but Polarization achieves the best performance, demonstrating that the improvement is due to polarization features rather than increased channel count.
We present PicoEyes, a unified gaze estimation framework that directly predicts all key attributes of gaze, including 3D eye parameters, eye-region segmentation, optical axis, visual axis, and depth maps, from either monocular or binocular inputs. The framework simultaneously addresses calibration, gaze forecasting, and varying device postures, while also supporting 3D eye reconstruction via joint estimation of eye parameters and depth maps in an end-to-end manner. In addition, we introduce a large-scale multi-view near-eye dataset containing comprehensive 2D and 3D annotations under diverse conditions, including train, test, rewear-test, and calibration sessions. Extensive experiments demonstrate that PicoEyes achieves state-ofthe-art performance, consistently outperforming both academic and industrial gaze tracking methods across nocalibration, calibration, rewear-after-calibration, and forecasting settings. This work establishes a practical, end-toend paradigm for robust and generalizable gaze estimation in mixed reality (MR) applications.
AI glasses typically require explicit, user-specific eye-tracker calibration, and accuracy can drift when the device shifts during use. We present GazeAgent, a VLM-guided framework that continuously and adaptively improves gaze-calibration accuracy from naturally occurring behavior without asking users to interrupt their activities. GazeAgent identifies precise visual attention (PVA) events from multimodal streams of inertial measurement unit (IMU), gaze, and wearer-hand signals, validates their task relevance, and localizes functional targets as pseudo-calibration targets. In a 10-participant scripted study, the system detected all 820 predefined PVA opportunities and, under device disturbance, reduced median gaze error by 44.1% compared with explicit calibration. In a five-participant case study, autonomous calibration produced the lowest workload and interruption ratings, and all participants preferred it for routine use. By deriving task-critical targets from natural multimodal behavior, GazeAgent demonstrates the potential for continuous, adaptive, and low-burden gaze interaction on egocentric AI glasses.
Yuhong Zhang, Khanh An Nguyen, Xinyu Tian +6
Department of Bioengineering, University of California San Diego · Swartz Center for Computational Neuroscience, University of California San Diego · Tsinghua University, University of California San Diego +2
Practical webcam gaze tracking is constrained not only by error, but also by calibration burden, robustness to head motion and session drift, runtime footprint, and browser use. We therefore target a deployment-oriented operating point rather than the image large-backbone regime. We cast landmark-based point-of-regard estimation as session-wise adaptation: a shared geometric encoder produces embeddings that can be aligned to a new session from a small calibration set. We present Equivariant Meta-Calibrated Gaze (EMC-Gaze), a lightweight landmark-only method combining an E(3)-equivariant landmark-graph encoder, local eye geometry, binocular emphasis, auxiliary 3D gaze-direction supervision, and a closed-form ridge calibrator differentiated through episodic meta-training. To reduce pose leakage, we use a two-view canonicalization consistency loss. The deployed predictor uses only facial landmarks and fits a per-session ridge head from brief calibration. In a fixation-style interactive evaluation over 33 sessions at 100 cm, EMC-Gaze achieves 5.79 +/- 1.81 deg RMSE after 9-point calibration versus 6.68 +/- 2.34 deg for Elastic Net; the gain is larger on still-head queries (2.92 +/- 0.75 deg vs. 4.45 +/- 0.30 deg). Across three subject holdouts of 10 subjects each, EMC-Gaze retains an advantage (5.66 +/- 0.19 deg vs. 6.49 +/- 0.33 deg). On MPIIFaceGaze with short per-session calibration, the eye-focused model reaches 8.82 +/- 1.21 deg at 16-shot calibration, ties Elastic Net at 1-shot, and outperforms it from 3-shot onward. The exported eye-focused encoder has 944,423 parameters, is 4.76 MB in ONNX, and supports calibrated browser prediction in 12.58/12.58/12.90 ms per sample (mean/median/p90) in Chromium 145 with ONNX Runtime Web. These results position EMC-Gaze as a calibration-friendly operating point rather than a universal state-of-the-art claim against heavier appearance-based systems.
Chenkai Zhang
Independent Researcher, Wenzhou, Zhejiang 325000, China