GazeAgent: VLM-Guided Implicit Gaze Calibration from Egocentric Multimodal Behavior for AI Glasses
Authors: Yuhong Zhang, Khanh An Nguyen, Xinyu Tian, KungYi Chang, Abraham Padua, Yuchen Xu, Ying Choon Wu, Gert Cauwenberghs, Tzyy-Ping Jung
Organizations: Department of Bioengineering, University of California San Diego · Swartz Center for Computational Neuroscience, University of California San Diego · Tsinghua University, University of California San Diego · Institute for Neural Computation, University of California San Diego · Department of Bioengineering, Institute for Neural Computation, University of California San Diego
AI glasses typically require explicit, user-specific eye-tracker calibration, and accuracy can drift when the device shifts during use. We present GazeAgent, a VLM-guided framework that continuously and adaptively improves gaze-calibration accuracy from naturally occurring behavior without asking users to interrupt their activities. GazeAgent identifies precise visual attention (PVA) events from multimodal streams of inertial measurement unit (IMU), gaze, and wearer-hand signals, validates their task relevance, and localizes functional targets as pseudo-calibration targets. In a 10-participant scripted study, the system detected all 820 predefined PVA opportunities and, under device disturbance, reduced median gaze error by 44.1% compared with explicit calibration. In a five-participant case study, autonomous calibration produced the lowest workload and interruption ratings, and all participants preferred it for routine use. By deriving task-critical targets from natural multimodal behavior, GazeAgent demonstrates the potential for continuous, adaptive, and low-burden gaze interaction on egocentric AI glasses.
Figures & tables
Figure 1. Overview of the GazeAgent implicit-calibration pipeline. A qualifying attention event provides a pseudo-calibration target, defined as the center of its functional region of interest (ROI), enabling estimation of the offset between the system-predicted gaze point and the intended target. Stage 1 combines head-motion stability, gaze stability, and wearer-hand evidence to generate candidate intervals and representative frames. Stage 2 uses human annotators and open-source and closed-source vision-language models (VLMs) to validate each candidate and localize its functional-target ROI; the ROI center is the single pseudo-calibration keypoint. Validated events are processed in successive batches to support continuous, one-step-ahead gaze calibration. Overview of the GazeAgent pipeline. A conceptual example illustrates the offset between the predicted gaze point and an event-based pseudo-calibration target. A multimodal candidate detector combines IMU, gaze, and hand signals. Human annotators and vision-language models then validate candidate events and localize their functional-target ROIs and center keypoints. Successive batches support adaptive implicit calibration.
Accepted Events
Rejected Events
Peeling a tape strip
Applying toothpaste
Reaching for a large object
Reaching for a pillow
Accept
Accept
Reject
Reject
Target: Tape edge
Target: Toothbrush tip
Target: None
Target: None
H1: The tape edge is a spatially precise functional target. H2: The head and hands remain stable. H3: Both hands perform a fine peeling action. H4: The action involves sustained manipulation and an elevated reach.
H1: Toothpaste must be accurately placed on the toothbrush head. H2: The head, hands, tube, and toothbrush remain stable. H3: One hand stabilizes the toothbrush while the other squeezes the tube. H4: The action requires sustained attention and precise alignment.
H1: No spatially precise functional target is visible. H2: The user appears to be moving toward the object. H3: The hand performs a coarse reaching action. H4: The action is brief, familiar, and visually undemanding.
H1: The large target does not require precise foveal vision. H2: The frame shows repositioning rather than stable fixation. H3: The hand performs a coarse grasp. H4: The action is familiar and low precision.
Table 1. Representative accepted and rejected examples of PVA events. Accepted events provide localized pseudo-calibration targets, whereas rejected events do not require precise foveal attention.
Figure 2. Overview of the GazeAgent workflow, which integrates multimodal behavioral streams, iterative VLM reasoning and tool use, evidence-grounded implicit calibration, and optional feedback-driven personalization. GazeAgent synchronizes egocentric video, motion, gaze, and hand signals; validates candidate PVA events through an iterative thought, action, and observation loop; performs evidence-grounded calibration; and can optionally collect feedback for personalization.
Figure 3. Example output of the high-recall multimodal PVA candidate detector. The left panel shows the representative keyframe, wearer-hand overlay, modality scores, and combined ranking score. The displayed gaze marker is used only for post-hoc validation and is removed from every frame supplied to the VLM. The right panel shows the synchronized head-motion, gaze-dynamics, and hand-activity signals. The green region denotes the candidate interval, and the orange line marks its highest-scoring keyframe. The detector proposes a possible event; the VLM subsequently determines its semantic validity without access to absolute gaze position. A representative egocentric keyframe is displayed beside synchronized plots of head motion, gaze stability, and hand-interaction signals. A green vertical band indicates the detected candidate interval, while an orange line identifies the selected representative keyframe.
Figure 4. Human annotation interface. Annotators selected a candidate from the script list and event strip, inspected its keyframe and context clip, drew a tight ROI around the functional target, and completed the criterion-level judgments. The point shown inside the ROI is its computed center and serves as the single reference keypoint. A screenshot of the annotation interface with labeled regions for the script list, keyframe, context clip, event-answer panel, event strip, and functional-target ROI.
Figure 5. Content and annotation characteristics of human-consensus PVA events. Results are reported for 621 candidates accepted by all five annotators. (a) Most frequent activity categories. (b) Functional target categories used as pseudo-calibration references. (c) Wearer-hand interaction types. Fine target-directed actions include tapping, pinching, pressing, aligning, inserting, peeling, and precise placement. (d) Annotation-criterion selection rates among accepted judgments. Activity, target, and interaction categories are mutually exclusive within their respective panels; multiple annotation criteria could be selected for the same event. Four horizontal bar charts summarize the activity categories, functional targets, hand-interaction types, and annotation criteria of 621 human-consensus PVA events.
Figure 6. Spatial reliability of the human-annotated pseudo-targets. Five annotators independently marked the functional target in each accepted PVA event. (a) Angular offsets from the leave-one-out annotator consensus. The median offset was 1.93∘ , approximately one foveal diameter, while the 95th-percentile offset was 10.1∘ . (b) Cumulative distributions of annotator-to-consensus and pairwise annotator disagreement; median pairwise disagreement was 2.39∘ . (c) Annotation disagreement relative to the foveal diameter, target width, camera field of view, and calibration-error scales. (d) Residual correction bias decreases as more pseudo-targets are aggregated, from 1.89∘ for one target to 0.78∘ for eight and 0.50∘ for twenty. (e) Median disagreement between each annotator pair ranged from 1.5∘ to 3.3∘ . (f) Annotated points were substantially more concentrated than their corresponding ROIs: the median point spread was 5.9% of the target extent. Six panels quantify agreement among five annotators, compare annotation error with relevant visual-angle scales, and show how aggregating pseudo-targets reduces residual correction bias.
Figure 7. Performance of closed-source and open-source VLMs on PVA validation and target localization. Left: Task 1 PVA recall across event conditions; darker cells indicate higher recall. Right: Task 1 validation accuracy versus Task 2 ROI-localization AP. Circles and diamonds represent closed-source and open-source models, respectively. Task 2 AP is conditional on correct Task 1 classification. A heatmap of condition-level PVA recall and a scatter plot comparing PVA-event validation accuracy with ROI-localization average precision across nine vision-language models.
Figure 8. Experimental protocol and representative PVA events across two scripted routes. Participants completed two continuous multimodal recordings while egocentric RGB, eye gaze, IMU, and frontal electroencephalography (EEG) were recorded. Script 1 covered four spaces and contained 21 PVA events, whereas Script 2 covered three spaces and contained 20 events. Both scripts included seven matched pen-and-paper tasks. The representative egocentric frames illustrate PVA-relevant activities across locations, hallway transitions, and hand-object interactions. Each script was performed once under a control condition and once under a disturbance condition, during which the glasses were adjusted, participants jumped or jogged, and the device was removed and re-worn at predefined points. Diagram of two experimental routes. Script 1 proceeds through the front door, a room, the library, and the lounge; Script 2 proceeds through the library, kitchen, and lounge. Both include a shared pen-and-paper block. Eight egocentric images show representative tasks, followed by timelines for the control and device-disturbance conditions.
Figure 9. Gaze-estimation accuracy for the general SocialEye CNN, explicit calibration, and implicit calibration. (a) Angular error distributions under control and disturbance conditions. (b) Error histograms, with red dashed lines indicating method-specific two-standard-deviation thresholds. (c) Target and predicted gaze directions in yaw-pitch coordinates under the control condition. (d) Batch-level angular errors. Batch 1 initializes the implicit model; implicit-calibration results from Batch 2 onward are one-step-ahead evaluations using the model fitted on the preceding batch. All implicit updates use VLM-predicted ROI centers. Four panels compare three gaze-estimation methods using condition-level boxplots, error histograms, yaw-pitch prediction plots, and batch-level error trends.
Figure 10. GazeAgent interface and feedback-driven personalization. (a) The mobile interface defers low-confidence queries until a natural pause, presents the candidate PVA event and model prediction, and allows the participant to confirm the prediction, reject the event as non-PVA, or correct the functional target by selecting its precise location. (b) Changes in Task 1 PVA-event validation, Task 2 ROI-localization, and median angular gaze error as additional user-reviewed events are incorporated into personalization. Thick green curves show the participant mean with personalization, thin green curves show individual participant trajectories, shaded regions indicate the participant range, and dashed orange lines show matched performance without personalization. The upper panel shows three mobile interface screens for recording a session, reviewing a candidate PVA event, and correcting its target location. The lower panel shows three adaptation plots comparing performance with and without personalization as the number of user-reviewed events increases.
Measure
Aria explicit calibration
Autonomous implicit calibration
Personalized feedback
NASA-TLX, median [IQR], 0 to 100
47.5 [46.7 to 55.0]
41.7 [40.8 to 44.2]
45.8 [43.3 to 48.3]
Interruption rating, median [IQR], 1 to 7
5 [4 to 6]
1 [1 to 2]
3 [2 to 4]
Preferred for routine use
0/5
5/5
0/5
Main perceived benefit
Predictable and transparent
No explicit action required
Allows users to correct errors
Main concern
Interrupts the ongoing task
Background updates are less visible
Queries still require attention
Table 2. Within-subject workload, interruption, and preference across the three calibration conditions ( n=5 ). Quantitative values are reported as median [interquartile range, IQR]. Lower NASA Task Load Index (NASA-TLX) and interruption scores indicate lower burden; bold values mark the most favorable condition.
Figure 11. Representative pre-fine-tuning VLM validation errors on AEA. (a) The model attributes another person’s pouring action to the wearer; all five annotators rejected the event. (b) The model treats eating as a routine, coarse action and misses the precise fork-to-food contact point. (c) The model evaluates the entire coffee-maker reservoir rather than its functional opening. All five annotators accepted the events in (b) and (c). Three egocentric AEA frames illustrating one false-positive and two false-negative PVA validation decisions.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12. Image-grounded few-shot exemplars used for Task 1 validation. The validation prompt includes two accepted and two rejected examples from recordings different from the candidate under evaluation. The accepted examples require precise alignment with a small functional target: the measuring-cap opening in (a) and the item pinched between the fingertips in (b). The rejected examples show a coarse interaction with a broad door panel in (c) and a manipulation performed by another person rather than the wearer in (d). Four egocentric few-shot example frames arranged in two rows. The top row contains accepted PVA examples: pouring detergent into a small cap and pinching a small item. The bottom row contains rejected examples: pushing a broad door panel and watching another person manipulate a fruit container.
Figure 13. Detector-conditioned operating profiles and temporal context. (a) Gaze dispersion and (b) the head-gaze stability relationship are conditioned by the stability gates in Eq. 2 . (c) Hand-presence comparisons are likewise conditioned by the visible-hand requirement; the spatial gaze to hand relationship is evaluated independently in Section 5.2 . (d) Event duration and pre-, during-, and post-event motion provide temporal context not used to define proposal eligibility. Panels (a) to (c) summarize the resulting detector-conditioned profiles. Four panels show detector-conditioned gaze concentration, head-gaze stability, hand-presence profiles, and descriptive event-duration and motion context.
Hypothesis
Independent evidence
Detector-conditioned characteristic
Conclusion
H1
Necessity was selected in 90.6% of accepted judgments. Median target width was 10.8∘ , approximately 9.8% of the 110∘ camera field of view, and 14.2% of targets were at most 5∘ .
Median gaze dispersion was 2.19∘ for PVA and 7.47∘ for controls, characterizing the stability profile induced by the proposal gate.
Task necessity and a functional target that is relatively small within the complete scene are supported.
H2
Stage 1 required concurrent IMU-based head stability and raw-gaze stability for every proposal, and it recalled all 820 scripted opportunities.
Stable gaze occurred in 70% of head-stable samples and 25% of head-unstable samples; the association was positive in 142 of 143 recordings.
H2 is operationally supported: the empirically effective stability gate identifies usable candidate intervals.
H3
C4 was selected in 77.4% of accepted judgments. Gaze-hand distance decreased from 13.2∘ for unanimously rejected events to 3.2∘ for unanimously accepted events; it was also lower for PVA than control intervals ( pHolm=2.4×10−17 ).
Hand-presence is required for proposal generation; stable-gaze comparisons by hand visibility are therefore descriptive.
Supported by spatial gaze-hand proximity and human endorsement, with hand presence serving as the proposal gate.
H4.1
Fixation duration was 0.840 s for PVA and 0.047 s for controls ( pHolm=1.9×10−23 ); event duration increased from 1.4 s to 4.8 s with consensus. Head angular rate changed from 0.49 to 0.22 to 0.52 rad/s (approach versus event: pHolm=8.1×10−24 ).
None.
Sustained attention and motion settling are supported.
H4.2
Reachability was selected in only 6.6% of accepted judgments and showed no significant association with event duration ( pHolm=0.633 ); these events were uncommon in AEA.
None.
The current data do not support H4.2; the low event frequency left the test underpowered, and a dedicated reachability protocol is needed.
Appendix
Table 3. Independent evidence and detector-conditioned characteristics for the four PVA hypotheses. Inferential (p)-values are reported only for measurements not used in candidate-generation and are corrected using the Holm procedure across the seven retained primary comparisons.
Figure 14. Exploratory ERP and PSD analysis of high- and low-visual-attention events. Left: Wearable recording setup combining Meta Aria Gen 1 AI glasses with a two-channel CGX EEG patch. Right: Trial-sorted, smoothed EEG amplitude maps and averaged ERP waveforms for high-visual-attention PVA events (563 trials) and low-visual-attention rejected candidates (208 trials) at AFz and Fp2. Time zero denotes the annotated event keyframe, and the black curves indicate the estimated latency lag used to order trials. Bottom: Participant-averaged PSD for the high- and low-attention conditions (nine participants each). These frontal measurements characterize event-related activity and are not used for cortical source localization. A participant wears Meta Aria glasses and a two-channel EEG patch. Four trial-sorted EEG heat maps and averaged ERP traces compare high- and low-visual-attention events at AFz and Fp2. Two PSD plots compare the conditions across nine participants.
Practical webcam gaze tracking is constrained not only by error, but also by calibration burden, robustness to head motion and session drift, runtime footprint, and browser use. We therefore target a deployment-oriented operating point rather than the image large-backbone regime. We cast landmark-based point-of-regard estimation as session-wise adaptation: a shared geometric encoder produces embeddings that can be aligned to a new session from a small calibration set. We present Equivariant Meta-Calibrated Gaze (EMC-Gaze), a lightweight landmark-only method combining an E(3)-equivariant landmark-graph encoder, local eye geometry, binocular emphasis, auxiliary 3D gaze-direction supervision, and a closed-form ridge calibrator differentiated through episodic meta-training. To reduce pose leakage, we use a two-view canonicalization consistency loss. The deployed predictor uses only facial landmarks and fits a per-session ridge head from brief calibration. In a fixation-style interactive evaluation over 33 sessions at 100 cm, EMC-Gaze achieves 5.79 +/- 1.81 deg RMSE after 9-point calibration versus 6.68 +/- 2.34 deg for Elastic Net; the gain is larger on still-head queries (2.92 +/- 0.75 deg vs. 4.45 +/- 0.30 deg). Across three subject holdouts of 10 subjects each, EMC-Gaze retains an advantage (5.66 +/- 0.19 deg vs. 6.49 +/- 0.33 deg). On MPIIFaceGaze with short per-session calibration, the eye-focused model reaches 8.82 +/- 1.21 deg at 16-shot calibration, ties Elastic Net at 1-shot, and outperforms it from 3-shot onward. The exported eye-focused encoder has 944,423 parameters, is 4.76 MB in ONNX, and supports calibrated browser prediction in 12.58/12.58/12.90 ms per sample (mean/median/p90) in Chromium 145 with ONNX Runtime Web. These results position EMC-Gaze as a calibration-friendly operating point rather than a universal state-of-the-art claim against heavier appearance-based systems.
Chenkai Zhang
Independent Researcher, Wenzhou, Zhejiang 325000, China
We present PicoEyes, a unified gaze estimation framework that directly predicts all key attributes of gaze, including 3D eye parameters, eye-region segmentation, optical axis, visual axis, and depth maps, from either monocular or binocular inputs. The framework simultaneously addresses calibration, gaze forecasting, and varying device postures, while also supporting 3D eye reconstruction via joint estimation of eye parameters and depth maps in an end-to-end manner. In addition, we introduce a large-scale multi-view near-eye dataset containing comprehensive 2D and 3D annotations under diverse conditions, including train, test, rewear-test, and calibration sessions. Extensive experiments demonstrate that PicoEyes achieves state-ofthe-art performance, consistently outperforming both academic and industrial gaze tracking methods across nocalibration, calibration, rewear-after-calibration, and forecasting settings. This work establishes a practical, end-toend paradigm for robust and generalizable gaze estimation in mixed reality (MR) applications.
Gaze estimation under natural head-eye motion underpins applications from driver monitoring to human-computer interaction. Single-frame methods predict each frame independently, so consecutive outputs fluctuate as jitter. Multi-frame methods reduce this, but they learn motion implicitly inside appearance features, so the gaze trajectory is never an explicit variable. We propose EyeTAG (Eye Trajectory-Aware Gaze Estimation), a causal multi-frame framework built around an explicit first-order gaze prior: at each step it differentiates its own recent predictions and feeds the resulting trajectory back as a compact kinematic token. Because differencing is translation-invariant in gaze space, this token carries subject-invariant motion rather than personal gaze offsets. Face and eye streams supply visual evidence, fused by cross-attention and a causal Transformer decoder. EyeTAG reduces the mean angular error by about 1.0∘ on Gaze360 and performs on par with the strongest baseline on EVE (2.56∘ vs. 2.58∘). Within-model ablations, which keep the encoder and the rest of the architecture fixed and vary only the gaze history, show that the differential formulation, rather than temporal context alone, removes the systematic saccade bias that persists even with an absolute gaze-history prior. Our code is available at https://github.com/peter8366/EyeTAG.
Jungmin Lee, Niamat Ullah, Yoseob Han
Department of Information and Telecommunication Engineering Soongsil University Seoul, Republic of Korea · Department of Electronic Engineering Soongsil University Seoul, Republic of Korea