cs.HCOct 17, 2026
SaveGazeAgent: VLM-Guided Implicit Gaze Calibration from Egocentric Multimodal Behavior for AI Glasses
Organizations: Department of Bioengineering, University of California San Diego · Swartz Center for Computational Neuroscience, University of California San Diego · Tsinghua University, University of California San Diego · Institute for Neural Computation, University of California San Diego ·
Abstract
AI glasses typically require explicit, user-specific eye-tracker calibration, and accuracy can drift when the device shifts during use. We present GazeAgent, a VLM-guided framework that continuously and adaptively improves gaze-calibration accuracy from naturally occurring behavior without asking users to interrupt their activities. GazeAgent identifies precise visual attention (PVA) events from multimodal streams of inertial measurement unit (IMU), gaze, and wearer-hand signals, validates their task relevance, and localizes functional targets as pseudo-calibration targets. In a 10-participant scripted study, the system detected all 820 predefined PVA opportunities and, under device disturbance, reduced median gaze error by 44.1% compared with explicit calibration. In a five-participant case study, autonomous calibration produced the lowest workload and interruption ratings, and all participants preferred it for routine use. By deriving task-critical targets from natural multimodal behavior, GazeAgent demonstrates the potential for continuous, adaptive, and low-burden gaze interaction on egocentric AI glasses.
Figures & tables
Figure 1. Overview of the GazeAgent implicit-calibration pipeline. A qualifying attention event provides a pseudo-calibration target, defined as the center of its functional region of interest (ROI), enabling estimation of the offset between the system-predicted gaze point and the intended target. Stage 1 combines head-motion stability, gaze stability, and wearer-hand evidence to generate candidate intervals and representative frames. Stage 2 uses human annotators and open-source and closed-source vision-language models (VLMs) to validate each candidate and localize its functional-target ROI; the ROI center is the single pseudo-calibration keypoint. Validated events are processed in successive batches to support continuous, one-step-ahead gaze calibration. Overview of the GazeAgent pipeline. A conceptual example illustrates the offset between the predicted gaze point and an event-based pseudo-calibration target. A multimodal candidate detector combines IMU, gaze, and hand signals. Human annotators and vision-language models then validate candidate events and localize their functional-target ROIs and center keypoints. Successive batches support adaptive implicit calibration.
| Accepted Events | Rejected Events | ||
|---|---|---|---|
| Peeling a tape strip | Applying toothpaste | Reaching for a large object | Reaching for a pillow |
| Accept | Accept | Reject | Reject |
| Target: Tape edge | Target: Toothbrush tip | Target: None | Target: None |
| H1: The tape edge is a spatially precise functional target. H2: The head and hands remain stable. H3: Both hands perform a fine peeling action. H4: The action involves sustained manipulation and an elevated reach. | H1: Toothpaste must be accurately placed on the toothbrush head. H2: The head, hands, tube, and toothbrush remain stable. H3: One hand stabilizes the toothbrush while the other squeezes the tube. H4: The action requires sustained attention and precise alignment. | H1: No spatially precise functional target is visible. H2: The user appears to be moving toward the object. H3: The hand performs a coarse reaching action. H4: The action is brief, familiar, and visually undemanding. | H1: The large target does not require precise foveal vision. H2: The frame shows repositioning rather than stable fixation. H3: The hand performs a coarse grasp. H4: The action is familiar and low precision. |
Table 1. Representative accepted and rejected examples of PVA events. Accepted events provide localized pseudo-calibration targets, whereas rejected events do not require precise foveal attention.
Figure 2. Overview of the GazeAgent workflow, which integrates multimodal behavioral streams, iterative VLM reasoning and tool use, evidence-grounded implicit calibration, and optional feedback-driven personalization. GazeAgent synchronizes egocentric video, motion, gaze, and hand signals; validates candidate PVA events through an iterative thought, action, and observation loop; performs evidence-grounded calibration; and can optionally collect feedback for personalization.
Figure 3. Example output of the high-recall multimodal PVA candidate detector. The left panel shows the representative keyframe, wearer-hand overlay, modality scores, and combined ranking score. The displayed gaze marker is used only for post-hoc validation and is removed from every frame supplied to the VLM. The right panel shows the synchronized head-motion, gaze-dynamics, and hand-activity signals. The green region denotes the candidate interval, and the orange line marks its highest-scoring keyframe. The detector proposes a possible event; the VLM subsequently determines its semantic validity without access to absolute gaze position. A representative egocentric keyframe is displayed beside synchronized plots of head motion, gaze stability, and hand-interaction signals. A green vertical band indicates the detected candidate interval, while an orange line identifies the selected representative keyframe.
Figure 4. Human annotation interface. Annotators selected a candidate from the script list and event strip, inspected its keyframe and context clip, drew a tight ROI around the functional target, and completed the criterion-level judgments. The point shown inside the ROI is its computed center and serves as the single reference keypoint. A screenshot of the annotation interface with labeled regions for the script list, keyframe, context clip, event-answer panel, event strip, and functional-target ROI.
Figure 5. Content and annotation characteristics of human-consensus PVA events. Results are reported for 621 candidates accepted by all five annotators. (a) Most frequent activity categories. (b) Functional target categories used as pseudo-calibration references. (c) Wearer-hand interaction types. Fine target-directed actions include tapping, pinching, pressing, aligning, inserting, peeling, and precise placement. (d) Annotation-criterion selection rates among accepted judgments. Activity, target, and interaction categories are mutually exclusive within their respective panels; multiple annotation criteria could be selected for the same event. Four horizontal bar charts summarize the activity categories, functional targets, hand-interaction types, and annotation criteria of 621 human-consensus PVA events.
Figure 6. Spatial reliability of the human-annotated pseudo-targets. Five annotators independently marked the functional target in each accepted PVA event. (a) Angular offsets from the leave-one-out annotator consensus. The median offset was , approximately one foveal diameter, while the 95th-percentile offset was . (b) Cumulative distributions of annotator-to-consensus and pairwise annotator disagreement; median pairwise disagreement was . (c) Annotation disagreement relative to the foveal diameter, target width, camera field of view, and calibration-error scales. (d) Residual correction bias decreases as more pseudo-targets are aggregated, from for one target to for eight and for twenty. (e) Median disagreement between each annotator pair ranged from to . (f) Annotated points were substantially more concentrated than their corresponding ROIs: the median point spread was 5.9% of the target extent. Six panels quantify agreement among five annotators, compare annotation error with relevant visual-angle scales, and show how aggregating pseudo-targets reduces residual correction bias.
Figure 7. Performance of closed-source and open-source VLMs on PVA validation and target localization. Left: Task 1 PVA recall across event conditions; darker cells indicate higher recall. Right: Task 1 validation accuracy versus Task 2 ROI-localization AP. Circles and diamonds represent closed-source and open-source models, respectively. Task 2 AP is conditional on correct Task 1 classification. A heatmap of condition-level PVA recall and a scatter plot comparing PVA-event validation accuracy with ROI-localization average precision across nine vision-language models.
Figure 8. Experimental protocol and representative PVA events across two scripted routes. Participants completed two continuous multimodal recordings while egocentric RGB, eye gaze, IMU, and frontal electroencephalography (EEG) were recorded. Script 1 covered four spaces and contained 21 PVA events, whereas Script 2 covered three spaces and contained 20 events. Both scripts included seven matched pen-and-paper tasks. The representative egocentric frames illustrate PVA-relevant activities across locations, hallway transitions, and hand-object interactions. Each script was performed once under a control condition and once under a disturbance condition, during which the glasses were adjusted, participants jumped or jogged, and the device was removed and re-worn at predefined points. Diagram of two experimental routes. Script 1 proceeds through the front door, a room, the library, and the lounge; Script 2 proceeds through the library, kitchen, and lounge. Both include a shared pen-and-paper block. Eight egocentric images show representative tasks, followed by timelines for the control and device-disturbance conditions.
Figure 9. Gaze-estimation accuracy for the general SocialEye CNN, explicit calibration, and implicit calibration. (a) Angular error distributions under control and disturbance conditions. (b) Error histograms, with red dashed lines indicating method-specific two-standard-deviation thresholds. (c) Target and predicted gaze directions in yaw-pitch coordinates under the control condition. (d) Batch-level angular errors. Batch 1 initializes the implicit model; implicit-calibration results from Batch 2 onward are one-step-ahead evaluations using the model fitted on the preceding batch. All implicit updates use VLM-predicted ROI centers. Four panels compare three gaze-estimation methods using condition-level boxplots, error histograms, yaw-pitch prediction plots, and batch-level error trends.
Figure 10. GazeAgent interface and feedback-driven personalization. (a) The mobile interface defers low-confidence queries until a natural pause, presents the candidate PVA event and model prediction, and allows the participant to confirm the prediction, reject the event as non-PVA, or correct the functional target by selecting its precise location. (b) Changes in Task 1 PVA-event validation, Task 2 ROI-localization, and median angular gaze error as additional user-reviewed events are incorporated into personalization. Thick green curves show the participant mean with personalization, thin green curves show individual participant trajectories, shaded regions indicate the participant range, and dashed orange lines show matched performance without personalization. The upper panel shows three mobile interface screens for recording a session, reviewing a candidate PVA event, and correcting its target location. The lower panel shows three adaptation plots comparing performance with and without personalization as the number of user-reviewed events increases.
| Measure | Aria explicit calibration | Autonomous implicit calibration | Personalized feedback |
|---|---|---|---|
| NASA-TLX, median [IQR], 0 to 100 | 47.5 [46.7 to 55.0] | 41.7 [40.8 to 44.2] | 45.8 [43.3 to 48.3] |
| Interruption rating, median [IQR], 1 to 7 | 5 [4 to 6] | 1 [1 to 2] | 3 [2 to 4] |
| Preferred for routine use | 0/5 | 5/5 | 0/5 |
| Main perceived benefit | Predictable and transparent | No explicit action required | Allows users to correct errors |
| Main concern | Interrupts the ongoing task | Background updates are less visible | Queries still require attention |
Table 2. Within-subject workload, interruption, and preference across the three calibration conditions ( ). Quantitative values are reported as median [interquartile range, IQR]. Lower NASA Task Load Index (NASA-TLX) and interruption scores indicate lower burden; bold values mark the most favorable condition.
Figure 11. Representative pre-fine-tuning VLM validation errors on AEA. (a) The model attributes another person’s pouring action to the wearer; all five annotators rejected the event. (b) The model treats eating as a routine, coarse action and misses the precise fork-to-food contact point. (c) The model evaluates the entire coffee-maker reservoir rather than its functional opening. All five annotators accepted the events in (b) and (c). Three egocentric AEA frames illustrating one false-positive and two false-negative PVA validation decisions.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12. Image-grounded few-shot exemplars used for Task 1 validation. The validation prompt includes two accepted and two rejected examples from recordings different from the candidate under evaluation. The accepted examples require precise alignment with a small functional target: the measuring-cap opening in (a) and the item pinched between the fingertips in (b). The rejected examples show a coarse interaction with a broad door panel in (c) and a manipulation performed by another person rather than the wearer in (d). Four egocentric few-shot example frames arranged in two rows. The top row contains accepted PVA examples: pouring detergent into a small cap and pinching a small item. The bottom row contains rejected examples: pushing a broad door panel and watching another person manipulate a fruit container.
Figure 13. Detector-conditioned operating profiles and temporal context. (a) Gaze dispersion and (b) the head-gaze stability relationship are conditioned by the stability gates in Eq. 2 . (c) Hand-presence comparisons are likewise conditioned by the visible-hand requirement; the spatial gaze to hand relationship is evaluated independently in Section 5.2 . (d) Event duration and pre-, during-, and post-event motion provide temporal context not used to define proposal eligibility. Panels (a) to (c) summarize the resulting detector-conditioned profiles. Four panels show detector-conditioned gaze concentration, head-gaze stability, hand-presence profiles, and descriptive event-duration and motion context.
| Hypothesis | Independent evidence | Detector-conditioned characteristic | Conclusion |
|---|---|---|---|
| H1 | Necessity was selected in 90.6% of accepted judgments. Median target width was , approximately 9.8% of the camera field of view, and 14.2% of targets were at most . | Median gaze dispersion was for PVA and for controls, characterizing the stability profile induced by the proposal gate. | Task necessity and a functional target that is relatively small within the complete scene are supported. |
| H2 | Stage 1 required concurrent IMU-based head stability and raw-gaze stability for every proposal, and it recalled all 820 scripted opportunities. | Stable gaze occurred in 70% of head-stable samples and 25% of head-unstable samples; the association was positive in 142 of 143 recordings. | H2 is operationally supported: the empirically effective stability gate identifies usable candidate intervals. |
| H3 | C4 was selected in 77.4% of accepted judgments. Gaze-hand distance decreased from for unanimously rejected events to for unanimously accepted events; it was also lower for PVA than control intervals ( ). | Hand-presence is required for proposal generation; stable-gaze comparisons by hand visibility are therefore descriptive. | Supported by spatial gaze-hand proximity and human endorsement, with hand presence serving as the proposal gate. |
| H4.1 | Fixation duration was 0.840 s for PVA and 0.047 s for controls ( ); event duration increased from 1.4 s to 4.8 s with consensus. Head angular rate changed from 0.49 to 0.22 to 0.52 rad/s (approach versus event: ). | None. | Sustained attention and motion settling are supported. |
| H4.2 | Reachability was selected in only 6.6% of accepted judgments and showed no significant association with event duration ( ); these events were uncommon in AEA. | None. | The current data do not support H4.2; the low event frequency left the test underpowered, and a dedicated reachability protocol is needed. |
Table 3. Independent evidence and detector-conditioned characteristics for the four PVA hypotheses. Inferential (p)-values are reported only for measurements not used in candidate-generation and are corrected using the Holm procedure across the seven retained primary comparisons.
Figure 14. Exploratory ERP and PSD analysis of high- and low-visual-attention events. Left: Wearable recording setup combining Meta Aria Gen 1 AI glasses with a two-channel CGX EEG patch. Right: Trial-sorted, smoothed EEG amplitude maps and averaged ERP waveforms for high-visual-attention PVA events (563 trials) and low-visual-attention rejected candidates (208 trials) at AFz and Fp2. Time zero denotes the annotated event keyframe, and the black curves indicate the estimated latency lag used to order trials. Bottom: Participant-averaged PSD for the high- and low-attention conditions (nine participants each). These frontal measurements characterize event-related activity and are not used for cortical source localization. A participant wears Meta Aria glasses and a two-channel EEG patch. Four trial-sorted EEG heat maps and averaged ERP traces compare high- and low-visual-attention events at AFz and Fp2. Two PSD plots compare the conditions across nine participants.
More in this topic
Eye Tracking
Eye tracking measures where people look and how their eyes move to study visual attention, behavior, and gaze-controlled systems.
See all 61 papers