cs.HCApr 9, 2026
SaveFrom Gaze to Guidance: Interpreting and Adapting to Users' Cognitive Needs with Multimodal Gaze-Aware AI Assistants
Organizations: Microsoft Research Cambridge, MA, USA · MIT Media Lab Cambridge, MA, USA · Microsoft Research Cambridge, Massachusetts, USA
Abstract
Current LLM assistants are powerful at answering questions, but they have limited access to the behavioral context that reveals when and where a user is struggling. We present a gaze-grounded multimodal LLM assistant that uses egocentric video with gaze overlays to identify likely points of difficulty and target follow-up retrospective assistance. We instantiate this vision in a controlled study (n=36) comparing the gaze-aware AI assistant to a text-only LLM assistant. Compared to a conventional LLM assistant, the gaze-aware assistant was rated as significantly more accurate and personalized in its assessments of users' reading behavior and significantly improved people's ability to recall information. Users spoke significantly fewer words with the gaze-aware assistant, indicating more efficient interactions. Qualitative results underscored both perceived benefits in comprehension and challenges when interpretations of gaze behaviors were inaccurate. Our findings suggest that gaze-aware LLM assistants can reason about cognitive needs to improve cognitive outcomes of users.
Figures & tables
Figure 1. Overview of the gaze-aware cognitive AI assistant. (a) A user wearing wearable AI glasses which streams eye gaze data in to a gaze-aware cognitive LLM assistant. (b-c) An example of the AI providing assistance after reading a difficult book, (d) User study evaluating AI interaction and accuracy. Four photographs illustrating the gaze-aware cognitive assistant prototype and study setup. (a) A participant wearing smart glasses looks thoughtful; text notes that physiological data such as egocentric video and eye tracking are captured, and the LLM assistant interprets needs. (b) The system identifies the user’s gaze point on a textbook while the assistant interprets needs from gaze behavior. (c) The assistant provides retrospective support, asking if the user would like an analogy after detecting difficulty with genetic algorithms. (d) Another participant, wearing glasses and headphones, sits at a computer during a user study measuring comprehension.
Figure 2. System overview of the gaze-aware AI assistant. Diagram of the gaze-aware AI assistant. (a) A pipeline shows how egocentric images with gaze points are processed through LLM-based gaze object recognition to generate a list of gaze objects, which are then analyzed by the LLM. (b) Illustration of a participant wearing Aria glasses with speakers and a microphone while reading a difficult text. The AI assistant detects repeated gaze on a word and asks, “You were repeatedly going back to this word. Do you want me to explain it?”
Figure 3. Real examples of gaze-reading behavior, the object list generated from LLM object recognition, the LLM eye tracking analysis with the object list as input, and a log of the resulting AI interaction. Composite figure showing how gaze data informs AI support. On the left, six sequential screenshots (1–6s) illustrate a participant’s gaze as they read text on a monitor. In the middle, a gaze object list logs words or objects fixated on, such as “asthenosphere,” with timestamps. Next, an LLM eye tracking analysis summarizes behaviors: fixations, regressions, off-text pauses, and an assessment noting difficulty with the asthenosphere’s physical state. On the right, a conversation log shows the AI asking if the user wants to clarify how the asthenosphere can be both solid and flowing, and the user confirming difficulty with lithosphere–asthenosphere interaction before the AI provides an explanation.
Figure 4. Bar plots of results for main variables across conditions for (a) learning performance, (b) LLM analysis ratings, and (c) the number of words uttered per interactions by each user, with standard errors. * , ** , *** . Three sets of bar charts comparing control and gaze-aware AI assistant conditions. (a) Learning performance: recall and definition probe scores are higher with the gaze-aware AI assistant, while concept inventory scores are slightly higher but closer across conditions. (b) Understanding: perceived understanding ratings are similar between conditions. (c) Perceived analysis accuracy: participants rated the gaze-aware AI assistant higher for perceived accuracy and personalization, and more participants expressed a preference for the gaze-aware AI assistant over control. Error bars indicate variability.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Item Type | Topic | Question Stem | A | B | C | D |
|---|---|---|---|---|---|---|
| Recall | Inflation | Inflation refers to the rate at which … | wages fall | money supply shrinks | prices rise* | debt grows |
| Recall | Plate Tectonics | The layer beneath the lithosphere is called the … | lower crust | asthenosphere* | outer core | biosphere |
| Definition Probe | Inflation | An inflation hawk is … | trader buying gold in recessions | citizen anti-food-tax lobby | policymaker prioritizing low inflation even over growth* | energy market index |
| Definition Probe | Plate Tectonics | In geology, mafic rocks are … | rich in Mg and Fe (dark)* | silica-rich and light-colored | frozen seawater crystals | formed only by asteroid impacts |
| Concept Inventory | Inflation | Cost-push inflation arises chiefly when … | demand outpaces supply | input costs (e.g., wages/oil) rise* | taxes are cut | exports surge |
| Concept Inventory | Plate Tectonics | Plate motion is primarily driven by … | Earth rotation | mantle convection and slab pull* | lunar tides | magnetic storms |
Table 1. Samples of multiple-choice assessment items by inventory type (Recall, Definition Probe, Concept Inventory) for the main study topics. Correct options are marked with an asterisk (*). Full items can be found in [GitHub repository link anonymized for review]
Figure 5. Exploratory results showing NASA TLX across conditions, with standard errors. NASA TLX workload ratings: the gaze-aware AI assistant shows lower scores than control for effort, frustration, mental demand, and temporal demand, with performance rated about equal. Error bars represent variability.
Figure 6. Exploratory results for sanity check questions and HLMIQ responses. A series of eight bar charts comparing control (gray) and experimental (orange) conditions on subjective ratings, including perceived usefulness, cognitive understanding, reaction awareness, timing, helpfulness, ease, enjoyment, satisfaction, and responsiveness. Ratings are generally similar across conditions, with small variations: the experimental condition is slightly higher on some dimensions (e.g., reaction awareness, satisfaction) and slightly lower on others. Error bars represent standard errors.
Figure 7. Rated accuracy of AI analysis in phase 3 between control and experimental differed based on text difficulty, with the easier texts showing bigger differences between conditions.
Figure 8. Conceptual overview of system user experience with AI interpretations of gaze behavior. Top row: What is captured with the eye tracking and front-facing cameras. Middle: LLM analysis about the user’s current cognitive-emotional reactions to the content. Bottom: Suggested AI interventions. Timeline of eye-gaze based reading analysis with AI feedback. At 5 seconds, the system detects the user is in the flow of reading; at 8 seconds, no change is made. At 30 seconds, the user looks away after reading “double slit,” interpreted as heavy thinking. At 34 seconds, the user is back in the flow of reading. At 40 seconds, no change is made. At 60 seconds, the user skips back and forth between words, suggesting fatigue. The AI assistant responds contextually: offering to explain “double slit” more simply and later suggesting a break.
Figure 9. Overview of study procedure and timing across phases. Flow diagram of the experimental procedure. Participants enroll and wear Aria glasses. In Condition 1, they read a passage followed by a gaze-informed AI conversation, then complete recall, understanding, concept, and user experience surveys. In Condition 2, they read another passage followed by a text-only informed AI conversation, then complete the same surveys. Finally, all participants complete post-experiment surveys during the debrief.