cs.CVFeb 18, 2026
SaveIRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models
Organizations: Department of Computer Science, UC Santa Barbara · Department of Psychological & Brain Sciences, UC Santa Barbara · DEVCOM Army Research Laboratory
Abstract
We introduce IRIS (Intent Resolution via Inference-time Saccades), a novel training-free approach that uses eye-tracking data in real-time to resolve ambiguity in open-ended VQA. Through a comprehensive user study with 500 unique image-question pairs, we demonstrate that fixations closest to the time participants start verbally asking their questions are the most informative for disambiguation in Large VLMs, more than doubling the accuracy of responses on ambiguous questions (from 35.2% to 77.2%) while maintaining performance on unambiguous queries. We evaluate our approach across state-of-the-art VLMs, showing consistent improvements when gaze data is incorporated in ambiguous image-question pairs, regardless of architectural differences. We release a new benchmark dataset to use eye movement data for disambiguated VQA, a novel real-time interactive protocol, and an evaluation suite.
Figures & tables
Figure 1: IRIS overview . Participant asks an ambiguous question about an image while their eyes are being tracked. The VLM uses the fixation data (marked as a white cross) to disambiguate the query and provide an accurate response in real-time.
Figure 2: Experimental procedure. A central fixation check was enforced, after which participants freely viewed each image and asked any question aloud about it. Once 1.5s of silence elapsed following the question, the VLM was prompted with (i) the image, (ii) the transcribed question, and (iii) the same image with fixation data superimposed. Finally, participants reported the object they queried about (location of interest) by clicking the corresponding region of the image.
Figure 3: Qualitative results showing successful (A) and failed (B) disambiguation using gaze data . Black circles with white crosses mark temporally and spatially filtered fixation locations.
Figure 4: Temporal-spatial filtering of eye gaze data. Fixations colored by time. Black lines on the color bar mark speech onset and end. Diamonds represent fixations within ±1s of speech onset; all others are circles. Any diamond within 2 dva of the diamonds’ median (red +) is spatially filtered and rendered as white crosses on black circles and passed onto IRIS.
Figure 5: Temporal dynamics of gaze informativeness for ambiguous questions. (A) Model performance increases as the temporal window expands around speech onset time , converging to the "all-fixations" baseline similarity at ±4500ms around speech onset. The bottom panel shows decreasing distance between the fixation median and the Location of Interest (LOI) in dark blue and increasing fixation count with larger windows in red. (B) Peak performance a few milliseconds before speech onset is revealed in sliding window analysis (600ms window-width, 400ms sliding step-size), aligning with minimum fixation-to-LOI distance. Gray shaded region indicates the interquartile range of speech end times. Error bars represent SEM. (A) and (B) top: green - "LOI (perfect gaze)" upper bound; orange - "all-fixations" baseline; purple - "image-only" baseline; pink - "wrong answer" lower bound (see Section 2.7 for details).
Figure 6: Effect of gaze input on performance compared to Image-Only and Image with Location of Interest (LOI). (A) By adding gaze to the input, accuracy increases from 35.2% to 77.2%, ) for ambiguous questions, while remaining non-significant for unambiguous questions ( ). (B) Semantic similarity shows similar significant improvements only in ambiguous conditions. Error bars represent SEM shown on the upper side.
Figure 7: Semantic similarity to ground truth across VLMs . Models above the diagonal line show improvement with the inclusion of eye gaze data in ambiguous trials.
| Model | Image Only (%) | Image+Gaze (%) | (pp) | -value |
|---|---|---|---|---|
| GPT-5 Mini | 49.7 | 83.0 | +33.2 | 0.001 |
| Gemini 2.5 Flash | 54.2 | 83.3 | +29.0 | 0.001 |
| Gemini 2.5 Pro | 59.3 | 82.3 | +23.0 | 0.001 |
| GPT-5 | 53.7 | 76.5 | +22.8 | 0.001 |
| Claude Sonnet 4 | 54.2 | 74.0 | +19.8 | 0.001 |
| Ovis 2.5 9B | 52.5 | 69.5 | +17.0 | 0.001 |
Table 1: VLM Accuracies on ambiguous trials. “Image+Gaze” uses fixations from ±1s around speech onset; is the difference in percentage points. -values test Image+Gaze vs. Image-Only .
| Method | Similarity |
|---|---|
| Crosses (Ours) | 0.830 0.020 |
| Heatmap | 0.820 0.019 |
| Bounding Box | 0.780 0.021 |
| Coordinates (Text) | 0.748 0.022 |
| Cropped | 0.703 0.023 |
Table 2: Ablation study on gaze representation methods. Semantic similarity to ground truth using GPT-5-Mini on ambiguous trials. Our cross-marker approach outperforms alternative representations.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Sample unambiguous and ambiguous questions asked by four participants on four example images.
Figure 9: Visualizing and filtering fixations: examples of "successful" ambiguous trials. Inclusion of gaze data successfully disambiguated the object in question. The colored circles and diamonds are fixations shaded by their temporal order (yellow - more recent, blue - older). The magenta circle represents the mouse click location registered by the participant at the end of the trial. The red dashed circle indicates a region of 2 dva radius around the median of all fixations within ±1s around speech onset (shown as diamonds). Any diamond within the red dashed circle was sent to the VLM as eye gaze data. The x and y-axes indicate distance to the mouse click location. The times at which the participant started and ended their question are marked as black lines on the color bar. The fixation markers are enlarged for illustrative purposes.
Figure 10: Visualizing and filtering fixations: examples of "unsuccessful" ambiguous trials. Inclusion of eye gaze data did not successfully disambiguate the object in these trials. Plotting conventions follow those described in Figure 9 .
Figure 11: Effect of spatial filtering. Visualizing ambiguous Image+Gaze trials where spatial filtering resulted in zero retained fixations. In these examples, removing the spatial filter (but retaining the temporal filter) led to accurate disambiguation (2% trials). In other words, all fixations marked as diamonds were sent to the VLM during inference. Plotting conventions follow those described in Figure 9 .
Figure 12: Participants’ speech (questioning) behavior. The left panel shows the mean time taken by participants to formulate and ask a question from the time of stimulus onset. The right panel shows the mean duration between speech onset and end, as a function of experiment condition. The error bars indicate the standard error of the mean.
Figure 13: Fixation metrics. From left to right, the panels show average fixation count, fixation duration, and first fixation latency for Unambiguous and Ambiguous conditions. The error bars indicate the standard error of the mean.
Figure 14: Types of errors. Example trials showing Inappropriate Question, Detection Error, Detail/Hallucination Error, Referent Bias, and Eye Data Error (discussed in Section A.4 in detail). The bottom right panel shows an example of a trial where none of the types of errors occurred. The red cross in each image indicates the mouse click location, depicting the object asked about in a given trial.
Figure 15: Quantification of error types. Proportion of trials that suffered from the error types discussed above and exemplified in Figure 14 . The three rows show data from Image+LOI, Image+Gaze, and Image Only input categories from top to bottom, respectively. The four columns show data from Unambiguous and Ambiguous conditions, each separated into subject-wise and across-subject measures. InapptQ - Error Type 1 (Inappropriate Question), Detection - Error Type 2, Detail - Error Type 3A (Detail/Hallucination Error), Referent Bias - Error Type 3B, Eye Data - Error Type 4.
More in this topic
Visual Question Answering
Visual Question Answering (VQA) develops systems that answer natural-language questions about images, videos, and visual documents.
See all 253 papers