cs.CVNov 17, 2026

Learning Grounded Reasoning from Human Gaze

Authors: Gabriel Sarch, Declan Campbell, Nicholas Budny, Kia Ghods, Katerina Fragkiadaki, Jonathan D. Cohen, Brenden M. Lake, Taylor W. Webb

Organizations: Princeton University · Perceptron AI · Carnegie Mellon University

Abstract

Human visual reasoning is driven by systematic exploration and extraction of task-relevant visual evidence. Human gaze provides a window into this process, making gaze a useful source of guidance for visually-grounded reasoning in vision-language models (VLMs). However, human gaze also includes noise and redundancies that may transfer poorly to VLMs. In this work, we investigate how human gaze data can be effectively leveraged to improve visual reasoning in VLMs, while maintaining alignment with human behavior. We collect 7,109 human gaze trials from a diverse and challenging set of visual reasoning tasks, and train grounded reasoning models with different forms of gaze guidance. Across evaluations in chart understanding, visual search, and counting, we find that using human gaze as a prior for grounded reasoning consistently outperforms supervision using the full gaze sequence. Based on these findings, we introduce GGR (Gaze-Guided Reasoner), which achieves greater human alignment and more reliable visual grounding while matching or outperforming leading grounded-reasoning baselines in accuracy. Reinforcement learning further improves task accuracy, gaze alignment, and grounding quality despite using no gaze supervision. Behavioral analyses show that gaze supervised models exhibit more visual exploration before arriving at an answer. Our results suggest that the value of human gaze lies in guiding spatial exploration and shaping reasoning behavior without requiring complete imitation of human behavior.

Explore similar work

CardsList
  1. Thinking with Visual Grounding

    Jun 15, 2026Junkai Zhang, Yihe Deng, Kai-Wei Chang +1Visual Spatial ReasoningVision-Language Grounding

  2. Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs

    Mar 5, 2026Yiwei Li, Yifan Zhou, Huaqin Zhao +6Radiology VLMsMedical VLMs

  3. GazeVLM: Active Vision via Internal Attention Control for Multimodal Reasoning

    May 8, 2026Brown Ebouky, Gabriele Carrino, Niccolo Avogaro +3Active PerceptionVLM Reasoning