Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs

Yiwei Li; Yifan Zhou; Huaqin Zhao; Zihao Wu; Zhengliang Liu; Xiang Li; Quanzheng Li; Tianming Liu

cs.CVMar 5, 2026

Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs

Authors: Yiwei Li, Yifan Zhou, Huaqin Zhao, Zihao Wu, Zhengliang Liu, Xiang Li, Quanzheng Li, Tianming Liu,

Organizations: School of Computing, University of Georgia · Massachusetts General Hospital, Harvard Medical School · Department of Biomedical Engineering, New Jersey Institute of Technology

Abstract

Vision--language models (VLMs) process images as visual tokens, yet their intermediate reasoning is often carried out in text, which can be suboptimal for visually grounded radiology tasks. Radiologists instead diagnose via sequential visual search; eye-tracking captures this process as time-ordered gaze trajectories that reveal how evidence is acquired over time. We use eye-gaze as supervision to guide VLM reasoning by introducing a small set of dedicated gaze tokens. These tokens are trained to predict gaze-selected image patch indices in temporal order, encouraging the model to follow human-like evidence acquisition and integration. Experiments on MIMIC-EYE and multiple external zero-shot benchmarks show consistent gains over baselines, achieving state-of-the-art in-domain performance and improved out-of-domain robustness. These results highlight temporally ordered gaze as an effective supervision signal for learning visually grounded medical reasoning.

Figures & tables

Figure 1: Method overview. We fine-tune a pretrained VLM with MIMIC-EYE by injecting gaze supervision as discrete patch indices. Stage 1 learns four dedicated gaze tokens via a lightweight projection head that predicts gaze-selected patch IDs (cross-entropy). Stage 2 adds a 14-label classifier head to predict radiographic findings (binary cross-entropy) while enforcing a strict fixed-format yes/no output.

Figure 2: Eye-gaze reasoning trajectories. Two MIMIC-EYE examples showing temporally ordered gaze heatmaps overlaid on the chest X-ray. Each sequence visualizes how attention evolves from Step 1 to Step 4, illustrating radiologists’ sequential evidence acquisition during interpretation.

Method	AUROC $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$
Vanilla	49.74	42.15	43.09
SFT	87.60	86.03	84.18
SFT-Heatmap	87.51	86.51	84.23
MedCLIP [ 28 ]	87.37	86.63	84.32
EGMA [ 17 ]	89.49	88.11	86.20
Random-Gaze	86.45	85.59	81.06

Table 1: In-domain performance on MIMIC-EYE. Area under ROC curve (AUROC), Accuracy (Acc.) and F1-score (F1) are reported. Best results are in bold and second-best results are underlined .

Method	CheXpert 5 $\times$ 200		RSNA		SIIM-ACR
Method	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$	Acc. $\uparrow$	F1 $\uparrow$
Vanilla	43.96	41.56	60.77	37.80	50.12	37.66
SFT	55.62	51.37	68.83	43.77	60.03	55.45
SFT-Heatmap	58.39	55.99	71.10	44.73	58.40	57.85
MedCLIP [ 28 ]	57.50	55.97	73.09	41.01	58.76	59.16
EGMA [ 17 ]	59.30	60.38	76.97	46.12	63.62	61.46

Table 2: Comparison results of zero-shot classification tasks on CheXpert 5

\times

200 [ 10 ] , RSNA [ 20 ] , and SIIM-ACR [ 32 ] datasets. Accuracy (Acc.) and F1-score (F1) are reported. Vanilla: Qwen2.5VL-7B. SFT: Qwen2.5VL-7B-SFT. SFT-Heatmap: Qwen2.5VL-7B-SFT with Eye-Gazed heatmap. Random-Gaze: replace gaze patch IDs with random indices, preserving list sizes. Shuffled-Gaze: permute the gaze lists. Best results are in bold and second-best results are underlined .

Radiology VLMs

Radiology vision-language models (VLMs) align medical images and text to support grounded diagnosis, reporting, retrieval, and clinical reasoning.

See all 109 papers

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering
Apr 10, 2026Suyang Xi, Songtao Hu, Yuxiang Lai +4Medical VQA Medical VLMs
Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA
Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models
Apr 10, 2026Oliver McLaughlin, Daniel Shubin, Carsten Eickhoff +3Medical VLMs VLM Reasoning
Brown University · University of Washington · University of Tübingen +1
GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI
Apr 25, 2026Duaa Alim, Mogtaba Alim, Liam ChalcroftVLM Evaluation Rare Disease Diagnosis
Imperial College London, UK · University of Toronto, Canada · University College London, UK

Apr 10, 2026J/K move · Enter open · S save

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Suyang Xi, Songtao Hu, Yuxiang Lai, Wangyun Dan+3

Department of Radiation Oncology and Winship Cancer Institute, Emory University School of Medicine, Atlanta, 30322, GA, USA · Department of Biostatistics and Bioinformatics, Emory University, Atlanta, 30322, GA, USA

Medical VQA Medical VLMs Latent Visual Reasoning RL for VLMs

Open paper AskPDF arXiv

Medical vision--language models (VLMs) have shown strong potential for medical visual question answering (VQA), yet their reasoning remains largely text-centric: images are encoded once as static context, and subsequent inference is dominated by language. This paradigm is fundamentally limited in clinical scenarios, where accurate answers often depend on subtle, localized visual evidence that cannot be reliably preserved in static embeddings. We propose MedLVR, a latent visual reasoning framework that introduces an explicit visual evidence state into autoregressive decoding. Instead of relying solely on text-based intermediate reasoning, MedLVR interleaves a short latent reasoning segment within the decoder by reusing hidden states as continuous latent steps, enabling iterative preservation and refinement of query-relevant visual evidence before answer generation. To support effective visual supervision, we adopt a two-stage training strategy: region of interest (ROI)-supervised fine-tuning aligns latent states with clinically relevant image evidence, and Visual-Latent Policy Optimization (VLPO) further optimizes latent reasoning and answer generation under outcome-level rewards. Experiments on OmniMedVQA and five external medical VQA benchmarks show that MedLVR consistently outperforms recent reasoning baselines and improves the average score over the Qwen2.5-VL-7B backbone from 48.3% to 53.4%. These results show that latent visual reasoning provides an effective mechanism for preserving diagnostically relevant visual evidence and improving the reliability of medical VQA.

Abstract

Radiology VLMs

Explore similar work

MedLVR: Latent Visual Reasoning for Reliable Medical Visual Question Answering

Is There Knowledge Left to Extract? Evidence of Fragility in Medically Fine-Tuned Vision-Language Models

GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI