cs.AIApr 15, 2026
SaveSeeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning
Organizations: Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK · National Heart and Lung Institute, Imperial College London, London SW7 2AZ, UK · Cardiovascular Research Centre, Royal Brompton Hospital, London SW3 6NP, UK · School of Biomedical Engineering & Imaging Sciences, King’s College London, London WC2R 2LS, UK
Abstract
Large scale vision language models have shown promise in automating chest Xray interpretation, yet their clinical utility remains limited by a gap between model outputs and radiologist reasoning. Most systems optimize for semantic information without emulating how experts visually examine medical images, often overlooking critical findings or diverging from established diagnostic workflows. Radiologists follow structured protocols (e.g., the ABCDEF approach) that ensure all clinically relevant regions are systematically examined, reducing missed findings and supporting reliable diagnostic reasoning. We introduce GazeX, a vision language model that leverages radiologists' eye tracking data as a behavioral prior to model expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, GazeX learns to follow the spatial and temporal structure of radiologist attention and integrates observations in a clinically meaningful sequence. Using a curated dataset of over 30,000 gaze key frames from five radiologists, we demonstrate that GazeX produces more accurate, interpretable, and expert consistent outputs across radiology report generation, disease grounding, and visual question answering, utilizing 231,835 radiographic studies, 780,014 question answer pairs, and 1,162 image sentence pairs with bounding boxes. Unlike autonomous reporting systems, GazeX produces verifiable evidence artifacts, including inspection trajectories and finding linked localized regions, enabling efficient human verification and safe human AI collaboration. Learning through expert eyes provides a practical route toward more trustworthy, explainable, and diagnostically robust AI systems for radiology and beyond.
Figures & tables
Figure 1: Radiologist-guided diagnostic reasoning patterns for structured chest X-ray interpretation. a , Radiologists’ gaze behaviors demonstrate structured and reproducible diagnostic workflows. Eye-tracking captures fixation trajectories from multiple experts, which reveal common attention patterns aligned with established reading protocols of clinicians. Statistical analysis of inter-expert trajectories shows consistent correlations, highlighting shared visual strategies during chest X-ray interpretation. Enabling large vision–language models to learn such standardized reading workflows has the potential to enhance the performance across diverse clinical downstream tasks. b , Attention-guided visual observations align with clinically defined categories (i.e., airways, breathing, circulation, diaphragm, external devices, and foreign materials), thereby providing a semantic mapping between image features (visual clues from different regions within the examined area) and diagnostic reasoning. c , Visual clues from key anatomical regions are integrated during multi-step diagnostic processes to support structured report generation. d , Downstream tasks such as segmentation, visual question answering (VQA), and report generation benefit from the model’s alignment with expert gaze. This figure illustrates how expert eye movement data capture the temporal and spatial logic of radiological decision-making, serving as supervision signals to guide AI models toward clinically grounded and interpretable report generation. e , Clinician-in-the-loop workflow enabled by GazeX. GazeX provides inspection cues and finding-linked regions to support systematic review and human verification, allowing radiologists to verify or edit model suggestions before finalizing the report.
Figure 2: Evaluating GazeX performance on chest radiology report generation, visual question answering and visual grounding. a , Comparison of different report generation methods across various evaluation metrics in MIMIC-CXR [ 24 ] and IU X-ray [ 11 ] . The shown results include evaluations on the MIMIC-CXR testing set using the GazeX model fine-tuned on MIMIC-CXR, zero-shot evaluations on the entire IU X-ray dataset using the GazeX model fine-tuned on MIMIC-CXR, and evaluations on the IU X-ray testing set using the model fine-tuned on the IU X-ray dataset. The Wilcoxon signed-rank test was performed to assess the statistical significance of the improvements compared to other models. ** indicates , * indicates , and ** indicates the reverse situation. b , Radar chart showing foundation models performance on different perspective of the visual question answering dataset, including model size, overall performance, localization, qualitative and quantitative diagnosis aspect and presence diagnosis aspect. c , Quantitative analysis, including mean Intersection over Union (mIOU) and mean Average Precision (mAP), of the accuracy of the visual grounding task across all disease categories for different model types (e.g., foundation models, CLIP-based models and generative models). Additional quantitative comparisons are provided in Supplementary Tables 1-3 .
Figure 3: Consistent inspection patterns among radiologists. a , Total time spent per radiologist across a grid of patches, illustrating consistent fixation patterns and dwell times, with the central thoracic regions receiving the greatest attention. b , Similarity of fixation trajectories quantified with Pearson correlation for the horizontal and vertical coordinates, indicating moderate spatial agreement. c , Total time spent per radiologist across a grid of patches, stratified by representative case distributions across clinical categories. d , Case-level dispersion of fixation coordinates along both axes, revealing comparable spatial variability when attention is directed to similar anatomical regions.
Figure 4: GazeX emulates radiologists’ inspection patterns across spatial, temporal, and disease-specific contexts. a , Scatter plots showing agreement between GazeX predicted gaze centroids and those from radiologists, quantified by intraclass correlation coefficients (ICC) for X and Y coordinates under two evaluation settings. b , Pairwise cosine similarity of attention maps derived from gaze clusters, comparing GazeX with each radiologist and inter-radiologist pairs. High similarity values indicate that the model replicates individual viewing behaviors with fidelity comparable to expert–expert agreement. c , Disease-specific similarity analysis across the 14 CheXbert categories. The fused category heatmap was computed by averaging the heatmaps from five radiologists. For most conditions, GazeX achieves correlation levels on par with, or approaching, the highest inter-radiologist similarity, and remains consistent with aggregated radiologist patterns.
Figure 5: Case-level comparison of disease-specific inspection patterns between GazeX and radiologists. Representative examples from five CheXbert disease categories showing model-generated inspection sequences (right) alongside corresponding radiologists’ inspection patterns (middle).
Figure 6: Quantitative evaluation of GazeX against baseline and ablated variants. a , Correlation between predicted and ground-truth gaze centroids for GazeX and the baseline Qwen2-VL without eye-tracking guided pretrain, reported as Pearson correlation coefficients (r) with corresponding p-values. Results are shown for all disease categories combined and for selected clinically significant categories. b , Ablation study results. The top row reports classification metrics (Precision, Recall, and F1) for GazeX without LoRA adaptation. The bottom row reports report-generation metrics (BLEU-1, BLEU-4) and clinical correctness (CheXbert F1) for fine-tuned GazeX with LoRA adaptation, under progressive removal of core components (FVP, GTM, and SDA). c , Condition-specific ablation results for 12 representative medical categories spanning lung, pleura, heart/mediastinum, bone/chest wall, and device-related findings. The radial plot summarizes the overall relative performance of GazeX before downstream task adaptation across disease categories, excluding Support Devices. Bars indicate the comparison between all ablated modules for GazeX and LoRA fine-tuned GazeX for the report generation task.
Figure 7: Case study analysis demonstrating the advantages of inspection-guided modeling in GazeX. a , Comparison of radiologists’ gaze clusters and GazeX-predicted gaze clusters on the same case. Colored bounding boxes indicate clinically relevant findings, with “Missing” labels marking unobserved findings. b , Alignment between GazeX-generated reports and individual radiologists’ reports. Colored highlights indicate matched findings; “Observed by” and “Missed by” tags quantify agreement with each radiologist. c , Quantitative comparison of false positives (FP), false negatives (FN), and true positives (TP) for GazeX, Qwen2-VL baseline, and individual radiologists, using majority-vote consensus reports as reference. d , Qualitative error analysis comparing baseline and GazeX reports. e , Visual question answering comparisons across multiple medical vision–language models. f , Localization performance for descriptive findings, with predicted bounding boxes and Intersection-over-Union (IoU) values against ground-truth annotations.
Figure 8: Overview of the GazeX framework and its components. a , Preliminaries: The REFLACX dataset provides synchronized radiology reports, frontal chest X-ray images, and eye-tracking data. Gaze clusters are extracted from fixation points, with each cluster characterized by its centroid and bounding box. These are assembled into cluster attention videos (per finding) and a summary attention video (aggregating all findings). b , GazeX architecture: Built on the Qwen2-Vision-Language backbone with optimized structured text encoder, vision encoder, and multimodal projector. c , Core modules: Fine-grained Visual Perception (FVP): Maps textual findings to gaze clusters, centroids, and bounding boxes via medical reasoning, enabling local-level spatial grounding. Gaze Trajectory Mimicking (GTM): Trains the model to follow radiologist-like inspection patterns using both cluster attention videos and summary attention videos. Sequential Dependency Awareness (SDA): Recovers the correct temporal order of shuffled attention videos, fostering global-level interpretive reasoning.
Figure 9: Illustrative example of the data processing pipeline. Given a chest X-ray image, following the pipeline defined by the pseudocode Algorithm 1 in supplementary data, gaze-guided attention videos are generated through three phases for model training. Phase 1: Acquire radiologist gaze records and extract fixations based on their verbal descriptions of the regions of interest. Phase 2: For each specific verbal description, apply Gaussian attention map to the corresponding fixation data, followed by DBSCAN clustering to identify attention clusters. Phase 3: Refine the extracted attention clusters and construct the final gaze-guided attention videos for use in model training.