GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans

Joy T Wu; Daniel Beckmann; Sarah Miller; Alexander Lee; Elizabeth Theng; Stephan Altmayer; Ken Chang; David Kersting

eess.IVFeb 26, 2026

GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans

Authors: Joy T Wu, Daniel Beckmann, Sarah Miller, Alexander Lee, Elizabeth Theng, Stephan Altmayer, Ken Chang, David Kersting, +10 more

Organizations: Stanford University/Stanford Health Care, Radiology Department, Stanford, CA 94305, USA · University of Münster, Computer Vision and Machine Learning Systems, Heisenbergstraße 2, Münster 48149, Germany · University of Illinois Urbana-Champaign, School of Information Sciences, Champaign, IL 61820, USA · Google, AI Infrastructure, Mountain View, CA 94043, USA

Abstract

[18F]FDG-PET/CT is a cornerstone imaging modality for guiding oncology therapies, yet human expert shortages necessitate more efficient diagnostic aids. While standalone AI models for automatic lesion detection exist, clinical translation remains hindered by AI explainability, reliability, and workflow integration. Meanwhile, human-computer-interaction in radiology remain limited to keyboard, mouse and voice, ignoring experts' faster, natural gaze signal. We present GazeXPErT, a 4D eye-tracking dataset with annotated expert decision windows for tumor detection and measurement on 346 dual-read FDG-PET/CTs. The dataset contributes 9,030 gaze-to-lesion trajectories derived from 3,948 minutes of 60 Hz eye-tracking data, rendered in COCO-style format. GazeXPErT captures experts' visual reasoning patterns when adjudicating suspicious lesions. It aims to facilitate development of trusted, explainable and interactive AI models through understanding expert gaze patterns. Baseline feasibility experiments suggest salient signal is extractable from routinely collected expert gaze (3D nnU-Net Dice: 0.6819 versus 0.6008 without), that gaze-trained vision transformers may aid dynamic lesion localization (74.95% predicted gaze closer to tumor), and that experts' intent may be predictable from raw gaze (Accuracy 67.53%, AUROC 0.747).

Figures & tables

Figure 1: Data collection platform. The PET/CT study was shown to readers in a full-screen application alongside some information about the patient and a keybindings legend.

Panel of Figure 2 — Figure 2: GazeXPErT dataset structure overview. Figure 2a depicts the top-level folder structure, Figure 2b illustrates the structure of raw recordings, following the Tübingen dataset structure.

Figure 3: Content overview for “gazedots_tobii.json”.

Panel of Figure 4 — Figure 4: Content overview for “gaze_lesions.json” (a) and “key_press_events.json” (b) files, respectively. These files contain information regarding annotated lesions and all keystrokes during the annotation process.

Figure 5: First (simplified) entry of each field in “coco_style_info_*.json”, all drawn from the same example study read, observer, and index lesion. (a) File-level metadata ( info , licenses ) and study/image-level records ( volumes , images ), plus the slice-level annotations entry for the lesion’s root (index) slice. (b) The study-read-level volume_annotations entry (linking the observer’s raw gaze/annotation/keypress files to the volume) and the object-level lesion_annotations entry for the same lesion. Identifiers are simplified/generalized from the real anonymized values for readability (e.g. “study_001” in place of the full source study path); numeric fields (bbox, threshold, slice number, timestamp) are real values from the “test” split. “…” denotes fields omitted for space (e.g. remaining view_parameters , repeated folder paths).

device_time_ stamp	slice_number	center_gaze_point_ on_pixel_image_x	center_gaze_point_ on_pixel_image_y	left_pupil_ diameter	paused
1000000	458	239	272	3.60	False
1016613	458	240	272	3.59	False
1033653	458	240	271	3.58	False
1049870	458	240	272	3.58	True
1066527	458	239	275	3.57	True
1083573	458	240	276	3.56	True

Figure 6: Example consecutive rows from “gaze_data_test.csv” (real values, test split), all sharing volume_annotation_id = “study_001.observer_12” and slice_number = 458, i.e. the same study read and root slice as Figure 5 . device_time_stamp values are shown as simplified relative device-clock values (shifted to a round base for readability) but preserve the true

\sim

16.7 ms (60 Hz) sampling interval between rows. The bolded row’s timestamp (1049870) exactly matches gaze_time_stamp in the Figure 5 (b) lesion_annotations entry: this is the instant the reader pressed the lesion “select” key, after which paused flips to “True” (consistent with the pause-tagging behavior described in Raw Gaze Data JSONs Explained). At that instant, the gaze pixel coordinates (240, 272) fall close to the annotated bbox center (

\approx

239, 262), consistent with the gaze accuracy reported in Technical Validation.

	Mean (std)	Median	Min	Max
Last Gaze Point	1.068 $\pm 0.672$	0.754	0.483	3.056
Closest Gaze Point	0.837 $\pm 0.648$	0.604	0.269	2.811

Table 1: Distance of the last and closest gaze point to the lesion center, across all participants and lesions.

			Precision	Recall	% Agreement	ICC [95% CI]
Trainee	$\leftrightarrow$	Experienced	0.8130	0.8165	0.6748	0.76 [0.75-0.78]
Trainee	$\leftrightarrow$	Tübingen	0.8355	0.8371	0.7140	0.70 [0.68-0.73]
Experienced	$\leftrightarrow$	Tübingen	0.8560	0.8668	0.7600	0.74 [0.72-0.76]
GazeXPErT Union	$\leftrightarrow$	Tübingen	0.8458	0.8519	0.7369	0.73 [0.71-0.75]

Table 2: Three Way Inter-reader Lesion Annotation Agreement Analysis between GazeXPErT and Tübingen annotations.

Panel of Figure 7 — Figure 7: Inter-Reader Agreement across lesion sizes.

		Lesion Level Metrics
	Dice Score	Precision	Recall
w/o Gaze	$0.601\pm 0.023$	$0.506\pm 0.053$	$0.774\pm 0.023$
with Gaze	0.682 $\pm$ 0.012	0.665 $\pm$ 0.033	0.857 $\pm$ 0.035

Table 3: 3D nnU-Net model lesion segmentation performance with and without expert static gaze insight.

Figure 8: Transformer-based Gaze + Vision architecture. Gaze tokens derived from the trajectory are encoded into a [GAZE] token, then processed jointly by pretrained ViT.

	On Mask? ( $\%$ )	Improved Prediction? ( $\%$ )	Distance ( ∘ )
Raw Last Gaze	6.11%	-	1.134 ∘
Predicted Gaze	18.12%	74.95%	0.856 ∘

Table 4: Gaze correction experiment results. Average performance over 10 randomly initialized runs.

Accuracy	Balanced (Macro) Accuracy	F1-Score	ROC AUC
$67.53\%\pm 1.63$	$67.51\%\pm 1.64$	$0.673\pm 0.016$	$0.747\pm 0.024$

Table 5: Intention prediction experiment results. Average performance over 10 randomly initialized runs.

Explore similar work

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning
Apr 15, 2026Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang +5Radiology VLMs Explainable Medical Image Analysis
Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK +3
Predicting Radiologist Expertise from 3D Gaze Patterns During CT Interpretation
Aug 24, 2026Leila Khaertdinova, Anna Anikina, Claudia Mello-Thoms +1Computed Tomography 3D ViTs
Department of Computer Science, University of Copenhagen, Copenhagen, Denmark · Department of Radiology, University of Iowa, Iowa, United States
FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification
Mar 24, 2026Daniel Beckmann, Benjamin RisseChest X-Ray Classification Medical Image Classification
Institute for Geoinformatics, Department of Computer Science, University of Münster, Münster, Germany

Apr 15, 2026J/K move · Enter open · S save

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang, Yue Yang+4

Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK+3

Radiology VLMs Explainable Medical Image Analysis Radiology Report Generation Medical VQA

Open paper AskPDF arXiv

Large scale vision language models have shown promise in automating chest Xray interpretation, yet their clinical utility remains limited by a gap between model outputs and radiologist reasoning. Most systems optimize for semantic information without emulating how experts visually examine medical images, often overlooking critical findings or diverging from established diagnostic workflows. Radiologists follow structured protocols (e.g., the ABCDEF approach) that ensure all clinically relevant regions are systematically examined, reducing missed findings and supporting reliable diagnostic reasoning. We introduce GazeX, a vision language model that leverages radiologists' eye tracking data as a behavioral prior to model expert diagnostic reasoning. By incorporating gaze trajectories and fixation patterns into pretraining, GazeX learns to follow the spatial and temporal structure of radiologist attention and integrates observations in a clinically meaningful sequence. Using a curated dataset of over 30,000 gaze key frames from five radiologists, we demonstrate that GazeX produces more accurate, interpretable, and expert consistent outputs across radiology report generation, disease grounding, and visual question answering, utilizing 231,835 radiographic studies, 780,014 question answer pairs, and 1,162 image sentence pairs with bounding boxes. Unlike autonomous reporting systems, GazeX produces verifiable evidence artifacts, including inspection trajectories and finding linked localized regions, enabling efficient human verification and safe human AI collaboration. Learning through expert eyes provides a practical route toward more trustworthy, explainable, and diagnostically robust AI systems for radiology and beyond.

Abstract

Explore similar work

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

Predicting Radiologist Expertise from 3D Gaze Patterns During CT Interpretation

FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification