Predicting Radiologist Expertise from 3D Gaze Patterns During CT Interpretation

Leila Khaertdinova; Anna Anikina; Claudia Mello-Thoms; Bulat Ibragimov

cs.CVAug 24, 2026

Predicting Radiologist Expertise from 3D Gaze Patterns During CT Interpretation

Authors: Leila Khaertdinova, Anna Anikina, Claudia Mello-Thoms, Bulat Ibragimov

Organizations: Department of Computer Science, University of Copenhagen, Copenhagen, Denmark · Department of Radiology, University of Iowa, Iowa, United States

Abstract

Accurate interpretation of volumetric CT requires efficient navigation of 3D image volumes and attention to diagnostically relevant regions. While eye-tracking has been widely studied in 2D medical imaging, its use for expertise assessment in CT settings remains limited. We propose a gaze-informed transformer framework for expertise classification in thoracic CT. Using a DINOv2 backbone, radiologist fixation patterns are integrated into volumetric feature learning through (1) a learnable log-space bias in self-attention and (2) gaze-weighted pooling of patch embeddings. We trained and evaluated our approach on 182 CT reading sessions from five radiologists with varying levels of experience. On a held-out test set, the model achieves an ROC-AUC of 0.91 and F1 score of 0.86, outperforming adapted methods. These findings suggest that incorporating visual search behavior into transformers may support objective, process-based expertise assessment in radiology. Code is available via https://github.com/leiluk1/GazeToSkill.

Figures & tables

Figure 1: Example scanpaths from an expert (a) and a novice (b). The trajectories indicate the sequence of fixations over time, with the time bar normalized. Lung nodules are indicated by orange segmentation masks .

Figure 2: Overview of the proposed gaze-informed transformer model. The input CT is partitioned into patches and projected into tokens, while the heatmap volume is downsampled and normalized to obtain per-patch gaze weights. The weights are injected into each self-attention layer as a log-space bias, reweighting attention toward fixated regions. Final slice embeddings are obtained via gaze-weighted pooling, aggregated across slices, and fed to a MLP classifier that predicts the expertise class.

Model	Input	Prediction	ROC-AUC	F1	Sens.	Spec.
TF-CNN [ 21 ]	SGP, GTI	Skill	0.7793	0.8372	0.9231	0.6071
TF-CNN [ 21 ]	FO-CT+Gaze, SGP, GTI	Skill	0.7454	0.7568	0.7179	0.7500
IF-CNN [ 21 ]	SGP, GTI	Skill	0.7308	0.6866	0.5897	0.8214
IF-CNN [ 21 ]	FO-CT+Gaze, SGP, GTI	Skill	0.7363	0.7532	0.7436	0.6786
LF-CNN [ 21 ]	SGP, GTI	Skill	0.7463	0.7838	0.7436	0.7857
LF-CNN [ 21 ]	FO-CT+Gaze, SGP, GTI	Skill	0.7637	0.8205	0.8205	0.7500

Table 1: Comparison against prior methods on the held-out test set using ROC-AUC, F1, Sensitivity, and Specificity metrics. SGP: Spatial Gaze Map; GTI: Gaze Trajectory Image; FO-CT+Gaze: fixation-ordered CT slices augmented with gaze information.

Attention	Pooling strategies	Mean Val AUC	AUC	F1	Sens.	Spec.
Gaze-Bias	Gaze-Weighted	0.796 $\pm$ 0.091	0.909	0.861	0.795	0.931
Gaze-Bias	CLS	0.828 $\pm$ 0.101	0.897	0.824	0.751	0.892
None	Gaze-Weighted	0.632 $\pm$ 0.090	0.856	0.759	0.623	0.969
Fix-Mask	Gaze-Weighted	0.871 $\pm$ 0.101	0.842	0.783	0.688	0.892

Table 2: Attention variants: Gaze-Bias - soft additive bias injected into self-attention layers; Fix-Mask - hard binary mask on attention, meaning patches with zero gaze weight are blocked from being attended to; None - standard DINOv2 self-attention, no gaze signal enters the transformer. Pooling strategies: Gaze-Weighted - weighted average of all patch tokens using normalized gaze weights, i.e., patches the radiologist looked at more contribute more to the session embedding; CLS - uses only CLS token.

Explore similar work

FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification
Mar 24, 2026Daniel Beckmann, Benjamin RisseChest X-Ray Classification Medical Image Classification
Institute for Geoinformatics, Department of Computer Science, University of Münster, Münster, Germany
Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning
Apr 15, 2026Kinhei Lee, Peiyuan Jing, Zhenxuan Zhang +5Radiology VLMs Explainable Medical Image Analysis
Bioengineering Department and Imperial-X, Imperial College London, London W12 7SL, UK · College of physics and information engineering, Fuzhou University, Fuzhou, 350108, China · Department of Surgery and Cancer, Imperial College London, London W12 0NN, UK +3
GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans
Feb 26, 2026Joy T Wu, Daniel Beckmann, Sarah Miller +15Explainable Medical Image Analysis Medical Image Benchmarks
Stanford University/Stanford Health Care, Radiology Department, Stanford, CA 94305, USA · University of Münster, Computer Vision and Machine Learning Systems, Heisenbergstraße 2, Münster 48149, Germany · University of Illinois Urbana-Champaign, School of Information Sciences, Champaign, IL 61820, USA +1

Mar 24, 2026J/K move · Enter open · S save

FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification

Daniel Beckmann, Benjamin Risse

Institute for Geoinformatics, Department of Computer Science, University of Münster, Münster, Germany

Chest X-Ray Classification Medical Image Classification Vision Transformer

Open paper AskPDF arXiv

Expert eye movements provide a rich, passive source of domain knowledge in radiology, offering a powerful cue for integrating diagnostic reasoning into computer-aided analysis. However, direct integration into CNN-based systems, which historically have dominated the medical image analysis domain, is challenging: gaze recordings are sequential, temporally dense yet spatially sparse, noisy, and variable across experts. As a consequence, most existing image-based models utilize reduced representations such as heatmaps. In contrast, gaze naturally aligns with transformer architectures, as both are sequential in nature and rely on attention to highlight relevant input regions. In this work, we introduce FixationFormer, a transformer-based architecture that represents expert gaze trajectories as sequences of tokens, thereby preserving their temporal and spatial structure. By modeling gaze sequences jointly with image features, our approach addresses sparsity and variability in gaze data while enabling a more direct and fine-grained integration of expert diagnostic cues through explicit cross-attention between the image and gaze token sequences. We evaluate our method on three publicly available benchmark chest X-ray datasets and demonstrate that it achieves state-of-the-art classification performance, highlighting the value of representing gaze as a sequence in transformer-based medical image analysis.

Abstract

Explore similar work

FixationFormer: Direct Utilization of Expert Gaze Trajectories for Chest X-Ray Classification

Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning

GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans