XGLASS: X-Ray, Gaze and Language Assisted Slice Sequencing for CT Volumes

Michael Glöckler; Timo Ropinski

cs.CVOct 19, 2026

XGLASS: X-Ray, Gaze and Language Assisted Slice Sequencing for CT Volumes

Authors: Michael Glöckler, Timo Ropinski

Organizations: Ulm University, Germany

Abstract

Vision-language models (VLMs) achieve remarkable performance on 2D medical images, such as X-rays or pathology images, while 3D VLMs that directly process CT/MRI volumes lag behind due to limited 3D training data and the predominance of 2D foundation architectures. To leverage powerful 2D VLMs for analyzing medical volumes, which contain hundreds of slices, it is therefore necessary to focus only on those few slices, which are diagnostically relevant. To enable such a prioritization, we propose XGLASS, a method that transfers medical expertise from 2D X-rays to 3D CT for task-aware slice prioritization. XGLASS uses disease-conditioned importance maps derived from language-aligned radiologist gaze on X-rays and spatially aligns them with coronal CT projections, bringing gaze priors and CT content into a shared 2D representation. The fused gaze-prior/CT projection is then used to predict a disease-specific ordering of axial slices, prioritizing diagnostically relevant slices. XGLASS significantly improves budgeted retrieval over 2D and 3D-native baselines, achieving a top-10 hit rate up to 75.9% for the best-performing disease. In practice, this means that after inspecting only the top 10 slices (4.17% of a 240-slice scan), XGLASS identifies a diagnostically relevant slice in more than 75% of cases. For downstream slice-wise finding classification, an XGLASS-selected 30-slice candidate set reduces model calls by 87.5% while improving F1 by 11.3 percentage points over all-slice inference, 4.2 points over the strongest 2D selector, and remains competitive with 3D-native selectors while requiring up to 130.8 times less time and 10.9 times less GPU memory per finding, supporting its use as a lightweight preselector for volumetric CT analysis. The code and pretrained checkpoint are available on https://github.com/michaelgloeckler/XGLASS.

Figures & tables

Figure 1: Overview of the XGLASS Pipeline. Given a chest CT and chest X-ray eye-tracking data with matching report transcripts as input, we first normalize CT volumes and compute coronal CT intensity projections, while also matching gaze and disease mentions to derive disease-conditioned importance maps. These importance maps are spatially aligned to the CT projection and fused via alpha blending. The gaze-enriched intensity projection is input to the ViT-based regression head trained with MSE to predict a slice-importance vector. For inference, the model uses the precomputed importance map for the desired disease fused with the CT intensity projection and outputs a disease-specific relevance ordering of axial slices, ranking diagnostically important slices at the top.

Method	Hit@1	Hit@3	Hit@5	Hit@10	AR@10	AUC	Spear. ( $\rho$ )
Random	0.101 [0.070, 0.136]	0.250 [0.203, 0.298]	0.342 [0.291, 0.396]	0.491 [0.437, 0.547]	0.043 [0.035, 0.051]	0.504 [0.497, 0.511]	0.003 [-0.004, 0.010]
Gaussian Centered	0.190 [0.149, 0.234]	0.237 [0.190, 0.285]	0.278 [0.231, 0.329]	0.323 [0.272, 0.377]	0.082 [0.064, 0.101]	0.718 [0.698, 0.738]	0.208 [0.185, 0.233]
Gaze prior only	0.158 [0.120, 0.199]	0.177 [0.136, 0.222]	0.209 [0.165, 0.253]	0.263 [0.215, 0.313]	0.068 [0.051, 0.088]	0.638 [0.615, 0.661]	0.163 [0.141, 0.186]
Vote-MI [ 41 ]	0.044 [0.022, 0.070]	0.190 [0.146, 0.234]	0.339 [0.288, 0.392]	0.465 [0.411, 0.519]	0.033 [0.027, 0.038]	0.472 [0.454, 0.491]	-0.025 [-0.036, -0.015]
BioMedCLIP [ 46 ]	0.212 [0.168, 0.256]	0.282 [0.234, 0.332]	0.339 [0.288, 0.392]	0.443 [0.389, 0.500]	0.070 [0.058, 0.083]	0.658 [0.627, 0.688]	0.164 [0.142, 0.187]
3D U-Net [ 9 ]	0.231 [0.187, 0.278]	0.275 [0.225, 0.326]	0.301 [0.250, 0.351]	0.361 [0.307, 0.415]	0.089 [0.071, 0.107]	0.785 [0.763, 0.806]	0.228 [0.204, 0.253]

Table 1: Main Results. Metrics are reported as point estimate with 95% confidence interval below. Intervals are computed using the nonparametric percentile bootstrap [ 13 ] over the test set with 10,000 resamples. The last rows report paired absolute differences in percentage points between XGLASS and the strongest 2D or 3D baseline for each metric; bold indicates statistical significance.

Category (N)	Hit@1	Hit@3	Hit@5	Hit@10	AR@10	AUC	Spear. ( $\rho$ )
Atelectasis (39)	0.205	0.385	0.436	0.564	0.075	0.773	0.265
Consolidation (29)	0.483	0.586	0.655	0.759	0.114	0.810	0.386
Groundglass opacity (95)	0.284	0.400	0.516	0.674	0.114	0.728	0.272
High lung volume (29)	0.172	0.241	0.276	0.414	0.055	0.657	0.161
Lung nodule or mass (112)	0.107	0.205	0.286	0.446	0.074	0.722	0.129
Pleural abnormality (12)	0.500	0.583	0.667	0.667	0.139	0.779	0.345

Table 2: Disease-Specific Results. The “Overall” row corresponds to the “Ours (w/ gaze cues)” model from Table 1 . Counts (N) are shown in parentheses. We report hit-rate metrics Hit@K, the averaged recall at 10 (AR@10), AUC and rank correlation (Spearman’s

\rho

Figure 4: Downstream slice-wise VLM screening under XGLASS budgets. For each CT-condition pair, XGLASS selects the Top-

K

axial slices and the downstream model is evaluated only on this subset. Precision, recall, F1, and runtime are reported for MedGemma 4B, Lingshu 7B, and BioMedCLIP under the same single-GPU setting. Smaller XGLASS budgets consistently improve precision and F1, while reducing runtime by decreasing the number of inspected slices. Comparison to other slice selectors are in Supplementary Sec. G.4 .

Target	AUC	Spear.	Hit@1	Hit@5	Hit@10
SE-Binary	0.791	0.223	0.168	0.364	0.490
SE-Smooth	0.792	0.227	0.220	0.413	0.524
Rel-Rank	0.793	0.212	0.187	0.417	0.524
Slice-Area	0.781	0.224	0.228	0.421	0.563

Table 3: Supervision Target Ablation. Comparison of dense Slice-Area supervision with only start/end slice annotations.

Projection	AUC	Spear.	Hit@1	Hit@5	Hit@10
AIP (lung)	0.784	0.222	0.206	0.361	0.462
Beer-Lambert	0.795	0.232	0.196	0.427	0.535
AIP (soft tissue)	0.783	0.223	0.228	0.383	0.519
AIP (wide)	0.781	0.224	0.228	0.421	0.563

Table 4: CT Projection Ablation. Comparison of Beer–Lambert and AIP projections with lung, soft-tissue, and wide CT window.

Channel	$\boldsymbol{\alpha}$	AUC	Spear.	Hit@1	Hit@5	Hit@10
	0.00	0.768	0.212	0.199	0.342	0.468
	0.25	0.787	0.221	0.187	0.392	0.506
Red	0.50	0.785	0.220	0.196	0.418	0.509
	1.00	0.797	0.227	0.199	0.415	0.528
	0.75	0.781	0.224	0.228	0.421	0.563
Green	0.75	0.754	0.212	0.193	0.345	0.459

Table 5: Gaze fusion ablation. Effect of blending opacity

\alpha

and RGB injection channel for the disease-specific importance map.

Figure 5: Text–gaze alignment for disease phrases. Examples of text–gaze alignment used to construct disease-conditioned importance maps for the six target findings: Atelectasis, Consolidation, Groundglass opacity, Lung nodule or mass, High lung volume, and Pleural abnormality. Each panel shows a REFLACX chest X-ray with the chest bounding box (green dashed line), disease-specific annotation ellipses (black), and gaze trajectories for the words in the corresponding disease phrase (color-coded by word with the pre-speech buffer in orange). Fixations that fall outside the chest bounding box, visible for the High lung volume and Groundglass opacity examples, are removed during preprocessing as described in Section 3.1.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Qualitative Results. Slice retrieval across six chest CT findings: Atelectasis, Consolidation, Ground-glass Opacity, Lung Nodule/Mass, High Lung Volume, and Pleural Abnormality. For each category, we show the axial slice indices ( $z$ ) predicted by a Random baseline, Vote-MI, and our XGLASS selector. XGLASS not only identifies a ground-truth slice (green border) earlier, but also provides broader coverage of the ground-truth region by retrieving multiple slices that fall within it, rather than a single correct slice.

Figure 7: REFLACX sessions per disease category. Number of REFLACX sessions contributing to each disease-conditioned gaze heatmap. Bars show the number of matching gaze–report pairs for each category defined in REFLACX. Categories highlighted in blue correspond to the six findings used in XGLASS, whereas orange bars denote additional categories for which we obtained gaze heatmaps but did not use them due to the lack of corresponding findings in the CT dataset.

Figure 8: CT disease category distribution. Counts of CT–finding cases for the six ReXGroundingCT categories selected for XGLASS, stacked by train/validation/test split. These categories were retained because they overlap with the REFLACX-derived gaze priors and provide sufficient CT cases for training and evaluation.

Figure 9: CT volume projections in XGLASS. Example coronal slice projections of the same chest CT under different projection methods and windowing settings. AIP (lung) emphasizes parenchymal findings and saturates mediastinal structures, AIP (soft) suppresses low-attenuation lung parenchyma, so bony structures dominate the appearance, AIP (wide) exposes a broader range of intensities, and the Beer–Lambert projection yields a more radiograph-like appearance with stronger contrast in high-attenuation structures.

Method	Hit@1	Hit@3	Hit@5	Hit@10	Hit@20	AR@10	AUC	Spearman ( $\rho$ )
Random	0.080	0.125	0.179	0.286	0.446	0.038	0.491	-0.007
Vote-MI [ 41 ]	0.018	0.107	0.179	0.313	0.545	0.035	0.478	-0.017
BioMedCLIP [ 46 ]	0.080	0.143	0.214	0.286	0.455	0.062	0.585	0.093
3D Merlin + MLP [ 6 ]	0.080	0.107	0.143	0.259	0.420	0.057	0.688	0.125
3D U-Net [ 9 ]	0.080	0.143	0.161	0.232	0.384	0.079	0.698	0.132
XGLASS	0.107	0.205	0.286	0.446	0.607	0.074	0.722	0.129

Appendix

Table 6: Lung-nodule-specific baseline comparison. Results on all 112 lung nodule/mass test cases, the most spatially sparse and variable finding category. XGLASS achieves the strongest Hit@K and AUC compared to the other 2D and 3D baselines, indicating more reliable early retrieval despite the difficulty of this disease.

Setting	Hit@5	Hit@10	Hit@20	$\boldsymbol{\overline{\mathrm{Hit}}}$	AUC	$\boldsymbol{\rho}$
Backbone
ViT-B/16	0.441	0.559	0.709	0.570	0.796	0.250
RadioDino-s8	0.435	0.539	0.644	0.539	0.804	0.256
Swin-Tiny	0.405	0.464	0.578	0.482	0.798	0.244
CT projection
AIP-wide	0.441	0.559	0.709	0.570	0.796	0.250

Appendix

Table 7: Validation-based hyperparameter selection. All values are computed on the validation split.

\overline{\mathrm{Hit}}

denotes the mean of Hit@5, Hit@10, and Hit@20. Gray rows indicate the settings selected for the final model; bold values indicate the best performance within each ablation.

Channels	AUC	Spearman	Hit@1	Hit@5	Hit@10	Hit@20
R	0.781	0.224	0.228	0.421	0.563	0.693
G	0.754	0.212	0.193	0.345	0.459	0.582
B	0.761	0.209	0.190	0.418	0.519	0.655
R+G	0.774	0.220	0.184	0.383	0.503	0.639
R+B	0.784	0.226	0.241	0.443	0.547	0.661
G+B	0.749	0.212	0.222	0.367	0.468	0.617

Appendix

Table 8: Channel Injection Strategy Ablation. Gaze-prior injection into individual RGB channels and their combinations.

Backbone	AUC	Spear.	Hit@1	Hit@5	Hit@10
RadioDino-s8	0.777	0.227	0.180	0.361	0.487
Swin-Tiny	0.784	0.222	0.203	0.358	0.472
ViT-B/16	0.781	0.224	0.228	0.421	0.563

Appendix

Table 9: Backbone Ablation. Under AIP with wide window projection and

\alpha{=}0.75

Method	AUC	Spear.	Hit@1	Hit@5	Hit@10	Hit@20
Gaze prior only	0.638	0.163	0.158	0.209	0.263	0.358
CT location prior only	0.734	0.229	0.250	0.329	0.453	0.522
XGLASS w/o gaze cues	0.768	0.212	0.199	0.342	0.468	0.619
XGLASS + CT loc. prior	0.722	0.199	0.199	0.424	0.535	0.658
XGLASS w/ gaze cues	0.781	0.224	0.228	0.421	0.563	0.693

Appendix

Table 10: Prior-Only and Prior-Injection Ablations. Comparison of static CXR gaze priors, empirical CT location priors, CT-only XGLASS, CT-prior injection, and full gaze-guided XGLASS.

Condition	AUC	Spear.	Hit@1	Hit@5	Hit@10
Baseline	0.781	0.224	0.228	0.421	0.563
Random shift $\pm 16$ px	0.778	0.221	0.206	0.405	0.528
Noise $\sigma{=}0.1$	0.775	0.220	0.215	0.427	0.557
Left–right flip	0.786	0.224	0.203	0.396	0.563

Appendix

Table 11: Robustness to Gaze-Prior Perturbations. Test-time perturbations of the disease-conditioned gaze map.

Loss	AUC	Spear.	Hit@1	Hit@5	Hit@10
MSE	0.781	0.224	0.228	0.421	0.563
Focal	0.790	0.218	0.180	0.392	0.516
Slice-bal. MSE	0.709	0.202	0.206	0.421	0.519
Size-wt. MSE	0.785	0.221	0.215	0.411	0.519

Appendix

Table 12: Loss-function ablation. We compare Area MSE with binary and lesion-size-aware alternatives.

Loss	Size	Hit@1	Hit@3	Hit@5	Hit@10	AR@10
MSE	Small	0.057	0.123	0.189	0.340	0.103
	Medium	0.114	0.276	0.352	0.543	0.079
	Large	0.514	0.619	0.724	0.810	0.090
Focal	Small	0.019	0.075	0.104	0.226	0.065
	Medium	0.162	0.352	0.476	0.581	0.108
	Large	0.438	0.600	0.667	0.752	0.090

Appendix

Table 13: Loss retrieval metrics by finding size. Test cases are grouped into tertiles by total ground-truth slice-area mass. We report retrieval metrics within each size group.

Figure 10: Matched-budget downstream screening across slice selectors. For each CT–finding pair, different selectors choose the Top-

K

inspected axial slices under the same slice budget. XGLASS is compared with BioMedCLIP retrieval, Vote-MI, 3D u-Net, 3D Merlin and Random selection across MedGemma 4B, Lingshu 7B, and BioMedCLIP downstream screening.

Figure 11: Effect of the gaze–heatmap blending strength $\alpha$ on the AIP-wide CT projections. Rows correspond to the six disease categories used in XGLASS (one representative CT per category), and columns sweep the blending coefficient from left to right: $\alpha\in\{0.0,0.25,0.5,0.75,1.0\}$ . The leftmost column ( $\alpha=0$ ) shows the raw CT projection without gaze information, while increasing $\alpha$ progressively injects the REFLACX-derived gaze importance map into the red channel within the chest bounding box.

Figure 12: AP–PA comparison of disease-conditioned gaze priors, part 1. For each finding, we construct separate AP-only and PA-only REFLACX gaze heatmaps and compare their superior–inferior profiles. The superior–inferior axis encodes the axial slices and therefore it is most relevant for XGLASS.

Figure 13: AP–PA comparison of disease-conditioned gaze priors, part 2. Continuation of the AP–PA comparison for the remaining findings. Similar superior–inferior profiles across views support combining AP and PA sessions into more stable disease-conditioned gaze priors.

Figure 14: CXR–CT comparison of disease-conditioned gaze priors, part 1. We compare REFLACX CXR-derived gaze priors with sentence-matched CT-ScanGaze fixation heatmaps. Although CT reading follows different fixation dynamics, the superior–inferior profiles show similar coarse disease-specific attention patterns, supporting the idea to transfer CXR gaze priors to CTs.

Figure 15: CXR–CT comparison of disease-conditioned gaze priors, part 2. Continuation of the CXR–CT comparison for the remaining findings. The aligned superior–inferior profiles support transferring CXR-derived disease-specific search priors to CT slice selection.

Medical Image Analysis

Medical image analysis uses computational methods to interpret clinical images, measure anatomy and disease, and support diagnosis and treatment.

See all 291 papers

Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
Mar 2, 2026Simon Ging, Philipp Arnold, Sebastian Walter +6Medical Image Retrieval Medical VLMs
Computer Vision Group, University of Freiburg, Germany · Adaptive & Agentic AI · Adaptive & Agentic AI (A3) Lab, Aarhus University, Denmark +5
OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis
Feb 18, 2026Tianwei Lin, Zhongwei Qiu, Wenqiao Zhang +12Radiology VLMs Medical VLMs
Zhejiang University · DAMO Academy, Alibaba Group · Hupan Lab
Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays
Mar 27, 2026Kang Liu, Zhuoqi Ma, Siyu Liang +5Radiology VLMs Medical VLMs
Xidian University, Xi’an, China · Wuhan University, Wuhan, China

Mar 2, 2026J/K move · Enter open · S save

Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT

Simon Ging, Philipp Arnold, Sebastian Walter, Hani Alnahas+5

Computer Vision Group, University of Freiburg, Germany · Adaptive & Agentic AI · Adaptive & Agentic AI (A3) Lab, Aarhus University, Denmark+5

Medical Image Retrieval Medical VLMs 3D VLMs Medical Image Localization Medical Image Classification

Open paper AskPDF arXiv

Recent 3D CT vision-language models align volumes with reports via contrastive pretraining, but typically rely on limited public data and provide only coarse global supervision. We train a 3D CT vision-language model on 98k report-volume pairs (50k patients) collected at a single hospital, combined with public datasets, using SigLIP-style contrastive pretraining together with prompt-based disease supervision in the shared vision-text embedding space. On CT-RATE, our model achieves state-of-the-art text-to-image retrieval (R@10 31.5 vs. 22.2) and competitive disease classification (AUC 83.8 vs. 83.8), with consistent results on Rad-ChestCT (AUC 77.0 vs. 77.3). We further observe that radiologists routinely reference specific images within their reports (e.g., ``series X, image Y''), linking textual descriptions to precise axial locations. We automatically mine 262k such snippet-slice pairs and introduce the task of intra-scan snippet localization -- predicting the axial depth referred to by a text snippet -- reducing mean absolute error to 36.3 mm at 12 mm feature resolution, compared with 67.0 mm for the best baseline. Adding this localization objective leaves retrieval and classification broadly unchanged within confidence bounds, yielding a single unified model for retrieval, classification, and intra-scan grounding.

Abstract

Medical Image Analysis

Explore similar work

Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT

OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis

Seeing Like Radiologists: Context- and Gaze-Guided Vision-Language Pretraining for Chest X-rays