cs.CVNov 2, 2026

Findings-aware Language-Image Pretraining for CT Volume Interpretation

Authors: Takuro Shimaya, Junya Inoh, Masahiro Saiko

Organizations: Biometrics Research Labs, NEC corporation, Kawasaki, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Computed tomography (CT) provides diverse patient information and various clinical tasks, for which numerous deep learning models have been investigated. Recent years have seen the development of vision-language foundation models that can comprehensively extract clinical features, driven by representation learning that combines CT volumes with radiology reports. Existing approaches typically adopt Contrastive Language-Image Pre-training (CLIP)-based representation learning, where pairs from different cases are, in principle, treated as negative pairs such that the volume and report embeddings are pushed apart. In the biomedical imaging, however, treating different cases as negative pairs is not always appropriate, particularly when clinical similarity exists. In this study, we propose a supervised representation learning framework grounded in clinical knowledge from language foundation models, rather than the weak supervision used in CLIP. Specifically, we utilized a large language model to extract abnormal finding phrases from reports. We then explicitly constructed the target features that the image encoder should output, as the semantic representation of the extracted findings using a biomedical text embedding model. Our Findings-aware Language-Image Pre-training (FiLIP) outperformed weakly supervised approaches in multi-label zero-shot finding classification in both in-domain and external-domain evaluations, and also proved effective when fine-tuned for the downstream task of radiology report generation.

Figures & tables

Figure 1: Schematic comparison of conventional methods and the proposed FiLIP. While contrastive learning may treat similar cases as negative pairs, our method directly supervises the model using text embeddings of extracted finding phrases from reports. We leverage a pretrained biomedical embedding model and an LLM as external clinical knowledge sources.
Figure 2: Overview of the proposed strategy. (a) Feature extraction processes. See also Fig. 3 . (b) Loss function calculations. See Methods for details.
Figure 3: Finding phrases extraction via an LLM.
Figure 4: Performance evaluation methods. (a) Zero-shot classification of finding presence. (b) Fine-tuning for radiology report generation as a downstream task. See Methods for details.
Figure 5: Zero-shot finding classification performance for CT-RATE. (a) AUC for each finding. See the supplementary text for correspondence between labels and organs. (b) Dependency of the mean AUC on the similarity threshold SthrS_{\mathrm{thr}} for computing Linter\mathcal{L}^{\mathrm{inter}} . Violin plots show the AUC distributions, with bootstrap-estimated 95% confidence intervals and interquartile ranges overlaid.
 Evaluation dataset
ModelCT-RATERad-ChestCT
CT-CLIP [ 12 ]0.733 [0.727-0.740]0.632 [0.625-0.640]
(VocabFine) †0.7600.657
(ClassFine) †0.7610.647
Merlin † [ 5 ]0.7280.644
fVLM [ 33 ]0.778 [0.770-0.786]0.680 [0.674-0.686]
Table 1: The mean AUC for zero-shot finding classification. 95% confidence intervals are shown in the brackets. †\dagger indicates models requiring additional data beyond radiology reports for training, and values are from ref. [ 7 ] . FiLIP significantly outperformed fVLM on both evaluation datasets (see the main and supplementary text for details).
Lintra\mathcal{L^{\mathrm{intra}}}tbg\bm{t}_{\mathrm{bg}}TopKFocalLinter\mathcal{L^{\mathrm{inter}}}AUC
✓\checkmark    0.761
✓\checkmark✓\checkmark   0.764
✓\checkmark✓\checkmark✓\checkmark  0.768
✓\checkmark✓\checkmark ✓\checkmark 0.774
✓\checkmark✓\checkmark✓\checkmark✓\checkmark 0.779
    ✓\checkmark0.788
Table 2: Ablation studies on losses evaluated using CT-RATE.
 LLMembedding modelAUC
# 1GPT-4.1CXR-BERT0.744 [0.734-0.754]
# 2GPT-4.1text-embedding-3-large0.787 [0.778-0.795]
# 3gpt-oss-120BBioLORD-20230.796 [0.787-0.804]
# 4GPT-4.1BioLORD-20230.799 [0.791-0.808]
Table 3: Ablation studies on foundation models evaluated using CT-RATE. # 4 is the default setting in this study. Statistical significance was observed between # 1 vs # 4, and # 2 vs # 4, while no significant difference was found between # 3 vs # 4 (see the main and supplementary text for details).
Table 4: Results on the downstream task of report generation using CT-RATE. Standard errors are shown in the brackets. For FiLIP scores, *1, *2, and *3 indicate statistically significant differences from ImageNet, fVLM, and SimCroP, respectively. *0 indicates no significant difference from any model. Metrics without an significance marker were not statistically tested. See p-values for the supplementary text.
Figure 6: Visualization of voxel-wise similarity map between target finding embeddings and volume features. The arrows indicate the position of each finding. The brighter color corresponds to the higher similarity.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

 Report gen. metricsGREEN subcategoryZero-shot AUC
modelsBLEU-4GREENg-1g-2g-3g-4g-5CT-RATERadChestCT
FiLIP vs ImageNet8.5×10−48.5\times 10^{-4}7.9×10−47.9\times 10^{-4}<10−4<10^{-4}1.1×10−11.1\times 10^{-1}<10−4<10^{-4}4.1×10−14.1\times 10^{-1}4.5×10−24.5\times 10^{-2}--
FiLIP vs CT-CLIP-------<10−4<10^{-4}<10−4<10^{-4}
FiLIP vs fVLM<10−4<10^{-4}1.9×10−31.9\times 10^{-3}<10−4<10^{-4}3.0×10−13.0\times 10^{-1}1.3×10−31.3\times 10^{-3}1.5×10−11.5\times 10^{-1}7.6×10−17.6\times 10^{-1}<10−4<10^{-4}1.1×10−31.1\times 10^{-3}
FiLIP vs SimCroP1.8×10−11.8\times 10^{-1}6.5×10−26.5\times 10^{-2}<10−4<10^{-4}2.6×10−12.6\times 10^{-1}<10−4<10^{-4}8.9×10−38.9\times 10^{-3}1.7×10−11.7\times 10^{-1}<10−4<10^{-4}<10−4<10^{-4}
FiLIP (w/ BioLORD) vs FiLIP (w/ CXR-BERT)-------<10−4<10^{-4}-
Appendix
Table S1 : P-values for the comparison of different models on various metrics BEFORE Holm correction.
 Report gen. metricsGREEN subcategoryZero-shot AUC
modelsBLEU-4GREENg-1g-2g-3g-4g-5CT-RATERadChestCT
FiLIP vs ImageNet1.4×10−2‾\underline{\mathbf{1.4\times 10^{-2}}}1.4×10−2‾\underline{\mathbf{1.4\times 10^{-2}}}<10−4‾\underline{\mathbf{<10^{-4}}}>0.99>0.996.5×10−4‾\underline{\mathbf{6.5\times 10^{-4}}}>0.99>0.994.9×10−14.9\times 10^{-1}--
FiLIP vs CT-CLIP-------<10−4‾\underline{\mathbf{<10^{-4}}}<10−4‾\underline{\mathbf{<10^{-4}}}
FiLIP vs fVLM<10−4‾\underline{\mathbf{<10^{-4}}}2.4×10−2‾\underline{\mathbf{2.4\times 10^{-2}}}<10−4‾\underline{\mathbf{<10^{-4}}}>0.99>0.991.8×10−2‾\underline{\mathbf{1.8\times 10^{-2}}}>0.99>0.997.6×10−17.6\times 10^{-1}<10−4‾\underline{\mathbf{<10^{-4}}}1.7×10−2‾\underline{\mathbf{1.7\times 10^{-2}}}
FiLIP vs SimCroP>0.99>0.996.5×10−16.5\times 10^{-1}1.7×10−4‾\underline{\mathbf{1.7\times 10^{-4}}}>0.99>0.99<10−4‾\underline{\mathbf{<10^{-4}}}1.1×10−11.1\times 10^{-1}>0.99>0.99<10−4‾\underline{\mathbf{<10^{-4}}}<10−4‾\underline{\mathbf{<10^{-4}}}
FiLIP (w/ BioLORD) vs FiLIP (w/ CXR-BERT)-------<10−4‾\underline{\mathbf{<10^{-4}}}-
Appendix
Table S2 : P-values for the comparison of different models on various metrics AFTER Holm correction. The bold and underlined values indicate statistical significance at a 5% significance level.
Finding labelOrgan
Arterial wall calcificationaorta
Cardiomegalyheart
Pericardial effusion
Coronary artery wall calcification
Hiatal herniaesophagus
Emphysemalung
Appendix
Table S3 : Correspondence between finding labels and organs, which is equivalent to that adopted in fVLM.
More in this topic

Medical VLMs

Medical vision-language models connect clinical images with text for diagnosis, reporting, question answering, and image grounding.

See all 487 papers

Explore similar work

CardsList

  1. Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT

    Mar 2, 2026Simon Ging, Philipp Arnold, Sebastian Walter +6Medical Image RetrievalMedical VLMs

    Computer Vision Group, University of Freiburg, Germany · Adaptive & Agentic AI · Adaptive & Agentic AI (A3) Lab, Aarhus University, Denmark +5

  2. Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Tasks

    Apr 5, 2026Rubén Moreno-Aguado, Alba Magallón, Victor Moreno +2Medical Imaging Foundation ModelsMedical Image Representation Learning

    Bioengineering Department and Imperial-X, Imperial College London, London, UK · Department of Computer Science, University of Manchester, Manchester, UK · Oncology Data Analytics Program, Catalan Institute of Oncology, Barcelona, Spain +6

  3. Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

    Jun 24, 2026Bowen Shi, Weiwei Cao, Ruifeng Yuan +5Contrastive Learning3D Medical Imaging

    DAMO Academy, Alibaba Group · Hupan Lab, 310023, Hangzhou, China · Shanghai Jiao Tong University, China +2