ARC-CT: Anatomy-Routed Contrastive Vision-Language Learning for 3D Chest CT
Authors: Huseyin Umut Isik, Mehmet Alp Ozaydin, Sila Kurugol, Şeyda Ertekin
Organizations: Department of Computer Engineering, METU, Ankara, Turkey · Quantitative Intelligent Imaging Lab, Boston Children’s Hospital and Harvard Medical School, Boston, MA, USA · METU-DTX Digital Transformation and Innovation Center, Ankara, Turkey
Contrastive vision-language learning uses paired chest CT volumes and radiology reports to learn abnormality classifiers without manually annotated labels. However, two characteristics of chest CT challenge conventional global contrastive learning. First, many critical abnormalities are small or anatomically localized, and pooling an en- tire volume into a single embedding may dilute their visual evidence. Second, the standard contrastive objective treats every other scan in a batch as a negative. Because many chest CTs share abnormalities, this objective incorrectly pushes co-positive pairs apart. We propose Anatomy-Routed Contrastive Learning for 3D Chest CT (ARC-CT), a region-aware framework that addresses these limitations using only la- bels extracted from reports by an LLM, with no manual annotations or bounding boxes. ARC-CT combines three components: (1) an Anato- myQFormer localizing evidence via queries constrained by automatically generated organ masks; (2) a label-Jaccard soft InfoNCE objective in- tegrating the standard one-hot target with the label-set overlap of each pair, which reduces false-negative penalties between studies that share clinical findings; and (3) an organ-level alignment loss connecting mask- pooled visual features to organ-specific report text extracted offline with a large language model. ARC-CT achieves a 0.86 mask-free macro AUC across 18 abnormalities using a compact 3D ResNet-18 backbone. Over- all, ARC-CT outperforms both comparable efficient baselines and sev- eral larger transformer models. Our code and weights are available at https://github.com/arc-ct/arc-ct.
Figures & tables
Figure 1: ARC-CT overview. Stage 1 pretrains a 3D ResNet-18 on 18 LLM-extracted abnormality labels, warm-starting Stage 2. In Stage 2, the encoder’s feature map is cross-attended by the AnatomyQFormer’s anatomy, pathology, and global queries, while an LLM parses reports into per-organ sentences. Embeddings are projected into a shared space and trained with a label-Jaccard soft InfoNCE contrastive loss with per-organ alignment and query supervision.
CT-RATE (in-domain)
RAD-ChestCT (external)
Method ∗
AUC
Prec
Acc
F1
AUC
Prec
Acc
F1
ARC-CT (ours)
0.855
0.455
0.787
0.809
0.734
0.392
0.679
0.716
GreenRFM [ 4 ]
0.848
0.444
0.777
0.800
0.780
0.461
0.730
0.757
MPS-CT [ 3 ]
0.838
0.435
0.774
0.796
0.773
0.450
0.719
0.748
BrgSA [ 5 ]
0.792
0.385
0.733
0.762
0.742
0.422
0.686
0.720
ViSD-Boost [ 7 ]
0.790
0.387
0.731
0.759
0.694
0.342
0.652
0.693
Table 1: Prompt-based abnormality classification (no manual labels) on CT-RATE (in-domain) and RAD-ChestCT (external). The best per column in bold , second underlined .
Class
ARC-CT
GreenRFM
MPS-CT
CT-CLIP
Pleural effusion
0.97
0.97
0.96
0.90
Cardiomegaly
0.93
0.93
0.94
0.86
Arterial wall calcification
0.94
0.94
0.93
0.85
Coronary artery wall calcification
0.93
0.94
0.93
0.85
Consolidation
0.91
0.91
0.89
0.72
Medical material
0.94
0.91
0.89
0.72
Table 2: Per-class AUC on CT-RATE validation (3,039 volumes), three-seed mean. GreenRFM [ 4 ] , MPS-CT [ 3 ] , and CT-CLIP [ 1 ] are the baselines with a published 18-class breakdown. Best per row in bold , second underlined .
Figure 2: Mask-free Grad-CAM for four positive findings (cardiomegaly, arterial wall calcification, pleural effusion, consolidation) on one CT-RATE validation volume. Top row (a–d): ARC-CT; bottom row (e–h): CT-CLIP [ 1 ] . ARC-CT’s activation is sharply localized to the relevant structure, while CT-CLIP’s is diffuse and largely unchanged between findings.
Configuration
Macro AUC
Δ
95% CI
ARC-CT
0.855±0.001
–
[0.848,0.862]
− Stage-1 warm-start (Kinetics only)
0.826±0.001
−0.028
[−0.032,−0.024]
− anatomy routing (plain Q-Former)
0.837±0.004
−0.017
[−0.019,−0.014]
− label-Jaccard soft target
0.845±0.001
−0.009
[−0.011,−0.007]
− learnable temperature (fixed)
0.847±0.002
−0.007
[−0.009,−0.006]
− per-organ alignment
0.850±0.002
−0.004
[−0.006,−0.002]
Table 3: Leave-one-out ablation on CT-RATE (three-seed mean ± s.d.; 95% CIs from a paired patient bootstrap; all p<0.001 ). The weakly supervised Stage-1 warm-start contributes the largest gain, followed by anatomy routing and the label-Jaccard target.
Image → image (MAP@ K )
Report → image (R@ K )
Method
@5
@10
@50
@5
@10
@50
@100
ARC-CT (ours)
71.3
61.7
54.3
11.4
18.0
40.4
52.1
GreenRFM [ 4 ]
–
–
–
9.5
16.0
39.9
54.3
MPS-CT [ 3 ]
71.3
61.5
53.7
10.0
16.3
39.0
52.2
BrgSA [ 5 ]
69.2
58.5
50.5
5.8
10.1
28.6
42.0
CT-CLIP [ 1 ]
68.3
57.2
48.9
2.9
5.0
18.0
28.7
Table 4: Cross-modal image-to-image and report-to-image retrieval on CT-RATE validation, no fine-tuning. ARC-CT leads on all metrics but R@100. The best per column in bold , second underlined .
Stanford Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA, USA · Department of Radiology, Stanford University, Stanford, CA, USA