Vision language models (VLMs) have achieved strong performance in 3D computed tomography (CT) classification, but high accuracy does not reflect whether their predictions rely on the correct anatomical evidence. We introduce ORGANIC3D, a framework for evaluating and improving organ level image text alignment through controlled anatomical interventions. Our benchmark assesses 12 vision encoder checkpoints and 7 generative VLMs using matched patient comparisons, organ erasure, lesion replacement, and laterality perturbations, revealing substantial alignment failures even among top performing models. To learn organ level image text correspondence, ORGANIC3D aligns each finding sentence with the anatomy aware image features and trains on edited scans to anchor this alignment in the organ's image evidence. We further extend this framework to generative VLMs, promoting evidence dependent question answering and report generation. Trained exclusively on public data, ORGANIC3D achieves SOTA classification AUROC on CTRATE, Merlin, and the heldout RADChestCT dataset and reaches 77% laterality accuracy, compared with at most 66% for prior models. Its generative VLM matches or exceeds all evaluated publicly released VLMs on visual question answering, achieves the highest report accuracy, and retracts 96 to 99% of target organ findings after organ removal, versus a maximum of 69% for other models. Together, these results establish ORGANIC3D as a framework for testing whether 3D CT vision language models ground their predictions in anatomical evidence and for improving organ level alignment alongside predictive accuracy.
Figures & tables
Figure 1: Alignment tests with expected responses. Each test checks whether predictions follow anatomical evidence. Clinical finding independence: among patients matched on all other report labels, the finding should score higher when present. Organ dependence: erasing the finding’s organ should lower its score toward its prior; erasing a control organ should leave it unchanged. Laterality: on the same scan, the correct-side sentence should score higher than its side-swapped version.
Figure 2: ORGANIC-3D overview. (a) Cross-attention forms 144 organ and global tokens from 3D ViT features. CXR-BERT text features condition pooling over the named organs’ tokens for image–text scoring. (b) The frozen vision encoder supplies all anatomy tokens in a fixed organ order to Qwen3.5-4B through an MLP. Only the MLP and LoRA adapters are trained.
Classification
Alignment
Lesion Erasure
MLC
Tumor
CTR
RAD
MER
LE
Model
CTR
RAD
MER
MER-r
ReX
AA3
MP
OE
Side
LL
MP
OE
MP
OE
Side
ReX
AA3
CT-CLIP
73.2
62.5
57.1
59.8
62.9
59.6
65.5
1.9
55.2
8.5
59.4
3.3
58.4
0.4
50.9
55.9
46.9
CT-VocabFine
76.0
65.3
57.0
59.1
64.6
60.8
65.9
5.0
49.5
19.5
61.6
3.9
58.2
0.3
50.6
51.1
44.8
HLIP
77.1
70.4
58.0
62.3
67.0
44.0
71.3
11.0
60.2
12.6
68.9
12.0
59.7
− 0.5
52.1
66.6
56.6
ItemizedCLIP
82.7
74.4
58.6
64.1
75.0
54.7
76.2
11.8
65.7
48.9
71.9
14.6
57.5
2.0
50.9
69.2
57.8
Table 1: Encoders evaluation (all values in %). Classification: AUROC, mean over each dataset’s findings (Secs. 3.4 and 4.1 ) on CTR (CT-RATE), RAD (RAD-ChestCT), MER (Merlin), MER-r (the released Merlin labels) and ReX (ReXGroundingCT); AA3: zero-shot tumor detection on AbdomenAtlas 3.0. Alignment (Sec. 3.4 ): matched pair AUROC (MP), organ erasure Δ AUROC (OE), laterality accuracy (Side) and CT-RATE lung lobe erasure hit rate (LL). Lesion Erasure: lesion erasure hit rate (LE). COLIPRI-FT: the COLIPRI backbone fine-tuned on the same public data, without ORGANIC losses. Bold: best; underlined: second best.
CT-RATE
Merlin
ReXGroundingCT
Model
AUROC
BACC
F1
MP
OEH
AUROC
BACC
F1
MP
OEH
AUROC
BACC
F1
MP
OEH
Hulu-Med
80.8
63.5
38.5
73.1
59.8
70.0
55.2
12.4
69.6
10.0
72.9
60.4
34.5
66.8
66.9
OmniCT
80.5
68.1
45.5
73.5
57.7
66.9
56.2
19.1
66.6
10.2
71.2
62.9
39.8
66.3
69.1
MedGemma 1.5
71.5
60.0
34.2
64.7
28.6
66.9
57.2
20.3
66.0
17.9
59.6
55.0
26.2
58.0
28.9
M3D-LaMed
57.9
53.6
31.8
52.8
40.0
53.8
51.1
14.0
53.6
-0.7
54.2
51.4
27.5
50.7
54.0
RadFM
54.2
50.4
21.4
52.4
18.6
53.7
50.2
7.9
56.7
4.9
53.9
50.4
19.9
48.6
17.2
Table 2: VLM evaluation (all values in %). Visual question answering on the full CT-RATE (3,002) and Merlin (5,109) test sets and 464 ReXGroundingCT scans. Reported metrics include answer AUROC, BACC and F1 of the stated answers; MP and OEH as in Sec. 3.4 . Bold: best; underlined: second best.
Free text report
Finding-based report
Findings
Similarity
Findings
Model
F1
CRG
GREEN
RadGraph
F1
IA-F1
CT-RATE
Hulu-Med
31.2
39.4
36.6
14.0
53.3
30.0
OmniCT
38.4
41.5
46.9
25.6
52.5
28.8
MedGemma 1.5
19.9
36.5
15.8
10.8
44.1
18.3
Table 3: Report generation (all values in %) on the CT-RATE and Merlin test sets. Free text report: finding accuracy (F1, CRG) and similarity to the reference (GREEN [ 40 ] , RadGraph F1). Finding-based report: F1 and image-attributable F1 (IA-F1, Sec. 3.4 ). Bold: best; underlined: second best.
Figure 3: Evidence maps (occlusion sensitivity) for nine findings (columns) and ORGANIC-3D and public encoders, one per family (rows; all checkpoints in the supplement). Brighter: removing the region lowers the finding’s score more, and each map is scaled to its own maximum. Cyan: annotated lesion. Cases from ReXGroundingCT and AbdomenAtlas 3.0.
MLC-AUROC
Lesion
Tumor
OE
Side
Encoder
CTR
MER
RAD
ReX
AA3
CTR
MER
CTR
MER
COLIPRI
82.2
65.5
74.8
73.4
62.7
10.5
3.2
51.0
49.5
+ fine-tuning
86.4
82.7
77.4
79.1
72.9
12.1
5.8
52.7
52.0
+ anat. queries
87.1
85.5
79.4
79.9
74.4
14.0
8.0
68.7
66.2
+ organ routing
87.5
87.8
79.6
79.0
75.6
13.5
9.4
74.9
63.8
+ ORGANIC (final)
87.7
87.7
79.7
79.5
75.0
28.5
21.3
76.9
63.8
Table 4: Ablation studies (all values in %). We compare COLIPRI and plain fine-tuning in encoder and VLM training to the anatomical-aware study design. OE indicates organ erasure Δ AUROC, computed on the finding score Δf for the encoder and on P(Yes) for the VLM. Bold: best; underlined: second best.
Recent 3D CT vision-language models align volumes with reports via contrastive pretraining, but typically rely on limited public data and provide only coarse global supervision. We train a 3D CT vision-language model on 98k report-volume pairs (50k patients) collected at a single hospital, combined with public datasets, using SigLIP-style contrastive pretraining together with prompt-based disease supervision in the shared vision-text embedding space. On CT-RATE, our model achieves state-of-the-art text-to-image retrieval (R@10 31.5 vs. 22.2) and competitive disease classification (AUC 83.8 vs. 83.8), with consistent results on Rad-ChestCT (AUC 77.0 vs. 77.3). We further observe that radiologists routinely reference specific images within their reports (e.g., ``series X, image Y''), linking textual descriptions to precise axial locations. We automatically mine 262k such snippet-slice pairs and introduce the task of intra-scan snippet localization -- predicting the axial depth referred to by a text snippet -- reducing mean absolute error to 36.3 mm at 12 mm feature resolution, compared with 67.0 mm for the best baseline. Adding this localization objective leaves retrieval and classification broadly unchanged within confidence bounds, yielding a single unified model for retrieval, classification, and intra-scan grounding.
Simon Ging, Philipp Arnold, Sebastian Walter +6
Computer Vision Group, University of Freiburg, Germany · Adaptive & Agentic AI · Adaptive & Agentic AI (A3) Lab, Aarhus University, Denmark +5
Computed tomography (CT) vision-language pretraining from paired volumes and radiology reports is a scalable yet challenging task. Existing methods commonly adopt global scan-report contrast, which is scalable but obscures heterogeneous organ evidence. Meanwhile, direct organ-level alignment remains coarse, since the same anatomy can exhibit multiple distinct radiological appearances. Therefore, pretraining requires a finer alignment unit: the organ-conditioned radiological pattern. In this work, we propose OCP-CT, an organ-conditioned pattern-token alignment framework for CT vision-language pretraining. Specifically, OCP-CT preserves a stable global CT-report contrastive branch and introduces an organ pattern interface: sparse Mixture-of-Experts (MoE) routes image and text tokens according to latent radiological patterns, learnable slots query the routed tokens into continuous pattern tokens, and paired token contrast aligns image-text pattern tokens with structured soft targets built from report-derived clinical similarity. On the publicly available CT-RATE and RAD-ChestCT benchmarks, OCP-CT achieves average AUROCs of 84.5% and 69.9% for zero-shot abnormality diagnosis, respectively. Compared with the strongest prior reported results, these results yield absolute AUROC gains of 6.7 and 0.8 percentage points.
Medical vision-language pretraining (VLP) from paired CT images and radiology reports enables scalable representation learning, but most existing methods align either whole scans with entire reports or local image regions with text fragments. These formulations underuse a key property of radiology reports: findings are organized around anatomical structures, with abnormalities described by organs, disease concepts, locations, and severity-related attributes. We propose OKA-CT, an organ-hierarchical knowledge-augmented framework for CT-report VLP. OKA-CT first converts free-text reports into organ-conditioned knowledge using radiology report parsing and LLM-assisted semantic structuring. The extracted hierarchy is used across two learning stages. Stage1 injects anatomy-grounded evidence into the CT visual representation through fine-grained organ-conditioned supervision, while Stage2 uses organ-specific report evidence to guide structured report-CT contrastive learning, where hierarchy-derived semantic soft targets treat non-paired cases with shared organ-level findings as weak semantic positives rather than uniform negatives. A lightweight query-based global branch further aggregates disease-relevant volumetric evidence for whole-scan representation. On CT-RATE and RAD-ChestCT datasets, OKA-CT achieves zero-shot abnormality diagnosis AUROCs of 84.9 and 72.2, outperforming prior CT VLP baselines. Retrieval and patch-occlusion analyses further show improved report-image alignment and stronger sensitivity to disease-associated anatomical regions.
Guoliang You, Hongming Li, Yuanwang Zhang +1
Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA · School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA 19104, USA