Vision language models (VLMs) have achieved strong performance in 3D computed tomography (CT) classification, but high accuracy does not reflect whether their predictions rely on the correct anatomical evidence. We introduce ORGANIC3D, a framework for evaluating and improving organ level image text alignment through controlled anatomical interventions. Our benchmark assesses 12 vision encoder checkpoints and 7 generative VLMs using matched patient comparisons, organ erasure, lesion replacement, and laterality perturbations, revealing substantial alignment failures even among top performing models. To learn organ level image text correspondence, ORGANIC3D aligns each finding sentence with the anatomy aware image features and trains on edited scans to anchor this alignment in the organ's image evidence. We further extend this framework to generative VLMs, promoting evidence dependent question answering and report generation. Trained exclusively on public data, ORGANIC3D achieves SOTA classification AUROC on CTRATE, Merlin, and the heldout RADChestCT dataset and reaches 77% laterality accuracy, compared with at most 66% for prior models. Its generative VLM matches or exceeds all evaluated publicly released VLMs on visual question answering, achieves the highest report accuracy, and retracts 96 to 99% of target organ findings after organ removal, versus a maximum of 69% for other models. Together, these results establish ORGANIC3D as a framework for testing whether 3D CT vision language models ground their predictions in anatomical evidence and for improving organ level alignment alongside predictive accuracy.
Figures & tables
Figure 1: Alignment tests with expected responses. Each test checks whether predictions follow anatomical evidence. Clinical finding independence: among patients matched on all other report labels, the finding should score higher when present. Organ dependence: erasing the finding’s organ should lower its score toward its prior; erasing a control organ should leave it unchanged. Laterality: on the same scan, the correct-side sentence should score higher than its side-swapped version.
Figure 2: ORGANIC-3D overview. (a) Cross-attention forms 144 organ and global tokens from 3D ViT features. CXR-BERT text features condition pooling over the named organs’ tokens for image–text scoring. (b) The frozen vision encoder supplies all anatomy tokens in a fixed organ order to Qwen3.5-4B through an MLP. Only the MLP and LoRA adapters are trained.
Classification
Alignment
Lesion Erasure
MLC
Tumor
CTR
RAD
MER
LE
Model
CTR
RAD
MER
MER-r
ReX
AA3
MP
OE
Side
LL
MP
OE
MP
OE
Side
ReX
AA3
CT-CLIP
73.2
62.5
57.1
59.8
62.9
59.6
65.5
1.9
55.2
8.5
59.4
3.3
58.4
0.4
50.9
55.9
46.9
CT-VocabFine
76.0
65.3
57.0
59.1
64.6
60.8
65.9
5.0
49.5
19.5
61.6
3.9
58.2
0.3
50.6
51.1
44.8
HLIP
77.1
70.4
58.0
62.3
67.0
44.0
71.3
11.0
60.2
12.6
68.9
12.0
59.7
− 0.5
52.1
66.6
56.6
ItemizedCLIP
82.7
74.4
58.6
64.1
75.0
54.7
76.2
11.8
65.7
48.9
71.9
14.6
57.5
2.0
50.9
69.2
57.8
Table 1: Encoders evaluation (all values in %). Classification: AUROC, mean over each dataset’s findings (Secs. 3.4 and 4.1 ) on CTR (CT-RATE), RAD (RAD-ChestCT), MER (Merlin), MER-r (the released Merlin labels) and ReX (ReXGroundingCT); AA3: zero-shot tumor detection on AbdomenAtlas 3.0. Alignment (Sec. 3.4 ): matched pair AUROC (MP), organ erasure Δ AUROC (OE), laterality accuracy (Side) and CT-RATE lung lobe erasure hit rate (LL). Lesion Erasure: lesion erasure hit rate (LE). COLIPRI-FT: the COLIPRI backbone fine-tuned on the same public data, without ORGANIC losses. Bold: best; underlined: second best.
CT-RATE
Merlin
ReXGroundingCT
Model
AUROC
BACC
F1
MP
OEH
AUROC
BACC
F1
MP
OEH
AUROC
BACC
F1
MP
OEH
Hulu-Med
80.8
63.5
38.5
73.1
59.8
70.0
55.2
12.4
69.6
10.0
72.9
60.4
34.5
66.8
66.9
OmniCT
80.5
68.1
45.5
73.5
57.7
66.9
56.2
19.1
66.6
10.2
71.2
62.9
39.8
66.3
69.1
MedGemma 1.5
71.5
60.0
34.2
64.7
28.6
66.9
57.2
20.3
66.0
17.9
59.6
55.0
26.2
58.0
28.9
M3D-LaMed
57.9
53.6
31.8
52.8
40.0
53.8
51.1
14.0
53.6
-0.7
54.2
51.4
27.5
50.7
54.0
RadFM
54.2
50.4
21.4
52.4
18.6
53.7
50.2
7.9
56.7
4.9
53.9
50.4
19.9
48.6
17.2
Table 2: VLM evaluation (all values in %). Visual question answering on the full CT-RATE (3,002) and Merlin (5,109) test sets and 464 ReXGroundingCT scans. Reported metrics include answer AUROC, BACC and F1 of the stated answers; MP and OEH as in Sec. 3.4 . Bold: best; underlined: second best.
Free text report
Finding-based report
Findings
Similarity
Findings
Model
F1
CRG
GREEN
RadGraph
F1
IA-F1
CT-RATE
Hulu-Med
31.2
39.4
36.6
14.0
53.3
30.0
OmniCT
38.4
41.5
46.9
25.6
52.5
28.8
MedGemma 1.5
19.9
36.5
15.8
10.8
44.1
18.3
Table 3: Report generation (all values in %) on the CT-RATE and Merlin test sets. Free text report: finding accuracy (F1, CRG) and similarity to the reference (GREEN [ 40 ] , RadGraph F1). Finding-based report: F1 and image-attributable F1 (IA-F1, Sec. 3.4 ). Bold: best; underlined: second best.
Figure 3: Evidence maps (occlusion sensitivity) for nine findings (columns) and ORGANIC-3D and public encoders, one per family (rows; all checkpoints in the supplement). Brighter: removing the region lowers the finding’s score more, and each map is scaled to its own maximum. Cyan: annotated lesion. Cases from ReXGroundingCT and AbdomenAtlas 3.0.
MLC-AUROC
Lesion
Tumor
OE
Side
Encoder
CTR
MER
RAD
ReX
AA3
CTR
MER
CTR
MER
COLIPRI
82.2
65.5
74.8
73.4
62.7
10.5
3.2
51.0
49.5
+ fine-tuning
86.4
82.7
77.4
79.1
72.9
12.1
5.8
52.7
52.0
+ anat. queries
87.1
85.5
79.4
79.9
74.4
14.0
8.0
68.7
66.2
+ organ routing
87.5
87.8
79.6
79.0
75.6
13.5
9.4
74.9
63.8
+ ORGANIC (final)
87.7
87.7
79.7
79.5
75.0
28.5
21.3
76.9
63.8
Table 4: Ablation studies (all values in %). We compare COLIPRI and plain fine-tuning in encoder and VLM training to the anatomical-aware study design. OE indicates organ erasure Δ AUROC, computed on the finding score Δf for the encoder and on P(Yes) for the VLM. Bold: best; underlined: second best.
Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA · School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA 19104, USA