cs.CVNov 17, 2026

ORGANIC3D: Grounding 3D CT Vision Language Models in Organ Level Image Text Coupling

Authors: Zelong Liu, Hamza Ahmed, Wynn Wingyi Lee, Mickael Torjman, Alexander Dimitrakakis, Jiahua Lu, Timothy Deyer, Bachir Taouli, 

Organizations: Icahn School of Medicine at Mount Sinai, New York, NY, USA

Abstract

Vision language models (VLMs) have achieved strong performance in 3D computed tomography (CT) classification, but high accuracy does not reflect whether their predictions rely on the correct anatomical evidence. We introduce ORGANIC3D, a framework for evaluating and improving organ level image text alignment through controlled anatomical interventions. Our benchmark assesses 12 vision encoder checkpoints and 7 generative VLMs using matched patient comparisons, organ erasure, lesion replacement, and laterality perturbations, revealing substantial alignment failures even among top performing models. To learn organ level image text correspondence, ORGANIC3D aligns each finding sentence with the anatomy aware image features and trains on edited scans to anchor this alignment in the organ's image evidence. We further extend this framework to generative VLMs, promoting evidence dependent question answering and report generation. Trained exclusively on public data, ORGANIC3D achieves SOTA classification AUROC on CTRATE, Merlin, and the heldout RADChestCT dataset and reaches 77% laterality accuracy, compared with at most 66% for prior models. Its generative VLM matches or exceeds all evaluated publicly released VLMs on visual question answering, achieves the highest report accuracy, and retracts 96 to 99% of target organ findings after organ removal, versus a maximum of 69% for other models. Together, these results establish ORGANIC3D as a framework for testing whether 3D CT vision language models ground their predictions in anatomical evidence and for improving organ level alignment alongside predictive accuracy.

Figures & tables

Explore similar work

Mar 2, 2026

Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT

Recent 3D CT vision-language models align volumes with reports via contrastive pretraining, but typically rely on limited public data and provide only coarse global supervision. We train a 3D CT vision-language model on 98k report-volume pairs (50k patients) collected at a single hospital, combined with public datasets, using SigLIP-style contrastive pretraining together with prompt-based disease supervision in the shared vision-text embedding space. On CT-RATE, our model achieves state-of-the-art text-to-image retrieval (R@10 31.5 vs. 22.2) and competitive disease classification (AUC 83.8 vs. 83.8), with consistent results on Rad-ChestCT (AUC 77.0 vs. 77.3). We further observe that radiologists routinely reference specific images within their reports (e.g., ``series X, image Y''), linking textual descriptions to precise axial locations. We automatically mine 262k such snippet-slice pairs and introduce the task of intra-scan snippet localization -- predicting the axial depth referred to by a text snippet -- reducing mean absolute error to 36.3 mm at 12 mm feature resolution, compared with 67.0 mm for the best baseline. Adding this localization objective leaves retrieval and classification broadly unchanged within confidence bounds, yielding a single unified model for retrieval, classification, and intra-scan grounding.
Jul 15, 2026

Fine-Grained Vision-Language Pretraining with Organ-Conditioned Pattern Tokens for CT Understanding

Computed tomography (CT) vision-language pretraining from paired volumes and radiology reports is a scalable yet challenging task. Existing methods commonly adopt global scan-report contrast, which is scalable but obscures heterogeneous organ evidence. Meanwhile, direct organ-level alignment remains coarse, since the same anatomy can exhibit multiple distinct radiological appearances. Therefore, pretraining requires a finer alignment unit: the organ-conditioned radiological pattern. In this work, we propose OCP-CT, an organ-conditioned pattern-token alignment framework for CT vision-language pretraining. Specifically, OCP-CT preserves a stable global CT-report contrastive branch and introduces an organ pattern interface: sparse Mixture-of-Experts (MoE) routes image and text tokens according to latent radiological patterns, learnable slots query the routed tokens into continuous pattern tokens, and paired token contrast aligns image-text pattern tokens with structured soft targets built from report-derived clinical similarity. On the publicly available CT-RATE and RAD-ChestCT benchmarks, OCP-CT achieves average AUROCs of 84.5% and 69.9% for zero-shot abnormality diagnosis, respectively. Compared with the strongest prior reported results, these results yield absolute AUROC gains of 6.7 and 0.8 percentage points.
Jul 12, 2026

Learning Anatomy-Grounded CT Vision-Language Representations with Organ-Hierarchical Report Knowledge

Medical vision-language pretraining (VLP) from paired CT images and radiology reports enables scalable representation learning, but most existing methods align either whole scans with entire reports or local image regions with text fragments. These formulations underuse a key property of radiology reports: findings are organized around anatomical structures, with abnormalities described by organs, disease concepts, locations, and severity-related attributes. We propose OKA-CT, an organ-hierarchical knowledge-augmented framework for CT-report VLP. OKA-CT first converts free-text reports into organ-conditioned knowledge using radiology report parsing and LLM-assisted semantic structuring. The extracted hierarchy is used across two learning stages. Stage1 injects anatomy-grounded evidence into the CT visual representation through fine-grained organ-conditioned supervision, while Stage2 uses organ-specific report evidence to guide structured report-CT contrastive learning, where hierarchy-derived semantic soft targets treat non-paired cases with shared organ-level findings as weak semantic positives rather than uniform negatives. A lightweight query-based global branch further aggregates disease-relevant volumetric evidence for whole-scan representation. On CT-RATE and RAD-ChestCT datasets, OKA-CT achieves zero-shot abnormality diagnosis AUROCs of 84.9 and 72.2, outperforming prior CT VLP baselines. Retrieval and patch-occlusion analyses further show improved report-image alignment and stronger sensitivity to disease-associated anatomical regions.