A 3D foundation model unifying CT interpretation and segmentation
Organizations: Department of Statistics and Data Science, Tsinghua University, Beijing, China · Alibaba DAMO Academy, Hangzhou, China · School of Psychological and Cognitive Sciences and Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China · Department of Head and Neck Surgery, Sir Run Run Shaw Hospital, Zhejiang University School of Medicine, Hangzhou, China ·
Abstract
Radiologists interpret computed tomography (CT) scans by jointly determining what a finding represents and where it lies in three-dimensional (3D) anatomy. However, CT models have treated CT interpretation and 3D segmentation as separate tasks, leaving textual findings spatially unverifiable and overlooking potential synergy between clinical understanding and spatial perception. Here we present HoloCT, a generalist 3D foundation model unifying CT interpretation and voxel-level segmentation by jointly modelling whole-volume context and fine-grained anatomical detail, providing 3D masks as evidence. We curated HoloDATA, comprising 195,921 CT scans from 73 public datasets and six clinical cohorts, with 6.2 million instruction dialogues and 2.8 million CT-mask pairs. Across 346 tasks spanning segmentation, lesion screening, visual question answering, and report generation on 61 internal and 12 external datasets, HoloCT outperformed specialist and foundation models. In multicentre validation across 11,455 clinical CT scans from five hospitals, HoloCT outperformed competing models in lesion segmentation and report generation by 18.2 and 15.3 points. It also achieved 85.1% mean accuracy in screening thyroid tumours and colon lesions, exceeding all four participating radiologists in a retrospective reader study. Further analyses revealed bidirectional semantic-spatial synergy: language-guided segmentation delineated unseen anatomical structures and lesions as well as diffuse free-text findings, outperforming the leading approach by 22.1 points, whereas prompting the model to generate corresponding 3D masks alongside its textual outputs significantly improved interpretation performance. HoloCT shows that clinical understanding and spatial perception can be learned together within a single foundation model and offers a path towards more accurate and reliable CT analysis.
Figures & tables
| \textbf ID | \textbf Dataset Name | \textbf 3D CTs | \textbf Classes | \textbf Masks | \textbf Body Parts | \textbf Split Rule | \textbf Dataset Type | \textbf Download Links |
|---|---|---|---|---|---|---|---|---|
| 001 | AbdomenCT-1k Ma et al. 2022 | 957 | 4 | 3,606 | Abdomen, Genitourinary | VISTA3D | Internal | Download |
| 002 | AMOS22 Ji et al. 2022 | 588 | 15 | 5,167 | Abdomen, Thorax | VISTA3D | Internal | Download |
| 003 | MSD-colon Antonelli et al. 2022 | 190 | 1 | 126 | Abdomen | VISTA3D | Internal | Download |
| 004 | MSD-hepatic-vessel Antonelli et al. 2022 | 443 | 2 | 606 | Abdomen, Vascular | VISTA3D | Internal | Download |
| 005 | MSD-liver Antonelli et al. 2022 | 201 | 2 | 249 | Abdomen | VISTA3D | Internal | Download |
| 006 | MSD-lung Antonelli et al. 2022 | 95 | 1 | 63 | Thorax | VISTA3D | Internal | Download |
| \textbf ID | \textbf Dataset Name | \textbf 3D CTs | \textbf Label Name | \textbf Masks | \textbf Positive | \textbf Negative | \textbf Split Rule | \textbf Dataset Type | \textbf Download Links |
|---|---|---|---|---|---|---|---|---|---|
| 001 | left kidney cyst | 30 | 30 | 1,198 | |||||
| 002 | Total-SegmentatorV2 Wasserthal et al. 2023 | 1,228 | right kidney cyst | 25 | 25 | 1,203 | VISTA3D | Internal | Download |
| 003 | KiTS23 Heller et al. 2023 | 489 | kidney cyst | 248 | 248 | 241 | SAT | Internal | Download |
| 004 | CT-ICH Hssayeni 2019 | 75 | intracranial hemorrhage | 36 | 36 | 39 | random | Internal | Download |
| 005 | PanTS Li et al. 2025 | 9,901 | pancreas lesion | 1,032 | 1,032 | 8,869 | random | Internal | Download |
| 006 | kidney lesion | 2,222 | 2,222 | 7,040 |
| \textbf Dataset Name | \textbf 3D CTs | \textbf Single-turn QAs | \textbf Body Parts | \textbf Split Rule | \textbf Dataset Type | \textbf Download Links |
|---|---|---|---|---|---|---|
| CT-RATE-option Hamamci et al. 2026 | 46,609 | 135,384 | Thorax, Vascular, Musculoskeletal | official | Internal | Download |
| CT-RATE-long&short Hamamci et al. 2026 | 50,175 | 2,472,480 (within 669,798 multi-turn QAs) | Thorax, Vascular, Musculoskeletal | official | Internal | Download |
| 3D-RAD Gai et al. 2025 | 16,188 | 170,105 | Thorax, Vascular, Musculoskeletal | official | Internal | Download |
| RadGenome-VQA Zhang et al. 2025 | 25,692 | 481,752 | Thorax, Vascular, Musculoskeletal | official | Internal | Download |
| AMOS-MM-VQA Ji et al. 2024 | 1,687 | 16,538 | Abdomen, Genitourinary, Vascular, Musculoskeletal | AMOS-22-aligned | Internal | Download |
| FLARE-2025-VQA FLARE-MedFM 2025 | 3,917 | 37,795 | Abdomen, Genitourinary, Vascular, Musculoskeletal | AMOS-MM-aligned | Internal | Download |
| \textbf Dataset Name | \textbf 3D CTs | \textbf Single-turn QAs | \textbf Body Parts | \textbf Split Rule | \textbf Dataset Type | \textbf Download Links |
|---|---|---|---|---|---|---|
| CT-RATE-RG Hamamci et al. 2026 | 50,175 | 50,175 | Thorax, Vascular, Musculoskeletal | official | Internal | Download |
| RadGenome Zhang et al. 2025 | 25,692 | 223,643 | Thorax, Vascular, Musculoskeletal | official | Internal | Download |
| INSPECT Huang et al. 2023 | 23,226 | 23,226 | Thorax, Vascular, Musculoskeletal | random | Internal | Download |
| Merlin Blankemeier et al. 2026 | 25,372 | 328,318 | Abdomen, Genitourinary, Vascular, Musculoskeletal | random | Internal | Download |
| AMOS-MM-RG Ji et al. 2024 | 1,687 | 5,350 | Abdomen, Genitourinary, Vascular, Musculoskeletal | AMOS-22-aligned | Internal | Download |
| FLARE-2025-RG FLARE-MedFM 2025 | 3,917 | 35,295 | Abdomen, Genitourinary, Vascular, Musculoskeletal | AMOS-MM-aligned | Internal | Download |
| \textbf Dataset Name | \textbf 3D CTs | \textbf Masks | \textbf Single-turn QAs | \textbf Body Parts | \textbf Split Rule | \textbf Dataset Type | \textbf Download Links |
|---|---|---|---|---|---|---|---|
| AMOS-MM-VQA-grounded Ji et al. 2024 | 207 | 1,642 | 1,438 | Abdomen, Genitourinary, Vascular, Musculoskeletal | AMOS-22-aligned | Internal | Download |
| DeepTumorVQA-grounded Chen et al. 2025 | 9,169 | 329,929 | 218,226 | Abdomen, Genitourinary, Vascular, Musculoskeletal | AbdomenAtlas3.0-aligned | Internal | Download |
| RadGenome-VQA-grounded Zhang et al. 2025 | 25,687 | 162,480 | 468,643 | Thorax, Vascular, Musculoskeletal | CT-RATE-aligned | Internal | Download |
| RadGenome-grounded Zhang et al. 2025 | 25,690 | 192,242 | 217,932 | Thorax, Vascular, Musculoskeletal | CT-RATE-aligned | Internal | Download |
| ReXGroundingCT Baharoon et al. 2025 | 3,042 | 7,802 | 10,844 | Thorax, Vascular, Musculoskeletal | CT-RATE-aligned | Internal | Download |
| \textbf TOTAL | \textbf 63,795 | \textbf 694,095 | \textbf 917,083 | \textbf | \textbf | \textbf | \textbf |
| \textbf Training stage | \textbf Supervision stream | \textbf Source data | \textbf Dialogue types included | \textbf Stage-specific selection | \textbf No. of dialogues |
|---|---|---|---|---|---|
| \textbf Stage 1: 3D CT vision–language alignment | Language | Supplementary Table 4 | Report generation | High-quality report-generation dialogues | \textbf 0.68M |
| \textbf Stage 2: 3D CT multitask pre-training | Language | Supplementary Tables 2 – 4 | Single-target and multi-target lesion screening without grounding; closed-ended and open-ended VQA; narrative and structured report generation | All eligible training dialogues from the corresponding training splits | \textbf 2.70M |
| Segmentation | Supplementary Tables 1 , 2 and 5 | Target-level and finding-level referring segmentation; single-target and multi-target lesion screening with grounding; grounded VQA; anatomy-grounded and finding-grounded report generation | All eligible training dialogues from the corresponding training splits | \textbf 3.50M | |
| \textbf Stage 3: 3D CT high-quality multitask tuning | Language | Supplementary Tables 2 – 4 | Single-target and multi-target lesion screening without grounding; closed-ended and open-ended VQA; narrative and structured report generation | Single-region report-generation dialogues were excluded; RadGenome-VQA dialogues were excluded | \textbf 1.69M |
| Segmentation | Supplementary Tables 1 , 2 and 5 | Target-level and finding-level referring segmentation; single-target and multi-target lesion screening with grounding; grounded VQA; anatomy-grounded and finding-grounded report generation | Low-quality segmentation data including PASTA-Gen-TotalSeg were excluded; RadGenome grounded-VQA dialogues were excluded; single-region grounded-report dialogues were excluded | \textbf 1.15M |
| \textbf Standard Terms and Counts | |||
|---|---|---|---|
| \textbf Primary Category | \textbf Secondary Category | \textbf ID | \textbf OOD |
| Brain & CNS | brain(962), brainstem(210), optic chiasm(210), pituitary gland(162), left temporal lobe(120), right temporal lobe(120), left hippocampus(120), right hippocampus(120), intracranial hemorrhage(357) | epidural hemorrhage(32), subdural hemorrhage(158), subarachnoid hemorrhage(142), intraparenchymal hemorrhage(142), intraventricular hemorrhage(120) | |
| Sensory Organs | left cochlea(162), right cochlea(162), left anterior eyeball(42), right anterior eyeball(42), left posterior eyeball(42), right posterior eyeball(42), left lacrimal gland(42), right lacrimal gland(42), left optic nerve(210), right optic nerve(210), left eyeball(652), right eyeball(648), left lens(120), right lens(120), left middle ear(120), right middle ear(120), left internal auditory canal(120), right internal auditory canal(120), left tympanic cavity(120), right tympanic cavity(120), left vestibule semicircular canal(120), right vestibule semicircular canal(120), left eustachian tube bone(120), right eustachian tube bone(120), left mastoid process(120), right mastoid process(120) | – | |
| Upper Aerodigestive Tract | arytenoid(42), buccal mucosa(42), oral cavity(162), cricopharyngeal inlet(42), cervical esophagus(42), larynx glottis(162), larynx supraglottis(162), lips(42), nasal cavity(533), larynx(120), pharynx constrictor muscle(120), nasopharyngeal tumor(120) | laryngeal cancer(439) | |
| Head & Neck | Neck Glands & Maxillofacial | thyroid gland(704), skull(1,006), mandible(82), left submandibular gland(203), right submandibular gland(198), thyroid(162), left parotid gland(210), right parotid gland(210), left thyroid(533), right thyroid(532), left cheek(468), right cheek(451), left mandible(120), right mandible(120), left temporomandibular joint(120), right temporomandibular joint(120), nasopharyngeal lymph node(120), face(259) | malignant thyroid tumor(193), benign thyroid tumor(432), thyroid tumor(549) |
| Lung & Pleura | lung tumor(2,664), lung(435), left lung upper lobe(21,740), left lung lower lobe(27,320), right lung upper lobe(6,386), right lung middle lobe(20,878), right lung lower lobe(27,582), pulmonary nodule(3,306), left lung(20,152), right lung(19,936), covid-19 infection(20), lung effusion(78) | lung cancer(1,191) | |
Medical Imaging Foundation Models
Medical imaging foundation models learn reusable representations for tasks across clinical images, modalities, and institutions.
See all 248 papers