cs.AIAug 20, 2026
SaveVolumetric Radiology AI in the Era of Multimodal Large Language Models
Organizations: Southern Medical University · The Chinese University of Hong Kong · Fudan University · Northwestern Polytechnical University ·
Abstract
Advances in multimodal large language models (MLLMs) are extending radiological artificial intelligence (AI) beyond task-specific image analysis toward multimodal understanding and reasoning. Volumetric radiology, however, presents a fundamental representational mismatch: clinical interpretation often requires full-volume spatial context and acquisition-dependent quantitative information, whereas current MLLMs are commonly conditioned on selected two-dimensional (2D) images, compressed visual representations, or report-derived text. Reliable volumetric radiology AI therefore requires representations that preserve task-relevant three-dimensional (3D) information and systems that can access, verify, and integrate this information across clinical workflows. In this Review, we examine more than 200 publications through July 2026. We organize the literature around volumetric representation and multimodal understanding at the model level, agentic orchestration at the system level, and their links to clinical applications and evaluation. We review volumetric foundation models, language alignment and compression strategies, and agentic systems that extend MLLMs through planning, tools, memory, and workflow interaction. We distinguish settings in which selected 2D views or report-mediated reasoning may suffice from those that warrant native volumetric modeling. We also introduce a Claim-Design-Validation framework to assess whether technical, workflow, and clinical claims are matched by appropriate design and validation. Across the literature, native volumetric modeling and agentic capabilities depend on the spatial, quantitative, contextual, and workflow requirements of the intended task. Clinical credibility requires faithful volumetric representation, traceable system behavior, claim-aligned validation, and clearly defined human oversight in realistic workflows.
Figures & tables
Figure 1: Co-development of model-level approaches and agentic systems in volumetric radiology AI. The figure presents two complementary research directions. Model-level approaches, shown in green, include medical MLLMs, volumetric foundation models, and volumetric radiology MLLMs that learn or expose reusable spatial and vision–language representations. Agentic systems, shown in magenta, coordinate these representations through iterative planning, specialized tools, context management, and workflow interaction.
Figure 2: Organization of this review. The review connects model-level volumetric representation and multimodal interpretation with system-level agentic orchestration, and relates both to clinical applications and evaluation. Volumetric foundation models determine what anatomical, quantitative, and clinical information is represented. MLLMs and agentic systems determine how that information is queried, supplemented, verified, and used within clinical and technical workflows. Clinical applications define the intended claims, evidence requirements, and evaluation conditions under which these capabilities should be interpreted.
Figure 3: The Claim–Design–Validation alignment framework. The framework evaluates the alignment among a study’s stated claim, technical design, and validation setting. The claim defines the intended capability or clinical use; the design specifies how the necessary evidence is represented and used; and the validation setting determines the strength and scope of the conclusions that can be supported.
Figure 4: Technical scope of this review. The reviewed literature spans volumetric self-supervised pre-training, vision–language alignment, volumetric radiology MLLMs across 2D-native or selected-view, slice-sequence, hybrid 2D/3D, and native 3D designs, agentic radiology systems, and clinical applications and evaluation. Representative methods and resources are organized according to the corresponding sections and tables.
Figure 5: Model-level foundation and MLLM components for volumetric radiology. The model landscape is organized into three linked functional stages. Volumetric self-supervised representation learning acquires anatomy-aware priors from unlabeled scans through masked, restorative, deformation-based, and contrastive objectives. Vision–language alignment makes these representations addressable through clinical language at global, lesion-aware, region-grounded, and knowledge-enhanced levels. MLLM-based interpretation couples vision encoders, vision–language interfaces, generative backbones, and training curricula.
| Encoder paradigm | Dominant prior | Visual evidence emphasized | Characteristic bottleneck | Key design implication |
| 2D-native encoders | Biomedical image–text alignment and multimodal instruction following | In-plane appearance and medically aligned semantics | No native modeling of through-plane continuity | Natural fit for 2D or selected-view tasks; volumetric use requires explicit cross-slice aggregation |
| Slice-sequence encoders | 2D image–language representations extended across ordered slices | Ordered slice content with broader volumetric coverage | Long token sequences; cross-slice structure depends on positional modeling and token reduction | Preserve slice order and select task-relevant tokens before aggressive compression |
| Hybrid 2D/3D encoders | 2D semantic features combined with 3D spatial features | Fine in-plane detail and global volumetric context | Explicit fusion and alignment across heterogeneous feature streams | Use task-conditioned fusion when both fine detail and 3D context are required |
| Native 3D encoders | Volumetric structure learned through 3D SSL or image–text alignment | Through-plane continuity, lesion extent, and global anatomy | High-dimensional inputs, limited 3D data, and costly token processing | Prioritize when spatial continuity or lesion extent is central; compress without discarding spatial structure |
Table 3: Comparison of visual encoder paradigms for volumetric radiology MLLMs, ordered by increasing native volumetric integration. The rows summarize recurring architectural trade-offs reflected in representative systems rather than a universal performance ranking.
| Generative backbone paradigm | Capability inherited | Volumetric dependency or limitation | Key evaluation implication |
| Report-generation or unified token decoders | Task-specific structured report generation or joint visual–text token modeling | Task-specific output scope; grounding depends on the encoded visual states delivered directly to the decoder | Assess report completeness, factuality, and spatial fidelity for the intended task |
| Text-decoder-centric MLLMs | Language fluency, instruction following, and medical terminology | Volumetric grounding remains external to the decoder and depends on encoder-interface delivery | Assess grounded factuality and hallucination jointly with encoder-interface quality rather than parameter scale alone |
| Pretrained VLM/MLLM foundation stacks | Multimodal conditioning, visual-token interfaces, and broader context handling | Inherited token-format and positional assumptions may be optimized for 2D images or video | Verify 3D token compatibility, spatial provenance, and grounding after volumetric adaptation |
Table 4: Comparison of generative backbone paradigms for volumetric radiology MLLMs. The rows summarize the capability inherited by each paradigm, its dependence on the encoder-interface pipeline, and the corresponding evaluation priority.
Figure 6: Agentic workflow and modules for volumetric radiology analysis. The four capability families correspond to the main operations required for volumetric analysis: selecting and sequencing evidence, converting clinical intent into spatial or quantitative actions, retaining context across analysis steps and examinations, and coordinating intermediate outputs within supervised workflows. Solid arrows denote workflow progression, dashed arrows denote feedback, and red tags indicate supervision or operational constraints.
Figure 7: Clinical applications and evaluation for volumetric radiology. The figure maps four application families to their claim types, validation demands, benchmark protocols, and metric evidence, emphasizing that evaluation standards should follow the clinical claim and intended output.
| Systems/resources | Clinical claim | Representative tasks/outputs | Common evaluation evidence | Principal validation gap |
| Diagnostic interpretation and reporting | ||||
| CT-RATE [ 19 ] , CT2Rep [ 115 ] , M 3 D [ 117 ] , Merlin [ 3 ] , MedVista3D [ 89 ] , CT-Agent [ 35 ] , and MedScribe [ 133 ] | Identify, localize, characterize, and communicate clinically relevant findings | Segmentation, detection, classification, VQA, grounded findings, and report generation | Dice, IoU, HD95, and FROC; AUROC and calibration; BLEU/ROUGE, RadGraph, RadCliQ, and grounding IoU | Clinical correctness and traceability to volumetric evidence remain incompletely assessed |
| Prognosis and treatment response | ||||
| CLIP-Lung [ 95 ] , LoV3D [ 179 ] , CTPA-Agent [ 140 ] , Neuro-Oracle [ 148 ] , TheraAgent [ 150 ] , and BT-RADS Agent [ 151 ] | Predict future risk or clinically meaningful change over time | Malignancy and survival prediction, longitudinal response assessment, and cognitive outcomes | AUROC, AUPRC, C-index, time-dependent AUC, Brier score, calibration, and response agreement | Temporal validity, including calibration and lesion correspondence, remains insufficiently established |
| Image-derived planning and clinical decision support | ||||
| DosimeTron [ 155 ] , GPT-Plan [ 156 ] , SAGE [ 159 ] , AgentMRI [ 142 ] , Scan-do Attitude [ 160 ] , and the PET/CT Agent [ 161 ] | Translate imaging evidence into technically valid planning or management outputs | Dosimetry, radiotherapy planning, reconstruction and protocol control, and incidental finding management | Dose agreement, dose–volume constraints, reconstruction quality, workflow completion, and expert correction | End-to-end error propagation and recovery remain poorly characterized |
Table 6: Clinical application families and validation requirements in volumetric radiology AI. Representative systems and evaluation resources are summarized by clinical claim, task scope, commonly reported evidence, and principal validation gap. Dataset-level details are provided separately in Table 7 . AUC, area under the curve; AUPRC, area under the precision–recall curve; AUROC, area under the receiver operating characteristic curve; C-index, concordance index; DVH, dose–volume histogram; FROC, free-response receiver operating characteristic; HD95, 95th percentile Hausdorff distance; IoU, intersection over union; PE, pulmonary embolism.
Figure 8: Core-corpus landscape of volumetric radiology AI. The figure summarizes the deduplicated core corpus of volumetric-radiology publications and resources included in the quantitative landscape analysis. Each work is assigned once to one of four primary review roles: foundation models/vision–language pre-training, volumetric radiology MLLMs, agentic systems, or clinical resources/evaluation. The right panel cross-tabulates the same assignments by dominant modality. General AI, review articles, and 2D-only background references are excluded.
Figure 9: A research agenda for volumetric radiology intelligence. The top panel illustrates the field’s transition from isolated, 2D-adapted task models toward reusable 3D foundations, multimodal interpretation, and supervised agentic workflows. To drive this evolution, the bottom panel outlines a research agenda structured across four core pillars of intelligence: (1) volumetric and acquisition-aware, (2) workflow-centric clinical, (3) self-evolving and embodied, and (4) traceable and clinical-grade, contrasting current limitations with future directions.