cs.AIAug 20, 2026

Volumetric Radiology AI in the Era of Multimodal Large Language Models

Authors: Zanting Ye, Shengyuan Liu, Xin Liu, Chenhui Wang, Zhisong Wang, Jiashuai Liu, Zipei Wang, Cheng Wang, 

Organizations: Southern Medical University · The Chinese University of Hong Kong · Fudan University · Northwestern Polytechnical University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Advances in multimodal large language models (MLLMs) are extending radiological artificial intelligence (AI) beyond task-specific image analysis toward multimodal understanding and reasoning. Volumetric radiology, however, presents a fundamental representational mismatch: clinical interpretation often requires full-volume spatial context and acquisition-dependent quantitative information, whereas current MLLMs are commonly conditioned on selected two-dimensional (2D) images, compressed visual representations, or report-derived text. Reliable volumetric radiology AI therefore requires representations that preserve task-relevant three-dimensional (3D) information and systems that can access, verify, and integrate this information across clinical workflows. In this Review, we examine more than 200 publications through July 2026. We organize the literature around volumetric representation and multimodal understanding at the model level, agentic orchestration at the system level, and their links to clinical applications and evaluation. We review volumetric foundation models, language alignment and compression strategies, and agentic systems that extend MLLMs through planning, tools, memory, and workflow interaction. We distinguish settings in which selected 2D views or report-mediated reasoning may suffice from those that warrant native volumetric modeling. We also introduce a Claim-Design-Validation framework to assess whether technical, workflow, and clinical claims are matched by appropriate design and validation. Across the literature, native volumetric modeling and agentic capabilities depend on the spatial, quantitative, contextual, and workflow requirements of the intended task. Clinical credibility requires faithful volumetric representation, traceable system behavior, claim-aligned validation, and clearly defined human oversight in realistic workflows.

Figures & tables

Figure 1: Co-development of model-level approaches and agentic systems in volumetric radiology AI. The figure presents two complementary research directions. Model-level approaches, shown in green, include medical MLLMs, volumetric foundation models, and volumetric radiology MLLMs that learn or expose reusable spatial and vision–language representations. Agentic systems, shown in magenta, coordinate these representations through iterative planning, specialized tools, context management, and workflow interaction.
Figure 2: Organization of this review. The review connects model-level volumetric representation and multimodal interpretation with system-level agentic orchestration, and relates both to clinical applications and evaluation. Volumetric foundation models determine what anatomical, quantitative, and clinical information is represented. MLLMs and agentic systems determine how that information is queried, supplemented, verified, and used within clinical and technical workflows. Clinical applications define the intended claims, evidence requirements, and evaluation conditions under which these capabilities should be interpreted.
Figure 3: The Claim–Design–Validation alignment framework. The framework evaluates the alignment among a study’s stated claim, technical design, and validation setting. The claim defines the intended capability or clinical use; the design specifies how the necessary evidence is represented and used; and the validation setting determines the strength and scope of the conclusions that can be supported.
Figure 4: Technical scope of this review. The reviewed literature spans volumetric self-supervised pre-training, vision–language alignment, volumetric radiology MLLMs across 2D-native or selected-view, slice-sequence, hybrid 2D/3D, and native 3D designs, agentic radiology systems, and clinical applications and evaluation. Representative methods and resources are organized according to the corresponding sections and tables.
Figure 5: Model-level foundation and MLLM components for volumetric radiology. The model landscape is organized into three linked functional stages. Volumetric self-supervised representation learning acquires anatomy-aware priors from unlabeled scans through masked, restorative, deformation-based, and contrastive objectives. Vision–language alignment makes these representations addressable through clinical language at global, lesion-aware, region-grounded, and knowledge-enhanced levels. MLLM-based interpretation couples vision encoders, vision–language interfaces, generative backbones, and training curricula.
Encoder paradigmDominant priorVisual evidence emphasizedCharacteristic bottleneckKey design implication
2D-native encodersBiomedical image–text alignment and multimodal instruction followingIn-plane appearance and medically aligned semanticsNo native modeling of through-plane continuityNatural fit for 2D or selected-view tasks; volumetric use requires explicit cross-slice aggregation
Slice-sequence encoders2D image–language representations extended across ordered slicesOrdered slice content with broader volumetric coverageLong token sequences; cross-slice structure depends on positional modeling and token reductionPreserve slice order and select task-relevant tokens before aggressive compression
Hybrid 2D/3D encoders2D semantic features combined with 3D spatial featuresFine in-plane detail and global volumetric contextExplicit fusion and alignment across heterogeneous feature streamsUse task-conditioned fusion when both fine detail and 3D context are required
Native 3D encodersVolumetric structure learned through 3D SSL or image–text alignmentThrough-plane continuity, lesion extent, and global anatomyHigh-dimensional inputs, limited 3D data, and costly token processingPrioritize when spatial continuity or lesion extent is central; compress without discarding spatial structure
Table 3: Comparison of visual encoder paradigms for volumetric radiology MLLMs, ordered by increasing native volumetric integration. The rows summarize recurring architectural trade-offs reflected in representative systems rather than a universal performance ranking.
Generative backbone paradigmCapability inheritedVolumetric dependency or limitationKey evaluation implication
Report-generation or unified token decodersTask-specific structured report generation or joint visual–text token modelingTask-specific output scope; grounding depends on the encoded visual states delivered directly to the decoderAssess report completeness, factuality, and spatial fidelity for the intended task
Text-decoder-centric MLLMsLanguage fluency, instruction following, and medical terminologyVolumetric grounding remains external to the decoder and depends on encoder-interface deliveryAssess grounded factuality and hallucination jointly with encoder-interface quality rather than parameter scale alone
Pretrained VLM/MLLM foundation stacksMultimodal conditioning, visual-token interfaces, and broader context handlingInherited token-format and positional assumptions may be optimized for 2D images or videoVerify 3D token compatibility, spatial provenance, and grounding after volumetric adaptation
Table 4: Comparison of generative backbone paradigms for volumetric radiology MLLMs. The rows summarize the capability inherited by each paradigm, its dependence on the encoder-interface pipeline, and the corresponding evaluation priority.
Figure 6: Agentic workflow and modules for volumetric radiology analysis. The four capability families correspond to the main operations required for volumetric analysis: selecting and sequencing evidence, converting clinical intent into spatial or quantitative actions, retaining context across analysis steps and examinations, and coordinating intermediate outputs within supervised workflows. Solid arrows denote workflow progression, dashed arrows denote feedback, and red tags indicate supervision or operational constraints.
Figure 7: Clinical applications and evaluation for volumetric radiology. The figure maps four application families to their claim types, validation demands, benchmark protocols, and metric evidence, emphasizing that evaluation standards should follow the clinical claim and intended output.
Systems/resourcesClinical claimRepresentative tasks/outputsCommon evaluation evidencePrincipal validation gap
Diagnostic interpretation and reporting
CT-RATE [ 19 ] , CT2Rep [ 115 ] , M 3 D [ 117 ] , Merlin [ 3 ] , MedVista3D [ 89 ] , CT-Agent [ 35 ] , and MedScribe [ 133 ]Identify, localize, characterize, and communicate clinically relevant findingsSegmentation, detection, classification, VQA, grounded findings, and report generationDice, IoU, HD95, and FROC; AUROC and calibration; BLEU/ROUGE, RadGraph, RadCliQ, and grounding IoUClinical correctness and traceability to volumetric evidence remain incompletely assessed
Prognosis and treatment response
CLIP-Lung [ 95 ] , LoV3D [ 179 ] , CTPA-Agent [ 140 ] , Neuro-Oracle [ 148 ] , TheraAgent [ 150 ] , and BT-RADS Agent [ 151 ]Predict future risk or clinically meaningful change over timeMalignancy and survival prediction, longitudinal response assessment, and cognitive outcomesAUROC, AUPRC, C-index, time-dependent AUC, Brier score, calibration, and response agreementTemporal validity, including calibration and lesion correspondence, remains insufficiently established
Image-derived planning and clinical decision support
DosimeTron [ 155 ] , GPT-Plan [ 156 ] , SAGE [ 159 ] , AgentMRI [ 142 ] , Scan-do Attitude [ 160 ] , and the PET/CT Agent [ 161 ]Translate imaging evidence into technically valid planning or management outputsDosimetry, radiotherapy planning, reconstruction and protocol control, and incidental finding managementDose agreement, dose–volume constraints, reconstruction quality, workflow completion, and expert correctionEnd-to-end error propagation and recovery remain poorly characterized
Table 6: Clinical application families and validation requirements in volumetric radiology AI. Representative systems and evaluation resources are summarized by clinical claim, task scope, commonly reported evidence, and principal validation gap. Dataset-level details are provided separately in Table 7 . AUC, area under the curve; AUPRC, area under the precision–recall curve; AUROC, area under the receiver operating characteristic curve; C-index, concordance index; DVH, dose–volume histogram; FROC, free-response receiver operating characteristic; HD95, 95th percentile Hausdorff distance; IoU, intersection over union; PE, pulmonary embolism.
Figure 8: Core-corpus landscape of volumetric radiology AI. The figure summarizes the deduplicated core corpus of volumetric-radiology publications and resources included in the quantitative landscape analysis. Each work is assigned once to one of four primary review roles: foundation models/vision–language pre-training, volumetric radiology MLLMs, agentic systems, or clinical resources/evaluation. The right panel cross-tabulates the same assignments by dominant modality. General AI, review articles, and 2D-only background references are excluded.
Figure 9: A research agenda for volumetric radiology intelligence. The top panel illustrates the field’s transition from isolated, 2D-adapted task models toward reusable 3D foundations, multimodal interpretation, and supervised agentic workflows. To drive this evolution, the bottom panel outlines a research agenda structured across four core pillars of intelligence: (1) volumetric and acquisition-aware, (2) workflow-centric clinical, (3) self-evolving and embodied, and (4) traceable and clinical-grade, contrasting current limitations with future directions.
More in this topic

Medical AI Agents

See all 39 papers

Explore similar work

CardsList
Jul 15, 2026

Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in 2D medical image understanding, their extension to 3D volumetric imaging remains hindered by prohibitive annotation costs and dataset opacity. Current data formats, predominantly consisting of rigid Visual Question Answering (VQA) pairs or unstructured final clinical reports, typically fail to capture explicit clinical reasoning. To address this limitation, we introduce a large-scale structured reasoning dataset constructed via a novel slice-wise data synthesis paradigm. Inspired by the genuine diagnostic workflow of radiologists, this paradigm models visual cognition by decomposing the complex 3D reading process, translating global clinical priors into fine-grained, per-slice observations that are subsequently synthesized into an interpretable Chain-of-Thought (CoT). Crucially, this synthesized reasoning framework enforces essential clinical principles: sequential spatial tracking, multi-slice spatial awareness for artifact mitigation, and differential exclusion. To validate this approach, we instruction-tune a standard 2D-pretrained MLLM baseline using the synthesized data to enhance its volumetric comprehension. Comprehensive evaluations across multiple 3D medical benchmarks demonstrate that our method yields significant performance improvements over the 2D baseline. Furthermore, the resulting model exhibits robust spatial reasoning capabilities and rivals resource-intensive native 3D architectures, effectively bridging the performance gap. Ultimately, this data-centric strategy unlocks deep volumetric understanding and highly interpretable clinical logic without requiring computationally expensive 3D-specific pre-training. The complete repository, including datasets and training workflows, is publicly available at https://github.com/2020420145009/hounsfield.

Zhuoyuan Fu, Zeshang Li, Yiqiong Zhang +5

University of International Relations, Beijing, China · University of Trento, Trento, Italy

3D Spatial Reasoning3D Medical ImagingMultimodal Large Language ModelsMedical VLMsMedical Reasoning
AskPDF
Apr 9, 2026

Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models

Medical vision-language models (VLMs) show strong performance on radiology tasks but often produce fluent yet weakly grounded conclusions due to over-reliance on a dominant modality. We introduce a context-aligned reasoning framework that enforces agreement across heterogeneous clinical evidence before generating diagnostic conclusions. The proposed approach augments a frozen VLM with structured contextual signals derived from radiomic statistics, explainability activations, and vocabulary-grounded semantic cues. Instead of producing free-form responses, the model generates structured outputs containing supporting evidence, uncertainty estimates, limitations, and safety notes. We observe that auxiliary signals alone provide limited benefit; performance gains emerge only when these signals are integrated through contextual verification. Experiments on chest X-ray datasets demonstrate that context alignment improves discriminative performance (AUC 0.918 to 0.925) while maintaining calibrated uncertainty. The framework also substantially reduces hallucinated keywords (1.14 to 0.25) and produces more concise reasoning explanations (19.4 to 15.3 words) without increasing model confidence (0.70 to 0.68). Cross-dataset evaluation on CheXpert further reveals that modality informativeness significantly influences reasoning behavior. These results suggest that enforcing multi-evidence agreement improves both reliability and trustworthiness in medical multimodal reasoning, while preserving the underlying model architecture.

Sumra Khan, Sagar Chhabriya, Aizan Zafar +5

Department of Computer Science, Salim Habib University, Karachi, Pakistan · Computer Science, Institute of Business Administration Sukkur, Pakistan · Center for Research in Computer Vision, University of Central Florida, USA +2

VLM Hallucination MitigationMedical VLMsVLM ReasoningRadiology VLMsVision-Language ModelsMultimodal Reasoning
AskPDF
Mar 14, 2026

Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space

Visual spatial intelligence is critical for medical image interpretation, yet remains largely unexplored in Multimodal Large Language Models (MLLMs) for 3D imaging. This gap persists due to a systemic lack of datasets featuring structured 3D spatial annotations beyond basic labels. In this study, we introduce an agentic pipeline that autonomously synthesizes spatial visual question-answering (VQA) data by orchestrating computational tools such as volume estimation and bounding boxes extraction with multi-agent collaboration and expert radiologist validation. We present SpatialMed, the first comprehensive benchmark for evaluating 3D spatial intelligence in medical MLLMs, comprising 31,253 question-answer pairs across multiple organs and tumor types. Our evaluations on 24 state-of-the-art MLLMs and extensive analyses reveal that current models lack robust spatial reasoning capabilities for medical imaging.

Quoc-Huy Trinh, Xi Ding, Yang Liu +7

3D Spatial ReasoningMultimodal Large Language ModelsMedical VQAMedical Image Benchmarks
AskPDF