3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis

Ziyue Wang; Linghan Cai; Chang Han Low; Haofeng Liu; Junde Wu; Jingyu Wang; Rui Wang; Lei Song

cs.CVFeb 20, 2026

3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis

Authors: Ziyue Wang, Linghan Cai, Chang Han Low, Haofeng Liu, Junde Wu, Jingyu Wang, Rui Wang, Lei Song,

Organizations: National University of Singapore · Microsoft Research · TUD Dresden University of Technology · PuzzleLogic ·

Abstract

3D CT analysis spans a continuum from low-level perception to high-level clinical understanding. Existing 3D-oriented analysis methods adopt either isolated task-specific modeling or task-agnostic end-to-end paradigms to produce one-hop outputs, impeding the systematic accumulation of perceptual evidence for downstream reasoning. In parallel, recent multimodal large language models (MLLMs) exhibit improved visual perception and can integrate visual and textual information effectively, yet their predominantly 2D-oriented designs fundamentally limit their ability to perceive and analyze volumetric medical data. To bridge this gap, we propose 3DMedAgent, a unified agent that enables 2D MLLMs to perform general 3D CT analysis without 3D-specific fine-tuning. 3DMedAgent coordinates heterogeneous visual and textual tools through a flexible MLLM agent, progressively decomposing complex 3D analysis into tractable subtasks that transition from global to regional views, from 3D volumes to informative 2D slices, and from visual evidence to structured textual representations. Central to this design, 3DMedAgent maintains a long-term structured memory that aggregates intermediate tool outputs and supports query-adaptive, evidence-driven multi-step reasoning. We further introduce the DeepChestVQA benchmark for evaluating unified perception-to-understanding capabilities in 3D thoracic imaging. Experiments across over 40 tasks demonstrate that 3DMedAgent consistently outperforms general, medical, and 3D-specific MLLMs, highlighting a scalable path toward general-purpose 3D clinical assistants.Code and data are available at https://github.com/jinlab-imvr/3DMedAgent.

Figures & tables

Figure 1: The illustration of existing tasks and methods in 3D medical image analysis. 3DMedAgent adaptively invokes visual tools for task-specific perception and analysis, and summarizes the outputs as compact evidence stored in shared memory for subsequent reasoning.

Dataset Overview
CT scans	892
VQA pairs	1020
Question Taxonomy
Question type	#Subtypes	#VQA pairs
Recognition	3	180
Visual Reasoning	8	480

Table 1: Overview of the DeepChestVQA benchmark.

Figure 2: The overall framework of 3DMedAgent. A 2D MLLM agent iteratively interacts with heterogeneous tools, distills their outputs into structured evidence, and updates a shared memory for reliable 3D medical image analysis. Subtitle colors follow the same agentic action-type scheme, and colors denote agentic action types: tool use (purple) , memory (blue) , and reasoning (red) .

Figure 3: 3DMedAgent’s reasoning trajectories across representative cases. 3DMedAgent addresses 3D CT queries by progressively gathering and reusing evidence in shared memory, illustrated on (a) measurement, (b) recognition, and (c) medical reasoning tasks.

Type	Subtype	Rand	GPT-5	Qwen3-VL (30B)	MedGemma (27B)	HuatuoGPT (34B)	M3D	RadFM	3DMedAgent (GPT-5)	3DMedAgent (Qwen3-VL)
Measurement	lesion volume measurement	0.25	0.25	0.30	0.24	0.20	0.18	0.25	0.42	0.35
	organ HU measurement	0.25	0.35	0.35	0.21	0.35	0.33	0.37	0.82	0.78
	organ volume measurement	0.25	0.39	0.41	0.41	0.24	0.32	0.31	0.63	0.63
	Average	0.25	0.33	0.35	0.29	0.26	0.28	0.31	0.63	0.59
Recognition	colon lesion existence	0.50	0.47	0.47	0.50	0.57	0.60	0.48	0.82	0.80
	kidney cyst existence	0.50	0.57	0.48	0.50	0.57	0.57	0.48	0.82	0.78

Table 2: The comparison results on the DeepTumorVQA benchmark. The “Rand” column indicates the random accuracy of the corresponding question subtype. Best results are bold and second best results are underlined .

Figure 4: Performance comparison across source datasets in DeepTumorVQA.

Type	Subtype	Rand	GPT-5	Qwen3-VL (30B)	MedGemma (27B)	Huatuo-GPT (34B)	M3D	RadFM	3DMedAgent (GPT-5)	3DMedAgent (Qwen3-VL)
Recognition	bronchus lesion existence	0.50	0.53	0.55	0.50	0.55	0.58	0.45	0.63	0.65
	lung lesion existence	0.50	0.52	0.53	0.50	0.50	0.52	0.48	0.68	0.65
	pleura lesion existence	0.50	0.50	0.50	0.50	0.55	0.45	0.48	0.77	0.73
	Average	0.50	0.52	0.53	0.50	0.53	0.52	0.47	0.69	0.68
Visual Reasoning	largest lesion diameter	0.25	0.38	0.27	0.10	0.27	0.22	0.28	0.38	0.32
	largest lesion location	0.20	0.18	0.38	0.23	0.27	0.17	0.20	0.38	0.40

Table 3: The comparison results on the DeepChestVQA benchmark. The “Rand” column indicates the random accuracy of the corresponding question subtype. Best results are bold and second best results are underlined .

Figure 5: Performance comparison across different organs.

Setting			MLLM Input	DeepTumorVQA				DeepChestVQA
OAMI	CFLT	T1S-Loop	Text / Image	Mea.	Rec.	Visual Reason.	Medical Reason.	Rec.	Visual Reason.	Medical Reason.
			I. + T.	0.33	0.49	0.38	0.43	0.52	0.38	0.39
✓			T.	0.54	0.49	0.43	0.62	0.51	0.44	0.45
✓	✓		T.	0.60	0.73	0.54	0.66	0.67	0.46	0.50
✓	✓	✓	I. + T.	0.63	0.76	0.56	0.70	0.69	0.49	0.53

Table 4: Ablation on our three key components on the DeepTumorVQA and DeepChestVQA datasets. “I.” and “T.” are short of Image and Text. GPT-5 is utilized as a baseline in the ablation experiment.

Figure 6: Slice-selection agreement with radiologists.

Figure 7: Analysis of T1S-Loop’s gain across iteration turns, results are computed on the same set of cases that require $k$ iterations under T1S-Loop.

3D Medical Imaging

3D medical imaging develops methods to analyze, reconstruct, and visualize volumetric scans such as CT, MRI, PET, and ultrasound.

See all 173 papers

Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis
Apr 11, 2026Yang Yu, Dunyuan Xu, Yaoqian Li +3Multimodal Large Language Models 3D VLMs
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China. · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China. +1
CT-Flow: Orchestrating CT Interpretation Workflow with Model Context Protocol Servers
Feb 23, 2026Yannian Gu, Xizhuo Zhang, Linjie Mu +4Medical Image Analysis Medical VLMs
Qing Yuan Research Institute, Shanghai Jiao Tong University, Shanghai, China · Sensetime Research, Shanghai, China · Shanghai Innovation Institute, Shanghai, China
Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models
Jul 15, 2026Zhuoyuan Fu, Zeshang Li, Yiqiong Zhang +53D Spatial Reasoning 3D Medical Imaging
University of International Relations, Beijing, China · University of Trento, Trento, Italy

Apr 11, 2026J/K move · Enter open · S save

Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis

Yang Yu, Dunyuan Xu, Yaoqian Li, Xiaomeng Li+2

Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China. · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China.+1

Multimodal Large Language Models 3D VLMs Medical Report Generation Medical VQA Mixture of Experts

Open paper AskPDF arXiv

3D medical image analysis is of great importance in disease diagnosis and treatment. Recently, multimodal large language models (MLLMs) have exhibited robust perceptual capacity, strong cross-modal alignment, and promising generalizability. Therefore, they have great potential to improve the performance of medical report generation (MRG) and medical visual question answering (MVQA), which serve as two important tasks in clinical scenarios. However, due to the scarcity of 3D medical images, existing 3D medical MLLMs suffer from insufficiently pretrained vision encoder and inability to extract customized image features for different kinds of tasks. In this paper, we propose to first transfer a 2D MLLM, which is well trained with 2D natural images, to support 3D medical volumetric inputs while reusing all of its pre-trained parameters. To enable the vision encoder to extract tailored image features for various tasks, we then design a Text-Guided Hierarchical MoE (TGH-MoE) framework, which can distinguish tasks under the guidance of the text prompt. Furthermore, we propose a two-stage training strategy to learn both task-shared and task-specific image features. As demonstrated empirically, our method outperforms existing 3D medical MLLMs in both MRG and MVQA tasks. Our code will be released once this paper is accepted.

Abstract

3D Medical Imaging

Explore similar work

Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis

CT-Flow: Orchestrating CT Interpretation Workflow with Model Context Protocol Servers

Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models