Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis

Yang Yu; Dunyuan Xu; Yaoqian Li; Xiaomeng Li; Jinpeng Li; Pheng-Ann Heng

cs.CVApr 11, 2026

Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis

Authors: Yang Yu, Dunyuan Xu, Yaoqian Li, Xiaomeng Li, Jinpeng Li, Pheng-Ann Heng

Organizations: Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China. · Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong, China. · Institute of Medical Intelligence and XR, The Chinese University of Hong Kong, Hong Kong, China.

Abstract

3D medical image analysis is of great importance in disease diagnosis and treatment. Recently, multimodal large language models (MLLMs) have exhibited robust perceptual capacity, strong cross-modal alignment, and promising generalizability. Therefore, they have great potential to improve the performance of medical report generation (MRG) and medical visual question answering (MVQA), which serve as two important tasks in clinical scenarios. However, due to the scarcity of 3D medical images, existing 3D medical MLLMs suffer from insufficiently pretrained vision encoder and inability to extract customized image features for different kinds of tasks. In this paper, we propose to first transfer a 2D MLLM, which is well trained with 2D natural images, to support 3D medical volumetric inputs while reusing all of its pre-trained parameters. To enable the vision encoder to extract tailored image features for various tasks, we then design a Text-Guided Hierarchical MoE (TGH-MoE) framework, which can distinguish tasks under the guidance of the text prompt. Furthermore, we propose a two-stage training strategy to learn both task-shared and task-specific image features. As demonstrated empirically, our method outperforms existing 3D medical MLLMs in both MRG and MVQA tasks. Our code will be released once this paper is accepted.

Figures & tables

Fig. 1: (a) Existing method pre-trains a 3D vision encoder with only 3D medical images in CLIP fashion. It also extracts the same image features for different tasks. (b) Our method adapts a 2D vision encoder to 3D medical images, which is pretrained with massive natural images and aligned with LLMs through instruction-tuning. We further propose TGH-MoE to extract task-specific image features for MRG and MVQA.

Fig. 2: Illustration of adapting 2D ViT for 3D medical images. We split a 3D image into multiple 3-slice images for patch embedding processing. We modify 2D attention blocks in higher layers into 3D attention blocks to capture 3D spatial information. RoPE is added to 3D attention blocks to distinguish positions along the $D$ dimension.

Fig. 3: We propose a text-guided hierarchical MoE (TGH-MoE) and a two-stage training strategy to learn task-shared and -specific image features for MRG and MVQA tasks. (a) In Stage 1, we use only token-level MoE so that the vision encoder can learn task-shared features for MRG and MVQA. (b) In Stage 2, we replicate token-level MoE as the experts of task-level MoE to learn task-specific image features for MRG and MVQA under the guidance of text prompt. (c) TGH-MoE is a two-level MoE. Task-level MoE first routes all image tokens to a specific expert under the guidance of the text prompt. Token-level MoE provides a sparse path toward a larger model by routing image tokens to different experts.

Method	Model Size	Medical VQA					Report Generation
Method	Model Size	BLEU-1	ROUGE-1	METEOR	BERT	Accuracy	BLEU-1	ROUGE-1	METEOR	BERT
LLAVA-Med-1.5 $\dagger$	7B	34.53	36.54	25.43	89.06	64.41	14.25	16.08	12.74	82.53
RadFM $\dagger$	14B	16.39	26.13	21.33	88.72	19.79	12.23	16.49	11.57	87.93
M3D $\star$	4B	52.10	55.19	35.89	92.14	79.35	13.90	18.02	12.77	83.63
Med3DVLM $\star$	7B	47.54	50.98	32.68	91.32	76.89	13.86	18.01	12.64	83.70
Ours	4B	54.64	57.97	37.85	92.57	81.06	14.23	19.03	13.40	84.03

TABLE I: Comparison with existing methods. All results are reported in percentages (

\%

). For Medical VQA, we report the average of five different topics.

\dagger

indicates that the existing methods are not trained on M3D-Data, while

\star

is on the contrary.

No.	2D to 3D Adap.	TGH-MoE	Two-stage Train. Stra.	Medical VQA					Report Generation
No.	2D to 3D Adap.	TGH-MoE	Two-stage Train. Stra.	B-1	R-1	M	BERT	Accuracy	B-1	R-1	M	BERT
1				52.10	55.35	35.62	92.15	78.41	14.67	19.54	13.97	83.88
2		$\checkmark$		52.83	56.05	36.44	92.30	78.96	14.78	19.48	14.08	83.94
3		$\checkmark$	$\checkmark$	52.87	56.08	36.43	92.29	79.11	14.73	19.87	13.87	83.98
4	$\checkmark$			52.74	55.95	36.20	92.30	79.13	14.69	20.02	14.09	84.05
5	$\checkmark$	$\checkmark$		53.02	56.17	36.52	92.33	79.28	14.91	19.55	14.11	83.92

TABLE II: Components analysis of our method. We conduct this ablation to illustrate the importance of our three key components, i.e., 2D to 3D adaptation, TGH-MoE and two-stage training strategy.

Design Choices			Medical VQA					Report Generation
Description	$N_{Stage}$	MoE Type	B-1	R-1	M	BERT	Accuracy	B-1	R-1	M	BERT
Only Task-shared	1	Token-level	52.77	56.05	36.29	92.29	79.24	14.96	20.03	14.07	84.06
Only Task-specific	1	TGH-MoE	53.02	56.17	36.52	92.33	79.28	14.91	19.55	14.11	83.92
Finetune LLM in Stage 2	2	TGH-MoE	52.73	55.98	36.47	92.26	79.29	15.10	19.62	13.91	83.89
Task-shared and -specific (Ours)	2	TGH-MoE	53.10	56.39	36.61	92.33	79.32	15.10	20.31	14.23	84.12

TABLE III: Ablation on training-strategy. We compare our training strategy with strategies that can learn only task-specific or shared image features. For “Only Task-shared”, we abandon task-level MoE so that different tasks share the same expert which is a token-level MoE. For “Only Task-specific”, we skip Stage 1 training and keep Stage 2 training.

Fig. 4: Ablation on the choice of the vision encoder. We compare the frozen and optimized vision encoder during training. We also compare our vision encoder with other 2D and 3D alternatives.

Fig. 5: Ablation on TGH-MoE. We compare our TGH-MoE using a hard router with TGH-MoE using a soft router and token-level MoE.

Fig. 6: Qualitative comparisons with our model and ground truth on report generation and visual question answering. Green represents matched text. Orange represents unmatched text. Blue represents excessive text.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis
Feb 20, 2026Ziyue Wang, Linghan Cai, Chang Han Low +83D Medical Imaging Medical VLMs
National University of Singapore · Microsoft Research · TUD Dresden University of Technology +2
MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation
Jan 14, 2026Yang Xing, Jiong Wu, Savas Ozdemir +4Interactive Medical Image Segmentation Unified Multimodal Models
Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA · Department of Radiology, University of Florida, Jacksonville, FL, USA · Research Computing, University of Florida, Gainesville, FL, USA +2
Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models
Jul 15, 2026Zhuoyuan Fu, Zeshang Li, Yiqiong Zhang +53D Spatial Reasoning 3D Medical Imaging
University of International Relations, Beijing, China · University of Trento, Trento, Italy

Feb 20, 2026J/K move · Enter open · S save

3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis

Ziyue Wang, Linghan Cai, Chang Han Low, Haofeng Liu+7

National University of Singapore · Microsoft Research · TUD Dresden University of Technology+2

3D Medical Imaging Medical VLMs 3D VQA Medical VQA LLM Agents

Open paper AskPDF arXiv

3D CT analysis spans a continuum from low-level perception to high-level clinical understanding. Existing 3D-oriented analysis methods adopt either isolated task-specific modeling or task-agnostic end-to-end paradigms to produce one-hop outputs, impeding the systematic accumulation of perceptual evidence for downstream reasoning. In parallel, recent multimodal large language models (MLLMs) exhibit improved visual perception and can integrate visual and textual information effectively, yet their predominantly 2D-oriented designs fundamentally limit their ability to perceive and analyze volumetric medical data. To bridge this gap, we propose 3DMedAgent, a unified agent that enables 2D MLLMs to perform general 3D CT analysis without 3D-specific fine-tuning. 3DMedAgent coordinates heterogeneous visual and textual tools through a flexible MLLM agent, progressively decomposing complex 3D analysis into tractable subtasks that transition from global to regional views, from 3D volumes to informative 2D slices, and from visual evidence to structured textual representations. Central to this design, 3DMedAgent maintains a long-term structured memory that aggregates intermediate tool outputs and supports query-adaptive, evidence-driven multi-step reasoning. We further introduce the DeepChestVQA benchmark for evaluating unified perception-to-understanding capabilities in 3D thoracic imaging. Experiments across over 40 tasks demonstrate that 3DMedAgent consistently outperforms general, medical, and 3D-specific MLLMs, highlighting a scalable path toward general-purpose 3D clinical assistants.Code and data are available at https://github.com/jinlab-imvr/3DMedAgent.

Abstract

Multimodal Large Language Models

Explore similar work

3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis

MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation

Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models