OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis

Tianwei Lin; Zhongwei Qiu; Wenqiao Zhang; Jiang Liu; Yihan Xie; Mingjian Gao; Zhenxuan Fan; Zhaocheng Li

cs.CVFeb 18, 2026

OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis

Authors: Tianwei Lin, Zhongwei Qiu, Wenqiao Zhang, Jiang Liu, Yihan Xie, Mingjian Gao, Zhenxuan Fan, Zhaocheng Li, +7 more

Organizations: Zhejiang University · DAMO Academy, Alibaba Group · Hupan Lab

Abstract

Computed Tomography (CT) is one of the most widely used and diagnostically information-dense imaging modalities, covering critical organs such as the heart, lungs, liver, and colon. Clinical interpretation relies on both slice-driven local features (e.g., sub-centimeter nodules, lesion boundaries) and volume-driven spatial representations (e.g., tumor infiltration, inter-organ anatomical relations). However, existing Large Vision-Language Models (LVLMs) remain fragmented in CT slice versus volumetric understanding: slice-driven LVLMs show strong generalization but lack cross-slice spatial consistency, while volume-driven LVLMs explicitly capture volumetric semantics but suffer from coarse granularity and poor compatibility with slice inputs. The absence of a unified modeling paradigm constitutes a major bottleneck for the clinical translation of medical LVLMs. We present OmniCT, a powerful unified slice-volume LVLM for CT scenarios, which makes three contributions: (i) Spatial Consistency Enhancement (SCE): volumetric slice composition combined with tri-axial positional embedding that introduces volumetric consistency, and an MoE hybrid projection enables efficient slice-volume adaptation; (ii) Organ-level Semantic Enhancement (OSE): segmentation and ROI localization explicitly align anatomical regions, emphasizing lesion- and organ-level semantics; (iii) MedEval-CT: the largest slice-volume CT dataset and hybrid benchmark integrates comprehensive metrics for unified evaluation. OmniCT consistently outperforms existing methods with a substantial margin across diverse clinical tasks and satisfies both micro-level detail sensitivity and macro-level spatial reasoning. More importantly, it establishes a new paradigm for cross-modal medical imaging understanding. Our project is available at https://github.com/ZJU4HealthCare/OmniCT.

Figures & tables

Figure 1: (a) is the statistics of the proposed MedEval-CT-Dataset. (b) describes the simplified architecture of proposed OmniCT. (c) shows that OmniCT consistently surpasses all baselines on both slice-driven and volume-driven CT benchmarks.

Figure 2: The architecture of OmniCT, a unified slice–volume LVLM paradigm.

Figure 3: (a) and (b) illustrate the data distribution of MedEval-CT-Bench at the slice and volume levels, respectively, encompassing both the clinical-based categorization (4 types: GIR, MAI, AII, and CRD) and the organ-level distribution (13 organs). (c) presents the data engineering pipeline.

SCE	OSE	Public Bench.		MedEval-CT-Bench
SCE	OSE	2D	3D	Organ	Task	Avg.
-	-	78.68	62.17	76.51	78.41	77.62
✓	-	80.14	63.68	76.79	78.69	78.06
-	✓	80.74	65.37	77.02	79.42	78.62
✓	✓	81.45	66.15	78.24	80.27	79.62

Table 1: Ablation analysis of OmniCT.

Model	#Params	SLAKE		VQA-RAD		OmniMedVQA		RadFig-VQA			Avg.
Model	#Params	Close	Open	Close	Open	Task1	Task2	Easy	Medium	Hard	Avg.
Med-LVLM (Slice-centric)
HealthGPT	4B	74.74	56.33	71.88	33.45	57.36	54.20	70.81	71.22	72.90	62.54
HuatuoGPT-V-Qwen2.5	7B	72.68	44.19	72.92	35.68	71.07	83.07	76.56	73.76	71.74	66.85
MedGemma-4B-IT	4B	68.04	53.95	56.25	33.55	61.42	67.00	64.59	65.40	64.20	59.38
MedVLM-R1-2B	2B	-	-	-	-	59.90	67.37	55.50	54.51	55.56	-

Table 2: The comparison of OmniCT with other LVLMs on 2D CT benchmarks.

Model	#Params	M3D			CT-RATE			3D-RAD					Avg.
Model	#Params	Cap	Close	Open	Multi choice	Clinical Entity	Report	I.O.	A.D.	E.D.	STD.	LTD.	Avg.
Med-LVLM (Volume-centric)
M3D-LaMed-7B	7B	24.79	75.78	56.09	47.44	18.15	16.18	16.85	16.71	18.00	25.47	24.17	30.88
M3D-LaMed-4B	4B	46.30	75.08	53.83	59.29	13.66	13.46	17.60	17.49	40.25	25.40	24.31	35.15
CT-CHAT	8B	21.21	35.88	21.81	86.46	49.95	46.76	31.56	29.98	45.33	12.95	13.68	35.97
General-LVLM

Table 3: The comparison of OmniCT with other LVLMs on 3D CT benchmarks.

Figure 4: (a) Comparison of OmniCT with 2D/3D LVLMs on 2D/3D benchmarks using 30%, 100% training data of 2D, 3D, and mixed 2D/3D. (b) The study of using a 3D vision encoder, 2D vision encoders by different pre-training ways. (c) Per-organ performance heatmap of 2D/3D models and OmniCT on 2D/3D MedEval-CT-Bench. (d) Performance heatmaps by clinical task category and bar charts comparing performance with clinical knowledge requirements across task categories.

Encoder	Token Budget Ratio	Plane	Phase	Organ	Abnormality	Location	Avg.
M3D-CLIP	$1.26\times$	99.0	84.8	77.1	78.2	63.1	80.4
VideoMAEv2	$0.97\times$	91.3	74.7	76.9	75.3	62.8	76.2
Wan2.1-VAE	$1.42\times$	97.9	76.4	77.1	74.7	63.9	78.0
DINOv3	$0.61\times$	99.5	88.0	77.8	78.9	65.3	81.9
SigLip	$1.00\times$	99.5	90.2	78.4	79.2	67.4	82.9

Table 4: 2D vs. 3D Encoder Comparison.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
$\mathcal{V}\in\mathbb{R}^{D\times H\times W}$	3D CT volume with depth $D$ , height $H$ , and width $W$
$\mathcal{V}_{j}\in\mathbb{R}^{1\times H\times W}$	$j$ -th 2D slice extracted from the 3D volume
$s_{i}\in\mathbb{R}^{1\times H\times W}$	Independent 2D slice input
$S=\{s_{1},\dots,s_{n}\}$	Collection of independent 2D slice inputs
$\hat{s}_{i}=\mathrm{Concat}(\mathcal{V}_{3i-2},\mathcal{V}_{3i-1},\mathcal{V}_{3i})$	Reassembled volumetric unit
$\hat{\mathcal{S}}=\{\hat{s}_{i}\mid i\in[1,n]\}$	Set of all reassembled slice units

Appendix

Table 5: Notations used throughout this paper.

	OmniCT-3B		OmniCT-7B
Hyperparameter	Stage-1	Stage-2	Stage-1	Stage-2
Optimizer	AdamW	AdamW	AdamW	AdamW
Learning Rate of Adapter	2e-4	5e-5	2e-4	5e-5
Learning Rate of LLM	-	5e-5	-	5e-5
Global Batch Size	256	256	256	256
Weight Decay	0	0	0	0

Appendix

Table 6: Overview of the hyperparameter settings used for training OmniCT-3B and OmniCT-7B across two stages.

Base Model	#Params	SLAKE	VQA-RAD	OmniMed VQA	RadFig -VQA	M3D	CT-RATE	3D-RAD	MedEval-CT -Bench
Qwen2.5-3B	3B	81.6	50.4	97.4	80.2	57.1	67.5	64.9	75.9
Phi-4-mini	3.8B	81.1	55.3	96.9	84.4	56.4	67.4	66.9	76.2

Appendix

Table 7: Performance of OmniCT with different LLM backbones on 2D and 3D CT benchmarks.

Dataset	Task Type	Train Size	Test Size	Source	License
Volume-driven Datasets
M3D-CAP	Report	116065	100	Radiopaedia	Apache-2.0
M3D-VQA	Multiple Choice	240929	5000	Radiopaedia	Apache-2.0
M3D-VQA	Short Answer	240929	5000	Radiopaedia	Apache-2.0
CT-RATEv2	Report	93822	6076	Istanbul Medipol University, Mega Hospital	CC-BY-NC-SA-4.0
CT-RATEv2	Free-form QA	693760	24149	Istanbul Medipol University, Mega Hospital	CC-BY-NC-SA-4.0

Appendix

Table 8: Summary of the datasets included in MedEval-CT, with their task types, training/test sizes, sources, and licenses.

Figure 5: Prompt template of data orchestration engine for generating MedEval-CT.

Figure 6: MedEval-CT-Factory enables standardized preprocessing and fair, consistent evaluation of medical LVLMs across benchmarks.

Figure 7: t-SNE plot showing distinct clusters of 2D and 3D expert features after MHP module.

Model	Qwen2.5-VL	InternVL3-8B	RadFM	Lingshu	HealthGPT	MedVLM-R1	MedGemma-4B	GPT-5-mini	Ours
lungs	72.50	66.25	18.99	70.00	63.75	70.00	66.25	71.25	74.68
heart	69.76	65.85	5.39	77.07	68.29	61.95	66.83	76.47	80.39
liver	76.95	70.17	10.20	86.44	75.25	62.71	68.47	75.59	82.31
spleen	57.38	54.10	10.00	68.85	59.02	47.54	60.66	67.21	68.33
kidneys	72.48	75.19	9.34	80.62	65.12	59.30	60.47	78.29	82.10
pancreas	72.00	66.00	8.08	84.00	71.00	62.00	66.00	75.00	78.79

Appendix

Table 9: MedEval-CT-Bench-2D

Model	Qwen2.5-VL	MiniCPM-V-4.5	CT-CHAT	M3D-LaMed-Phi-3-4B	M3D-LaMed-Llama-2-7B	GPT-5-mini	Ours
lungs	52.07	47.66	52.07	54.27	62.26	53.72	67.22
heart	54.45	59.16	51.83	58.64	63.87	54.45	86.91
liver	55.14	55.39	60.40	75.94	67.42	62.91	80.20
spleen	51.88	45.94	64.38	76.88	74.06	48.44	71.25
kidneys	47.87	41.60	55.89	71.68	80.20	44.11	78.20
pancreas	41.38	40.52	56.47	76.72	70.69	43.53	75.00

Appendix

Table 10: MedEval-CT-Bench-3D

Training Strategy	Perf. 2D	Perf. 3D	Avg.
SigLip + M3D-CLIP (w/o MHP)	34.57	30.58	32.58
SigLip + Siglip (w/ MHP)	55.30	48.61	51.96

Appendix

Table 11: Ablation of MoE Hybrid Projection.

Figure 8: A case study comparing CT findings generated by different medical LVLMs with the clinical ground truth.

ROI Strategy	Perf. 2D	Perf. 3D	Avg.
No OSE	78.68	62.17	70.43
OSE w/ native ROI	78.37	62.24	70.31
OSE w/ random ROI	80.13	64.22	72.18
OSE w/ adaptive ROI	81.45	66.15	73.80

Appendix

Table 12: Ablation analysis of adaptive organ-level feature aggregation.

$\mathbf{m_{2D}}$	$\mathbf{m_{3D}}$	Perf. 2D	Perf. 3D	Avg.
0	0	78.68	62.17	70.43
36	40	80.66	63.81	72.24
81	90	81.45	66.15	73.80
144	160	81.23	66.04	73.64
225	250	80.64	65.48	73.06

Appendix

Table 13: Ablation of the OSE aggregation ratios for 2D and 3D tokens.

m_{2D}

and

m_{3D}

denote the numbers of aggregated organ-level tokens for 2D slices and 3D volumes, respectively.

Organ	2D Normal	2D Mismatch	2D Noise	3D Normal	3D Mismatch	3D Noise
lungs	74.7	23.1	19.5	67.2	29.4	27.3
heart	80.4	24.0	14.9	86.9	31.8	20.1
liver	82.3	19.9	20.9	80.2	30.5	25.8
spleen	68.3	13.7	17.4	71.3	28.4	29.0
kidneys	82.1	25.1	21.1	78.2	22.3	22.6
pancreas	78.8	24.9	15.2	75.0	30.2	23.4

Appendix

Table 14: Organ-level accuracy on MedEval-CT-Bench under normal, image–question mismatch, and noise settings

Training Strategy	Ratio	SLAKE	VQA-RAD	RadFig-VQA	M3D-VQA	CT-RATE	3D-RAD
2D-Only	25%	70.6	62.5	77.2	-	-	-
3D-Only	25%	-	-	-	65.8	84.5	65.4
Mixed	25%	72.2	62.5	78.2	69.9	84.9	66.9
2D-Only	50%	75.8	66.7	79.4	-	-	-
3D-Only	50%	-	-	-	72.7	86.2	67.2
Mixed	50%	77.3	70.8	79.8	73.6	87.1	68.1

Appendix

Table 15: Ablation study of unified representation gains. Compared to 2D-only and 3D-only training, mixed training consistently improves performance across all data scales while preserving the same 2D/3D ratio.

Model	Precision	Recall	F1
RadFM	13.1	6.4	7.2
M3D-LaMed-7B	8.1	2.5	3.5
M3D-LaMed-4B	16.5	8.4	9.6
CT-CHAT	24.3	38.8	27.2
CT2Rep	41.6	38.1	36.7
CT-AGRG	37.8	55.4	42.1

Appendix

Table 16: Performance of 18 types of anomaly label prediction.

Explore similar work

Segmentation, Detection and Explanation: A Unified Framework for CT Appearance Reasoning
May 15, 2026Yuyuan Liu, Can Peng, Yingyu Yang +3Image Segmentation Medical VLMs
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford
Representation geometry shapes task performance in vision-language modeling for CT enterography
Apr 14, 2026Cristian Minoccheri, Emily Wittrup, Kayvan Najarian +1Medical VLMs Medical Image Classification
Gilbert S. Omenn Department of Computational Medicine and Bioinformatics, University of Michigan, 1109 Geddes Avenue, Ann Arbor, 48104, MI, USA. · Department of Emergency Medicine, University of Michigan, 1109 Geddes Avenue, Ann Arbor, 48104, MI, USA. · Department of Electrical Engineering and Computer Science, University of Michigan, 1109 Geddes Avenue, Ann Arbor, 48104, MI, USA. +1
Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
Mar 2, 2026Simon Ging, Philipp Arnold, Sebastian Walter +6Medical VLMs 3D VLMs
Computer Vision Group, University of Freiburg, Germany · Adaptive & Agentic AI · Adaptive & Agentic AI (A3) Lab, Aarhus University, Denmark +5

May 15, 2026J/K move · Enter open · S save

Segmentation, Detection and Explanation: A Unified Framework for CT Appearance Reasoning

Yuyuan Liu, Can Peng, Yingyu Yang, Qianye Yang+2

Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford

Image Segmentation Medical VLMs Medical Image Segmentation Computed Tomography Medical Image Localization

Open paper AskPDF arXiv

Recent progress in deep learning has significantly advanced CT image analysis, particularly for segmentation tasks. However, these advances are largely confined to image-level pattern recognition, with most methods lacking explicit anatomical or contextual reasoning. Large vision-language models introduce linguistic context into image analysis, yet most approaches typically focus on a single task, which is insufficient for clinical workflow analysis that requires multiple fine-grained types of analysis, such as anatomy detection and segmentation. In this paper, we propose a unified autoregressive framework that integrates language-guided visual reasoning into CT interpretation. Our method introduces task-routing tokens that trigger detection and segmentation heads conditioned on the hidden states of a large vision-language model, enabling coherent generation of visual outputs (e.g., masks and bounding boxes) and textual reasonings. To progressively enhance localisation accuracy and semantic clarity, we further design a "closer-look" mechanism that allows the model to perform progressive coarse-to-fine visits to regions of interest under refined fields of view. To support model training and evaluation, we curated a new multimodal CT dataset containing pixel-wise masks, bounding boxes, spatial prompts, and structured descriptions for visual objects constructed through an AI-assisted annotation process with human verification. Experiments on public benchmarks demonstrate consistent improvements over the SoTA, achieving up to 1.0% Dice on BTCV and 1.7% Dice on MosMed+, while additionally providing appearance reasoning outputs. The code and dataset will be available.

Abstract

Explore similar work

Segmentation, Detection and Explanation: A Unified Framework for CT Appearance Reasoning

Representation geometry shapes task performance in vision-language modeling for CT enterography

Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT