U-VLM: Hierarchical Vision Language Modeling for Report Generation

Pengcheng Shi; Minghui Zhang; Kehan Song; Jiaqi Liu; Yun Gu; Xinglin Zhang

cs.CVFeb 28, 2026

U-VLM: Hierarchical Vision Language Modeling for Report Generation

Authors: Pengcheng Shi, Minghui Zhang, Kehan Song, Jiaqi Liu, Yun Gu, Xinglin Zhang

Organizations: Medical Image Insights Co. Ltd., Shanghai, China · Shanghai Jiao Tong University, Shanghai, China

Abstract

Automated radiology report generation is key for reducing radiologist workload and improving diagnostic consistency, yet generating accurate reports for 3D medical imaging remains challenging. Existing vision-language models face two limitations: they do not leverage segmentation-pretrained encoders, and they inject visual features only at the input layer of language models, losing multi-scale information. We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture: (1) progressive training from segmentation to classification to report generation, and (2) multi-layer visual injection that routes U-Net encoder features to corresponding language model layers. Each training stage can leverage different datasets without unified annotations. U-VLM achieves state-of-the-art performance on CT-RATE (F1: 0.414 vs 0.258, BLEU-mean: 0.349 vs 0.305) and AbdomenAtlas 3.0 (F1: 0.624 vs 0.518 for segmentation-based detection) using only a 0.1B decoder trained from scratch, demonstrating that well-designed vision encoder pretraining outweighs the benefits of 7B+ pre-trained language models. Ablation studies show that progressive pretraining significantly improves F1, while multi-layer injection improves BLEU-mean. Code is available at https://github.com/yinghemedical/U-VLM.

Figures & tables

Method	VE	LD	VE Init	LD Init	E2E
CT2Rep [ 8 ]	Causal Trans.	Trans. Dec.	Scratch	Scratch	✓
RadFM [ 21 ]	ViT+Perc.	MedLLaMA-13B	Scratch	Fine-tune	✓
M3D-LaMed [ 2 ]	ViT+Perc.	LLaMA-2-7B	Contrastive	LoRA	✓
Merlin [ 5 ]	3D ResNet	RadLlama-7B	Cls	LoRA	✓
CT-CHAT [ 10 ]	ViT+AP	LLaMA-3.1-70B	Contrastive	LoRA	✓
MedM-VL [ 18 ]	ViT+AP	Qwen2.5-3B	Contrastive	Fine-tune	✓

Table 1: Comparison with related work on 3D radiology report generation. Existing methods inject visual features only at the language model input layer, and none leverage segmentation-pretrained U-Net for end-to-end report generation. VE: Vision Encoder. LD: Language Decoder. E2E: End-to-End. VE Init: Contrastive (CLIP [ 16 ] -style), VQ Recon (vector-quantized reconstruction), Cls (classification), Seg (segmentation). Perc.: Perceiver. AP: Attention Pooling.

Figure 1: U-VLM framework. Stage 1: Segmentation pretraining for learning fine-grained spatial structures. Stage 2: Classification pretraining for disease pattern recognition. Stage 3: Report generation via multi-layer injection (deep encoder $\rightarrow$ early language layers, shallow encoder $\rightarrow$ later layers).

Figure 2: Qualitative results of segmentation and report generation on chest CT (CT-RATE) and abdominal CT (AbdomenAtlas 3.0). We visualize the input 3D CT volumes alongside segmentation predictions: Seg(F+L) for chest CT and Seg(C+L) for abdominal CT. For all reports, text is color-coded to highlight abnormalities, maintaining consistent colors for the same pathology, while normal descriptions are shown in black.

Method	F1	Prec.	Rec.	B-1	B-2	B-3	B-4	B-mean
CT2Rep [ 8 ]	0.160	0.435	0.128	0.372	0.292	0.243	0.213	0.280
Merlin [ 5 ]	0.160	0.295	0.112	0.231	0.163	0.124	0.099	0.154
CT-CHAT [ 10 ]	0.184	0.450	0.158	0.373	0.284	0.231	0.198	0.272
MedM-VL [ 18 ]	0.167	0.323	0.160	0.390	0.295	0.240	0.205	0.283
BTB3D-8 [ 9 ]	0.187	0.260	0.150	0.411	0.307	0.245	0.215	0.295
BTB3D-16 [ 9 ]	0.258	0.260	0.260	0.420	0.318	0.256	0.225	0.305

Table 2: Report generation on CT-RATE.

End-to-End Report Generation
	Pancreatic Lesion			Kidney Lesion			Liver Lesion
Model	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
M3D (zero-shot) [ 2 ]	11.8	3.6	5.6	23.5	9.2	13.2	16.5	10.7	13.0
RadFM (zero-shot) [ 21 ]	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
U-VLM Seg(C+L)	61.0	58.2	59.5	54.8	79.4	64.8	58.1	68.7	62.9
U-VLM Seg(F+L)	51.2	39.1	44.3	59.8	70.2	64.6	45.8	75.6	57.1

Table 3: Lesion detection on AbdomenAtlas 3.0.

	CT-RATE			AbdomenAtlas 3.0
Training	F1	Precision	Recall	F1	Precision	Recall
None $\rightarrow$ Cls	0.451	0.541	0.404	0.373	0.304	0.557
Seg(C) $\rightarrow$ Cls	0.546	0.587	0.535	0.617	0.568	0.679
Seg(C+L) $\rightarrow$ Cls	0.553	0.595	0.541	0.646	0.595	0.716
Seg(F+L) $\rightarrow$ Cls	0.555	0.598	0.539	0.578	0.567	0.589

Table 4: Classification ablation on CT-RATE and AbdomenAtlas 3.0.

					CT-RATE		Abdomen
Training	SC	Frz	VT	Dec.	F1	B-M	F1	B-M
None $\rightarrow$ Rep		✓	384	0.1B	0.062	0.251	0.264	0.336
Cls $\rightarrow$ Rep		✓	384	0.1B	0.292	0.294	0.384	0.391
Seg(C+L) $\rightarrow$ Rep		✓	384	0.1B	0.232	0.297	0.348	0.341
Seg(F+L) $\rightarrow$ Rep		✓	384	0.1B	0.238	0.278	0.398	0.392
Seg(C) $\rightarrow$ Cls $\rightarrow$ Rep		✓	384	0.1B	0.399	0.298	0.592	0.423

Table 5: Report generation ablation on CT-RATE and AbdomenAtlas 3.0.

Explore similar work

Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography
Jun 24, 2026Bowen Shi, Weiwei Cao, Ruifeng Yuan +5Contrastive Learning 3D Medical Imaging
DAMO Academy, Alibaba Group · Hupan Lab, 310023, Hangzhou, China · Shanghai Jiao Tong University, China +2
Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression
Mar 24, 2026V. K. Cody Bumgardner, Mitchell A. Klusty, Mahmut S. Gokmen +1Radiology Report Generation Computed Tomography
Center for Applied Artificial Intelligence, University of Kentucky, Lexington, KY 40506 USA
NV-Reason-CT: 3D Visual Language Model for CT Analysis
Sep 23, 2026Andriy Myronenko, Dong Yang, Yucheng Tang +15Radiology Report Generation Radiology VLMs
NVIDIA, Santa Clara, CA · NIH, Bethesda, MD · University of Oxford, Oxford, UK +4

Jun 24, 2026J/K move · Enter open · S save

Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

Bowen Shi, Weiwei Cao, Ruifeng Yuan, Wanxing Chang+4

DAMO Academy, Alibaba Group · Hupan Lab, 310023, Hangzhou, China · Shanghai Jiao Tong University, China+2

Contrastive Learning 3D Medical Imaging Medical Image Classification Medical VLMs Zero-Shot Medical Image Segmentation 3D VLMs

Open paper AskPDF arXiv

Vision-language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet existing methods struggle with 3D CT imaging due to inefficient visual backbones and coarse semantic alignment. To address these issues, we propose a tailored VLP framework featuring three key components: (1) a CNN-ViT hybrid encoder that replaces ViT's patch embedding with a 3D CNN backbone to efficiently capture local anatomical details while preserving global attention and compatibility with pre-trained cross-modal priors; (2) a disease-level contrastive learning mechanism using learnable query tokens to dynamically extract disease-specific semantics from full reports and align them with corresponding visual features, thereby disentangling distinct diseases within the same anatomical region; and (3) a diagnosis-aware prompt strategy that employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap and enhance zero-shot diagnostic reliability. Our model achieves state-of-the-art performance on CT-RATE (84.4% AUC, +5.1%) and Rad-ChestCT (75.4% AUC, +5.4%), with even larger gains (+9.8% AUC) on a challenging 60-disease benchmark, and demonstrates strong transferability to radiology report generation, underscoring the generality and clinical utility of our approach.

Abstract

Explore similar work

Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

NV-Reason-CT: 3D Visual Language Model for CT Analysis