MedUAG: Unified Understanding and Generation for Medical Multimodal Models

Zijie Meng; Yuncheng Zhang; Hualiang Wang; Yitian Tang; Xiaotang Gai; Chen Shen; Songtao Jiang; Shaosheng Cao

cs.CLAug 19, 2026

MedUAG: Unified Understanding and Generation for Medical Multimodal Models

Authors: Zijie Meng, Yuncheng Zhang, Hualiang Wang, Yitian Tang, Xiaotang Gai, Chen Shen, Songtao Jiang, Shaosheng Cao, +3 more

Organizations: Zhejiang University · Hong Kong University of Science and Technology · Tsinghua University · Tencent Jarvis Lab

Abstract

Recent Multimodal Large Language Models (MLLMs) are rapidly evolving into unified understanding and generation (UAG) frameworks. However, extending these unified paradigms to the medical domain is hindered by: the absence of comprehensive training and evaluation benchmarks, and the lack of broadly validated unified medical model. To address these gaps, we present a comprehensive foundation for medical UAG. First, we construct MedUAGCorpus, the largest unified medical understanding and generation dataset to date, comprising over 6 million instances across 14 imaging modalities. Second, we introduce MedUAGBench, a systematic benchmark that expands medical generation evaluation to 12 diverse tasks under standardized protocols. Finally, leveraging these resources, we develop MedUAG, an end-to-end trained unified medical model. Extensive experiments demonstrate that MedUAG achieves strong performance across a wide array of understanding and generation tasks, establishing a competitive baseline and paving the way for next-generation medical multimodal systems.

Figures & tables

Fig. 1: Comparison of different medical multimodal modeling paradigms. Our framework unifies understanding and generation within a single model.

	Modalities in Generation	Understanding		Generation Tasks (#)				Data Scale
	Modalities in Generation	VQA	MRG	Synthesis	Translation	Reconstruction	Prediction	Data Scale
HealthGPT [ 7 ]	11	✓	✓	1	2	2	✗	1.5M
UniMedVL [ 8 ]	10	✓	✓	2	1	1	1	5.6M
MedUAG (ours)	14	✓	✓	3	3	4	2	6.4M

TABLE I: Comparison with existing unified medical understanding and generation works.

Fig. 2: Overview of MedUAG. Our unified framework supports both medical understanding and generation. On the understanding side, it covers VQA and MRG. On the generation side, it supports four task categories: synthesis, translation, reconstruction, and prediction across diverse medical modalities. Representative demonstrations of each task type are provided.

Fig. 3: The construction process of datasets and model architecture of MedUAG. The dataset construction pipeline includes source data curation, task-specific construction, and unified sample standardization. The model adopts dual image encoders and mixture of transformer experts to support both medical image understanding and generation within a single framework.

Fig. 4: Statistical overview of dataset. (a) MedUAGCorpus: distribution of samples across anatomical systems and imaging modalities, together with the composition of the two training stages, including MedUAG-Corpus Align and MedUAG-Corpus SFT. “T2I”, “I2T”, “Recon”, “Und” and “Gen” indicate text-to-image, image-to-text, reconstruction, understanding and generation, respectively. (b) MedUAGBench: distribution of benchmark samples across anatomical systems and imaging modalities, along with the hierarchical task composition covering reconstruction, translation, synthesis, and prediction.

	Medical	Synthesis Avg.			Translation Avg.				Reconstruction Avg.				Prediction Avg.
	Medical	FID $\downarrow$	gFID $\downarrow$	BioCS $\uparrow$	LPIPS $\downarrow$	MSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	MSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
BLIP3-o [ 5 ]	✗	320.35	316.74	0.349	0.625	0.064	12.536	0.348	0.513	0.056	13.177	0.373	0.713	0.091	10.596	0.268
UniWorld-V1 [ 36 ]	✗	334.51	332.54	0.321	0.609	0.053	13.141	0.377	0.602	0.069	11.931	0.287	0.674	0.084	11.140	0.150
Bagel [ 4 ]	✗	261.39	248.72	0.361	0.613	0.186	9.147	0.307	0.523	0.123	12.301	0.329	0.602	0.165	8.496	0.249
HealthGPT [ 7 ]	✓	211.63	202.48	0.381	0.466	0.057	13.659	0.416	0.370	0.024	17.000	0.533	0.679	0.076	12.003	0.262
UniMedVL [ 8 ]	✓	192.21	179.32	0.403	0.407	0.054	13.616	0.486	0.365	0.063	13.439	0.455	0.531	0.129	11.237	0.354

TABLE II: Average results across the four generation categories. For synthesis, we report task-level macro averages of FID, gFID, and BioCS. For translation, reconstruction, and prediction, we report task-level macro averages of LPIPS, MSE, PSNR, and SSIM.

Model	Unified	VQA-RAD	SLAKE	PathVQA	OM.VQA	Avg.
Proprietary Baselines
GPT-4.1	✗	65.0	72.2	55.5	75.5	67.1
Claude Sonnet 4	✗	67.6	70.6	54.2	65.5	64.5
Gemini-2.5-Flash	✗	68.5	75.8	55.4	71.0	67.7
General-purpose Baslines
Qwen2.5-VL-32B	✗	71.8	71.2	41.9	68.2	63.3

TABLE III: Comparison among various models on understanding benchmarks, where OM.VQA indicates OmniMedVQA.

Fig. 5: Visualization of the MedUAG generation capability.

Align.	Init.	Synthesis	Translation	Reconstruction	Prediction
Align.	Init.	(gFID $\downarrow$ )	(LPIPS $\downarrow$ )	(PSNR $\uparrow$ )	(MSE $\downarrow$ )
✗	Scratch	332.40	0.478	22.449	0.053
✗	Bagel	128.28	0.396	24.476	0.040
✓	Scratch	309.50	0.475	23.078	0.057
✓	Bagel	167.72	0.364	25.357	0.039

TABLE IV: Ablation study of domain alignment and initialization.

Init.	Low-Dose CT	Low-Dose PET	Low-Field MRI	Undersampled MRI	Avg.
From Scratch	+1.13	+1.00	+0.33	+0.05	+0.63
From Bagel	+1.14	+0.05	+1.89	+0.44	+0.88

TABLE V: PSNR gains from domain alignment on reconstruction tasks.

Fig. 6: Comparison of different SFT data ratios.

#Real	#Synthetic	#Total	METEOR	BERTScore	LLM Judge	Average
1K	0	1K	0.098	0.394	0.037	0.176
1K	1K	2K	0.191	0.495	0.100	0.262
2K	0	2K	0.196	0.491	0.125	0.271

TABLE VI: Effect of synthetic data on medical report generation.

Explore similar work

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Medical Report Generation Multimodal Pretraining
LASA Team · DAMO Academy, Alibaba Group
SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment
May 9, 2026Weiren Zhao, Yi Dong, Cheng ChenHealthcare Unified Multimodal Models
The University of Hong Kong, Hong Kong, China.
MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation
Nov 17, 2025Junjie Yang, Yuhao Yan, Gang Wu +12Healthcare Multimodal Generation
South China University of Technology · Sun Yat-sen University · Hangzhou Dianzi University +10

Jun 8, 2025J/K move · Enter open · S save

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied+14

LASA Team · DAMO Academy, Alibaba Group

Medical Report Generation Multimodal Pretraining Multimodal QA Multimodal Reasoning Medical VLMs Multimodal Foundation Models+2

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in understanding common visual elements, largely due to their large-scale datasets and advanced training strategies. However, their effectiveness in medical applications remains limited due to the inherent discrepancies between data and tasks in medical scenarios and those in the general domain. Concretely, existing medical MLLMs face the following critical limitations: (1) limited coverage of medical knowledge beyond imaging, (2) heightened susceptibility to hallucinations due to suboptimal data curation processes, (3) lack of reasoning capabilities tailored for complex medical scenarios. To address these challenges, we first propose a comprehensive data curation procedure that (1) efficiently acquires rich medical knowledge data not only from medical imaging but also from extensive medical texts and general-domain data; and (2) synthesizes accurate medical captions, visual question answering (VQA), and reasoning samples. As a result, we build a multimodal dataset enriched with extensive medical knowledge. Building on the curated data, we introduce our medical-specialized MLLM: Lingshu. Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities progressively. Besides, we preliminarily explore the potential of applying reinforcement learning with verifiable rewards paradigm to enhance Lingshu's medical reasoning ability. Additionally, we develop MedEvalKit, a unified evaluation framework that consolidates leading multimodal and textual medical benchmarks for standardized, fair, and efficient model assessment. We evaluate the performance of Lingshu on three fundamental medical tasks, multimodal QA, text-based QA, and medical report generation. The results show that Lingshu consistently outperforms the existing open-source multimodal models on most tasks ...

Abstract

Explore similar work

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

SynerMedGen: Synergizing Medical Multimodal Understanding with Generation via Task Alignment

MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generation