Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-training

Peng Sun; Jun Xie; Tao Lin

cs.CVMar 17, 2026

Rethinking UMM Visual Generation: Masked Modeling for Efficient Image-Only Pre-training

Authors: Peng Sun, Jun Xie, Tao Lin

Organizations: Zhejiang University · Westlake University · Shanghai Innovation Institute

Abstract

Unified Multimodal Models (UMMs) are often constrained by the pre-training of their $\textbf{visual generation components}$ , which typically relies on inefficient paradigms and scarce, high-quality text-image paired data. In this paper, we systematically analyze pre-training recipes for $\textbf{UMM visual generation}$ and identify these two issues as the major bottlenecks. To address them, we propose $\textbf{Image-Only Training for UMMs (IOMM)}$ , a data-efficient two-stage training framework. The first stage pre-trains the visual generative component $\textbf{exclusively}$ using abundant unlabeled image-only data, thereby removing the dependency on paired data $\textbf{for this costly phase}$ . The second stage fine-tunes the model using a mixture of unlabeled images and a small curated set of text-image pairs, leading to improved instruction alignment and generative quality. Extensive experiments show that IOMM not only improves training efficiency but also achieves state-of-the-art (SOTA) performance. For example, our IOMM-B (3.6B) model was trained from scratch using only $\sim \textbf{1050}$ H800 GPU hours (with the vast majority, $\textbf{1000}$ hours, dedicated to the efficient $\textbf{image-only pre-training stage}$ ). It achieves $\textbf{0.89}$ on GenEval and $\textbf{0.55}$ on WISE--surpassing strong baselines such as BAGEL-7B (0.82 & 0.55) and BLIP3-o-4B (0.84 & 0.50). Code is available https://github.com/LINs-lab/IOMM.

Figures & tables

Panel of Figure 1 — Figure 1: An overview and validation of our proposed training paradigm. (a) Visual results of our IOMM -XL, demonstrating high-quality, multi-resolution image synthesis. Corresponding prompts are provided in App. C.7 . (b) An illustration of the six training recipes we investigate. (c) Quantitative results of six training recipes on the GenEval benchmark.

Panel of Figure 2 — Figure 2: Visualization of the IOMM framework. (a) The architecture of our proposed framework. (b) Ablation study demonstrating the effectiveness of architectural design choices, confirming that each component contributes positively to the final GenEval score. All variants utilize the same IOMM -XL architecture.

Gen. Only
METHOD	GenEval							DPGBench ( $\uparrow$ )	WISE ( $\uparrow$ )
METHOD	Single Obj.	Two Obj.	Counting	Colors	Position	Color Attri.	Overall ( $\uparrow$ )	DPGBench ( $\uparrow$ )	WISE ( $\uparrow$ )
SDv1.5 [ 42 ]	0.97	0.38	0.35	0.76	0.04	0.06	0.43	63.18	0.32
SDv2.1 [ 42 ]	0.98	0.51	0.44	0.85	0.07	0.17	0.50	-	0.32
SD3-Medium [ 14 ]	0.99	0.94	0.72	0.89	0.33	0.60	0.74	84.08	0.42
SDXL [ 40 ]	0.98	0.74	0.39	0.85	0.15	0.23	0.55	74.65	0.43

Table 1: Quantitative comparison on text-to-image generation benchmarks. The (

\uparrow

) symbol indicates that higher scores are better. † Results obtained using rewritten prompts from the original GenEval benchmark. ∗ Indicates the model was trained on an additional 30M proprietary image-text pairs.

METHOD	Res.	NFE	GenEval							WISE ( $\uparrow$ )
METHOD	Res.	NFE	Single Obj.	Two Obj.	Counting	Colors	Position	Color Attri.	Overall ( $\uparrow$ )	WISE ( $\uparrow$ )
OpenUni-L [ 50 ]	512	20 $\times$ 2	0.99	0.91	0.77	0.90	0.75	0.76	0.85	0.52
$\boldsymbol{\color[rgb]{0.7,0,0}\oplus}$ Image finetuning	512	20 $\times$ 2	1.00 ↑0.01	0.98 ↑0.07	0.22 ↓0.55	0.91 ↑0.01	0.60 ↓0.15	0.77 ↑0.01	0.74 ↓0.11	0.49 ↓0.03
$\boldsymbol{\color[rgb]{0.7,0,0}\oplus}$ Pair finetuning	512	20 $\times$ 2	0.99	0.94 ↑0.03	0.82 ↑0.05	0.91 ↑0.01	0.85 ↑0.10	0.76	0.88 ↑0.03	0.62 ↑0.10
$\boldsymbol{\color[rgb]{0.7,0,0}\oplus}$ Mix finetuning	512	20 $\times$ 2	0.99	0.91	0.78 ↑0.01	0.93 ↑0.03	0.87 ↑0.12	0.78 ↑0.02	0.88 ↑0.03	0.59 ↑0.07
Qwen-Image [ 48 ]	512	50 $\times$ 2	0.99	0.91	0.87	0.88	0.73	0.74	0.85	-

Table 2: Evaluating different fine-tuning strategies on various open-source UMMs. The notation A

\boldsymbol{\color[rgb]{0.7,0,0}\oplus}

B denotes applying fine-tuning method B to a pre-trained model A. The symbols ↓ / ↑ indicate the performance change relative to the baseline pre-trained model.

Trained with editing data
METHOD	ImgEdit-Bench
METHOD	Add	Adjust	Extract	Replace	Remove	Background	Style	Hybrid	Action	Overall ( $\uparrow$ )
MagicBrush [ 63 ]	2.84	1.58	1.51	1.97	1.58	1.75	2.38	1.62	1.22	1.90
Instruct-Pix2Pix [ 3 ]	2.45	1.83	1.44	2.01	1.50	1.44	3.55	1.20	1.46	1.88
AnyEdit [ 61 ]	3.18	2.95	1.88	2.47	2.23	2.24	2.85	1.56	2.65	2.45
UltraEdit [ 65 ]	3.44	2.81	2.13	2.96	1.45	2.83	3.76	1.91	2.98	2.70

Table 3: Image editing benchmark results. Methods highlighted in red are trained on specific editing datasets. Our IOMM , highlighted in blue , is evaluated in a training-free setting without any training on editing data.

Figure 3: Analysis of different data paradigms. Fine-tuning performance comparison of models pre-trained on different data compositions (image-only, text-image pair) across distinct datasets.

Panel of Figure 4 — Figure 4: Ablation studies of key components in IOMM . These experiments analyze the impact of our primary design choices: (a) the residual query adapter, (b) the mask ratio for sparse reconstruction, and (c) the data mixture ratio during fine-tuning.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Optimization
METHOD	IOMM -B	IOMM -L	IOMM -XL
Optimizer	AdamW		Muon
$\beta$	$(0.9,0.95)$		$(0.9,0.95)$
Learning rate	1e-4		1e-4
Max gradient norm	1.0		1.0
Weight decay	0.0		0.0

Appendix

Table 4: Pre-training settings.

METHOD	IOMM -B		IOMM -L		IOMM -XL
Resolution	512	1024	512	1024	512
Optimization
Optimizer	AdamW		AdamW		Muon
$\beta$	$(0.9,0.95)$		$(0.9,0.95)$		$(0.9,0.95)$
Learning rate	1e-4		1e-4		1e-4
Max gradient norm	1.0		1.0		1.0

Appendix

Table 5: Finetuning settings.

Optimization
METHOD	OpenUni-L	Qwen-Image-20B
Optimizer	AdamW	AdamW
$\beta$	(0.9,0.95)	(0.9,0.95)
Learning rate	1e-4	1e-4
Max gradient norm	1.0	1.0
Weight decay	0.0	0.0

Appendix

Table 6: UMM finetuning settings.

Gen. Only
METHOD	Global	Entity	Attribute	Relation	Other	Overall
SDv1.5 [ 42 ]	74.63	74.23	75.39	73.49	67.81	63.18
SD3-Medium [ 14 ]	87.90	91.01	88.83	80.70	88.68	84.08
SDXL [ 40 ]	83.27	82.43	80.91	86.76	80.41	74.65
PixArt- $\alpha$ [ 9 ]	74.97	79.32	78.60	82.57	76.96	71.11
FLUX.1-dev [ 2 ]	74.35	90.00	88.96	90.87	88.33	83.84

Appendix

Table 7: DPGBench evaluation results. Here BLIP3-o-8B* donates the model that is trained with an 30 million proprietary data.

Gen. Only
METHOD	Cultural	Time	Space	Biology	Physics	Chemistry	Overall
SDv1.5 [ 42 ]	0.34	0.35	0.32	0.28	0.29	0.21	0.32
SDv2.1 [ 42 ]	0.30	0.38	0.35	0.33	0.34	0.21	0.32
SD3-Medium [ 14 ]	0.42	0.44	0.48	0.39	0.47	0.29	0.42
SDXL [ 40 ]	0.43	0.48	0.47	0.44	0.45	0.27	0.43
SD3.5-Large [ 14 ]	0.44	0.50	0.58	0.44	0.52	0.31	0.46

Appendix

Table 8: WISE evaluation results. Here BLIP3-o-8B* donates the model that is trained with an 30 million proprietary data.

Finetuning Recipe	Single Obj.	Two Obj.	Counting	Colors	Position	Color Attri.	Overall ( $\uparrow$ )
Pre-trained with Text-Image Pair Data
Image	1.00	0.95	0.63	0.87	0.50	0.72	0.78
Pair	0.99	0.92	0.76	0.91	0.87	0.69	0.86
Mix	0.99	0.91	0.80	0.92	0.90	0.75	0.88
Pre-trained with Image-Only Data
Image	0.99	0.84	0.24	0.75	0.37	0.45	0.61

Appendix

Table 9: Training recipe comparison. The GenEval score of the models pre-trained with different training recipes. Bold denotes the best performance and underline denotes the second best performance.

Figure 5: Image editing ability with different pre-training method.

METHOD	Res.	NFEs	Cultural	Time	Space	Biology	Physics	Chemistry	Overall
OpenUni-L [ 50 ]	512	20 $\times$ 2	0.51	0.45	0.58	0.39	0.50	0.30	0.52
$\boldsymbol{\color[rgb]{0.7,0,0}\oplus}$ Image finetuning	512	20 $\times$ 2	0.46	0.52	0.66	0.49	0.51	0.29	0.49
$\boldsymbol{\color[rgb]{0.7,0,0}\oplus}$ Pair finetuning	512	20 $\times$ 2	0.63	0.58	0.74	0.57	0.71	0.44	0.62
$\boldsymbol{\color[rgb]{0.7,0,0}\oplus}$ Mix finetuning	512	20 $\times$ 2	0.60	0.58	0.70	0.51	0.64	0.46	0.59
Qwen-Image [ 48 ]	512	50 $\times$ 2	-	-	-	-	-	-	-
$\boldsymbol{\color[rgb]{0.7,0,0}\oplus}$ Image finetuning	512	50 $\times$ 2	0.39	0.42	0.56	0.32	0.50	0.28	0.41

Appendix

Table 10: UMM finetuning WISE results. Notation A

\boldsymbol{\color[rgb]{0.7,0,0}\oplus}

B denotes the result obtained by combining methods A and B.

Figure 6: Generation results of OpenUni-L before and after finetuning. The left one is the image generated by the original OpenUni-L, while the right one is generated by the OpenUni-L after finetuning.

Panel of Figure 7 — Figure 7: (a, b) Qualitative comparison between the original Qwen-Image model and our fine-tuned version. Our method enhances the model’s ability to generate images with richer visual detail and improved alignment to the textual prompt.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 290 papers

Reconstruction Alignment Improves Unified Multimodal Models
Sep 8, 2025Ji Xie, Trevor Darrell, Luke Zettlemoyer +1Unified Multimodal Models Cross-Modal Alignment
1UC Berkeley · University of Washington · 3Duke University
Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards
Jun 25, 2026Ritesh Thawkar, Shravan Venkatraman, Omkar Thawakar +5Unified Multimodal Models Multimodal Generation
1Mohamed bin Zayed University of Artificial Intelligence · 4Linköping University · 3Australian National University +1
Transferability Between Understanding and Generation in Unified Multimodal Models
Jul 5, 2026Jiwon Kang, Heeji Yoon, Jaewoo Jung +5Unified Multimodal Models Multimodal Generation
KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea

Sep 8, 2025J/K move · Enter open · S save

Reconstruction Alignment Improves Unified Multimodal Models

Ji Xie, Trevor Darrell, Luke Zettlemoyer, XuDong Wang

1UC Berkeley · University of Washington · 3Duke University

Unified Multimodal Models Cross-Modal Alignment Multimodal Generation Multimodal Foundation Models Multimodal Understanding

Open paper AskPDF arXiv

Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture. However, conventional training relies on image-text pairs (or sequences) whose captions are typically sparse and miss fine-grained visual details, even when they use hundreds of words to describe a simple image. We introduce Reconstruction Alignment (RECA), a resource-efficient post-training method that leverages visual understanding encoder embeddings as dense "text prompts", providing rich supervision without captions. Concretely, RECA conditions a UMM on its own visual understanding embeddings and optimizes it to reconstruct the input image with a self-supervised reconstruction loss, thereby realigning understanding and generation. Despite its simplicity, RECA is broadly applicable: across autoregressive, masked-autoregressive, and diffusion-based UMMs, it consistently improves generation and editing fidelity. With only 27 GPU hours, post-training with RECA substantially improves image generation performance on GenEval (0.73

\rightarrow

0.90) and DPGBench (80.93

\rightarrow

88.15), while also boosting editing benchmarks (ImgEdit 3.38

\rightarrow

3.75, GEdit 6.94

\rightarrow

7.27). Notably, RECA surpasses much larger open-source models and applies broadly across diverse UMM architectures, establishing it as an efficient and general post-training alignment strategy for UMMs.

Abstract

Unified Multimodal Models

Explore similar work

Reconstruction Alignment Improves Unified Multimodal Models

Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards

Transferability Between Understanding and Generation in Unified Multimodal Models