A Simple Baseline for Unifying Understanding, Generation, and Editing via Vanilla Next-token Prediction
Authors: Jie Zhu, Hanghang Ma, Jia Wang, Yayong Guan, Yanbing Zeng, Lishuai Gao, Junqiang Wu, Jie Hu, +1 more
Organizations: Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China · School of Computer Science, Peking University, Beijing, China · Meituan · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China
In this work, we introduce Wallaroo, a simple autoregressive baseline that leverages next-token prediction to unify multi-modal understanding, image generation, and editing at the same time. Moreover, Wallaroo supports multi-resolution image input and output, as well as bilingual support for both Chinese and English. We decouple the visual encoding into separate pathways and apply a four-stage training strategy to reshape the model's capabilities. Experiments are conducted on various benchmarks where Wallaroo produces competitive performance or exceeds other unified models, suggesting the great potential of autoregressive models in unifying multi-modality understanding and generation. Our code is available at https://github.com/JiePKU/Wallaroo.
Figures & tables
Figure 1: Some text-to-image generation showcases of our Wallaroo.
Figure 2: Illustration of our Wallaroo. We decouple visual encoding into separate pathways for visual understanding and image generation. For editing, we integrate two complementary types of visual representations to improve Wallaroo’s performance.
Figure 3: A four-stage training procedure of our Wallaroo based on Qwen2.5 VL. We use flame symbols to denote modules that update their parameters, and snowflake symbols to denote modules that keep their parameters fixed.
Hyperparameters
Stage 1
Stage 2
Stage 3
Stage 4
Learning rate
1×10−4
1×10−4
4×10−5 / 1×10−5
1×10−5
LR scheduler
Constant
Constant
Constant
Constant
Weight decay
0.0
0.0
0.0
0.0
Gradient clip
1.0
1.0
1.0
1.0
Optimizer
AdamW ( β1=0.9,β2=0.95 , ϵ=1e−8 )
Warm-up steps
1000
5000
0
0
Table 1: Detailed hyperparameters of each training stage. Data ratio refers to the ratio of multimodal understanding data, visual generation data, and editing data in a batch size.
Dataset
Type
Num
Source
LLAVA-OneVision
Modality
3M
lmms-lab/LLaVA-OneVision-Data
Llama-Nemotron-VLM
Modality
2.1M
nvidia/Llama-Nemotron-VLM-Dataset-v1
GQA (Balanced)
Modality
943K
lmms-lab/GQA
MMPR-v1.2
Modality
815K
OpenGVLab/MMPR-v1.2
LLAVA-Next
Modality
779K
lmms-lab/LLaVA-NeXT-Data
llava-v1_5-mix
Modality
665k
liuhaotian/LLaVA-Instruct-150K
Table 2: Detailed information about the understanding datasets we used in Stage 2 and Stage 3.
Model
Params
POPE ↑
MME-P ↑
MMB ↑
SEED ↑
GQA ↑
MMMU ↑
MM-Vet ↑
Multi-modal Understanding Models as Enhanced Conditional Encoders :
MetaQuery
7B + 1.6B
-
1685.2
83.5
76.9
58.6
66.6
Blip3-o
7B + 1.4B
-
1682.6
83.5
77.5
50.6
66.6
Ming-Lite-Uni
8B+1.6B
-
-
80.7
-
-
51.2
72.3
UniWorld-V1
7B + 12B
-
-
83.5
-
-
58.6
67.1
OmniGen2
3B + 4B
-
-
79.1
-
-
53.1
61.8
Table 3: Comparison with state-of-the-art unified model on multi-modal understanding benchmarks. ∗ indicates that we use the VLMEvalKit to evaluate the results.
Method
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri.
Overall ↑
Multi-modal Understanding Models as Enhanced Conditional Encoders :
MetaQuery ∗
-
-
-
-
-
-
0.80
Blip3-o ∗
-
-
-
-
-
-
0.84
Ming-Lite-Uni
0.99
0.76
0.53
0.87
0.26
0.30
0.62
UniWorld-V1
0.99
0.93
0.79
0.89
0.49
0.70
0.80
OmniGen2
1
0.95
0.64
0.88
0.55
0.76
0.80
Table 4: Comparison of text-to-image generation ability on GenEval benchmark.
Method
Global
Entity
Attribute
Relation
Other
Overall ↑
Multi-modal Understanding Models as Enhanced Conditional Encoders :
MetaQuery
-
-
-
-
-
82.05
Blip3-o
-
-
-
-
-
81.60
UniWorld-V1
83.64
88.39
88.44
89.27
87.22
81.38
OmniGen2
88.81
88.83
90.18
89.37
90.27
83.57
Integrating Autoregressive and Diffusion within Transformers :
Table 5: Comparison of text-to-image generation ability on DPG benchmark. We use CFG=2.5.
Method
Extract
Adjust
Background
Add
Replace
Remove
Style
Compose
Action
Overall ↑
AnyEdit
1.88
2.95
2.24
3.18
2.47
2.23
2.85
1.56
2.65
2.45
UltraEdit
2.13
2.81
2.83
3.44
2.96
1.45
3.76
1.91
2.98
2.70
OmniGen
1.71
3.04
3.21
3.47
2.94
2.43
4.19
2.24
3.38
2.96
Step1X-Edit
1.76
3.14
3.16
3.88
3.40
2.41
4.63
2.64
2.52
3.06
BAGEL
1.70
3.31
3.24
3.56
3.30
2.62
4.49
2.38
4.17
3.20
UniWorld-V1
2.27
3.64
2.99
3.82
3.47
3.24
4.21
2.96
2.74
3.26
Table 6: Comparison of image editing capability on ImgEdit benchmark.
VQ Tokenizer
Inception Score
FID
sFID
LlamaGen
199.06
11.04
15.04
MoVQGAN
136.10
14.72
25.36
Table 7: Comparison of different VQ tokenizer on generation performance.
Ratio
Extract
Adjust
Background
Add
Replace
Remove
Style
Compose
Action
Overall
0.5
1.86
2.63
2.93
2.92
2.39
1.33
4.06
1.91
2.77
2.53
0.6
2.05
3.19
2.88
2.74
2.22
1.41
4.44
2.09
3.08
2.67
0.75
1.82
2.61
2.98
2.94
2.26
1.41
3.91
1.83
2.76
2.50
Table 8: Comparison of different mask ratio on editing performance.