A Simple Baseline for Unifying Understanding, Generation, and Editing via Vanilla Next-token Prediction
Authors: Jie Zhu, Hanghang Ma, Jia Wang, Yayong Guan, Yanbing Zeng, Lishuai Gao, Junqiang Wu, Jie Hu, +1 more
Organizations: Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China · School of Computer Science, Peking University, Beijing, China · Meituan · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China
In this work, we introduce Wallaroo, a simple autoregressive baseline that leverages next-token prediction to unify multi-modal understanding, image generation, and editing at the same time. Moreover, Wallaroo supports multi-resolution image input and output, as well as bilingual support for both Chinese and English. We decouple the visual encoding into separate pathways and apply a four-stage training strategy to reshape the model's capabilities. Experiments are conducted on various benchmarks where Wallaroo produces competitive performance or exceeds other unified models, suggesting the great potential of autoregressive models in unifying multi-modality understanding and generation. Our code is available at https://github.com/JiePKU/Wallaroo.
Figures & tables
Figure 1: Some text-to-image generation showcases of our Wallaroo.
Figure 2: Illustration of our Wallaroo. We decouple visual encoding into separate pathways for visual understanding and image generation. For editing, we integrate two complementary types of visual representations to improve Wallaroo’s performance.
Figure 3: A four-stage training procedure of our Wallaroo based on Qwen2.5 VL. We use flame symbols to denote modules that update their parameters, and snowflake symbols to denote modules that keep their parameters fixed.
Hyperparameters
Stage 1
Stage 2
Stage 3
Stage 4
Learning rate
1×10−4
1×10−4
4×10−5 / 1×10−5
1×10−5
LR scheduler
Constant
Constant
Constant
Constant
Weight decay
0.0
0.0
0.0
0.0
Gradient clip
1.0
1.0
1.0
1.0
Optimizer
AdamW ( β1=0.9,β2=0.95 , ϵ=1e−8 )
Warm-up steps
1000
5000
0
0
Table 1: Detailed hyperparameters of each training stage. Data ratio refers to the ratio of multimodal understanding data, visual generation data, and editing data in a batch size.
Dataset
Type
Num
Source
LLAVA-OneVision
Modality
3M
lmms-lab/LLaVA-OneVision-Data
Llama-Nemotron-VLM
Modality
2.1M
nvidia/Llama-Nemotron-VLM-Dataset-v1
GQA (Balanced)
Modality
943K
lmms-lab/GQA
MMPR-v1.2
Modality
815K
OpenGVLab/MMPR-v1.2
LLAVA-Next
Modality
779K
lmms-lab/LLaVA-NeXT-Data
llava-v1_5-mix
Modality
665k
liuhaotian/LLaVA-Instruct-150K
Table 2: Detailed information about the understanding datasets we used in Stage 2 and Stage 3.
Model
Params
POPE ↑
MME-P ↑
MMB ↑
SEED ↑
GQA ↑
MMMU ↑
MM-Vet ↑
Multi-modal Understanding Models as Enhanced Conditional Encoders :
MetaQuery
7B + 1.6B
-
1685.2
83.5
76.9
58.6
66.6
Blip3-o
7B + 1.4B
-
1682.6
83.5
77.5
50.6
66.6
Ming-Lite-Uni
8B+1.6B
-
-
80.7
-
-
51.2
72.3
UniWorld-V1
7B + 12B
-
-
83.5
-
-
58.6
67.1
OmniGen2
3B + 4B
-
-
79.1
-
-
53.1
61.8
Table 3: Comparison with state-of-the-art unified model on multi-modal understanding benchmarks. ∗ indicates that we use the VLMEvalKit to evaluate the results.
Method
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri.
Overall ↑
Multi-modal Understanding Models as Enhanced Conditional Encoders :
MetaQuery ∗
-
-
-
-
-
-
0.80
Blip3-o ∗
-
-
-
-
-
-
0.84
Ming-Lite-Uni
0.99
0.76
0.53
0.87
0.26
0.30
0.62
UniWorld-V1
0.99
0.93
0.79
0.89
0.49
0.70
0.80
OmniGen2
1
0.95
0.64
0.88
0.55
0.76
0.80
Table 4: Comparison of text-to-image generation ability on GenEval benchmark.
Method
Global
Entity
Attribute
Relation
Other
Overall ↑
Multi-modal Understanding Models as Enhanced Conditional Encoders :
MetaQuery
-
-
-
-
-
82.05
Blip3-o
-
-
-
-
-
81.60
UniWorld-V1
83.64
88.39
88.44
89.27
87.22
81.38
OmniGen2
88.81
88.83
90.18
89.37
90.27
83.57
Integrating Autoregressive and Diffusion within Transformers :
Table 5: Comparison of text-to-image generation ability on DPG benchmark. We use CFG=2.5.
Method
Extract
Adjust
Background
Add
Replace
Remove
Style
Compose
Action
Overall ↑
AnyEdit
1.88
2.95
2.24
3.18
2.47
2.23
2.85
1.56
2.65
2.45
UltraEdit
2.13
2.81
2.83
3.44
2.96
1.45
3.76
1.91
2.98
2.70
OmniGen
1.71
3.04
3.21
3.47
2.94
2.43
4.19
2.24
3.38
2.96
Step1X-Edit
1.76
3.14
3.16
3.88
3.40
2.41
4.63
2.64
2.52
3.06
BAGEL
1.70
3.31
3.24
3.56
3.30
2.62
4.49
2.38
4.17
3.20
UniWorld-V1
2.27
3.64
2.99
3.82
3.47
3.24
4.21
2.96
2.74
3.26
Table 6: Comparison of image editing capability on ImgEdit benchmark.
VQ Tokenizer
Inception Score
FID
sFID
LlamaGen
199.06
11.04
15.04
MoVQGAN
136.10
14.72
25.36
Table 7: Comparison of different VQ tokenizer on generation performance.
Ratio
Extract
Adjust
Background
Add
Replace
Remove
Style
Compose
Action
Overall
0.5
1.86
2.63
2.93
2.92
2.39
1.33
4.06
1.91
2.77
2.53
0.6
2.05
3.19
2.88
2.74
2.22
1.41
4.44
2.09
3.08
2.67
0.75
1.82
2.61
2.98
2.94
2.26
1.41
3.91
1.83
2.76
2.50
Table 8: Comparison of different mask ratio on editing performance.
Unified Multimodal Modeling aims to integrate visual understanding and generation within a single system. However, existing approaches typically rely on two disparate visual tokenizers, which splits the representation space and hinders truly unified modeling. We propose UniAR, a unified autoregressive framework where a single discrete visual tokenizer serves as the key bridge between understanding and generation, enabling a shared context in which the model can directly interpret its own generated visual tokens without additional re-encoding. UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme, preserving both high-level semantics and low-level details while scaling the effective visual vocabulary at minimal cost. Building on this, the unified autoregressive model adopts parallel-bitwise-prediction to jointly predict spatially grouped, multi-level visual codes, substantially reducing visual sequence length and accelerating generation. Finally, a diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images. Through large-scale pre-training, followed by supervised fine-tuning and reinforcement learning, UniAR achieves state-of-the-art performance on image generation and image editing while remaining competitive on multimodal understanding benchmarks. The project page is available at https://sharelab-sii.github.io/uniar-web.
Wujian Peng, Lingchen Meng, Yuxuan Cai +7
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Qwen Team, Alibaba Inc.
Most unified large multimodal models (LMMs) that support both visual understanding and image generation still rely on curated post-training supervision, such as human annotations, preference labels, or external reward models. We ask whether a unified LMM can improve both abilities autonomously using only unlabeled images. We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a Solver that answers and evaluates them, and a Generator that synthesizes images. Training uses only self-derived consistency signals, without human annotations, preference labels, or task-trained external reward/judge models. To stabilize learning, we introduce Solver Token Entropy (STE), a continuous difficulty signal based on token-level prediction uncertainty that remains useful even when sample-level consistency becomes unreliable. For image generation, we design a multi-scale internal evaluation scheme that combines question-answer fidelity scoring with cycle-consistent captioning. This creates a solver-mediated coupling, where better visual understanding enables more reliable generation assessment and stronger internal training signals. The framework preserves the same role decomposition, reward logic, and training schedule across diffusion-based BLIP3o, rectified-flow BAGEL, and autoregressive VARGPT-v1.1 architectures, requiring only each backbone's native prompting and generation interface. Across eight understanding metrics, our method consistently improves over the corresponding base models. On BAGEL, it achieves a +3.5% absolute gain on MMMU and improves GenEval image generation performance from 82% to 85%. Code and models are publicly released.
We introduce Boogu-Image-0.1, an open-source unified multimodal understanding and generation model family, comprising Base, Turbo, Edit, and Edit-Turbo variants. It delivers competitive performance in high-quality text-to-image generation, fast inference, instruction-based editing, and bilingual (Chinese-English) text rendering. Closed-source multimodal systems like Nano-Banana-Pro and GPT-Image-2 achieve strong performance through system-level integration rather than a single model, yet their internal practices remain largely undisclosed. In this work, we demonstrate that targeted improvements in model understanding, data quality, and training pipelines, coupled with agentic inference-time scaling, can substantially enhance generation and editing performance even under highly constrained compute budgets. Comprehensive evaluations show that Boogu-Image-0.1 consistently matches or surpasses other open-source models across standard benchmarks, and achieves results approaching leading closed-source systems. Notably, this is accomplished with only 208.62 million unique images. The base model's theoretical training cost is only approximately $400K. We share practical discussions that we believe are valuable to the broader research community, and release weights, code, and recipes under Apache 2.0 to advance the open ecosystem for unified multimodal understanding and generation. Our code is available here: https://github.com/Boogu-Project/Boogu-Image.