We present UniSpark-9B, a unified multimodal model that investigates hybrid attention for visual understanding and generation. Built on a Mixture-of-Transformer-Experts backbone with approximately 9B parameters in each branch, UniSpark combines full attention for explicit token interactions with GatedDeltaNet linear attention for compact context propagation. Autoregressive text prediction and visual flow matching jointly support visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction. In interleaved generation, noisy visual targets are isolated from subsequent predictions, while clean visual representations provide context for later outputs. A blockwise protocol enforces this distinction through attention masks in full-attention layers and recurrent-state retention rules in linear-attention layers. The selected training sources contain 159.7M generation samples, 213.4M understanding samples, and 624.8K interleaved sequences. A four-stage curriculum jointly trains multimodal understanding and visual generation, refining answer accuracy and instruction following while extending image synthesis to 2K. UniSpark scores 2470.3 on MME-S and 0.83 on GenEval. Qualitative examples further illustrate its capabilities in interleaved image generation, video synthesis, and video editing. Together, these results demonstrate that hybrid attention provides an effective backbone for unified multimodal understanding and generation.
Figures & tables
Figure 4: Hybrid MoT Architecture of UniSpark. Token-type routing selects understanding or generation parameters, while full-attention and recurrent linear-attention layers exchange shared multimodal context for text prediction, image generation, editing, and interleaved outputs.
Figure 5: Information flow under hybrid attention. Visibility masks and recurrent-state rules preserve the same public/private boundary, allowing clean context reuse without propagating noisy targets.
Figure 6: Training-data composition and task examples. Source counts and examples summarize the task coverage of joint training, from multimodal understanding to image generation, editing, and interleaved prediction.
Figure 7: Video-data composition and examples. Generation sources (a) and aligned editing pairs (b) provide 24.24M and 3.29M samples, respectively, for post-SFT video adaptation. Counts refer to indexed samples; byte-identical Ditto shards are counted once.
Source identifier
Source samples
Panda1M
10.02 K
Panda6M
21.82 K
Koala clips
52.08 K
Editing
1.38 M
VCG
21.76 M
Pexel
32.79 K
Table 1: Video annotation sources. Source samples before filtering and annotation acceptance.
Figure 8: Video-based interleaved data construction. Event-selected video frames are paired with scene descriptions and transition instructions to form grounded text–image training sequences.
Figure 9: Multimodal sequence packing. Online packing combines task-specific samples within a token budget while preserving sample isolation and the public/private boundaries required by hybrid attention.
Figure 10: Data sampling across training stages. Representative task weights are shown for three pre-SFT stages; asterisks mark mandatory groups. These weights do not represent observed sample or token proportions.
Stage
Name
Generation scale
Primary objective
I
Joint Multimodal Foundation
256-scale
Establish joint text, visual understanding, and latent-generation modeling.
II
Progressive Resolution Scaling
512-scale
Expand spatial detail while continuing multimodal supervision.
III
High-Resolution Refinement
1024-scale to 2K
Extend visual generation to longer target sequences and finer detail.
IV
Supervised Capability Refinement (SFT)
Task-dependent
Refine understanding accuracy, instruction following, generation quality, and editing consistency.
Table 2: Four-stage training curriculum. Resolution labels describe training scales rather than fixed square image sizes.
Parameter
Foundation
Resolution scaling
High-resolution a
SFT b
Learning rate
10−4
10−4
10−4
10−4
LR schedule
Constant
Constant
Constant
Constant
Warmup updates
2,500
2,500
2,500
500
EMA decay
0.9999
0.9999
0.9999
0.995
Timestep shift
1.0
1.0
1.0
4.0
Text / visual loss weights
1 / 1
1 / 1
0.25 / 1
0.25 / 1
Table 3: Reference optimization settings by training stage. The packed-token target applies to one packed sequence, not a global batch. a High-resolution values refer to the 1024-scale setting. b SFT settings are preliminary. All learning-rate schedules are constant after warmup.
Parameter
Value
Learning rate
5×10−5
LR schedule
Constant
Warmup updates
1,000
EMA decay
0.999
Timestep shift
1.0
Text conditioning dropout
0.1
Table 4: Optimization settings for video adaptation stage. The understanding branch is frozen.
Type
Model
# LLM Params
MME-P ↑
MME-S ↑
MMBench ↑
MMMU ↑
MM-Vet ↑
MathVista ↑
MMVP ↑
Und. Only
InternVL2 ( 2024 )
1.8B
1440.0
1877.0
73.2
34.3
44.6
46.4
35.3
InternVL2.5 ( 2024 )
1.8B
–
2138.0
74.7
43.6
60.8
51.3
–
Qwen2-VL ( 2024b )
1.5B
–
1872.0
74.9
41.1
49.5
43.0
–
BLIP-3 ( 2024 )
4B
–
–
76.8
41.1
–
39.6
–
LLaVA-OV ( 2024 )
7B
1580.0
–
80.8
48.8
57.5
63.2
–
InternVL2 ( 2024 )
7B
1648.0
2210.0
81.7
49.3
54.2
58.3
51.3
Table 5: Bold and underlining indicate the best and second-best scores. MoT parameters are per branch; MoE parameters are active/total. MMBench answer extraction differs: UniSpark uses rules only, while Qwen3.5 and SenseNova-U1 use a Qwen fallback. † : frozen Qwen2.5-VL backbone. ∗∗ : scores taken from MetaMorph or MetaQuery in the source comparison. Dashes indicate unreported scores.
Type
Model
Architecture
Single object
Two objects
Counting
Colors
Position
Attribute binding
Overall ↑
Gen. Only
FLUX.1 [Dev] ( 2024 )
CD
0.98
0.81
0.74
0.79
0.22
0.45
0.66
Emu3-Gen ( 2024c )
AR
0.98
0.71
0.34
0.81
0.17
0.21
0.54
Lumina-mGPT 2.0 ( 2025b )
AR
0.99
0.87
0.44
0.85
0.44
0.54
0.69
Seedream 3.0 ( 2025 )
CD
0.99
0.96
0.91
0.93
0.47
0.80
0.84
Qwen-Image ( 2025a )
CD
0.99
0.92
0.89
0.88
0.76
0.77
0.87
LongCat-Image ( 2025 )
CD
0.99
0.98
0.86
0.86
0.75
0.73
0.87
Table 6: Text-to-image generation on GenEval. Scores range from 0 to 1; higher is better. The blue row reports UniSpark after unified SFT. AR, CD, and DD denote autoregressive, continuous diffusion, and discrete diffusion models, respectively; “+” indicates combined architectures.
Type
Model
Architecture
Global
Entity
Attribute
Relation
Other
Overall ↑
Gen. Only
FLUX.1 [Dev] ( 2024 )
CD
74.35
90.00
88.96
90.87
88.33
83.84
Emu3-Gen ( 2024c )
AR
85.21
86.68
86.84
90.22
83.15
80.60
Lumina-mGPT 2.0 ( 2025b )
AR
86.63
91.97
90.20
94.85
84.80
87.20
Seedream 3.0 ( 2025 )
CD
94.31
92.65
91.36
92.78
88.24
88.27
Qwen-Image ( 2025a )
CD
91.32
91.56
92.02
94.31
92.73
88.32
LongCat-Image ( 2025 )
CD
89.10
92.54
92.00
93.28
87.50
86.80
Table 7: Text-to-image generation on DPG-Bench. Higher is better. The blue row reports UniSpark after unified SFT. AR, CD, and DD denote autoregressive, continuous diffusion, and discrete diffusion models, respectively; “+” indicates combined architectures.
Figure 11: Text-only generation (top) and first-frame-conditioned editing (bottom). Generation predicts F1–F4 jointly, whereas editing conditions on F1 to predict F2–F4, illustrating visual continuity under two conditioning settings.
Figure 12: Qualitative examples of video generation. Chronological frames illustrate text-conditioned scene evolution, including camera movement, waves, and subtle human motion.
Figure 13: Qualitative examples of video editing. Temporally aligned source and edited frames illustrate object replacement and global stylization.
Resolution
Model
# LLM Params
Image tokens/s ↑
Median latency (s) ↓
P95 latency (s) ↓
Peak allocated memory (GB) ↓
BAGEL ( 2025 )
7B MoT
87.09
11.76
11.77
30.47
SenseNova-U1 ( 2026 )
8B MoT
39.80
6.43
6.52
35.49
512×512
UniSpark
9B MoT
35.19
7.27
7.36
37.39
BAGEL ( 2025 )
7B MoT
89.44
45.80
45.81
33.29
SenseNova-U1 ( 2026 )
8B MoT
73.90
13.86
13.91
35.84
1024×1024
UniSpark
9B MoT
72.33
14.16
14.21
42.47
Table 9: Image generation efficiency across output resolutions.
Visual tokens
Model
# LLM Params
Input
Text decode tokens/s ↑
First logits (s) ↓
Median latency (s) ↓
Peak allocated memory (GB) ↓
BAGEL ( 2025 )
7B MoT
224×224
36.09
0.176
3.70
29.58
SenseNova-U1 ( 2026 )
8B MoT
512×512
17.92
0.165
7.26
35.32
256
UniSpark
9B MoT
512×512
21.91
0.143
5.94
35.73
BAGEL ( 2025 )
7B MoT
448×448
36.60
0.263
3.76
29.77
SenseNova-U1 ( 2026 )
8B MoT
1024×1024
18.09
0.289
7.32
35.85
1,024
UniSpark
9B MoT
1024×1024
22.19
0.292
6.02
36.03
Table 10: Visual understanding efficiency at matched visual-token budgets.
Unified multimodal models are envisioned to bridge the gap between understanding and generation. Yet, to achieve competitive performance, state-of-the-art models adopt largely decoupled understanding and generation components. This design, while effective for individual tasks, weakens the connection required for mutual enhancement, leaving the potential synergy empirically uncertain. We propose to explicitly restore this synergy by introducing Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations. By incorporating objectives that encode semantic abstraction (captioning) and structural details (visual regression), we enable effective gradient flow from understanding to generation. Extensive experiments on image generation and editing demonstrate that understanding can serve as an effective catalyst for generation.
Zeyu Liu, Zanlin Ni, Yang Yue +5
1Tsinghua University · 2Kolors Team, Kuaishou Technology
We present Lance, a lightweight native unified model supporting multimodal understanding, generation, and editing for both images and videos. Rather than relying on model capacity scaling or text-image-dominant designs, Lance explores a practical paradigm for unified multimodal modeling via collaborative multi-task training. It is grounded in two core principles: unified context modeling and decoupled capability pathways. Specifically, Lance is trained from scratch and employs a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, enabling joint context learning while decoupling the pathways for understanding and generation. We further introduce modality-aware rotary positional encoding to mitigate interference among heterogeneous visual tokens and boost cross-task alignment. During training, Lance adopts a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling to strengthen both semantic comprehension and visual generation performance. Experimental results demonstrate that Lance substantially outperforms existing open-source unified models in image and video generation, while retaining strong multimodal understanding capabilities. The homepage is available at https://lance-project.github.io.
Unified Multimodal Modeling aims to integrate visual understanding and generation within a single system. However, existing approaches typically rely on two disparate visual tokenizers, which splits the representation space and hinders truly unified modeling. We propose UniAR, a unified autoregressive framework where a single discrete visual tokenizer serves as the key bridge between understanding and generation, enabling a shared context in which the model can directly interpret its own generated visual tokens without additional re-encoding. UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme, preserving both high-level semantics and low-level details while scaling the effective visual vocabulary at minimal cost. Building on this, the unified autoregressive model adopts parallel-bitwise-prediction to jointly predict spatially grouped, multi-level visual codes, substantially reducing visual sequence length and accelerating generation. Finally, a diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images. Through large-scale pre-training, followed by supervised fine-tuning and reinforcement learning, UniAR achieves state-of-the-art performance on image generation and image editing while remaining competitive on multimodal understanding benchmarks. The project page is available at https://sharelab-sii.github.io/uniar-web.
Wujian Peng, Lingchen Meng, Yuxuan Cai +7
Institute of Trustworthy Embodied AI, Fudan University · Shanghai Innovation Institute · Qwen Team, Alibaba Inc.