We present UniSpark-9B, a unified multimodal model that investigates hybrid attention for visual understanding and generation. Built on a Mixture-of-Transformer-Experts backbone with approximately 9B parameters in each branch, UniSpark combines full attention for explicit token interactions with GatedDeltaNet linear attention for compact context propagation. Autoregressive text prediction and visual flow matching jointly support visual question answering, text-to-image generation, instruction-guided editing, and interleaved text-image prediction. In interleaved generation, noisy visual targets are isolated from subsequent predictions, while clean visual representations provide context for later outputs. A blockwise protocol enforces this distinction through attention masks in full-attention layers and recurrent-state retention rules in linear-attention layers. The selected training sources contain 159.7M generation samples, 213.4M understanding samples, and 624.8K interleaved sequences. A four-stage curriculum jointly trains multimodal understanding and visual generation, refining answer accuracy and instruction following while extending image synthesis to 2K. UniSpark scores 2470.3 on MME-S and 0.83 on GenEval. Qualitative examples further illustrate its capabilities in interleaved image generation, video synthesis, and video editing. Together, these results demonstrate that hybrid attention provides an effective backbone for unified multimodal understanding and generation.
Figures & tables
Figure 4: Hybrid MoT Architecture of UniSpark. Token-type routing selects understanding or generation parameters, while full-attention and recurrent linear-attention layers exchange shared multimodal context for text prediction, image generation, editing, and interleaved outputs.
Figure 5: Information flow under hybrid attention. Visibility masks and recurrent-state rules preserve the same public/private boundary, allowing clean context reuse without propagating noisy targets.
Figure 6: Training-data composition and task examples. Source counts and examples summarize the task coverage of joint training, from multimodal understanding to image generation, editing, and interleaved prediction.
Figure 7: Video-data composition and examples. Generation sources (a) and aligned editing pairs (b) provide 24.24M and 3.29M samples, respectively, for post-SFT video adaptation. Counts refer to indexed samples; byte-identical Ditto shards are counted once.
Source identifier
Source samples
Panda1M
10.02 K
Panda6M
21.82 K
Koala clips
52.08 K
Editing
1.38 M
VCG
21.76 M
Pexel
32.79 K
Table 1: Video annotation sources. Source samples before filtering and annotation acceptance.
Figure 8: Video-based interleaved data construction. Event-selected video frames are paired with scene descriptions and transition instructions to form grounded text–image training sequences.
Figure 9: Multimodal sequence packing. Online packing combines task-specific samples within a token budget while preserving sample isolation and the public/private boundaries required by hybrid attention.
Figure 10: Data sampling across training stages. Representative task weights are shown for three pre-SFT stages; asterisks mark mandatory groups. These weights do not represent observed sample or token proportions.
Stage
Name
Generation scale
Primary objective
I
Joint Multimodal Foundation
256-scale
Establish joint text, visual understanding, and latent-generation modeling.
II
Progressive Resolution Scaling
512-scale
Expand spatial detail while continuing multimodal supervision.
III
High-Resolution Refinement
1024-scale to 2K
Extend visual generation to longer target sequences and finer detail.
IV
Supervised Capability Refinement (SFT)
Task-dependent
Refine understanding accuracy, instruction following, generation quality, and editing consistency.
Table 2: Four-stage training curriculum. Resolution labels describe training scales rather than fixed square image sizes.
Parameter
Foundation
Resolution scaling
High-resolution a
SFT b
Learning rate
10−4
10−4
10−4
10−4
LR schedule
Constant
Constant
Constant
Constant
Warmup updates
2,500
2,500
2,500
500
EMA decay
0.9999
0.9999
0.9999
0.995
Timestep shift
1.0
1.0
1.0
4.0
Text / visual loss weights
1 / 1
1 / 1
0.25 / 1
0.25 / 1
Table 3: Reference optimization settings by training stage. The packed-token target applies to one packed sequence, not a global batch. a High-resolution values refer to the 1024-scale setting. b SFT settings are preliminary. All learning-rate schedules are constant after warmup.
Parameter
Value
Learning rate
5×10−5
LR schedule
Constant
Warmup updates
1,000
EMA decay
0.999
Timestep shift
1.0
Text conditioning dropout
0.1
Table 4: Optimization settings for video adaptation stage. The understanding branch is frozen.
Type
Model
# LLM Params
MME-P ↑
MME-S ↑
MMBench ↑
MMMU ↑
MM-Vet ↑
MathVista ↑
MMVP ↑
Und. Only
InternVL2 ( 2024 )
1.8B
1440.0
1877.0
73.2
34.3
44.6
46.4
35.3
InternVL2.5 ( 2024 )
1.8B
–
2138.0
74.7
43.6
60.8
51.3
–
Qwen2-VL ( 2024b )
1.5B
–
1872.0
74.9
41.1
49.5
43.0
–
BLIP-3 ( 2024 )
4B
–
–
76.8
41.1
–
39.6
–
LLaVA-OV ( 2024 )
7B
1580.0
–
80.8
48.8
57.5
63.2
–
InternVL2 ( 2024 )
7B
1648.0
2210.0
81.7
49.3
54.2
58.3
51.3
Table 5: Bold and underlining indicate the best and second-best scores. MoT parameters are per branch; MoE parameters are active/total. MMBench answer extraction differs: UniSpark uses rules only, while Qwen3.5 and SenseNova-U1 use a Qwen fallback. † : frozen Qwen2.5-VL backbone. ∗∗ : scores taken from MetaMorph or MetaQuery in the source comparison. Dashes indicate unreported scores.
Type
Model
Architecture
Single object
Two objects
Counting
Colors
Position
Attribute binding
Overall ↑
Gen. Only
FLUX.1 [Dev] ( 2024 )
CD
0.98
0.81
0.74
0.79
0.22
0.45
0.66
Emu3-Gen ( 2024c )
AR
0.98
0.71
0.34
0.81
0.17
0.21
0.54
Lumina-mGPT 2.0 ( 2025b )
AR
0.99
0.87
0.44
0.85
0.44
0.54
0.69
Seedream 3.0 ( 2025 )
CD
0.99
0.96
0.91
0.93
0.47
0.80
0.84
Qwen-Image ( 2025a )
CD
0.99
0.92
0.89
0.88
0.76
0.77
0.87
LongCat-Image ( 2025 )
CD
0.99
0.98
0.86
0.86
0.75
0.73
0.87
Table 6: Text-to-image generation on GenEval. Scores range from 0 to 1; higher is better. The blue row reports UniSpark after unified SFT. AR, CD, and DD denote autoregressive, continuous diffusion, and discrete diffusion models, respectively; “+” indicates combined architectures.
Type
Model
Architecture
Global
Entity
Attribute
Relation
Other
Overall ↑
Gen. Only
FLUX.1 [Dev] ( 2024 )
CD
74.35
90.00
88.96
90.87
88.33
83.84
Emu3-Gen ( 2024c )
AR
85.21
86.68
86.84
90.22
83.15
80.60
Lumina-mGPT 2.0 ( 2025b )
AR
86.63
91.97
90.20
94.85
84.80
87.20
Seedream 3.0 ( 2025 )
CD
94.31
92.65
91.36
92.78
88.24
88.27
Qwen-Image ( 2025a )
CD
91.32
91.56
92.02
94.31
92.73
88.32
LongCat-Image ( 2025 )
CD
89.10
92.54
92.00
93.28
87.50
86.80
Table 7: Text-to-image generation on DPG-Bench. Higher is better. The blue row reports UniSpark after unified SFT. AR, CD, and DD denote autoregressive, continuous diffusion, and discrete diffusion models, respectively; “+” indicates combined architectures.
Figure 11: Text-only generation (top) and first-frame-conditioned editing (bottom). Generation predicts F1–F4 jointly, whereas editing conditions on F1 to predict F2–F4, illustrating visual continuity under two conditioning settings.
Figure 12: Qualitative examples of video generation. Chronological frames illustrate text-conditioned scene evolution, including camera movement, waves, and subtle human motion.
Figure 13: Qualitative examples of video editing. Temporally aligned source and edited frames illustrate object replacement and global stylization.
Resolution
Model
# LLM Params
Image tokens/s ↑
Median latency (s) ↓
P95 latency (s) ↓
Peak allocated memory (GB) ↓
BAGEL ( 2025 )
7B MoT
87.09
11.76
11.77
30.47
SenseNova-U1 ( 2026 )
8B MoT
39.80
6.43
6.52
35.49
512×512
UniSpark
9B MoT
35.19
7.27
7.36
37.39
BAGEL ( 2025 )
7B MoT
89.44
45.80
45.81
33.29
SenseNova-U1 ( 2026 )
8B MoT
73.90
13.86
13.91
35.84
1024×1024
UniSpark
9B MoT
72.33
14.16
14.21
42.47
Table 9: Image generation efficiency across output resolutions.
Visual tokens
Model
# LLM Params
Input
Text decode tokens/s ↑
First logits (s) ↓
Median latency (s) ↓
Peak allocated memory (GB) ↓
BAGEL ( 2025 )
7B MoT
224×224
36.09
0.176
3.70
29.58
SenseNova-U1 ( 2026 )
8B MoT
512×512
17.92
0.165
7.26
35.32
256
UniSpark
9B MoT
512×512
21.91
0.143
5.94
35.73
BAGEL ( 2025 )
7B MoT
448×448
36.60
0.263
3.76
29.77
SenseNova-U1 ( 2026 )
8B MoT
1024×1024
18.09
0.289
7.32
35.85
1,024
UniSpark
9B MoT
1024×1024
22.19
0.292
6.02
36.03
Table 10: Visual understanding efficiency at matched visual-token budgets.