Most unified multimodal models treat images and text asymmetrically: a pretrained language model generates text, an attached diffusion head generates images, and outputs that contain both are produced in a fixed order, so the first output cannot adapt to the second. We ask whether both modalities can instead be generated by one process that is trained on them jointly from scratch. Pretrained encoder spaces hold both modalities in one continuous form and carry knowledge learned at scale, and recent work finds them effective as generative targets for vision and language. We therefore model the joint distribution directly in these spaces. We introduce JUICE (Joint Unified multimodal modeling In Continuous Encoder spaces), a single flow matching Transformer over frozen image and text representations, in which each modality has its own noise level. Which modalities are clean and which are generated then decides the task: image generation, captioning, visual question answering, or joint generation. Controlled pretraining runs test each design choice. At equal guidance, generating text as continuous states leaves image generation intact (48.41 GenEval, against 47.86 for a model trained only for text-to-image generation), while captions lose some exact wording. Training on joint and conditional denoising together beats either alone, and semantic encoder spaces beat VAE latents and raw pixels as targets. At 2.1B parameters, with no language model initialization and at 224x224, JUICE reaches up to 0.88 on GenEval and 73.1 on VQAv2. By sampling both modalities from the joint distribution (coupled sampling), it can also draw a route on a map while writing out the matching moves. On mazes with several valid solutions, joint success rises from 54.6% with independent sampling to 93.7%, and no modality has to go first. On visual spatial planning, JUICE solves 99.8% of maps at its training sizes.
Figures & tables
Figure 1: JUICE overview. (a) Continuous image and text representations from frozen encoders share one MM-RiT backbone via bidirectional joint attention. Appendix A.3 explains the details of each block. (b) Conditional tasks (text-to-image, visual question answering) and joint reasoning (maze solving) use the same interface by assigning clean context ( t=1 ) and generation targets ( t<1 ).
Model
Text representation
Text process
Image representation
Image process
Chameleon [ 16 ]
Discrete tokens
AR
Discrete codes
AR
Transfusion [ 17 ]
Discrete tokens
AR
VAE latents
Diffusion
Show-o2 [ 1 ]
Discrete tokens
AR
VAE latents
FM
Tuna-2 [ 2 ]
Discrete tokens
AR
Pixels
FM
MMaDA [ 3 ]
Discrete tokens
Masked diffusion
Discrete codes
Masked diffusion
Muddit [ 4 ]
Discrete tokens
Masked diffusion
Discrete codes
Masked diffusion
Table 1: Generation spaces and processes in multimodal models. AR: autoregressive; FM: flow matching; repr.: representations.
Figure 2: Tasks are paths in the 2D time domain. Each modality has its own time, from noise ( t=0 ) to clean ( t=1 ). Conditional tasks run along the edges, where the condition is clean, and joint generation crosses the interior; its path bends because the image schedule keeps the image noisier for longer. Training covers both regions: conditional denoising on the edges and joint denoising in the interior.
Generation
Image generation
Captioning
Configuration
Text
Image
Text condition
GE (%)
DPG (%)
CIDEr
CLIP
JUICE
FM
FM
frozen T5 states
48.41
78.85
88.80
80.20
Transfusion-style [ 17 ]
AR
FM
token embeddings
44.23
76.84
160.52
80.37
D-DiT-style [ 18 ]
MD
FM
token embeddings
34.50
74.35
115.22
79.82
MMaDA-style [ 3 ]
MD
MD (VQ)
token embeddings
41.61
66.50
89.53
71.61
Transfusion-style
AR
FM
frozen T5 states
46.07
78.24
158.63
80.36
Table 2: Generative frameworks at 668M. All configurations share the trunk with two streams, the data, and the schedule. FM: flow matching; AR: autoregressive; MD: masked diffusion. The middle block gives the discrete text frameworks the same frozen T5 condition as JUICE, and the last row is a model trained only for text-to-image generation.
Figure 3: Continuous text generation gives the strongest image generation throughout training. Generative framework comparison at the 668M scale (Table 2 ).
Task ratio (%)
Image generation
Captioning
Recipe
I2T
T2I
Joint
GE (%)
DPG (%)
CIDEr
CLIP
T2I only
0
100
0
47.86
77.50
2.83
25.15
I2T only
100
0
0
0.21
9.95
107.22
77.81
Multi-task
50
50
0
36.30
77.71
79.41
75.07
Joint only
0
0
100
31.33
56.98
75.66
73.70
Mixing (default)
22.2
22.2
55.6
48.41
78.85
88.80
80.20
Table 3: Training task mixtures at 668M. Task ratios give the share of examples assigned to each denoising task. The controls hold the number of supervised targets or the cross-modal exchange fixed (Appendix B.2.1 ): matched DC (dual conditional) replaces each joint example with two conditional ones, blocked joint removes cross-modal attention on joint examples, single target joint supervises only one stream of each joint example, and Multi-task trained for 77.8k updates matches the target count of Mixing.
Figure 4: Joint plus conditional training leads conditional training alone throughout pretraining. Mixing combines joint, T2I, and I2T denoising; Multi-task uses T2I and I2T only.
Figure 5: Semantic encoder spaces outperform VAE latents and pixels, and T5 leads among text spaces. Top: image spaces with T5 text. Bottom: text spaces with SigLIP2 images.
Figure 6: Scaling the backbone from 179M to 2.1B parameters. DINO’s early lead in image generation disappears with scale, while SigLIP2 leads captioning at every size. DPG-Bench in this figure uses the time swap text reference (Appendix A.6.2 ).
(a) GenEval ↑
Model
Size
Single
Two
Count
Color
Position
Attribute
Overall
Muddit [ 4 ]
1B
0.98
0.72
0.54
0.82
0.19
0.41
0.61
Show-o [ 20 ]
1.3B
0.98
0.80
0.66
0.84
0.31
0.50
0.68
UniDisc [ 49 ]
1.4B
0.92
0.47
0.15
0.67
0.13
0.19
0.42
FUDOKI [ 50 ]
1.5B
0.96
0.85
0.56
0.88
0.68
0.67
0.77
Janus-Pro-1B [ 51 ]
1.5B
0.98
0.82
0.51
0.89
0.65
0.56
0.73
Table 4: Text-to-image generation against unified models of 1B to 3B parameters. GenEval on the [0,1] scale; DPG-Bench with the official script. JUICE samples 224×224 images with 25 steps; its size counts the MM-RiT backbone without the frozen encoders and decoders, and baseline sizes are as reported.
Figure 7: Text-to-image samples from both variants for the same prompts. (a) Counting, spatial relations, and unusual combinations of objects. (b) Prompts that ask for a specific artistic style. All samples use guidance w=5 with the generic caption reference; Appendix E.1 gives more samples.
Model
Size
Cult.
Time
Space
Bio.
Phys.
Chem.
Overall
Unified models of 1B to 3B parameters
OneFlow [ 23 ]
1B
0.63
–
–
–
–
–
–
JanusFlow [ 53 ]
1.3B
0.13
0.26
0.28
0.20
0.19
0.11
0.18
Show-o ( 5122 ) [ 20 ]
1.3B
0.28
0.40
0.48
0.30
0.46
0.30
0.35
Janus-Pro-1B [ 51 ]
1.5B
0.20
0.28
0.45
0.24
0.32
0.16
0.26
Show-o2 [ 1 ]
1.5B
0.30
0.38
0.47
0.32
0.42
0.28
0.35
Table 5: Text-to-image generation with world knowledge on WISE. WiScore under the original protocol (GPT-4o-2024-05-13 judge); Overall weights categories by prompt count. JUICE is scored on two images per prompt, and its 95% confidence interval on Overall is about ±0.02 . JUICE uses the same guidance as on GenEval ( w=5 , generic caption reference). Baseline values are from the WISE paper [ 13 ] or the models’ own papers; OneFlow reports only the cultural category.
Figure 8: WISE prompts that require world knowledge. Each prompt names what to draw only indirectly; the gray line gives the implied knowledge (abridged from the WISE explanation). The prompt alone is the condition, with guidance w=5 .
Model
Size
Init.
Res.
VQAv2
GQA
POPE
MME-P
MMB
AI2D
MMMU
RWQA
Muddit [ 4 ]
1B
T2I
5122
68.2
57.5
–
1107.4 †
–
–
27.6
–
OneFlow [ 23 ]
1B
LLM
5122
67.7
57.8
85.7
1497.1 †
69.0
58.5
29.8
50.6
Show-o [ 20 ]
1.3B
LLM
2562
64.7
54.2
76.2
1014.9 †
–
–
25.0
–
Show-o [ 20 ]
1.3B
LLM
5122
69.4
58.0
80.0
1097.2 †
–
–
26.7
–
JanusFlow [ 53 ]
1.3B
LLM
3842
79.8
60.3
88.0
1333.1
74.9
–
29.3
–
Harmon [ 52 ]
1.5B
LLM
5122
–
58.9
87.6
1155.0
65.5
–
38.9
–
Table 6: Visual understanding against unified models of 1B to 3B parameters. Init. gives the backbone initialization and Res. the input resolution (LLM: pretrained language model; T2I: pretrained text-to-image model). Baseline values and sizes are from each model’s paper; the size of JUICE counts the MM-RiT backbone without the frozen encoders and decoders (Appendix A.5 ). † The paper reports an unlabeled MME score, which may be the sum of perception and cognition. MME-P: MME perception score; MMB: MMBench; RWQA: RealWorldQA. JUICE uses VQAv2 test-dev and MMBench dev; Appendix C gives all 10 benchmarks.
Figure 9: Selected questions of different types, answered by both variants. Each panel gives the question type and benchmark, the question, both answers, and the reference answer. Appendix E gives more examples, including captions and long answers.
Figure 10: Joint samples of images and text. With no condition, both streams start from noise and are sampled together (64 steps, no guidance). Text is read out with the autoregressive caption decoder (Appendix A.4.2 ) and shown up to the point where it starts to repeat. Appendix E.3 gives more samples, and Figure 21 shows how both streams form during sampling.
Sampler
Order
SigLIP2
DINO
Coupled
none
93.7
81.7
Image first
image → text
95.2
82.7
Text first
text → image
85.3
59.9
Independent
none
54.6
42.7
Validity of one modality alone
Image FT only
–
95.1
83.3
Table 7: Joint success on mazes with multiple solutions. J: both outputs are shortest routes and describe the same route. Samplers are compared at 16 backbone calls; mean of two sampling seeds. Lower rows give the route validity of models finetuned on a single modality.
Figure 11: Coupled sampling brings the two outputs onto one route; independent sampling does not. Two mazes under coupled sampling at three of its 16 steps per stream, and under independent sampling with the same 16 steps per stream (32 backbone calls, twice the budget of Table 7 ) at its last step; both samplers start from the same state at step 0, and the coupled outputs do not change after the last step shown. The blue path is the decoded image, and the dashed line traces the decoded moves from the start. Underlined moves differ from the previous column.
Model
Size
Task FT
Lv3
Lv4
Lv5
Lv6
Avg.
Lv7
Lv8
InternVL3.5 [ 56 ]
38B
✗
–
–
–
–
0.20
–
–
ThinkMorph [ 57 ]
7B
✓
–
–
–
–
0.76
–
–
LatentUM, coarse [ 55 ]
4B
✓
1.00
0.85
0.83
0.71
0.85
–
–
LatentUM, fine [ 55 ]
4B
✓
1.00
1.00
1.00
0.97
0.99
–
–
JUICE (SigLIP2)
2.1B
✓
1.00
1.00
1.00
0.99
0.998
0.94
0.69
Table 8: Visual spatial planning on VSP. Success rate under the official rule; levels 3 to 6 match the training map sizes, and levels 7 and 8 are larger maps unseen in training. Baseline values are from LatentUM [ 55 ] , which reports InternVL3.5 and ThinkMorph only as averages; ThinkMorph does not state which levels its average covers. Task FT marks finetuning on maps in the style of VSP: JUICE uses 55.9k maps, ThinkMorph and coarse LatentUM 6k, and fine LatentUM 18k problems solved step by step.
Figure 12: VSP cases. Adaptation uses sizes 3×3 to 6×6 . (a) One solved map of each size. (b) Maps whose moves fail at the first of 16 sampling steps and solve the map at the last, with the drawn path fixed throughout. The red path is the decoded image, and the dashed line traces the decoded moves from the start; a cross marks where the moves fail, in a hole or short of the goal. A repeated move is written with its count ( L4 for LLLL), and underlined runs change between the two steps. Decoded images are matched in color to the input rendering.
Appendix figures & tables48 assets
Supplementary material from the paper’s appendix.
Appendix
Input
Frozen source and extraction
Tokens × channels
SigLIP2
SigLIP2-SO400M/14 , 2242 , final patch states
256×1152
DINO
WebSSL-DINO300M-full2b , 2242 , final patch states
256×1024
VAE
FLUX.1-schnell VAE , 2562 , packed latent mode
256×64
Pixel
None; 2562 RGB in [−1,1] , 16×16 patches
256×768
Appendix
Table 9: Image encoder sources and token shapes.
Frozen encoder
Layer
Positions
Channels
Flan-T5-Small [ 36 ]
Final (8)
128
512
Qwen3-Embedding-0.6B [ 47 ]
14
128
1,024
Qwen3-VL-Embedding-2B [ 48 ]
14
128
2,048
SigLIP2-SO400M/14 text [ 11 ]
13
64
1,152
Appendix
Table 10: Selected text encoder states.
Representation
Transformation
SigLIP2 / DINO image
Per token LayerNorm, without affine parameters
FLUX VAE image
z=(h−bVAE)aVAE using predefined factors
Pixel image
RGB values scaled to [−1,1] , then patchified
Flan-T5 text
z=h/0.2 , a fixed scalar for all channels
Qwen3-E / Qwen3-VL-E text
zc=(hc−μc)/sc using fixed corpus statistics
SigLIP2 text
zc=(hc−μc)/sc using fixed corpus statistics
Appendix
Table 11: Representation normalization before diffusion. h denotes an encoder output, c a feature channel, and (bVAE,aVAE) the shift and scale supplied by the pretrained FLUX VAE.
Configuration
Parameters
Width
Heads
Head dim.
Joint blocks
Total blocks
d12
178.6M
768
12
64
10
12
d15
335.0M
960
15
64
13
15
d19
668.3M
1,216
19
64
17
19
d24
1.33B
1,536
24
64
22
24
d28
2.12B
1,792
28
64
26
28
Appendix
Table 12: Backbone scaling varies width and depth (trainable parameters). Parameter counts exclude encoders and decoders. Each backbone also has two input layers for text only; d12 to d28 denote head counts.
Representation
D
V
Length
Parameters
Token accuracy
Qwen3-E
1,024
151,680
128
253M
0.9797
Qwen3-VL-E
2,048
151,936
128
409M
0.9748
SigLIP2 text
1,152
256,000
64
446M
0.9870
Appendix
Table 13: Parallel decoders of the other text representations. D is the state width, V the vocabulary size, and length the number of text positions. Token accuracy is measured from clean states on the 5,000 held out captions of Table 29 .
Decoder
τ
Token accuracy
String match
EOS error
Parallel (final)
1.0
0.9997
0.984
0.00
Autoregressive
1.0
0.9759
0.966
0.45
Parallel, first stage only
1.0
–
0.941
–
Linear head
1.0
0.9872
0.508
10.5
Parallel (final)
0.9
0.9997
0.983
0.00
Autoregressive
0.9
0.9689
0.953
0.45
Appendix
Table 14: Text decoder fidelity on 2,000 captions held out from training. τ is the clean fraction of each input state ( τ=1 : clean). String match compares the lowercased text up to the first EOS. EOS error is the mean distance, in positions, between the predicted and true end of the text. The row of the linear head differs slightly from Table 29 , which uses a different set of captions.
SigLIP2
DINO
Benchmark
Parallel
Autoregressive
Parallel
Autoregressive
VQAv2 (val, 1,000)
71.21
71.71
66.32
66.68
OK-VQA
33.87
35.32
28.65
29.69
POPE
83.97
83.97
82.29
82.29
GQA
58.58
58.42
55.68
55.76
MME-P
1115.7
1133.1
960.2
983.6
Appendix
Table 15: Parallel and autoregressive decoding of the same generated answer states (16 steps, w=3 ).
Text representation
Input D
Vocabulary V
Parameters (M)
Flan-T5-Small
512
32,128
16.48
Qwen3-Embedding-0.6B
1,024
151,680
155.47
Qwen3-VL-Embedding-2B
2,048
151,936
311.32
SigLIP2 text
1,152
256,000
295.17
Appendix
Table 16: Linear readout head sizes.
Component
Parameters
Used in
Principal image/text pipeline components
SigLIP2-SO400M image encoder
427.7M
I2T and VQA (SigLIP2 arm)
WebSSL-DINO300M image encoder
303.7M
I2T and VQA (DINO arm)
Flan-T5-Small encoder
35.3M
T2I, I2T, and VQA
SigLIP2 Scale-RAE image decoder
415.6M
T2I (SigLIP2 arm)
DINO Scale-RAE image decoder
415.4M
T2I (DINO arm)
Appendix
Table 17: Fixed component sizes. Parameters were counted from the recorded module state dictionaries. All text decoders are fixed during MM-RiT training.
Image arm
Scale
Trainable
T2I forward
I2T/VQA forward
All loaded
SigLIP2
d12
178.6M
629.5M
677.0M
≈ 1.09B
d15
335.0M
785.9M
833.4M
≈ 1.25B
d19
668.3M
≈ 1.12B
≈ 1.17B
≈ 1.58B
d24
1.33B
≈ 1.78B
≈ 1.83B
≈ 2.24B
d28
2.12B
≈ 2.57B
≈ 2.62B
≈ 3.03B
DINO
d12
178.6M
629.3M
553.0M
968.4M
Appendix
Table 18: Task active and fully loaded parameter counts. T2I is backbone + T5 encoder + matched image decoder. I2T/VQA is backbone + image encoder + T5 encoder + parallel text decoder (35.36M); the d24 and d28 entries are computed from the rounded backbone sizes, so their last digit is approximate. “All loaded” is the union.
Image representation
(μi,si)
Noise scale σi
Time shift αi
SigLIP2 [ 6 ]
(0,1)
1
8.485
DINO [ 6 ]
(0,1)
1
8
FLUX VAE [ 10 ]
(0,1)
1
2
Pixel [ 34 ]
(−0.8,0.8)
2
1
Appendix
Table 19: Image noise settings in the representation configurations. The base logit normal parameters, noise scales, and time shifts depend on the representation.
Recipe
Text reference
w=1
w=2
w=3
w=4
w=5
T2I only
generic caption
36.12
47.26
51.23
51.03
47.86
time swap
36.12
43.38
42.90
38.67
31.34
Multi-task
generic caption
32.36
43.44
44.29
40.85
36.30
time swap
32.36
24.67
5.75 †
2.16 †
0.92 †
Joint only
generic caption
10.09
21.54
27.43
31.06
31.33
time swap
10.09
3.30 †
0.78 †
0.31 †
0.25 †
Appendix
Table 20: GenEval of the pretraining task mixtures across guidance scales and text references (50k updates, 25 steps). At w=1 the reference is not used. Bold marks each model’s best score with the generic caption; † marks collapse into images that look like mosaics. I2T only cannot generate images (0.21 at every scale) and is omitted.
Model
Cult.
Time
Space
Bio.
Phys.
Chem.
Overall
JUICE (SigLIP2)
0.466
0.533
0.642
0.396
0.531
0.346
0.488 [0.468, 0.508]
JUICE (DINO)
0.379
0.481
0.607
0.344
0.456
0.313
0.424 [0.405, 0.443]
Appendix
Table 21: WISE with the time swap text reference at w=3 . 95% confidence intervals in brackets. The main text uses the generic caption reference at w=5 (Table 5 ).
Setting
Value
Updates / global batch
50,000 / 4,096
Optimizer
AdamW, (β1,β2)=(0.9,0.95) , weight decay 0.01
Learning rate
4×10−4 ; warmup over 5,000 steps, then constant
Gradient norm cap / precision
1.0 / bfloat16
EMA decay (slow / fast)
0.99995 / 0.9996
Image / text loss coefficient
1.0 / 0.25
Appendix
Table 22: Controlled comparisons share the same pretraining settings. The budget controls of Table 27 differ only in the number of updates.
Stage
Starts from
Updates
Global batch
GPUs
Evaluated weights
1. Pretraining
random initialization
60,000
4,096
8
fast EMA
2. SFT
stage 1, step 60,000
15,000
1,024
8
fast EMA
3a. Maze adaptation
stage 2, step 15,000
5,000
1,024
8
raw
3b. VSP adaptation
stage 2, step 15,000
5,000
1,024
4
raw
Appendix
Table 23: Training stages of the final models. Each later stage starts from the raw weights (without EMA) of the stage before it.
Table 24: SFT data. Row counts include repetition. K is the length of the generated answer window. Text-to-image rows train text-to-image generation; all other rows train image-to-text generation.
Initialization
Raw weights of the 2.1B pretraining run at step 60,000
AdamW, (β1,β2)=(0.9,0.95) ; 10−4 after 250 linear warmup updates, then constant
Weight decay / gradient cap
0.01 / 1.0
Precision / seed
bfloat16 / 4242
EMA decay (slow / fast)
0.99995 / 0.9996; we evaluate the fast EMA
Task assignment
text-to-image rows: text-to-image only; question, caption prompt, and long answer rows: image-to-text only
Appendix
Table 25: SFT recipe.
Configuration
Image space / process
Text space / process
Image / text decoder
JUICE
SigLIP2 features / FM
T5 features / FM
RAE / parallel decoder of T5 states
Transfusion-style
SigLIP2 features / FM
Discrete tokens / AR
RAE / token vocabulary head
D-DiT-style
SigLIP2 features / FM
Discrete tokens / MD
RAE / token vocabulary head
MMaDA-style
aMUSEd VQ codes [ 73 ] / MD
Discrete tokens / MD
VQ decoder / token vocabulary head
Appendix
Table 26: Configurations in the framework comparison. All use the 668M d19 backbone family and a schedule of 50k updates. FM denotes flow matching; AR denotes autoregressive; MD denotes masked diffusion.
Figure 13: Four generative configurations at a shared MM-RiT trunk scale. JUICE uses frozen SigLIP2 and T5 features for continuous image and text generation. The token controls use learned text embeddings without a T5 encoder; only the tokenizer and vocabulary are shared. The first three configurations use continuous SigLIP2 image states, whereas the fully discrete control uses VQ image codes. MSE and CE denote mean squared error and cross entropy, respectively.
Recipe
Updates (k)
Targets (M/mod.)
Work proxy
GE
DPG
CIDEr
CLIP
Mixing
50
159.33
1.000
48.41
78.85
88.80
80.20
DC (matched targets)
50
159.33
1.556
45.90
78.05
87.72
79.63
DC (matched work)
32.1
102.40
1.000
43.18
75.24
71.55
76.81
Blocked joint
50
159.33
1.000
45.49
78.14
86.58
78.66
Multi-task / 50k
50
102.40
1.000
36.30
77.71
79.41
75.07
Multi-task / 77.8k
77.8
159.33
1.556
43.16
78.51
89.31
79.14
Appendix
Table 27: Supervision and budget controls, 668M. Work proxy: backbone forward passes summed over training, relative to Mixing at 50k updates; DC spends two passes on each joint example.
Image representation
Text representation
GE
DPG
CIDEr
CLIP
SigLIP2
T5
48.41
78.85
88.80
80.20
DINO
T5
49.07
77.85
68.80
71.66
FLUX VAE
T5
35.42
74.05
49.95
64.91
Pixels
T5
24.48
71.64
49.57
65.51
SigLIP2
Qwen3-E
43.07
78.04
88.50
76.54
SigLIP2
Qwen3-VL-E
39.28
78.70
84.24
75.94
Appendix
Table 28: Configurations of representation spaces exhibit different strengths across generation and captioning.
Readout
Flan-T5
Qwen3-E
Qwen3-VL-E
SigLIP2 text
Linear head
.9849
.9686
.9652
.9813
Parallel decoder
.9922
.9797
.9748
.9870
Appendix
Table 29: Token accuracy of text readouts from clean states. 5,000 reference captions held out from training are encoded and immediately decoded, without MM-RiT generation. The linear heads are fitted to each representation; the parallel decoders are those of the pretraining study (Appendix A.4.2 ), whose linear branch starts from the linear head for Qwen3-E, Qwen3-VL-E, and SigLIP2 text. SigLIP2 text uses 64 positions and the other representations 128.
Image representation / decoder
rFID ImageNet
rFID YFCC
rFID RenderedText
FLUX VAE
.288
.410
.638
WebSSL-DINO / matched Scale-RAE decoder
.388
.558
1.372
SigLIP2 / matched Scale-RAE decoder
.435
.702
1.621
SDXL VAE (reconstruction reference)
.930
1.168
2.057
Pixels (identity reference)
.000
.000
.000
Appendix
Table 30: Reconstruction diagnostics of image encoders and decoders. Lower rFID is better. ImageNet, YFCC, and RenderedText each contain 50k reconstruction samples, following the Scale-RAE comparison [ 7 , Table 2] .
(a) GenEval
Model
Size
Single
Two
Count
Color
Position
Attr.
Overall
BLIP3-o [ 39 ]
8B
–
–
–
–
–
–
0.84
Janus-Pro-7B [ 51 ]
7B
0.99
0.89
0.59
0.90
0.79
0.66
0.80
Show-o2 [ 1 ]
7B
1.00
0.87
0.58
0.92
0.52
0.62
0.76
TUNA [ 21 ]
7B
1.00
0.97
0.81
0.91
0.88
0.83
0.90
TUNA-R [ 2 ]
7B
1.00
0.95
0.82
0.89
0.86
0.79
0.88
Appendix
Table 31: Image generation with larger models.
Model
Steps
Single
Two
Count
Color
Position
Attr.
Overall
JUICE (SigLIP2)
25
99.4
93.4
77.8
91.0
85.0
73.0
86.60
50
98.8
92.7
77.8
92.3
84.5
74.8
86.80
100
98.8
93.2
78.1
91.8
84.3
74.5
86.76
JUICE (DINO)
25
99.4
94.7
80.6
93.1
84.8
77.5
88.34
50
99.7
94.2
81.3
92.6
84.8
78.8
88.53
100
99.7
94.2
81.9
92.8
84.3
79.0
88.64
Appendix
Table 32: GenEval of JUICE at different numbers of sampling steps ( w=5 , ×100 ). The main text uses 25 steps.
Model
Size
Input
VQAv2 ↑
GQA ↑
MME-P ↑
POPE ↑
Models for understanding only
LLaVA-1.5 [ 64 ]
7B
3362
78.50
62.00
1510.7
85.90
Qwen-VL-Chat [ 74 ]
7B
4482
78.20
57.50
–
–
Composite unified models
BLIP3-o [ 39 ]
8B
Dyn.
83.10
–
1682.6
–
Native unified models
Appendix
Table 33: Visual understanding with larger models.
Benchmark
Split
Metric
SigLIP2
DINO
VQAv2 [ 63 ]
test-dev
VQA accuracy
73.10
68.75
yes/no
90.66
88.18
number
58.73
53.30
other
61.29
55.57
VQAv2
val2014, fixed 1,000
VQA accuracy
71.21
66.32
OK-VQA [ 75 ]
val2014
VQA accuracy
33.87
28.65
Appendix
Table 34: JUICE on 10 understanding benchmarks. VQAv2 is reported on two splits. Question answering with 16 sampling steps, guidance w=3 , and the parallel text decoder. OK-VQA is scored against the annotator answers as written (the raw_answer field of OK-VQA v1.1); the lemmatized answers of the same file give 29.15 and 24.63, since predictions are not lemmatized.
Model
Size
OK-VQA
SQA-IMG
D-DiT ( 2562 ) [ 18 ]
2.1B
28.5
–
D-DiT ( 5122 ) [ 18 ]
2.1B
25.3
–
LLaVA-1.5 [ 64 ]
7B
–
66.8
Qwen-VL-Chat [ 74 ]
7B
56.6
68.2
JUICE (SigLIP2)
2.1B
33.9
80.6
JUICE (DINO)
2.1B
28.7
73.7
Appendix
Table 35: Remaining understanding benchmarks. Only values reported in each model’s paper are listed. JUICE uses OK-VQA val and ScienceQA-IMG test, and its SFT data include the ScienceQA training split (Table 24 ).
SigLIP2
DINO
Sampler
Seed 1
Seed 2
Seed 1
Seed 2
Coupled
95.2
92.2
82.2
81.2
Image first
95.8
94.6
83.4
82.0
Text first
84.8
85.8
59.4
60.4
Independent (8 steps per stream)
56.0
53.2
42.6
42.8
Appendix
Table 36: Joint success of each sampling seed on mazes (16 backbone calls, 500 mazes).
J at 8 calls
Sampler
Steps at 16 calls
Steps at 8 calls
SigLIP2
DINO
Coupled
16 shared
8 shared
93.7
81.4
Image first
8 + 8
4 + 4
94.9
83.0
Text first
8 + 8
4 + 4
84.9
61.6
Independent
8 per stream
4 per stream
55.9
45.7
Appendix
Table 37: Maze samplers at 16 and at 8 backbone calls. A coupled step advances both streams with one call; independent and sequential sampling spend one call per stream per step. J at 16 calls is in Table 7 ; J at 8 calls is the mean of two sampling seeds.
Figure 14: Image generation conditioned on text: illustration and imagined scenes. Outputs for the same prompts show a wave beside a lighthouse, an astronaut riding a horse on the Moon, and an art nouveau peacock poster.
Figure 15: Image generation conditioned on text: varied visual styles. The prompts specify a fantasy matte painting, a watercolor sailboat, and an ink wash heron. Complete prompts accompany each pair.
Figure 16: Image generation conditioned on text: natural and architectural scenes. A fox in a snowy forest, a lamplit street, and a cathedral interior illustrate differences in composition, lighting, and texture under shared prompts.
Figure 17: Image captioning with shared visual inputs. SigLIP2+T5 and DINO+T5 describe a bedroom, a restaurant meal, and a workspace.
Figure 18: Image captioning with shared visual inputs (continued). Both models describe a road through tea plantations, a vintage family portrait, and a church with a clock tower.
Figure 19: Joint generation of images and text with SigLIP2+T5. Three generated pairs depict a winged cartoon boy, a person on a lakeside pier, and a woman in a flower garden.
Figure 20: Joint generation of images and text with DINO+T5. Three generated pairs depict a castle in a park, a woman at a bar, and a street parade.
Figure 21: How a joint sample forms. For four unconditional joint samples (64 steps), each column shows, at one step, the model’s current estimate of the clean image (decoded) and of the clean text (read out with the autoregressive caption decoder, first 28 words). The two streams follow different time grids, given under each step, and the text is further along at a given step: in SigLIP2 sample 25 it already names a snowboarding scene while the image has not yet formed. Later steps revise both streams; in DINO sample 5 the guessed location moves from London to Paris, Madrid, and Portugal as the facades sharpen. The second and fourth rows end in samples shown in Figure 10 .
Figure 22: Image captions with long answers. Detailed descriptions of jets trailing colored smoke, a wedding under a canopy, and a stone doorway.
Figure 23: Image captions with long answers (continued). Detailed descriptions of a portrait in a fur coat, a cartoon elf, and a 3D robot.
Figure 24: Image captions with long answers (continued). Detailed descriptions of a wall with a red side table, a street seen from above, and a person beside a road.
Figure 25: Image captions with long answers (continued). Detailed descriptions of a red stone texture and a bedroom. Each row compares the two models on the same image and prompt.
Figure 26: Visual question answering: attributes and quantities. Questions concern material, furniture identity, counting, and relative appliance size.
Figure 27: Visual question answering: spatial relations and actions. Questions concern left/right relations and the activities of people in the image.
Figure 28: Visual question answering: compositional and outside knowledge queries. Questions identify object materials using relational descriptions and ask for outside knowledge about the animal shown.
Figure 29: Visual question answering: outside knowledge and object existence. Questions ask where an airplane comes from and whether specified objects are present.
Figure 30: Visual question answering: additional object, action, and color examples. Questions ask what a man is riding, what a man is doing, and the color of a cat’s eyes and of a wave, without the instruction to answer briefly. Model responses retain their original wording.
Figure 31: Visual question answering: additional object and color examples. Questions ask what a man holds in his hand, the color of a fire hydrant and of the sky, and what a woman is floating on. Model responses retain their original wording.
A recent cutting-edge topic in multimodal modeling is to unify visual comprehension and generation within a single model. However, the two tasks demand mismatched decoding regimes and visual representations, making it non-trivial to jointly optimize within a shared feature space. In this work, we present Cheers, a unified multimodal model that decouples patch-level details from semantic representations, thereby stabilizing semantics for multimodal understanding and improving fidelity for image generation via gated detail residuals. Cheers includes three key components: (i) a unified vision tokenizer that encodes and compresses image latent states into semantic tokens for efficient LLM conditioning, (ii) an LLM-based Transformer that unifies autoregressive decoding for text generation and diffusion decoding for image generation, and (iii) a cascaded flow matching head that decodes visual semantics first and then injects semantically gated detail residuals from the vision tokenizer to refine high-frequency content. Experiments on popular benchmarks demonstrate that Cheers matches or surpasses advanced UMMs in both visual understanding and generation. Cheers also achieves 4x token compression, enabling more efficient high-resolution image encoding and generation. Notably, Cheers outperforms the Tar-1.5B on the popular benchmarks GenEval and MMBench, while requiring only 20% of the training cost, indicating effective and efficient (i.e., 4x token compression) unified multimodal modeling. We will release all code and data for future research.
Yichen Zhang, Da Peng, Zonghao Guo +19
Tsinghua University · Xi’an Jiaotong University · University of Chinese Academy of Sciences
We present Lance, a lightweight native unified model supporting multimodal understanding, generation, and editing for both images and videos. Rather than relying on model capacity scaling or text-image-dominant designs, Lance explores a practical paradigm for unified multimodal modeling via collaborative multi-task training. It is grounded in two core principles: unified context modeling and decoupled capability pathways. Specifically, Lance is trained from scratch and employs a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, enabling joint context learning while decoupling the pathways for understanding and generation. We further introduce modality-aware rotary positional encoding to mitigate interference among heterogeneous visual tokens and boost cross-task alignment. During training, Lance adopts a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling to strengthen both semantic comprehension and visual generation performance. Experimental results demonstrate that Lance substantially outperforms existing open-source unified models in image and video generation, while retaining strong multimodal understanding capabilities. The homepage is available at https://lance-project.github.io.
We present Multimodal Flow, a fully continuous generative model of language and vision. Most unified multimodal models either model both language and quantized images as discrete tokens or combine discrete language prediction with continuous image generation. The former introduces a visual quantization bottleneck. The latter requires modality-dependent objectives and sampling procedures. Fully continuous modeling avoids these trade-offs and enables a shared generative process, but remains underexplored for multimodal pretraining. Multimodal Flow introduces a unified continuous architecture that integrates multimodal continuous representations with a shared chunk-causal flow backbone. It organizes text blocks and images as ordered continuous hyperchunks, preserving textual token order and visual spatial structure. The backbone learns a single vector field over these hyperchunks through Flow Matching. Joint attention enables cross-modal interaction, while modality-specific feed-forward networks process each modality. The model predicts multiple target chunks in parallel during training and generates hyperchunks sequentially at inference. We instantiate MF-1 and pretrain it on multimodal data. Across 0.6B, 1.2B, and 1.6B scales, continued pretraining consistently improves multimodal modeling. With only 150B pretraining tokens, MF-1 achieves an average score of 82.8 across GenEval and DPG-Bench and 75.3 across VQAv2, MMBench, and POPE, remaining competitive with unified models trained on substantially more data. Under matched data, optimization, and parameter budgets, Multimodal Flow further outperforms representative hybrid and discrete models. These results establish continuous chunk-based embedding flow modeling as a new fully continuous paradigm for unified multimodal modeling. The related code and model are publicly released at https://github.com/hustvl/Multimodal-Flow.
Hongyuan Tao, Xinggang Wang, Lianghui Zhu +7
Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics