JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces

Jiahan Li; Huangyuan Su; Shengbang Tong; Shusheng Yang; Ruihan Guo; Suozhi Huang; Hexu Zhao; Pratyusha Sharma

cs.CVOct 20, 2026

JUICE: Unified Multimodal Modeling in Pretrained Representation Spaces

Authors: Jiahan Li, Huangyuan Su, Shengbang Tong, Shusheng Yang, Ruihan Guo, Suozhi Huang, Hexu Zhao, Pratyusha Sharma,

Organizations: New York University · Harvard University · University of Illinois Urbana-Champaign · Princeton University ·

Abstract

Most unified multimodal models treat images and text asymmetrically: a pretrained language model generates text, an attached diffusion head generates images, and outputs that contain both are produced in a fixed order, so the first output cannot adapt to the second. We ask whether both modalities can instead be generated by one process that is trained on them jointly from scratch. Pretrained encoder spaces hold both modalities in one continuous form and carry knowledge learned at scale, and recent work finds them effective as generative targets for vision and language. We therefore model the joint distribution directly in these spaces. We introduce JUICE (Joint Unified multimodal modeling In Continuous Encoder spaces), a single flow matching Transformer over frozen image and text representations, in which each modality has its own noise level. Which modalities are clean and which are generated then decides the task: image generation, captioning, visual question answering, or joint generation. Controlled pretraining runs test each design choice. At equal guidance, generating text as continuous states leaves image generation intact (48.41 GenEval, against 47.86 for a model trained only for text-to-image generation), while captions lose some exact wording. Training on joint and conditional denoising together beats either alone, and semantic encoder spaces beat VAE latents and raw pixels as targets. At 2.1B parameters, with no language model initialization and at 224x224, JUICE reaches up to 0.88 on GenEval and 73.1 on VQAv2. By sampling both modalities from the joint distribution (coupled sampling), it can also draw a route on a map while writing out the matching moves. On mazes with several valid solutions, joint success rises from 54.6% with independent sampling to 93.7%, and no modality has to go first. On visual spatial planning, JUICE solves 99.8% of maps at its training sizes.

Figures & tables

Figure 1: JUICE overview. (a) Continuous image and text representations from frozen encoders share one MM-RiT backbone via bidirectional joint attention. Appendix A.3 explains the details of each block. (b) Conditional tasks (text-to-image, visual question answering) and joint reasoning (maze solving) use the same interface by assigning clean context ( $t{=}1$ ) and generation targets ( $t{<}1$ ).

Model	Text representation	Text process	Image representation	Image process
Chameleon [ 16 ]	Discrete tokens	AR	Discrete codes	AR
Transfusion [ 17 ]	Discrete tokens	AR	VAE latents	Diffusion
Show-o2 [ 1 ]	Discrete tokens	AR	VAE latents	FM
Tuna-2 [ 2 ]	Discrete tokens	AR	Pixels	FM
MMaDA [ 3 ]	Discrete tokens	Masked diffusion	Discrete codes	Masked diffusion
Muddit [ 4 ]	Discrete tokens	Masked diffusion	Discrete codes	Masked diffusion

Table 1: Generation spaces and processes in multimodal models. AR: autoregressive; FM: flow matching; repr.: representations.

Panel of Figure 2 — Figure 2: Tasks are paths in the 2D time domain. Each modality has its own time, from noise ( $t{=}0$ ) to clean ( $t{=}1$ ). Conditional tasks run along the edges, where the condition is clean, and joint generation crosses the interior; its path bends because the image schedule keeps the image noisier for longer. Training covers both regions: conditional denoising on the edges and joint denoising in the interior.

	Generation			Image generation		Captioning
Configuration	Text	Image	Text condition	GE (%)	DPG (%)	CIDEr	CLIP
JUICE	FM	FM	frozen T5 states	48.41	78.85	88.80	80.20
Transfusion-style [ 17 ]	AR	FM	token embeddings	44.23	76.84	160.52	80.37
D-DiT-style [ 18 ]	MD	FM	token embeddings	34.50	74.35	115.22	79.82
MMaDA-style [ 3 ]	MD	MD (VQ)	token embeddings	41.61	66.50	89.53	71.61
Transfusion-style	AR	FM	frozen T5 states	46.07	78.24	158.63	80.36

Table 2: Generative frameworks at 668M. All configurations share the trunk with two streams, the data, and the schedule. FM: flow matching; AR: autoregressive; MD: masked diffusion. The middle block gives the discrete text frameworks the same frozen T5 condition as JUICE, and the last row is a model trained only for text-to-image generation.

Figure 3: Continuous text generation gives the strongest image generation throughout training. Generative framework comparison at the 668M scale (Table 2 ).

	Task ratio (%)			Image generation		Captioning
Recipe	I2T	T2I	Joint	GE (%)	DPG (%)	CIDEr	CLIP
T2I only	0	100	0	47.86	77.50	2.83	25.15
I2T only	100	0	0	0.21	9.95	107.22	77.81
Multi-task	50	50	0	36.30	77.71	79.41	75.07
Joint only	0	0	100	31.33	56.98	75.66	73.70
Mixing (default)	22.2	22.2	55.6	48.41	78.85	88.80	80.20

Table 3: Training task mixtures at 668M. Task ratios give the share of examples assigned to each denoising task. The controls hold the number of supervised targets or the cross-modal exchange fixed (Appendix B.2.1 ): matched DC (dual conditional) replaces each joint example with two conditional ones, blocked joint removes cross-modal attention on joint examples, single target joint supervises only one stream of each joint example, and Multi-task trained for 77.8k updates matches the target count of Mixing.

Figure 4: Joint plus conditional training leads conditional training alone throughout pretraining. Mixing combines joint, T2I, and I2T denoising; Multi-task uses T2I and I2T only.

Figure 5: Semantic encoder spaces outperform VAE latents and pixels, and T5 leads among text spaces. Top: image spaces with T5 text. Bottom: text spaces with SigLIP2 images.

Figure 6: Scaling the backbone from 179M to 2.1B parameters. DINO’s early lead in image generation disappears with scale, while SigLIP2 leads captioning at every size. DPG-Bench in this figure uses the time swap text reference (Appendix A.6.2 ).

(a) GenEval $\uparrow$
Model	Size	Single	Two	Count	Color	Position	Attribute	Overall
Muddit [ 4 ]	1B	0.98	0.72	0.54	0.82	0.19	0.41	0.61
Show-o [ 20 ]	1.3B	0.98	0.80	0.66	0.84	0.31	0.50	0.68
UniDisc [ 49 ]	1.4B	0.92	0.47	0.15	0.67	0.13	0.19	0.42
FUDOKI [ 50 ]	1.5B	0.96	0.85	0.56	0.88	0.68	0.67	0.77
Janus-Pro-1B [ 51 ]	1.5B	0.98	0.82	0.51	0.89	0.65	0.56	0.73

Table 4: Text-to-image generation against unified models of 1B to 3B parameters. GenEval on the [0,1] scale; DPG-Bench with the official script. JUICE samples

224{\times}224

images with 25 steps; its size counts the MM-RiT backbone without the frozen encoders and decoders, and baseline sizes are as reported.

Panel of Figure 7 — Figure 7: Text-to-image samples from both variants for the same prompts. (a) Counting, spatial relations, and unusual combinations of objects. (b) Prompts that ask for a specific artistic style. All samples use guidance $w{=}5$ with the generic caption reference; Appendix E.1 gives more samples.

Unified models of 1B to 3B parameters
Model	Size	Cult.	Time	Space	Bio.	Phys.	Chem.	Overall
OneFlow [ 23 ]	1B	0.63	–	–	–	–	–	–
JanusFlow [ 53 ]	1.3B	0.13	0.26	0.28	0.20	0.19	0.11	0.18
Show-o ( $512^{2}$ ) [ 20 ]	1.3B	0.28	0.40	0.48	0.30	0.46	0.30	0.35
Janus-Pro-1B [ 51 ]	1.5B	0.20	0.28	0.45	0.24	0.32	0.16	0.26
Show-o2 [ 1 ]	1.5B	0.30	0.38	0.47	0.32	0.42	0.28	0.35

Table 5: Text-to-image generation with world knowledge on WISE. WiScore under the original protocol (GPT-4o-2024-05-13 judge); Overall weights categories by prompt count. JUICE is scored on two images per prompt, and its 95% confidence interval on Overall is about

\pm 0.02

. JUICE uses the same guidance as on GenEval (

w{=}5

, generic caption reference). Baseline values are from the WISE paper [ 13 ] or the models’ own papers; OneFlow reports only the cultural category.

Panel of Figure 8 — Figure 8: WISE prompts that require world knowledge. Each prompt names what to draw only indirectly; the gray line gives the implied knowledge (abridged from the WISE explanation). The prompt alone is the condition, with guidance $w{=}5$ .

Model	Size	Init.	Res.	VQAv2	GQA	POPE	MME-P	MMB	AI2D	MMMU	RWQA
Muddit [ 4 ]	1B	T2I	$512^{2}$	68.2	57.5	–	1107.4 †	–	–	27.6	–
OneFlow [ 23 ]	1B	LLM	$512^{2}$	67.7	57.8	85.7	1497.1 †	69.0	58.5	29.8	50.6
Show-o [ 20 ]	1.3B	LLM	$256^{2}$	64.7	54.2	76.2	1014.9 †	–	–	25.0	–
Show-o [ 20 ]	1.3B	LLM	$512^{2}$	69.4	58.0	80.0	1097.2 †	–	–	26.7	–
JanusFlow [ 53 ]	1.3B	LLM	$384^{2}$	79.8	60.3	88.0	1333.1	74.9	–	29.3	–
Harmon [ 52 ]	1.5B	LLM	$512^{2}$	–	58.9	87.6	1155.0	65.5	–	38.9	–

Table 6: Visual understanding against unified models of 1B to 3B parameters. Init. gives the backbone initialization and Res. the input resolution (LLM: pretrained language model; T2I: pretrained text-to-image model). Baseline values and sizes are from each model’s paper; the size of JUICE counts the MM-RiT backbone without the frozen encoders and decoders (Appendix A.5 ). † The paper reports an unlabeled MME score, which may be the sum of perception and cognition. MME-P: MME perception score; MMB: MMBench; RWQA: RealWorldQA. JUICE uses VQAv2 test-dev and MMBench dev; Appendix C gives all 10 benchmarks.

Panel of Figure 9 — Figure 9: Selected questions of different types, answered by both variants. Each panel gives the question type and benchmark, the question, both answers, and the reference answer. Appendix E gives more examples, including captions and long answers.

Panel of Figure 10 — Figure 10: Joint samples of images and text. With no condition, both streams start from noise and are sampled together (64 steps, no guidance). Text is read out with the autoregressive caption decoder (Appendix A.4.2 ) and shown up to the point where it starts to repeat. Appendix E.3 gives more samples, and Figure 21 shows how both streams form during sampling.

Validity of one modality alone
Sampler	Order	SigLIP2	DINO
Coupled	none	93.7	81.7
Image first	image $\to$ text	95.2	82.7
Text first	text $\to$ image	85.3	59.9
Independent	none	54.6	42.7
Image FT only	–	95.1	83.3

Table 7: Joint success on mazes with multiple solutions. J: both outputs are shortest routes and describe the same route. Samplers are compared at 16 backbone calls; mean of two sampling seeds. Lower rows give the route validity of models finetuned on a single modality.

Figure 11: Coupled sampling brings the two outputs onto one route; independent sampling does not. Two mazes under coupled sampling at three of its 16 steps per stream, and under independent sampling with the same 16 steps per stream (32 backbone calls, twice the budget of Table 7 ) at its last step; both samplers start from the same state at step 0, and the coupled outputs do not change after the last step shown. The blue path is the decoded image, and the dashed line traces the decoded moves from the start. Underlined moves differ from the previous column.

Model	Size	Task FT	Lv3	Lv4	Lv5	Lv6	Avg.	Lv7	Lv8
InternVL3.5 [ 56 ]	38B	✗	–	–	–	–	0.20	–	–
ThinkMorph [ 57 ]	7B	✓	–	–	–	–	0.76	–	–
LatentUM, coarse [ 55 ]	4B	✓	1.00	0.85	0.83	0.71	0.85	–	–
LatentUM, fine [ 55 ]	4B	✓	1.00	1.00	1.00	0.97	0.99	–	–
JUICE (SigLIP2)	2.1B	✓	1.00	1.00	1.00	0.99	0.998	0.94	0.69

Table 8: Visual spatial planning on VSP. Success rate under the official rule; levels 3 to 6 match the training map sizes, and levels 7 and 8 are larger maps unseen in training. Baseline values are from LatentUM [ 55 ] , which reports InternVL3.5 and ThinkMorph only as averages; ThinkMorph does not state which levels its average covers. Task FT marks finetuning on maps in the style of VSP: JUICE uses 55.9k maps, ThinkMorph and coarse LatentUM 6k, and fine LatentUM 18k problems solved step by step.

Figure 12: VSP cases. Adaptation uses sizes $3{\times}3$ to $6{\times}6$ . (a) One solved map of each size. (b) Maps whose moves fail at the first of 16 sampling steps and solve the map at the last, with the drawn path fixed throughout. The red path is the decoded image, and the dashed line traces the decoded moves from the start; a cross marks where the moves fail, in a hole or short of the goal. A repeated move is written with its count ( $\mathrm{L}^{4}$ for LLLL), and underlined runs change between the two steps. Decoded images are matched in color to the input rendering.

Appendix figures & tables48 assets

Supplementary material from the paper’s appendix.

Appendix

Input	Frozen source and extraction	Tokens $\times$ channels
SigLIP2	SigLIP2-SO400M/14 , $224^{2}$ , final patch states	$256\times 1152$
DINO	WebSSL-DINO300M-full2b , $224^{2}$ , final patch states	$256\times 1024$
VAE	FLUX.1-schnell VAE , $256^{2}$ , packed latent mode	$256\times 64$
Pixel	None; $256^{2}$ RGB in $[-1,1]$ , $16\times 16$ patches	$256\times 768$

Appendix

Table 9: Image encoder sources and token shapes.

Frozen encoder	Layer	Positions	Channels
Flan-T5-Small [ 36 ]	Final (8)	128	512
Qwen3-Embedding-0.6B [ 47 ]	14	128	1,024
Qwen3-VL-Embedding-2B [ 48 ]	14	128	2,048
SigLIP2-SO400M/14 text [ 11 ]	13	64	1,152

Appendix

Table 10: Selected text encoder states.

Representation	Transformation
SigLIP2 / DINO image	Per token LayerNorm, without affine parameters
FLUX VAE image	$z=(h-b_{\mathrm{VAE}})a_{\mathrm{VAE}}$ using predefined factors
Pixel image	RGB values scaled to $[-1,1]$ , then patchified
Flan-T5 text	$z=h/0.2$ , a fixed scalar for all channels
Qwen3-E / Qwen3-VL-E text	$z_{c}=(h_{c}-\mu_{c})/s_{c}$ using fixed corpus statistics
SigLIP2 text	$z_{c}=(h_{c}-\mu_{c})/s_{c}$ using fixed corpus statistics

Appendix

Table 11: Representation normalization before diffusion.

h

denotes an encoder output,

c

a feature channel, and

(b_{\mathrm{VAE}},a_{\mathrm{VAE}})

the shift and scale supplied by the pretrained FLUX VAE.

Configuration	Parameters	Width	Heads	Head dim.	Joint blocks	Total blocks
d12	178.6M	768	12	64	10	12
d15	335.0M	960	15	64	13	15
d19	668.3M	1,216	19	64	17	19
d24	1.33B	1,536	24	64	22	24
d28	2.12B	1,792	28	64	26	28

Appendix

Table 12: Backbone scaling varies width and depth (trainable parameters). Parameter counts exclude encoders and decoders. Each backbone also has two input layers for text only; d12 to d28 denote head counts.

Representation	$D$	$V$	Length	Parameters	Token accuracy
Qwen3-E	1,024	151,680	128	253M	0.9797
Qwen3-VL-E	2,048	151,936	128	409M	0.9748
SigLIP2 text	1,152	256,000	64	446M	0.9870

Appendix

Table 13: Parallel decoders of the other text representations.

D

is the state width,

V

the vocabulary size, and length the number of text positions. Token accuracy is measured from clean states on the 5,000 held out captions of Table 29 .

Decoder	$\tau$	Token accuracy	String match	EOS error
Parallel (final)	1.0	0.9997	0.984	0.00
Autoregressive	1.0	0.9759	0.966	0.45
Parallel, first stage only	1.0	–	0.941	–
Linear head	1.0	0.9872	0.508	10.5
Parallel (final)	0.9	0.9997	0.983	0.00
Autoregressive	0.9	0.9689	0.953	0.45

Appendix

Table 14: Text decoder fidelity on 2,000 captions held out from training.

\tau

is the clean fraction of each input state (

\tau=1

: clean). String match compares the lowercased text up to the first EOS. EOS error is the mean distance, in positions, between the predicted and true end of the text. The row of the linear head differs slightly from Table 29 , which uses a different set of captions.

	SigLIP2		DINO
Benchmark	Parallel	Autoregressive	Parallel	Autoregressive
VQAv2 (val, 1,000)	71.21	71.71	66.32	66.68
OK-VQA	33.87	35.32	28.65	29.69
POPE	83.97	83.97	82.29	82.29
GQA	58.58	58.42	55.68	55.76
MME-P	1115.7	1133.1	960.2	983.6

Appendix

Table 15: Parallel and autoregressive decoding of the same generated answer states (16 steps,

w=3

Text representation	Input $D$	Vocabulary $V$	Parameters (M)
Flan-T5-Small	512	32,128	16.48
Qwen3-Embedding-0.6B	1,024	151,680	155.47
Qwen3-VL-Embedding-2B	2,048	151,936	311.32
SigLIP2 text	1,152	256,000	295.17

Appendix

Table 16: Linear readout head sizes.

Component	Parameters	Used in
Principal image/text pipeline components
SigLIP2-SO400M image encoder	427.7M	I2T and VQA (SigLIP2 arm)
WebSSL-DINO300M image encoder	303.7M	I2T and VQA (DINO arm)
Flan-T5-Small encoder	35.3M	T2I, I2T, and VQA
SigLIP2 Scale-RAE image decoder	415.6M	T2I (SigLIP2 arm)
DINO Scale-RAE image decoder	415.4M	T2I (DINO arm)

Appendix

Table 17: Fixed component sizes. Parameters were counted from the recorded module state dictionaries. All text decoders are fixed during MM-RiT training.

Image arm	Scale	Trainable	T2I forward	I2T/VQA forward	All loaded
SigLIP2	d12	178.6M	629.5M	677.0M	$\approx$ 1.09B
	d15	335.0M	785.9M	833.4M	$\approx$ 1.25B
	d19	668.3M	$\approx$ 1.12B	$\approx$ 1.17B	$\approx$ 1.58B
	d24	1.33B	$\approx$ 1.78B	$\approx$ 1.83B	$\approx$ 2.24B
	d28	2.12B	$\approx$ 2.57B	$\approx$ 2.62B	$\approx$ 3.03B
DINO	d12	178.6M	629.3M	553.0M	968.4M

Appendix

Table 18: Task active and fully loaded parameter counts. T2I is backbone

+

T5 encoder

+

matched image decoder. I2T/VQA is backbone

+

image encoder

+

T5 encoder

+

parallel text decoder (35.36M); the d24 and d28 entries are computed from the rounded backbone sizes, so their last digit is approximate. “All loaded” is the union.

Image representation	$(\mu_{\mathrm{i}},s_{\mathrm{i}})$	Noise scale $\sigma_{\mathrm{i}}$	Time shift $\alpha_{\mathrm{i}}$
SigLIP2 [ 6 ]	$(0,1)$	1	8.485
DINO [ 6 ]	$(0,1)$	1	8
FLUX VAE [ 10 ]	$(0,1)$	1	2
Pixel [ 34 ]	$(-0.8,0.8)$	2	1

Appendix

Table 19: Image noise settings in the representation configurations. The base logit normal parameters, noise scales, and time shifts depend on the representation.

Recipe	Text reference	$w=1$	$w=2$	$w=3$	$w=4$	$w=5$
T2I only	generic caption	36.12	47.26	51.23	51.03	47.86
	time swap	36.12	43.38	42.90	38.67	31.34
Multi-task	generic caption	32.36	43.44	44.29	40.85	36.30
	time swap	32.36	24.67	5.75 †	2.16 †	0.92 †
Joint only	generic caption	10.09	21.54	27.43	31.06	31.33
	time swap	10.09	3.30 †	0.78 †	0.31 †	0.25 †

Appendix

Table 20: GenEval of the pretraining task mixtures across guidance scales and text references (50k updates, 25 steps). At

w=1

the reference is not used. Bold marks each model’s best score with the generic caption; † marks collapse into images that look like mosaics. I2T only cannot generate images (0.21 at every scale) and is omitted.

Model	Cult.	Time	Space	Bio.	Phys.	Chem.	Overall
JUICE (SigLIP2)	0.466	0.533	0.642	0.396	0.531	0.346	0.488 [0.468, 0.508]
JUICE (DINO)	0.379	0.481	0.607	0.344	0.456	0.313	0.424 [0.405, 0.443]

Appendix

Table 21: WISE with the time swap text reference at

w=3

. 95% confidence intervals in brackets. The main text uses the generic caption reference at

w=5

(Table 5 ).

Setting	Value
Updates / global batch	50,000 / 4,096
Optimizer	AdamW, $(\beta_{1},\beta_{2})=(0.9,0.95)$ , weight decay 0.01
Learning rate	$4\times 10^{-4}$ ; warmup over 5,000 steps, then constant
Gradient norm cap / precision	1.0 / bfloat16
EMA decay (slow / fast)	0.99995 / 0.9996
Image / text loss coefficient	1.0 / 0.25

Appendix

Table 22: Controlled comparisons share the same pretraining settings. The budget controls of Table 27 differ only in the number of updates.

Stage	Starts from	Updates	Global batch	GPUs	Evaluated weights
1. Pretraining	random initialization	60,000	4,096	8	fast EMA
2. SFT	stage 1, step 60,000	15,000	1,024	8	fast EMA
3a. Maze adaptation	stage 2, step 15,000	5,000	1,024	8	raw
3b. VSP adaptation	stage 2, step 15,000	5,000	1,024	4	raw

Appendix

Table 23: Training stages of the final models. Each later stage starts from the raw weights (without EMA) of the stage before it.

Block	Sources (rows)	Task	$K$	Rows	Share
Text-to-image	BLIP3o-60k [ 39 ] (3,776,625), DALL-E 3 synthetic pairs (989,248), ShareGPT-4o-Image [ 62 ] (2,377,284)	caption $\to$ image	–	7,143,157	34.7%
Short QA	VQAv2 train [ 63 ] (3,010,000); LLaVA-1.5 mix [ 64 ] , twice: GQA [ 65 ] (1,887,666), COCO [ 66 ] (1,269,778), Visual Genome [ 67 ] (672,448), OCR-VQA [ 68 ] (635,350), VizWiz [ 69 ] (40,256)	image + question $\to$ short answer	12	7,515,498	36.5%
Multiple choice	Cambrian [ 40 ] multiple choice questions (433,570), ScienceQA train [ 70 ] (53,130)	image + question + options $\to$ letter	12	486,700	2.4%
Concise captions	COCO train2014 captions, twice (827,606), BLIP3-o short captions (1,250,000)	image + prompt (11 templates) $\to$ 1 to 40 tokens	48	2,077,606	10.1%
Detailed captions	BLIP3-o long captions (1,500,000), BLIP3o-60k and DALL-E 3 captions, four times (116,516)	image + prompt (16 templates) $\to$ 30 to 100 tokens	101	1,616,516	7.9%
Long answers	ALLaVA [ 71 ] via Cambrian (671,552), ShareGPT4V [ 72 ] (1,042,950), LLaVA (3,719)	image + question $\to$ 9 to 100 tokens	101	1,718,221	8.3%

Appendix

Table 24: SFT data. Row counts include repetition.

K

is the length of the generated answer window. Text-to-image rows train text-to-image generation; all other rows train image-to-text generation.

Initialization	Raw weights of the 2.1B pretraining run at step 60,000
Updates / global batch	15,000 / 1,024 (8 rows $\times$ 16 accumulation steps $\times$ 8 GPUs); 15.36M samples, about 0.75 epoch
Optimizer / learning rate	AdamW, $(\beta_{1},\beta_{2})=(0.9,0.95)$ ; $10^{-4}$ after 250 linear warmup updates, then constant
Weight decay / gradient cap	0.01 / 1.0
Precision / seed	bfloat16 / 4242
EMA decay (slow / fast)	0.99995 / 0.9996; we evaluate the fast EMA
Task assignment	text-to-image rows: text-to-image only; question, caption prompt, and long answer rows: image-to-text only

Appendix

Table 25: SFT recipe.

Configuration	Image space / process	Text space / process	Image / text decoder
JUICE	SigLIP2 features / FM	T5 features / FM	RAE / parallel decoder of T5 states
Transfusion-style	SigLIP2 features / FM	Discrete tokens / AR	RAE / token vocabulary head
D-DiT-style	SigLIP2 features / FM	Discrete tokens / MD	RAE / token vocabulary head
MMaDA-style	aMUSEd VQ codes [ 73 ] / MD	Discrete tokens / MD	VQ decoder / token vocabulary head

Appendix

Table 26: Configurations in the framework comparison. All use the 668M d19 backbone family and a schedule of 50k updates. FM denotes flow matching; AR denotes autoregressive; MD denotes masked diffusion.

Figure 13: Four generative configurations at a shared MM-RiT trunk scale. JUICE uses frozen SigLIP2 and T5 features for continuous image and text generation. The token controls use learned text embeddings without a T5 encoder; only the tokenizer and vocabulary are shared. The first three configurations use continuous SigLIP2 image states, whereas the fully discrete control uses VQ image codes. MSE and CE denote mean squared error and cross entropy, respectively.

Recipe	Updates (k)	Targets (M/mod.)	Work proxy	GE	DPG	CIDEr	CLIP
Mixing	50	159.33	1.000	48.41	78.85	88.80	80.20
DC (matched targets)	50	159.33	1.556	45.90	78.05	87.72	79.63
DC (matched work)	32.1	102.40	1.000	43.18	75.24	71.55	76.81
Blocked joint	50	159.33	1.000	45.49	78.14	86.58	78.66
Multi-task / 50k	50	102.40	1.000	36.30	77.71	79.41	75.07
Multi-task / 77.8k	77.8	159.33	1.556	43.16	78.51	89.31	79.14

Appendix

Table 27: Supervision and budget controls, 668M. Work proxy: backbone forward passes summed over training, relative to Mixing at 50k updates; DC spends two passes on each joint example.

Image representation	Text representation	GE	DPG	CIDEr	CLIP
SigLIP2	T5	48.41	78.85	88.80	80.20
DINO	T5	49.07	77.85	68.80	71.66
FLUX VAE	T5	35.42	74.05	49.95	64.91
Pixels	T5	24.48	71.64	49.57	65.51
SigLIP2	Qwen3-E	43.07	78.04	88.50	76.54
SigLIP2	Qwen3-VL-E	39.28	78.70	84.24	75.94

Appendix

Table 28: Configurations of representation spaces exhibit different strengths across generation and captioning.

Readout	Flan-T5	Qwen3-E	Qwen3-VL-E	SigLIP2 text
Linear head	.9849	.9686	.9652	.9813
Parallel decoder	.9922	.9797	.9748	.9870

Appendix

Table 29: Token accuracy of text readouts from clean states. 5,000 reference captions held out from training are encoded and immediately decoded, without MM-RiT generation. The linear heads are fitted to each representation; the parallel decoders are those of the pretraining study (Appendix A.4.2 ), whose linear branch starts from the linear head for Qwen3-E, Qwen3-VL-E, and SigLIP2 text. SigLIP2 text uses 64 positions and the other representations 128.

Image representation / decoder	rFID ImageNet	rFID YFCC	rFID RenderedText
FLUX VAE	.288	.410	.638
WebSSL-DINO / matched Scale-RAE decoder	.388	.558	1.372
SigLIP2 / matched Scale-RAE decoder	.435	.702	1.621
SDXL VAE (reconstruction reference)	.930	1.168	2.057
Pixels (identity reference)	.000	.000	.000

Appendix

Table 30: Reconstruction diagnostics of image encoders and decoders. Lower rFID is better. ImageNet, YFCC, and RenderedText each contain 50k reconstruction samples, following the Scale-RAE comparison [ 7 , Table 2] .

(a) GenEval
Model	Size	Single	Two	Count	Color	Position	Attr.	Overall
BLIP3-o [ 39 ]	8B	–	–	–	–	–	–	0.84
Janus-Pro-7B [ 51 ]	7B	0.99	0.89	0.59	0.90	0.79	0.66	0.80
Show-o2 [ 1 ]	7B	1.00	0.87	0.58	0.92	0.52	0.62	0.76
TUNA [ 21 ]	7B	1.00	0.97	0.81	0.91	0.88	0.83	0.90
TUNA-R [ 2 ]	7B	1.00	0.95	0.82	0.89	0.86	0.79	0.88

Appendix

Table 31: Image generation with larger models.

Model	Steps	Single	Two	Count	Color	Position	Attr.	Overall
JUICE (SigLIP2)	25	99.4	93.4	77.8	91.0	85.0	73.0	86.60
	50	98.8	92.7	77.8	92.3	84.5	74.8	86.80
	100	98.8	93.2	78.1	91.8	84.3	74.5	86.76
JUICE (DINO)	25	99.4	94.7	80.6	93.1	84.8	77.5	88.34
	50	99.7	94.2	81.3	92.6	84.8	78.8	88.53
	100	99.7	94.2	81.9	92.8	84.3	79.0	88.64

Appendix

Table 32: GenEval of JUICE at different numbers of sampling steps (

w=5

\times 100

). The main text uses 25 steps.

Models for understanding only
Model	Size	Input	VQAv2 $\uparrow$	GQA $\uparrow$	MME-P $\uparrow$	POPE $\uparrow$
LLaVA-1.5 [ 64 ]	7B	$336^{2}$	78.50	62.00	1510.7	85.90
Qwen-VL-Chat [ 74 ]	7B	$448^{2}$	78.20	57.50	–	–
Composite unified models
BLIP3-o [ 39 ]	8B	Dyn.	83.10	–	1682.6	–
Native unified models

Appendix

Table 33: Visual understanding with larger models.

Benchmark	Split	Metric	SigLIP2	DINO
VQAv2 [ 63 ]	test-dev	VQA accuracy	73.10	68.75
yes/no			90.66	88.18
number			58.73	53.30
other			61.29	55.57
VQAv2	val2014, fixed 1,000	VQA accuracy	71.21	66.32
OK-VQA [ 75 ]	val2014	VQA accuracy	33.87	28.65

Appendix

Table 34: JUICE on 10 understanding benchmarks. VQAv2 is reported on two splits. Question answering with 16 sampling steps, guidance

w{=}3

, and the parallel text decoder. OK-VQA is scored against the annotator answers as written (the raw_answer field of OK-VQA v1.1); the lemmatized answers of the same file give 29.15 and 24.63, since predictions are not lemmatized.

Model	Size	OK-VQA	SQA-IMG
D-DiT ( $256^{2}$ ) [ 18 ]	2.1B	28.5	–
D-DiT ( $512^{2}$ ) [ 18 ]	2.1B	25.3	–
LLaVA-1.5 [ 64 ]	7B	–	66.8
Qwen-VL-Chat [ 74 ]	7B	56.6	68.2
JUICE (SigLIP2)	2.1B	33.9	80.6
JUICE (DINO)	2.1B	28.7	73.7

Appendix

Table 35: Remaining understanding benchmarks. Only values reported in each model’s paper are listed. JUICE uses OK-VQA val and ScienceQA-IMG test, and its SFT data include the ScienceQA training split (Table 24 ).

	SigLIP2		DINO
Sampler	Seed 1	Seed 2	Seed 1	Seed 2
Coupled	95.2	92.2	82.2	81.2
Image first	95.8	94.6	83.4	82.0
Text first	84.8	85.8	59.4	60.4
Independent (8 steps per stream)	56.0	53.2	42.6	42.8

Appendix

Table 36: Joint success of each sampling seed on mazes (16 backbone calls, 500 mazes).

Sampler	Steps at 16 calls	Steps at 8 calls	SigLIP2	DINO
			J at 8 calls
Coupled	16 shared	8 shared	93.7	81.4
Image first	8 + 8	4 + 4	94.9	83.0
Text first	8 + 8	4 + 4	84.9	61.6
Independent	8 per stream	4 per stream	55.9	45.7

Appendix

Table 37: Maze samplers at 16 and at 8 backbone calls. A coupled step advances both streams with one call; independent and sequential sampling spend one call per stream per step. J at 16 calls is in Table 7 ; J at 8 calls is the mean of two sampling seeds.

Figure 14: Image generation conditioned on text: illustration and imagined scenes. Outputs for the same prompts show a wave beside a lighthouse, an astronaut riding a horse on the Moon, and an art nouveau peacock poster.

Figure 15: Image generation conditioned on text: varied visual styles. The prompts specify a fantasy matte painting, a watercolor sailboat, and an ink wash heron. Complete prompts accompany each pair.

Figure 16: Image generation conditioned on text: natural and architectural scenes. A fox in a snowy forest, a lamplit street, and a cathedral interior illustrate differences in composition, lighting, and texture under shared prompts.

Figure 17: Image captioning with shared visual inputs. SigLIP2+T5 and DINO+T5 describe a bedroom, a restaurant meal, and a workspace.

Figure 18: Image captioning with shared visual inputs (continued). Both models describe a road through tea plantations, a vintage family portrait, and a church with a clock tower.

Figure 19: Joint generation of images and text with SigLIP2+T5. Three generated pairs depict a winged cartoon boy, a person on a lakeside pier, and a woman in a flower garden.

Figure 20: Joint generation of images and text with DINO+T5. Three generated pairs depict a castle in a park, a woman at a bar, and a street parade.

Figure 21: How a joint sample forms. For four unconditional joint samples (64 steps), each column shows, at one step, the model’s current estimate of the clean image (decoded) and of the clean text (read out with the autoregressive caption decoder, first 28 words). The two streams follow different time grids, given under each step, and the text is further along at a given step: in SigLIP2 sample 25 it already names a snowboarding scene while the image has not yet formed. Later steps revise both streams; in DINO sample 5 the guessed location moves from London to Paris, Madrid, and Portugal as the facades sharpen. The second and fourth rows end in samples shown in Figure 10 .

Figure 22: Image captions with long answers. Detailed descriptions of jets trailing colored smoke, a wedding under a canopy, and a stone doorway.

Figure 23: Image captions with long answers (continued). Detailed descriptions of a portrait in a fur coat, a cartoon elf, and a 3D robot.

Figure 24: Image captions with long answers (continued). Detailed descriptions of a wall with a red side table, a street seen from above, and a person beside a road.

Figure 25: Image captions with long answers (continued). Detailed descriptions of a red stone texture and a bedroom. Each row compares the two models on the same image and prompt.

Figure 26: Visual question answering: attributes and quantities. Questions concern material, furniture identity, counting, and relative appliance size.

Figure 27: Visual question answering: spatial relations and actions. Questions concern left/right relations and the activities of people in the image.

Figure 28: Visual question answering: compositional and outside knowledge queries. Questions identify object materials using relational descriptions and ask for outside knowledge about the animal shown.

Figure 29: Visual question answering: outside knowledge and object existence. Questions ask where an airplane comes from and whether specified objects are present.

Figure 30: Visual question answering: additional object, action, and color examples. Questions ask what a man is riding, what a man is doing, and the color of a cat’s eyes and of a wave, without the instruction to answer briefly. Model responses retain their original wording.

Figure 31: Visual question answering: additional object and color examples. Questions ask what a man holds in his hand, the color of a fire hydrant and of the sky, and what a woman is floating on. Model responses retain their original wording.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 287 papers

Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation
Mar 13, 2026Yichen Zhang, Da Peng, Zonghao Guo +19Unified Multimodal Models Multimodal Generation
Tsinghua University · Xi’an Jiaotong University · University of Chinese Academy of Sciences
Lance: Unified Multimodal Modeling by Multi-Task Synergy
May 18, 2026Fengyi Fu, Mengqi Huang, Shaojin Wu +10Unified Multimodal Models Multi-Task Learning
Intelligent Creation Lab, ByteDance
Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces
Sep 30, 2026Hongyuan Tao, Xinggang Wang, Lianghui Zhu +7Unified Multimodal Models Multimodal Pretraining
Huazhong University of Science and Technology · Beijing Jiaotong University · Horizon Robotics

Mar 13, 2026J/K move · Enter open · S save

Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation

Yichen Zhang, Da Peng, Zonghao Guo, Zijian Zhang+18

Tsinghua University · Xi’an Jiaotong University · University of Chinese Academy of Sciences

Unified Multimodal Models Multimodal Generation Multimodal Understanding Multimodal Transformer Multimodal Disentangled Representation Learning

Open paper AskPDF arXiv

A recent cutting-edge topic in multimodal modeling is to unify visual comprehension and generation within a single model. However, the two tasks demand mismatched decoding regimes and visual representations, making it non-trivial to jointly optimize within a shared feature space. In this work, we present Cheers, a unified multimodal model that decouples patch-level details from semantic representations, thereby stabilizing semantics for multimodal understanding and improving fidelity for image generation via gated detail residuals. Cheers includes three key components: (i) a unified vision tokenizer that encodes and compresses image latent states into semantic tokens for efficient LLM conditioning, (ii) an LLM-based Transformer that unifies autoregressive decoding for text generation and diffusion decoding for image generation, and (iii) a cascaded flow matching head that decodes visual semantics first and then injects semantically gated detail residuals from the vision tokenizer to refine high-frequency content. Experiments on popular benchmarks demonstrate that Cheers matches or surpasses advanced UMMs in both visual understanding and generation. Cheers also achieves 4x token compression, enabling more efficient high-resolution image encoding and generation. Notably, Cheers outperforms the Tar-1.5B on the popular benchmarks GenEval and MMBench, while requiring only 20% of the training cost, indicating effective and efficient (i.e., 4x token compression) unified multimodal modeling. We will release all code and data for future research.

Abstract

Unified Multimodal Models

Explore similar work

Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation

Lance: Unified Multimodal Modeling by Multi-Task Synergy

Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces