Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation

Chao Li; Tianhong Li; Sai Vidyaranya Nuthalapati; Hong-You Chen; Satya Narayan Shukla; Jianpeng Cheng; Yonghuan Yang; Jun Xiao

cs.CVMar 3, 2026

Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation

Authors: Chao Li, Tianhong Li, Sai Vidyaranya Nuthalapati, Hong-You Chen, Satya Narayan Shukla, Jianpeng Cheng, Yonghuan Yang, Jun Xiao,

Organizations: MIT Computer Science & Artificial Intelligence Laboratory · Meta AI

Abstract

Unifying text-image contrastive learning and text-to-image (T2I) generation in a single end-to-end model is challenging because the two objectives demand opposing masking regimes: contrastive alignment needs near-complete visible tokens, while masked generative modeling needs heavy corruption. We introduce DREAM, a unified framework that resolves this conflict through Masking Warmup, a schedule that shifts the center of the masking distribution over training, so low and high masking ratios coexist at every step. This co-exposure lets a single jointly-trained encoder serve both objectives. The resulting stable optimization unlocks Semantically Aligned Decoding at inference: the text encoder, trained against visual embeddings at all masking ratios, can score partially generated images and select the best trajectory with as little as 12.5% of the image decoded, improving both FID and throughput. DREAM outperforms its single-objective baselines, CLIP and FLUID: on ImageNet linear-probing (+1.1%), 5-shot transfer (+4.1%), ADE20K segmentation (+1.9%), and NYU depth estimation (+6.25%) over CLIP, and on CC12M FID (+6.2%) over FLUID while maintaining CLIP Score. Together, these gains show that text-image contrastive and generative objectives, when properly unified, are synergistic rather than competing.

Figures & tables

Figure 1: Comparison of representation and T2I generation models on CC12M. DREAM (yellow) outperforms across all discriminative and generative axes, unifying visual understanding with T2I generation in a single stage model.

Figure 2: DREAM training framework. Images are encoded into continuous tokens via Stable Diffusion VAE and randomly masked and dropped following a masking warmup schedule. The vision encoder is trained contrastively with text, and the decoder conditions on text to predict masked tokens with a diffusion-based reconstructive loss. Text conditioning is introduced only in the decoder, ensuring the encoder learns visual representations without a text shortcut.

Figure 3: Semantically Aligned Decoding. The model spawns $K$ parallel candidates, each partially decoded to an intermediate timestep $t$ . The encoder scores each candidate by comparing its visual embedding to the prompt embedding, and only the top-scoring candidate is fully decoded.

	LP $\uparrow$	FID $\downarrow$
Baselines
FLUID	48.1	4.53
CLIP	71.6	N/A
DREAM (varying masking schedule)
FX	4.6	N/A
CD	73.1	5.80

Table 1: Component and Schedule Ablations. Isolating the contribution of masking warmup, diffusion reconstruction, and masking schedule design. DREAM (WM + reconstruction) achieves the best balance between representation (LP) and generation (FID).

	Visual Understanding											Text-to-Image Generation
	LP $\uparrow$	FT $\uparrow$										CC12M		MS-COCO
Model	IN-1K	IN-1K	IN-A	IN-R	IN-S	IN-H	MF	T7	Top	ON	Avg.	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$
MAR	50.7	80.4	20.6	44.6	29.4	21.1	69.0	77.0	81.7	37.4	51.2	✗	✗	✗	✗
FLUID	48.1	80.3	20.2	44.2	28.6	21.1	68.9	77.3	81.6	37.6	51.1	4.53	30.0	9.62	30.7
CLIP	71.6	81.1	24.3	53.9	40.8	23.3	69.3	77.1	81.7	37.8	54.4	✗	✗	✗	✗
REPA	62.5 †	81.7	27.7	51.2	37.2	23.1	71.0	79.1	83.1	39.1	54.8	4.42	29.9	10.0	30.7

Table 2: Unified evaluation of Visual Understanding and Text-to-Image generation. Comparison of models trained on CC12M across two paradigms: (a) Visual Understanding , measured by Top-1 accuracy (%) on ImageNet and its robustness benchmarks; and (b) Text-to-Image Generation , measured by FID and CLIP Score (CS) on CC12M and zero-shot MS-COCO. FID and CS for CC12M were computed over 50K samples, and FID and CS for MS-COCO were computed over 30K samples. ✗ indicates models inherently incapable of Text-to-Image generation.

	Model	Discriminative		T2I Generation
		IN-1K		CC12M-50K		MS-COCO-30K
		LP $\uparrow$	FT $\uparrow$	FID $\downarrow$	CS $\uparrow$	FID $\downarrow$	CS $\uparrow$
T2I	LPL Berrada et al. (2025)	–	–	6.22	25.1	–	–
	SDXL Ifriqi et al. (2024)	–	–	8.53	–	–	25.4
	mmDiT-SD3 Ifriqi et al. (2024)	–	–	7.54	–	–	24.8
	mmDiT-Imp Ifriqi et al. (2024)	–	–	6.79	–	–	26.6

Table 3: System-level comparison of T2I and unified models. We evaluate models on (1) discriminative representations measured by linear probing (“LP”) and fine-tuned (“FT”) accuracy on ImageNet-1K, and (2) T2I generation with FID and CLIP Score (CS) on CC12M and zero-shot MS-COCO. Metrics shown in blue are computed over 40K samples. “✗” indicates models without text-to-image capability, and “–” denotes unreported results. DREAM maintains competitive representation and generation quality against methods that use additional supervision † .

Method	Thr. $\uparrow$	FID $\downarrow$	CLIP $\uparrow$
64 NFE budget
DREAM (K=1, T=64)	3.39	4.57	29.1
128 NFE budget
DREAM (K=1, T=128)	1.77	4.34	29.4
DREAM (K=2) + Ext. CLIP †	1.69	4.50	29.6
DREAM (K=9) + SD	1.86	4.25	30.1

Table 4: Efficiency of DREAM decoding under fixed compute budgets (A100 GPU). We compare the throughput (images/sec), FID, and CLIP Score (CS) for varying candidate counts

K

against using pretrained CLIP as external reranker. SD : Semantically Aligned Decoding.

Figure 4: Performance of DREAM across different model sizes (B/L/H/G) for $\sigma{=}0.45$ . Top: Linear Probing on IN-1K. Bottom: FID on CC12M.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Examples of images generated by DREAM with and without Semantically Aligned Decoding (SD). Without Semantically Aligned Decoding, the outputs exhibit less coherent structure and more low-level blur. Applying Semantically Aligned Decoding produces images with clearer details and improved consistency with the prompt, in line with the gains observed in FID and CLIP scores.

Figure 6: Performance of DREAM across different masking standard deviations. Top: Linear Probing on IN-1K. Bottom: FID on CC12M-50K without Semantically Aligned Decoding. We do not plot the FID values for epoch 12 since the training has not converged and the FID values are too high (FID $>$ 50). The curves show that when $\sigma>0.35$ , both Linear Probing and FID performance are stable with increasing training duration even after masking warm-up ends.

Stage 2 Training	S1 Epochs	S2 Epochs	LP $\uparrow$	FID $\downarrow$
Generation-Only	12	37	1.2	N/A
	24	25	2.0	N/A
	36	13	8.5	N/A
Joint	12	37	1.8	N/A
	24	25	2.5	N/A
	36	13	9.1	N/A

Appendix

Table 5: Two-stage training results. FID reported as N/A where FID

>

50 (generation collapse). Stage 2 is either generation-only (MAR diffusion loss, no contrastive loss) or joint training (MAR diffusion loss + CLIP contrastive loss).

Model	DTD Cimpoi et al. (2014)	Caltech-256 Griffin et al. (2022)	SUN397 Xiao et al. (2010)	Food-101 Bossard et al. (2014)	VOC2007 Everingham et al. (2010)	STL-10 Coates et al. (2011)	Flowers Nilsback and Zisserman (2008)	UC Merced Yang and Newsam (2010)	EuroSAT Helber et al. (2017)	Country211 Thomee et al. (2016)	RESISC45 Cheng et al. (2017)	Dogs Khosla et al. (2011)	MIT Indoors Quattoni and Torralba (2009)	IN-1K Deng et al. (2009)	Average
MAR	64.9	56.1	73.0	42.3	47.2	53.1	74.6	77.3	74.9	28.9	73.8	37.7	61.2	62.3	59.1
FLUID	66.3	58.0	73.7	43.7	47.9	53.7	76.1	77.0	75.8	29.4	74.4	38.9	62.7	64.1	60.1
CLIP	81.0	95.1	95.8	86.0	79.6	96.1	96.2	92.6	82.1	42.9	90.4	78.2	93.7	94.3	86.0
REPA	73.2	73.6	88.1	60.2	61.2	74.6	86.5	82.4	79.2	33.8	80.4	50.2	81.2	78.9	71.7
DREAM	88.6	96.4	97.3	86.9	95.0	99.8	95.9	98.5	95.0	42.9	93.4	81.2	94.5	96.1	90.1

Appendix

Table 6: Few-shot transfer evaluation of different models . We report 5-way, 5-shot classification accuracy across 14 datasets. We highlight the best performance of each dataset in bold.

	Sem. Seg. (ADE20K)		Depth (NYU Depth V2)
Model	mIoU $\uparrow$	PixAcc $\uparrow$	RMSE $\downarrow$	ARel $\downarrow$
MAR	23.4	70.6	0.75	0.25
FLUID	22.1	69.5	0.76	0.26
CLIP	34.9	75.3	0.64	0.20
REPA	32.7	75.9	0.60	0.19
DREAM	36.8	76.7	0.60	0.19

Appendix

Table 7: Performance of frozen backbones on dense prediction tasks. Results are reported for semantic segmentation (on ADE20K Zhou et al. (2017) ) and depth estimation (on NYU Depth V2 Silberman et al. (2012) ) using linear probes Siméoni et al. (2025) mIoU: mean Intersection-over-Union; PixAcc: pixel accuracy; RMSE: root mean squared error; ARel: absolute relative error.

	4	6	8
LP $\uparrow$	58.0	62.5	66.1
FID $\downarrow$	4.71	4.42	4.76

Appendix

Table 8: Ablation on the layer to align with pretrained CLIP encoder. Applying the loss at earlier (shallower) layers yields better linear probing (LP) performance, whereas applying it at later (deeper) layers improves generation quality (lower FID), highlighting the trade-off between representation and generation quality.

	CLIP	DINOv2
LP $\uparrow$	62.5	68.4
FID $\downarrow$	4.42	4.67

Appendix

Table 9: Effect of Visual Backbone

Figure 7: Comparison of zero-shot accuracy for CLIP, CLIP with masking warmup (CLIP-M), and DREAM across masking ratios. The green shaded region marks the masking levels at which DREAM surpasses CLIP.

Condition	Avg. Intra-Prompt Diversity
DREAM, $K{=}1$ (no SD)	0.122
External CLIP reranker	0.117
DREAM (SD), $K{=}9$	0.119

Appendix

Table 10: Intra-prompt generation diversity measured by average pairwise cosine dissimilarity using an external CLIP ViT-B/32 encoder. Higher values indicate greater diversity.

	0.35	0.45	0.55	UNI
LP $\uparrow$	72.0	72.5	72.7	73.1
FID $\downarrow$	4.74	4.57	4.57	4.88

Appendix

Table 11: Standard Deviation of Masking Distribution (

\sigma

)

	30	36	42
LP $\uparrow$	72.5	72.5	72.4
FID $\downarrow$	4.68	4.57	4.71

Appendix

Table 12: Masking Warm-up Duration

	0	0.25	0.5	0.75
LP $\uparrow$	72.1	72.7	72.5	71.4
FID $\downarrow$	4.95	4.73	4.57	7.25

Appendix

Table 13: Varying Minimum Masking Ratio for Diffusion Loss (

\gamma

)

	0.25	0.5	0.75	1.0
LP $\uparrow$	63.4	70.0	72.5	72.5
FID $\downarrow$	8.65	4.81	4.57	4.71

Appendix

Table 14: Varying Maximum Masking Ratio for CLIP loss (

\phi

)

	0.002	0.005	0.01
LP $\uparrow$	72.1	72.5	72.0
FID $\downarrow$	4.76	4.57	4.64

Appendix

Table 15: Effect of CLIP Loss Weight (

\lambda

)

	All Tokens	Buffer Tokens
LP $\uparrow$	72.5	72.1
FID $\downarrow$	4.57	4.57

Appendix

Table 16: Application of CLIP loss.

	6	8	10	12
LP $\uparrow$	66.6	68.9	70.7	72.5
FID $\downarrow$	4.51	4.71	4.53	4.57

Appendix

Table 17: CLIP loss layer

Model	Total Params	Train GFLOPs/sample	Training Time	Peak GPU Mem
FLUID-L	570M	772.8	$\sim$ 48 hours	43,827 MB
REPA-L	578M	825.2	$\sim$ 48 hours	44,872 MB (+2.4%)
DREAM-L	636M (+11.6%)	877.2 (+13.5%)	$\sim$ 50 hours (+4.2%)	45,688 MB (+4.2%)

Appendix

Table 18: Training efficiency comparison of large-scale models. Training overhead and peak memory are measured on the same hardware setup.

Config	Value
Optimizer	AdamW
Base learning rate	$1\times 10^{-4}$
Weight decay	0.02
Momentum	(0.9, 0.95)
Batch size	2048
LR schedule	Constant

Appendix

Table 19: Pre-training settings for CLIP.

Config	Value
Optimizer	AdamW
Base learning rate	$1\times 10^{-4}$
Weight decay	0.02
Momentum	$(0.9,0.95)$
Batch size	2048
LR schedule	Constant

Appendix

Table 20: Pre-training settings for MAR, and FLUID.

Config	Value
Optimizer	AdamW
Base learning rate	$1\times 10^{-4}$
Weight decay	0.02
Momentum	$(0.9,0.95)$
Batch size	2048
LR schedule	Constant

Appendix

Table 21: Pre-training settings for REPA.

Config	Value
Optimizer	AdamW
Base learning rate	$1\times 10^{-4}$
Weight decay	0.04
Momentum	$(0.9,0.95)$
Batch size	2048
LR schedule	Constant

Appendix

Table 22: Pre-training settings for DREAM.

Config	Value
Optimizer	LARS You et al. (2017)
Base learning rate	0.1 (B), 0.05 (L, H), 0.02 (G)
Weight decay	0
Optimizer momentum	0.9
Batch size	4096
Learning rate schedule	Cosine decay Loshchilov and Hutter (2016)

Appendix

Table 23: Linear Probing Settings.

Config	Value
Optimizer	AdamW Loshchilov and Hutter (2017)
Base learning rate	$2.5\times 10^{-4}$
Weight decay	0.05
Optimizer momentum	$\beta_{1},\beta_{2}=0.9,0.999$
Layer-wise LR decay Bao et al. (2021)	0.75
Batch size	1024

Appendix

Table 24: Fine-tuning Settings.

Figure 8: Images generated by DREAM-L (0.57B) model from CC12M captions with CFG = 5.0.

Figure 9: Images generated by DREAM-H (1.1B) model from CC12M captions with CFG = 5.0.

Figure 10: Images generated by DREAM-G (2.4B) model from CC12M captions with CFG = 5.0.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 299 papers

Large Language Models are Universal Reasoners for Visual Generation
May 5, 2026Sucheng Ren, Chen Chen, Zhenbang Wang +5Diffusion Model Guidance Image Generation
1Johns Hopkins University · 2Apple
Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment
Jun 29, 2026Soyoun Won, Aryan Yazdan Parast, Basim Azam +2Text-Image Alignment T2I Generation
The University of Melbourne, Australia
RAISE: Requirement-Adaptive Evolutionary Refinement for Training-Free Text-to-Image Alignment
Feb 28, 2026Liyao Jiang, Ruichen Chen, Chao Gao +1T2I Generation Text-Image Alignment
Department of ECE, University of Alberta, Canada · Huawei Technologies, Canada

May 5, 2026J/K move · Enter open · S save

Large Language Models are Universal Reasoners for Visual Generation

Sucheng Ren, Chen Chen, Zhenbang Wang, Liangchen Song+4

1Johns Hopkins University · 2Apple

Diffusion Model Guidance Image Generation Compositional T2I Generation VLM Reasoning

Open paper AskPDF arXiv

Text-to-image generation has advanced rapidly with diffusion models, progressing from CLIP and T5 conditioning to unified systems where a single LLM backbone handles both visual understanding and generation. Despite the architectural unification, these systems frequently fail to faithfully align complex prompts during synthesis, even though they remain highly accurate at verifying whether an image satisfies those same prompts. We formalize this as the understanding-generation gap and propose UniReasoner, a framework that leverages the LLM as a universal reasoner to convert its understanding strength into direct generation guidance. Given a prompt, the LLM first produces a coarse visual draft composed of discrete vision tokens. It then performs a self-critique by evaluating the draft for prompt consistency, producing a grounded textual evaluation that pinpoints what needs to be corrected. Finally, a diffusion model is conditioned jointly on the prompt, the visual draft, and the evaluation, ensuring that generation is guided by explicit corrective signals. Each signal addresses a limitation of the other: the draft provides a concrete, scene-level anchor that reduces under-specification in text-only conditioning, while the evaluation turns verification into grounded, actionable constraints that correct omissions, hallucinations, and relational errors. Experiments show that UniReasoner improves compositional alignment and semantic faithfulness under the same diffusion backbone while maintaining image quality, demonstrating a practical way to exploit LLM reasoning to close the understanding-generation gap.

Abstract

T2I Generation

Explore similar work

Large Language Models are Universal Reasoners for Visual Generation

Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment

RAISE: Requirement-Adaptive Evolutionary Refinement for Training-Free Text-to-Image Alignment