A Language-Informed Loss for Visual Generation

Suchir Agarwal; Keshigeyan Chandrasegaran; Ziyu Chen; Kyle Sargent; Juan Carlos Niebles; Li Fei-Fei; Jiajun Wu

cs.CVOct 16, 2026

A Language-Informed Loss for Visual Generation

Authors: Suchir Agarwal, Keshigeyan Chandrasegaran, Ziyu Chen, Kyle Sargent, Juan Carlos Niebles, Li Fei-Fei, Jiajun Wu

Organizations: Stanford University

Abstract

End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced "lily"), a Language-Informed and Lossless Image Encoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 14 points (66.84 $\to$ 53.01) against a pixel-only baseline on the GPIC dataset. Our code, encoders, and generative models are available at https://suchiragarwal.com/lilie.

Figures & tables

Figure 1: Generation quality with LILIE. Upper left: Pixel and LILIE supervision at two generator sizes evaluated on GPIC Test-50K with 50 Euler steps. CFG scale is selected using GPIC Validation-50K. All models are trained for one epoch on GPIC Full ( Chandrasegaran et al., 2026 ) . Mosaic: selected $256\!\times\!256$ samples from the 1.1B diffusion model with LILIE supervision. Four samples include prompt excerpts. We include randomly selected samples from GPIC Test-50K in Appendix G .

Figure 2: Two perturbed image patches (additive Gaussian noise) with equal pixel Euclidean distance of 1000 from the original patches. Caption: “Three airplanes fly in the sky against a cloudy backdrop.”

Figure 3: Training and using LILIE. (a) We jointly train LILIE and VLM LoRA parameters on noise-corrupted features with next token prediction. (b) The diffusion model trains on Pixel and LILIE $L^{2}$ losses, comparing its prediction with the target image in pixel space and through the learned encoder.

Figure 4: LILIE training diagram. We expand on Figure 3 (a) to include TARFlow internals. We provide additional details on affine coupling and transformer internals in Appendix A.1 .

Figure 5

Figure 5: gFD over prompt length. We take three sets of 50K prompts from GPIC Test, each containing only short, medium, or long captions. We then sample from the Pixel, LILIE, and Pixel+LILIE JiT-T2I-g models at their respective optimal CFG scales (Table B.1 ). We report gFD over the GPIC Test set. Our optimal Pixel+LILIE configuration outperforms the Pixel baseline at every prompt length.

Figure 6: Effect of informative captions. gFD on GPIC Test-50K for JiT-T2I-XL with Pixel or Pixel+LILIE loss. “Random” indicates replacing ground-truth captions with random token sequences during LILIE training. We observe that this prevents LILIE from learning a representation that improves generation quality.

Figure 7: Effect of Euler sampling steps. Left: JiT-T2I-g gFD on GPIC Test-50K at 390,600 training steps, with each model’s CFG scale chosen at 50 steps and held constant. Right: three selected qualitative examples with Euler steps increasing across columns and Pixel, LILIE, and Pixel+LILIE down rows. Prompts and initial noise are shared across methods and step counts.

Figure 8: Qualitative samples over LILIE model size and training data. Selected qualitative JiT-T2I-g samples from GPIC use the same prompt and initial noise in each column. The highlighted row is LILIE-S/100M, our main configuration. Model size decreases above it, and training data decreases below it. All generators are trained for 200K steps.

Table 10

Figure 9: LILIE provides a stronger training signal closer to the data distribution. Average standard deviation ratios for LILIE versus pixel values

x_{t}

, and residuals from clean images

x

, using the LILIE-S encoder on GPIC Nano (1M train images).

x_{t}

denotes linear interpolation between pure noise (

t=0

) and real data (

t=1

) .

Loss	Supervised	SSL	Language	Training images	Not distilled	GPIC compliant
DINOv2	-	✓	-	142M	✗	✗
SigLIP 2	-	✓	✓	10B	✗	✗
LPIPS (VGG16)	✓	-	-	1.28M	✓	✓
LILIE	-	-	✓	100M	✓	✓

Table 4: Comparison of feature-space losses. “GPIC compliant” indicates compliance with the GPIC evaluation protocol. “SSL” refers to self-supervised learning, and “Language” refers to language-supervised learning. The VGG16 backbone is label-supervised, trained for ImageNet classification. SigLIP 2 is trained with both language-supervised and SSL objectives. Our method is only supervised by language.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: Encoder implementation details. (a) The block stack and autoregressive affine map. The tuple $\gamma_{t}=(\alpha_{t},\mu_{t})$ contains the log-scale and shift of patch $t$ . (b) The causal transformer within one block, with bottleneck dimension $H$ and $L$ layers. LN denotes layer normalization. The lower row expands one transformer layer.

Figure B.1: Left: Training loss for LILIE-S over one epoch on GPIC Full. Right: Training loss for JiT-T2I-XL and -g trained with Pixel+LILIE loss ( $\lambda=2$ ) over one epoch on GPIC Full.

architecture
	LILIE- $\mu$	LILIE-T	LILIE-S
flow backbone	Improved TARFlow ( Zhai et al., 2025 ; Lu et al., 2026b )
blocks $K$	2	4	4
layers per block $L$	2	4	4
bottleneck dim $H$	128	192	384
heads	2	3	6

Appendix

Table B.2: LILIE configurations.

Figure B.2: gFD over CFG on GPIC Validation-50K. We show gFD on the GPIC Validation-50K set against CFG for JiT-T2I-XL and JiT-T2I-g models for Pixel, LILIE, and Pixel+LILIE. Selected CFG scales in Table B.1 are indicated with dashed vertical lines.

architecture
	JiT-T2I-XL	JiT-T2I-g
depth	24	16
hidden dim	1024	1536
heads	16	24
bottleneck	128	256
parameters	700M	1.1B

Appendix

Table B.3: Generative model configurations.

Loss	Overall	Single	Two	Counting	Colors	Position	Color attr.
Pixel	33.57 $\,\pm\,$ 1.38	67.50 $\,\pm\,$ 4.34	28.28 $\,\pm\,$ 3.26	14.69 $\,\pm\,$ 2.42	53.72 $\,\pm\,$ 4.21	18.00 $\,\pm\,$ 2.78	19.25 $\,\pm\,$ 2.79
LILIE	32.49 $\,\pm\,$ 1.32	62.50 $\,\pm\,$ 4.00	26.52 $\,\pm\,$ 3.17	14.06 $\,\pm\,$ 2.63	56.12 $\,\pm\,$ 4.03	13.00 $\,\pm\,$ 2.32	22.75 $\,\pm\,$ 2.80
Pixel+LILIE	36.41 $\,\pm\,$ 1.32	77.50 $\,\pm\,$ 3.57	24.24 $\,\pm\,$ 3.16	15.62 $\,\pm\,$ 2.68	56.12 $\,\pm\,$ 3.89	23.75 $\,\pm\,$ 3.02	21.25 $\,\pm\,$ 2.92

Appendix

Table B.4: GenEval scores (%, higher is better) using the original prompts with the leading a photo of prefix removed, which improves scores for all 3 configurations. Errors are

\pm

one standard error over prompts. Bold indicates the best result per column.

Loss	Overall	Global	Entity	Attribute	Relation	Other
Pixel	73.00 $\,\pm\,$ 0.63	76.60 $\,\pm\,$ 2.31	82.21 $\,\pm\,$ 0.56	86.59 $\,\pm\,$ 0.53	92.30 $\,\pm\,$ 0.58	70.00 $\,\pm\,$ 2.91
LILIE	72.86 $\,\pm\,$ 0.61	76.90 $\,\pm\,$ 2.29	82.13 $\,\pm\,$ 0.58	86.25 $\,\pm\,$ 0.54	91.68 $\,\pm\,$ 0.60	64.40 $\,\pm\,$ 3.07
Pixel+LILIE	73.63 $\,\pm\,$ 0.61	79.33 $\,\pm\,$ 2.15	82.49 $\,\pm\,$ 0.57	86.15 $\,\pm\,$ 0.53	92.50 $\,\pm\,$ 0.57	68.80 $\,\pm\,$ 2.89

Appendix

Table B.5: DPG-Bench scores (%, higher is better). Errors are

\pm

one standard error over prompts. Bold indicates the best result per column.

Figure G.1: Selected sampling trajectories over 50 steps. We fix a 50 step sampling budget and show clean image predictions at intermediate Euler steps 0, 5, 10, and 25, followed by the final image at step 50. Each example uses the same prompt and initial noise across methods.

Figure G.2: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.

Figure G.3: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.

Figure G.4: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.

Figure G.5: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.

Figure G.6: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.

Figure G.7: A mosaic of lilies and water lilies sampled from JiT-T2I-g trained with Pixel+LILIE.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 299 papers

Let ViT Speak: Generative Language-Image Pre-training
May 1, 2026Yan Fang, Mengcheng Lan, Zilong Huang +7Cross-Modal Learning Vision Transformer
1Beijing Jiaotong University · 2ByteDance · 3Nanyang Technological University
Unified Pix Token And Word Token Generative Language Model
May 13, 2026Haun Leung, ZiNan WangImage Tokenization Self-Supervised Pre-Training
How Do Inpainting Artifacts Propagate to Language?
Feb 24, 2026Pratham Yashwante, Davit Abrahamyan, Shresth Grover +1VLM Robustness VLM Evaluation
UC San Diego

May 1, 2026J/K move · Enter open · S save

Let ViT Speak: Generative Language-Image Pre-training

Yan Fang, Mengcheng Lan, Zilong Huang, Weixian Lei+6

1Beijing Jiaotong University · 2ByteDance · 3Nanyang Technological University

Cross-Modal Learning Vision Transformer Multimodal Large Language Models Multimodal Foundation Models Vision-Language Pretraining

Open paper AskPDF arXiv

In this paper, we present Generative Language-Image Pre-training (GenLIP), a minimalist generative pretraining framework for Vision Transformers (ViTs) designed for multimodal large language models (MLLMs). To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder. This design offers three key advantages: (1) Simplicity: a single transformer jointly models visual and textual tokens; (2) Scalability: it scales effectively with both data and model size; and (3) Performance: it achieves competitive or superior results across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data. After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs.

Abstract

T2I Generation

Explore similar work

Let ViT Speak: Generative Language-Image Pre-training

Unified Pix Token And Word Token Generative Language Model

How Do Inpainting Artifacts Propagate to Language?