End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced "lily"), a Language-Informed and Lossless Image Encoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 14 points (66.84 → 53.01) against a pixel-only baseline on the GPIC dataset. Our code, encoders, and generative models are available at https://suchiragarwal.com/lilie.
Figures & tables
Figure 1: Generation quality with LILIE. Upper left: Pixel and LILIE supervision at two generator sizes evaluated on GPIC Test-50K with 50 Euler steps. CFG scale is selected using GPIC Validation-50K. All models are trained for one epoch on GPIC Full ( Chandrasegaran et al., 2026 ) . Mosaic: selected 256×256 samples from the 1.1B diffusion model with LILIE supervision. Four samples include prompt excerpts. We include randomly selected samples from GPIC Test-50K in Appendix G .
Figure 2: Two perturbed image patches (additive Gaussian noise) with equal pixel Euclidean distance of 1000 from the original patches. Caption: “Three airplanes fly in the sky against a cloudy backdrop.”
Figure 3: Training and using LILIE. (a) We jointly train LILIE and VLM LoRA parameters on noise-corrupted features with next token prediction. (b) The diffusion model trains on Pixel and LILIE L2 losses, comparing its prediction with the target image in pixel space and through the learned encoder.
Figure 4: LILIE training diagram. We expand on Figure 3 (a) to include TARFlow internals. We provide additional details on affine coupling and transformer internals in Appendix A.1 .
Figure 5
Figure 5: gFD over prompt length. We take three sets of 50K prompts from GPIC Test, each containing only short, medium, or long captions. We then sample from the Pixel, LILIE, and Pixel+LILIE JiT-T2I-g models at their respective optimal CFG scales (Table B.1 ). We report gFD over the GPIC Test set. Our optimal Pixel+LILIE configuration outperforms the Pixel baseline at every prompt length.
Figure 6: Effect of informative captions. gFD on GPIC Test-50K for JiT-T2I-XL with Pixel or Pixel+LILIE loss. “Random” indicates replacing ground-truth captions with random token sequences during LILIE training. We observe that this prevents LILIE from learning a representation that improves generation quality.
Figure 7: Effect of Euler sampling steps. Left: JiT-T2I-g gFD on GPIC Test-50K at 390,600 training steps, with each model’s CFG scale chosen at 50 steps and held constant. Right: three selected qualitative examples with Euler steps increasing across columns and Pixel, LILIE, and Pixel+LILIE down rows. Prompts and initial noise are shared across methods and step counts.
Figure 8: Qualitative samples over LILIE model size and training data. Selected qualitative JiT-T2I-g samples from GPIC use the same prompt and initial noise in each column. The highlighted row is LILIE-S/100M, our main configuration. Model size decreases above it, and training data decreases below it. All generators are trained for 200K steps.
Table 10
Figure 9: LILIE provides a stronger training signal closer to the data distribution. Average standard deviation ratios for LILIE versus pixel values xt , and residuals from clean images x , using the LILIE-S encoder on GPIC Nano (1M train images). xt denotes linear interpolation between pure noise ( t=0 ) and real data ( t=1 ) .
Loss
Supervised
SSL
Language
Training images
Not distilled
GPIC compliant
DINOv2
-
✓
-
142M
✗
✗
SigLIP 2
-
✓
✓
10B
✗
✗
LPIPS (VGG16)
✓
-
-
1.28M
✓
✓
LILIE
-
-
✓
100M
✓
✓
Table 4: Comparison of feature-space losses. “GPIC compliant” indicates compliance with the GPIC evaluation protocol. “SSL” refers to self-supervised learning, and “Language” refers to language-supervised learning. The VGG16 backbone is label-supervised, trained for ImageNet classification. SigLIP 2 is trained with both language-supervised and SSL objectives. Our method is only supervised by language.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Encoder implementation details. (a) The block stack and autoregressive affine map. The tuple γt=(αt,μt) contains the log-scale and shift of patch t . (b) The causal transformer within one block, with bottleneck dimension H and L layers. LN denotes layer normalization. The lower row expands one transformer layer.
Figure B.1: Left: Training loss for LILIE-S over one epoch on GPIC Full. Right: Training loss for JiT-T2I-XL and -g trained with Pixel+LILIE loss ( λ=2 ) over one epoch on GPIC Full.
LILIE- μ
LILIE-T
LILIE-S
architecture
flow backbone
Improved TARFlow ( Zhai et al., 2025 ; Lu et al., 2026b )
blocks K
2
4
4
layers per block L
2
4
4
bottleneck dim H
128
192
384
heads
2
3
6
Appendix
Table B.2: LILIE configurations.
Figure B.2: gFD over CFG on GPIC Validation-50K. We show gFD on the GPIC Validation-50K set against CFG for JiT-T2I-XL and JiT-T2I-g models for Pixel, LILIE, and Pixel+LILIE. Selected CFG scales in Table B.1 are indicated with dashed vertical lines.
JiT-T2I-XL
JiT-T2I-g
architecture
depth
24
16
hidden dim
1024
1536
heads
16
24
bottleneck
128
256
parameters
700M
1.1B
Appendix
Table B.3: Generative model configurations.
Loss
Overall
Single
Two
Counting
Colors
Position
Color attr.
Pixel
33.57 ± 1.38
67.50 ± 4.34
28.28 ± 3.26
14.69 ± 2.42
53.72 ± 4.21
18.00 ± 2.78
19.25 ± 2.79
LILIE
32.49 ± 1.32
62.50 ± 4.00
26.52 ± 3.17
14.06 ± 2.63
56.12 ± 4.03
13.00 ± 2.32
22.75 ± 2.80
Pixel+LILIE
36.41 ± 1.32
77.50 ± 3.57
24.24 ± 3.16
15.62 ± 2.68
56.12 ± 3.89
23.75 ± 3.02
21.25 ± 2.92
Appendix
Table B.4: GenEval scores (%, higher is better) using the original prompts with the leading a photo of prefix removed, which improves scores for all 3 configurations. Errors are ± one standard error over prompts. Bold indicates the best result per column.
Loss
Overall
Global
Entity
Attribute
Relation
Other
Pixel
73.00 ± 0.63
76.60 ± 2.31
82.21 ± 0.56
86.59 ± 0.53
92.30 ± 0.58
70.00 ± 2.91
LILIE
72.86 ± 0.61
76.90 ± 2.29
82.13 ± 0.58
86.25 ± 0.54
91.68 ± 0.60
64.40 ± 3.07
Pixel+LILIE
73.63 ± 0.61
79.33 ± 2.15
82.49 ± 0.57
86.15 ± 0.53
92.50 ± 0.57
68.80 ± 2.89
Appendix
Table B.5: DPG-Bench scores (%, higher is better). Errors are ± one standard error over prompts. Bold indicates the best result per column.
Figure G.1: Selected sampling trajectories over 50 steps. We fix a 50 step sampling budget and show clean image predictions at intermediate Euler steps 0, 5, 10, and 25, followed by the final image at step 50. Each example uses the same prompt and initial noise across methods.
Figure G.2: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Figure G.3: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Figure G.4: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Figure G.5: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Figure G.6: Random samples from the held-out GPIC Test-50K. Each example shows the real image from the GPIC corpus on the left, followed by the three generated samples from JiT-T2I-g models. The first generated sample is from the Pixel baseline, and the following two are from LILIE and Pixel+LILIE.
Figure G.7: A mosaic of lilies and water lilies sampled from JiT-T2I-g trained with Pixel+LILIE.
Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.
In this paper, we present Generative Language-Image Pre-training (GenLIP), a minimalist generative pretraining framework for Vision Transformers (ViTs) designed for multimodal large language models (MLLMs). To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder. This design offers three key advantages: (1) Simplicity: a single transformer jointly models visual and textual tokens; (2) Scalability: it scales effectively with both data and model size; and (3) Performance: it achieves competitive or superior results across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data. After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs.
Yan Fang, Mengcheng Lan, Zilong Huang +7
1Beijing Jiaotong University · 2ByteDance · 3Nanyang Technological University
Since the emergence of Vision Transformer (ViT), it has been widely used in generative language model and generative visual model. Especially in the current state-of-art open source multimodal models, ViT obtained by CLIP or SigLIP method serves as the vision encoder backbone to help them acquire visual understanding capabilities. But this method leads to limitations in visual understanding for details, such as difficulty in recognizing small text or numbers in images. To address these issues, we propose a new model to unify pix token and word token into the generative language model. The new model also features with each pix of image having its own token embedding, color folding, global conditional attention approximation and image unsupervised pretraining. We conducted image unsupervised pretraining experiments using our new model to explore its potential. The experimental results show that it has good performance even in small model and with limited training data. We believe our model also conforms to the scaling law, as long as model parameters and training data increased, its performance will continue to improve.
We study how visual artifacts introduced by diffusion-based inpainting affect language generation in vision-language models. We use a two-stage diagnostic setup in which masked image regions are reconstructed and then provided to captioning models, enabling controlled comparisons between captions generated from original and reconstructed inputs. Across multiple datasets, we analyze the relationship between reconstruction fidelity and downstream caption quality. We observe consistent associations between pixel-level and perceptual reconstruction metrics and both lexical and semantic captioning performance. Additional analysis of intermediate visual representations and attention patterns shows that inpainting artifacts lead to systematic, layer-dependent changes in model behavior. Together, these results provide a practical diagnostic framework for examining how visual reconstruction quality influences language generation in multimodal systems.