Text-to-Image Generative Modeling Without Language Pretraining

Keshigeyan Chandrasegaran; Yue Zhao; Armin W. Thomas; Ehsan Adeli; Jiajun Wu; Juan Carlos Niebles; Li Fei-Fei; Michael Poli

cs.LGOct 19, 2026

Text-to-Image Generative Modeling Without Language Pretraining

Authors: Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas, Ehsan Adeli, Jiajun Wu, Juan Carlos Niebles, Li Fei-Fei, Michael Poli

Organizations: Stanford University · Radical Numerics

Abstract

Modern text-to-image (T2I) systems use internet-scale language pretraining for text-conditioned image generation. We ask whether language pretraining is necessary for strong T2I generation. We introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs. Halcyon-0.52B outperforms LlamaGen-XL/8 (48.0 vs. 63.3 gFD) with 18.2 $\times$ less pretraining compute. Further, Halcyon-1.6B outperforms JiT-T2I-XXL/8++ (35.3 vs. 41.0 gFD) with 275 $\times$ less pretraining compute. Controlled experiments show that text-only pretraining does not improve T2I generation. Halcyon also learns image captioning by replacing caption dropout with I2T supervision, maintaining competitive T2I generation quality under the same pretraining budget. We show that strong T2I systems can be trained from scratch without internet-scale language pretraining. Code and models are available at https://cs.stanford.edu/~keshik/halcyon

Figures & tables

Figure 1: Text-to-image generation without internet-scale language pretraining. Left: Generation quality on GPIC ( Chandrasegaran et al., 2026 ) (gFD, log scale) vs. total pretraining FLOPs required to build each T2I system, including language model, image tokenizer, and image generator pretraining. Each $\star$ marks total pretraining FLOPs and gFD, while the donut breaks down language and vision compute. Halcyon (Ours) , trained entirely from scratch, achieves lower gFD with substantially lower total pretraining compute compared to: (i) LlamaGen ( Sun et al., 2024 ) , late-fusion autoregressive models with 0.78–0.90B image generators and pretrained FLAN-T5 encoders (0.11–1.22B); (ii) JiT-T2I ( Li and He, 2026 ; Ma et al., 2026 ) , late-fusion diffusion models with a 1.10B image generator and pretrained Qwen3-1.7B; and (iii) Qwen3-0.6B-CPT , an autoregressive model continually pretrained on image–text pairs, initialized from a pretrained Qwen3-0.6B language model ( Yang et al., 2025a ) . Right: $256\times 256$ samples generated by Halcyon-4B with prompts shown for two samples. Best viewed in color.

Panel of Figure 2 — Figure 2: Language vs. vision pretraining tokens. Language tokens count text-only pretraining. Emu3 and Tuna-2 values are estimated.

Panel of Figure 3 — Figure 3: Sequence formats. Text-only pretraining tokens , caption tokens , and image tokens occupy one joint vocabulary alongside special tokens. The top two formats are unimodal, while the bottom two represent paired image-caption examples. Rows $\mathrm{I}$ and $\mathrm{I2T}$ share the same [SOV] $\mathbf{x}$ [EOV] prefix, which is also obtained when dropping the caption from a T2I sequence used for classifier-free guidance. Image-caption sequences are padded to a fixed length, with [PAD] positions excluded from the training loss. Box widths are schematic and not proportional to token counts.

Panel of Figure 4 — Figure 4: Pretraining data composition and scheduling described in Section 3.3 . The four token types are: image tokens conditioned on text ( $\mathcal{I}\mid\mathcal{T}$ ); image tokens without text conditioning ( $\mathcal{I}\mid\varnothing$ ); text tokens conditioned on images ( $\mathcal{T}\mid\mathcal{I}$ ); and text-only tokens ( $\mathcal{T}\mid\varnothing$ ). Vertical boundaries denote training phases, while horizontal bands show the token composition within each phase. Bands represent interleaved examples. Band height and width are schematic.

Tokenizer	rFD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
LlamaGen-VQ-16x16	129.87	20.18	0.560	0.146
Cosmos-DI-8x8	37.93	25.38	0.776	0.153
Halcyon	18.86	25.61	0.792	0.062
Real-vs-Real	7.59	–	–	–

Table 1: Reconstruction results on GPIC Test-50K (

256\times 256

). Our tokenizer, trained from scratch on GPIC, outperforms Cosmos-DI-8x8 on all metrics.

Tokenizer	Vocab.	Bytes/token $\uparrow$	Tok./caption $\downarrow$
FLAN-T5	32,100	4.152	51.1
Qwen3	151,669	4.763	44.6
Halcyon	16,384	4.774	44.5

Table 2: Text tokenizer comparison on GPIC test captions. Higher bytes/token and lower tokens/caption indicate better compression.

Figure 5: Comparison of generated images. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon models from 0.1B to 4B parameters. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K. Additional samples are provided in Appendix D .

Configuration	$\alpha$	$\mathcal{T}\mid\varnothing$	$\mathcal{T}\mid\mathcal{I}$	$\mathcal{I}\mid\varnothing$	$\mathcal{I}\mid\mathcal{T}$	$\mathcal{T}\mid\varnothing$	$\mathcal{T}\mid\mathcal{I}$	$\mathcal{I}\mid\varnothing$	$\mathcal{I}\mid\mathcal{T}$	gFD $\downarrow$
		Text tokens (B)		Image tokens (B)		NLL $\downarrow$
Halcyon -0.52B	–	4.05	0	10.24	92.16	1.433	18.414	8.428	8.366	48.02
Staged: Language $\rightarrow$ T2I	0.10	16.44	0	9.22	82.94	1.447	19.098	8.442	8.382	51.56
	0.20	28.83	0	8.19	73.73	1.451	17.197	8.445	8.388	52.76
	0.50	65.80	0	5.12	46.08	1.454	22.850	8.505	8.451	65.13
Staged: Language $\rightarrow$ T2I †	0.33	68.03	0	10.24	92.16	1.445	18.809	8.428	8.371	49.22

Table 6: Effect of language pretraining on Halcyon -0.52B. Under a fixed pretraining budget, we measure the impact of allocating compute to external text-only pretraining on T2I generation. Each language-pretraining stage exceeds the Chinchilla reference of 20 text tokens per parameter. The † configuration preserves a full T2I epoch after additional language pretraining (

1.5\times

total pretraining compute), but does not improve gFD. Token counts exclude padding.

Model	LP	$\mathcal{T}\mid\varnothing$	$\mathcal{T}\mid\mathcal{I}$	$\mathcal{I}\mid\varnothing$	$\mathcal{I}\mid\mathcal{T}$	$\mathcal{T}\mid\varnothing$	$\mathcal{T}\mid\mathcal{I}$	$\mathcal{I}\mid\varnothing$	$\mathcal{I}\mid\mathcal{T}$	gFD $\downarrow$
		Text tokens (B)		Image tokens (B)		Text bpb $\downarrow$		Image bpb ( $\times 10^{-2}$ ) $\downarrow$
Halcyon -0.52B	0	4.05	0	10.24	92.16	0.433	5.565	6.333	6.286	48.02
Qwen3-0.6B	0	4.05	0	10.24	92.16	0.434	5.436	6.324	6.278	46.61
Qwen3-0.6B-CPT	36T	4.05	0	10.24	92.16	0.430	4.974	6.326	6.283	47.81
Qwen3-0.6B-CPT ∗	36T	4.05	0	10.24	92.16	0.424	6.743	6.409	6.367	63.62

Table 7: Continual pretraining from a language model. Qwen3-0.6B-CPT is initialized from Qwen3-0.6B-Base (pretrained on 36T text tokens), while Qwen3-0.6B is randomly initialized. Both share the same Transformer architecture as Halcyon -0.52B but use a larger vocabulary (216k vs. 80,640 for Halcyon ). We additionally evaluate continual pretraining with a lower peak learning rate ( ∗ :

1\times 10^{-4}

vs.

7.5\times 10^{-4}

) to test whether aggressive weight updates diminish the benefits of pretrained language weights. Since vocabularies differ, we report bits per byte (bpb) for both text and images. Neither configuration improves T2I generation over random initialization. All models are trained for one epoch on GPIC and evaluated with CFG 6.0. LP denotes language pretraining tokens. Token counts exclude padding.

		Text tokens (B)		Image tokens (B)		NLL $\downarrow$
Configuration	$\beta$ (or $\eta$ )	$\mathcal{T}\mid\varnothing$	$\mathcal{T}\mid\mathcal{I}$	$\mathcal{I}\mid\varnothing$	$\mathcal{I}\mid\mathcal{T}$	$\mathcal{T}\mid\varnothing$	$\mathcal{T}\mid\mathcal{I}$	$\mathcal{I}\mid\varnothing$	$\mathcal{I}\mid\mathcal{T}$	gFD $\downarrow$
Baseline	–	4.05	0	10.24	92.16	1.433	18.414	8.428	8.366	48.02
Staged: I2T $\rightarrow$ T2I	$\beta=0.1$	3.65	0.45	19.46	82.94	1.440	18.415	8.426	8.366	48.35
	$\beta=0.5$	2.03	2.25	56.32	46.08	1.435	20.437	8.421	8.370	54.22
Mixing: T2I + I2T	$\eta=0.1$	3.65	0.45	19.46	82.94	1.443	1.125	8.414	8.362	48.18
	$\eta=0.5$	2.03	2.25	56.32	46.08	1.475	1.009	8.453	8.417	75.97

Table 8: Effect of I2T supervision and scheduling on T2I generation for Halcyon -0.52B. Under a fixed pretraining budget, staged and mixed training use identical token counts at each I2T allocation, differing only in scheduling. At 10%, both schedules preserve comparable T2I generation, while at 50%, staged training achieves substantially lower gFD. However, only mixing substantially reduces captioning NLL, namely

\mathrm{NLL}(\mathcal{T}\mid\mathcal{I})

. NLL and gFD are evaluated on GPIC Test-50K; token counts exclude padding.

			NLL $\downarrow$				T2I generation			Captioning
Size	$\eta$	$p_{\mathrm{drop}}$	$\mathcal{T}\mid\varnothing$	$\mathcal{T}\mid\mathcal{I}$	$\mathcal{I}\mid\varnothing$	$\mathcal{I}\mid\mathcal{T}$	gFD $\downarrow$	GenEval $\uparrow$	DPG-Bench $\uparrow$	Alignment $\uparrow$
0.10B	–	0.1	1.601	19.066	8.774	8.727	144.16	27.31	66.36	–
0.10B	0.1	0	1.598	1.410	8.771	8.724	144.14	27.66	68.89	0.090
0.25B	–	0.1	1.497	21.472	8.566	8.509	72.55	35.72	72.66	–
0.25B	0.1	0	1.491	1.218	8.559	8.503	70.44	36.54	73.07	0.207
0.52B	–	0.1	1.433	18.414	8.428	8.366	48.02	44.91	76.84	–

Table 9: Replacing caption dropout with I2T supervision across model sizes. Replacing caption-dropped T2I examples with I2T preserves the pretraining budget and image-token counts, enabling image captioning while maintaining comparable T2I generation. We report mean Qwen3-VL-Reranker-8B alignment scores over 50K generated image–caption pairs from GPIC Test-50K, using the image as the query and caption as the document. GPIC Test-50k image-caption pairs obtain a mean alignment score of 0.891.

Figure 6: Image captioning examples from Halcyon -4B. For each example, the ground-truth caption is shown above the input image, and the generated caption is shown below. The model learns image captioning by replacing caption-dropped T2I examples with I2T supervision, using only 10% of the pretraining budget for captioning.

Figure 7: Scaling behavior of Halcyon across model sizes. Validation negative log-likelihood (NLL) for image tokens (left) and text tokens (right) throughout pretraining on GPIC. As model size increases from 0.1B to 4B, both image and text NLL consistently decrease, with larger models achieving lower NLL throughout training. All models are pretrained from scratch under the same training schedule.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

architecture
	0.1B	0.25B	0.52B	1.6B	4B
backbone	Qwen3 ( Yang et al., 2025a ) , decoder-only
layers	14	21	28	28	36
hidden dim	512	768	1024	2048	2560
FFN dim	1536	2304	3072	6144	9728
query / KV heads	8 / 4	12 / 6	16 / 8	16 / 8	32 / 8

Appendix

Table 11: Halcyon configurations.

language model (frozen)
	JiT-T2I-XXL/16	JiT-T2I-XXL/16++	JiT-T2I-XXL/8++
text encoder	Qwen3-1.7B
feature dim	2048
max caption tokens	300
image generator
backbone	JiT-T2I (pixel space, no image tokenizer)

Appendix

Table 12: JiT-T2I configurations.

language model (frozen)
	LlamaGen-XL	LlamaGen-XL/8	LlamaGen-XL-T5-Base/8
text encoder	FLAN-T5-XL	FLAN-T5-XL	FLAN-T5-Base
encoder params	1.22B	1.22B	0.11B
feature dim	2048	2048	768
max caption tokens	300
image tokenizer

Appendix

Table 13: LlamaGen configurations.

Image tokenizer
architecture	Cosmos ( Agarwal et al., 2025 ) encoder–decoder, FSQ ( Mentzer et al., 2024 )
decoder width	192 (128 in Cosmos)
spatial downsampling	$8\times$
codebook size	64,000 (FSQ levels $[8,8,8,5,5,5]$ )
Params	136M
tokens per image	1,024 ( $32\times 32$ , raster order)

Appendix

Table 14: Tokenizer training configurations.

Figure 9: Classifier-free guidance (CFG) sweeps for late-fusion T2I baselines. We evaluate LlamaGen variants (left) and JiT-T2I variants (right) across CFG scales on GPIC Validation-50K. Stars indicate the scale achieving the lowest gFD for each configuration, with the corresponding gFD and CFG scale reported in the figure. The selected CFG scales are used for all subsequent evaluations on GPIC Test-50K, GenEval, and DPG-Bench.

Params	Wall time (s) $\downarrow$	Throughput (img/s) $\uparrow$
0.1B	395.3	126.5
0.25B	507.4	98.5
0.52B	662.8	75.4
1.6B	712.2	70.2
4B	935.5	53.4

Appendix

Table 17: Inference cost of Halcyon. Wall-clock time and throughput for generating 50K images (GPIC Test-50K set) using 8xH100 GPUs with vLLM, averaged over three runs. We use temperature 1.0 without top-

k

/top-

p

filtering and each model’s CFG scale (two forward passes per image token). Image decoding takes 198.0

\pm

5.7 seconds (averaged over 3 runs) for all model sizes and is excluded from the reported results.

Figure 10: Effect of language pretraining on validation NLL. Validation NLL for image tokens conditioned on text (

\mathcal{I}\mid\mathcal{T}

, left) and text tokens without conditioning (

\mathcal{T}\mid\varnothing

, right) during training of Qwen3-0.6B with random vs. language-pretrained initialization. Random initialization consistently achieves lower image NLL, while language-pretrained initialization achieves lower text NLL.

Figure 11: Comparison of generated images with “tag” captions. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon from 0.1B to 4B. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K.

Figure 12: Comparison of generated images with “short” captions. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon from 0.1B to 4B. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K.

Figure 13: Comparison of generated images with “medium” captions. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon from 0.1B to 4B. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K.

Figure 14: Comparison of generated images with “long” captions. From left to right: JiT-T2I-XXL/8++, LlamaGen-XL/8, and Halcyon from 0.1B to 4B. For the baselines, we report the sizes of the language model and image generator separately. Image quality improves as Halcyon scales up. All prompts are from GPIC Test-50K.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 290 papers

Text-to-Image Synthesis: A Decade Survey
Nov 25, 2024Nonghai Zhang, Hao TangT2I Generation T2I Diffusion Models
School of Software & Microelectronics, Peking University, Beijing 102600, China. Beijing 100871, China. · School of Computer Science, Peking University, Beijing 100871, China.
Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models
May 20, 2026Dong Chen, Fangyun Wei, Ziyu Wan +18Multimodal Foundation Models T2I Diffusion Models
Microsoft Lens Team
Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards
Oct 2, 2026Yuanhao Ban, I-Hung Hsu, Anastasios Angelopoulos +3T2I Generation Reward Design for RL
Arena Intelligence Inc · UCLA

Nov 25, 2024J/K move · Enter open · S save

Text-to-Image Synthesis: A Decade Survey

Nonghai Zhang, Hao Tang

School of Software & Microelectronics, Peking University, Beijing 102600, China. Beijing 100871, China. · School of Computer Science, Peking University, Beijing 100871, China.

T2I Generation T2I Diffusion Models Controllable Image Generation Identity-Preserving Image Generation

Open paper AskPDF arXiv

Text-to-image synthesis (T2I) focuses on generating high-quality images from textual descriptions In this survey, we review 490 papers on T2I spanning the past decade, with representative advances covered through 2026. We start by briefly introducing how GANs, autoregressive models, and diffusion models have been used for image generation, and we additionally examine the shared components whose evolution has driven recent progress. Building on this foundation, we discuss the development of these models for T2I and organize cutting-edge research by problem rather than by chronology alone, covering controllability, personalized and identity-preserving generation, instruction-based image editing, preference alignment and post-training with human and AI feedback, efficiency, safety, and consistency in content and spatial relationships. We further review unified multimodal understanding-and-generation models, summarize the datasets and evaluation metrics commonly used in T2I research, and provide a comparative discussion of the three paradigms together with their respective trade-offs. Finally, we discuss the potential applications of T2I within AIGC, along with the challenges and forward-looking research opportunities in this field.

Abstract

T2I Generation

Explore similar work

Text-to-Image Synthesis: A Decade Survey

Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models

Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards