High-Fidelity Text-to-Image Generation from Pre-Trained Vision-Language Models via Distribution-Conditioned Diffusion Decoding

Ji Woo Hong; Hee Suk Yoon; Gwanhyeong Koo; Eunseop Yoon; SooHwan Eom; Qi Dai; Chong Luo; Chang D. Yoo

cs.CVMar 11, 2026

High-Fidelity Text-to-Image Generation from Pre-Trained Vision-Language Models via Distribution-Conditioned Diffusion Decoding

Authors: Ji Woo Hong, Hee Suk Yoon, Gwanhyeong Koo, Eunseop Yoon, SooHwan Eom, Qi Dai, Chong Luo, Chang D. Yoo

Organizations: Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea · Microsoft Research Asia (MSRA), Beijing, China

Abstract

Recent large-scale vision-language models (VLMs) have shown remarkable text-to-image generation capabilities, yet their visual fidelity remains constrained by the discrete image tokenization, which poses a major challenge. Although several studies have explored continuous representation modeling to enhance visual quality, adapting pre-trained VLM models to such representations requires large-scale data and training costs comparable to the original pre-training. To circumvent this limitation, we propose a diffusion-based decoding framework that enhances image fidelity by training only a diffusion decoder on the output image-token logits of pre-trained VLMs, thereby preserving the original model intact. At its core, Logit-to-Code Distributional Mapping converts the VLM's image-token logits into continuous, distribution-weighted code vectors with uncertainty features, providing an effective conditioning signal for diffusion decoding. A lightweight Logit Calibration aligns training-time proxy logits from the VQ-VAE encoder with VLM-generated logits, mitigating the train-inference gap. Conditioned on these representations, the Distribution-Conditioned Diffusion Decoder generates high-fidelity images. Achieved solely through short training on ImageNet-1K, our method consistently improves visual fidelity for both VQ-VAE reconstructions and text-to-image generations from VLM-predicted tokens.

Figures & tables

Figure 1: Comparison of visual reconstruction quality from VQ-VAE–encoded representations. While the VQ-VAE decoded output exhibits shape distortion and loss of detail, our diffusion-based method consistently improves image fidelity, preserving structural accuracy and fine-grained visual sharpness.

Figure 2: Conceptual comparison of different strategies for improving visual fidelity in VLM image generation. (a) Tokenizer replacement retrains the entire VLM with a new continuous or hybrid tokenizer. (b) Diffusion-assisted decoding jointly optimizes the VLM and diffusion model, altering the original VLM. (c) Ours train only the diffusion-based decoder, preserving VLM’s original capabilities.

Figure 3: Overview of our proposed framework. During inference, the pre-trained VLM produces Image-Token Logits, which are transformed in the Logit-to-Code Distributional Mapping stage into continuous Distribution-Weighted Code Vectors and complementary Uncertainty Features capturing the reliability and ambiguity of each token prediction. These continuous representations serve as conditioning inputs to the Distribution-Conditioned Diffusion Decoder, which refines localized visual structures to generate high-quality images. During training, instead of forwarding the VLM itself, the logits are approximated using the VQ-VAE Encoder from the VLM’s pre-training pipeline, followed by lightweight Logit Calibration to match the VLM’s logit distribution.

Statistic	VLM	VQ-VAE	VQ-VAE (Cal.)
Probability Distribution Statistics
Top-1 Probability	0.221627	0.000151	0.222283
Top-2 Probability	0.100532	0.000145	0.064229
Normalized Entropy	0.415303	0.993630	0.424155
Tail Entropy	0.560794	0.993661	0.520536
Otsu Thresholding Statistics

Table 1: Token-probability statistics and Otsu thresholding results for the VLM’s image-token logits, the raw VQ-VAE pseudo-logits, and our calibrated pseudo-logits. The probability statistics (top rows) summarize the sharpness and entropy characteristics of each distribution, while the Otsu-derived quantities (bottom rows) identify the number of statistically significant codebook entries (threshold rank) and the corresponding probability mass. Details on the computation and interpretation are provided in Appendix A .

ImageNet-1K Val.	rFID↓	SSIM↑	LPIPS↓
VQ-VAE Decoder	2.8511	0.5786	0.1541
LCDM + LC + DCDD (ours)	0.7535	0.5957	0.1374

Table 2: Quantitative Results on Image Reconstruction. We evaluate image reconstruction on ImageNet-1K and MJHQ-30K, reporting metrics averaged across all classes and categories. It compares the baseline VLM’s native VQ-VAE decoder and our diffusion-based decoder, evaluated with the recent publicly available Janus-Pro 7B (

384\times 384

) backbone. Results for additional VLM backbones, along with ablation studies analyzing each of our component’s effect on performance, are provided in the Supplementary . Superior results are highlighted .

MJHQ-30K	Overall	Animals	Art	Fashion	Food	Indoor	Landscape	Logo	People	Plants	Vehicles
VQ-VAE Decoder	14.2888	31.6866	35.6408	32.1410	33.1787	32.4016	30.5020	42.3565	37.6049	33.1228	27.4424
LCDM + LC + DCDD (ours)	9.4160	25.1060	28.9806	26.0182	28.1580	28.5172	26.7104	38.2032	30.2217	33.6617	25.0277

Table 3: Quantitative results on text-to-image generation. We evaluate MJHQ-30K and GenEval metrics, respectively: MJHQ measures generation FID (lower is better), while GenEval assesses semantic correctness across object existence, attributes, counting, and spatial relations (higher is better). Results of the Janus-Pro 7B backbone are reported, and additional VLM backbones are provided in the Supplementary . Superior results are highlighted .

ImageNet-1K Val.	rFID↓	SSIM↑	LPIPS↓
Diff. Dec. (baseline)	33.5081	0.4244	0.4132
DCDD + LCDM	31.2049	0.4275	0.4009
DCDD + LCDM + LC	30.5760	0.4360	0.3875

Table 4: Quantitative ablation results on ImageNet-1K image reconstruction. Each row corresponds to a diffusion model trained with a different conditioning representation. The best performance for each metric is highlighted .

Janus-Pro 7B
MJHQ-30K Benchmark	gFID↓	Latency (s)↓
Baseline (VLM + VQ-VAE)	14.28	11.86
Ours (VLM + Diffusion)	9.41	15.56
Trade-off	-34.12%	+31.19%

Table 5: End-to-end text-to-image comparison between our method and the baseline on the MJHQ-30K benchmark with Janus-Pro 7B. Using 30 denoising diffusion steps, our diffusion-based decoder achieves 34% lower FID while increasing end-to-end latency by only 31%.

Figure 4: Ablation results comparing the effect of discrete conditioning ( Diff. Dec. (baseline) ), continuous distribution-weighted conditioning ( DCDD + LCDM ), and calibrated continuous conditioning ( DCDD + LCDM + LC ) on ImageNet-1K image reconstruction.

Figure 5: Qualitative results on text-to-image generation. Comparison between images decoded from the VLM’s predicted tokens using its native VQ-VAE decoder and our diffusion-based decoder on prompts from the MJHQ-30K benchmark. Additional samples with corresponding prompts are provided in the Appendix.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Average top-30 token probability distributions for Janus-Pro 7B (VLM), the VQ-VAE encoder, and the calibrated VQ-VAE encoder. Each curve visualizes per-token probabilities sorted by rank and then averaged across 100 images. The dashed line marks the Otsu-derived threshold rank.

ImageNet-1K Val.	rFID↓	SSIM↑	LPIPS↓
VQ-VAE Decoder	23.1038	0.5756	0.1551
Ours (5 steps)	35.0169	0.5855	0.1918
Ours (10 steps)	20.8859	0.5995	0.1478
Ours (15 steps)	18.3279	0.6012	0.1417
Ours (20 steps)	17.6918	0.5993	0.1405
Ours (25 steps)	17.5659	0.5975	0.1404

Appendix

Table 6: Reconstruction performance of our diffusion-based decoder across different numbers of denoising steps, compared against the Janus-Pro 7B VQ-VAE baseline. The best results are highlighted .

ImageNet-1K Val.	rFID↓	SSIM↑	LPIPS↓
VQ-VAE Decoder	4.1919	0.5566	0.1388
LCDM + LC + DCDD (ours)	1.4087	0.5600	0.1404

Appendix

Table 7: Quantitative image reconstruction results (256

\times

256 resolution) using LlamaGen-XL. We evaluate image reconstruction on ImageNet-1K and MJHQ-30K, reporting metrics averaged across all classes and categories. It compares the LlamaGen-XL baseline’s native VQ-VAE decoder and our diffusion-based decoder. Superior results are highlighted .

MJHQ-30K	Overall	Animals	Art	Fashion	Food	Indoor	Landscape	Logo	People	Plants	Vehicles
VQ-VAE Decoder	59.0292	74.1959	82.9698	92.8457	86.4562	89.2189	82.0326	68.5263	105.9981	87.2854	82.0326
LCDM + LC + DCDD (ours)	61.0925	74.6621	87.1321	104.2776	93.6168	96.6309	90.0496	66.9729	114.4049	93.7683	90.0496

Appendix

Table 8: Quantitative text-to-image generation results (256

\times

256 resolution) using LlamaGen-XL. We evaluate MJHQ-30K and GenEval metrics, respectively: MJHQ measures generation FID (lower is better), while GenEval assesses semantic correctness across object existence, attributes, counting, and spatial relations (higher is better). Superior results are highlighted .

Figure 7: Illustrations of the limitation of our method in the Text-to-Image generation. While our diffusion-based decoder enhances visual fidelity compared to the VQ-VAE decoder of Janus-Pro 7B, it cannot correct semantic errors generated by the VLM.

Figure 8: Additional qualitative image reconstruction results (384 $\times$ 384 resolution) using Janus-Pro-7B. We compare the baseline VQ-VAE decoder with our diffusion-based decoder. The top two rows show samples from ImageNet-1K, and the bottom two rows show samples from MJHQ-30K.

Figure 9: Additional qualitative text-to-image samples (384 $\times$ 384 resolution) generated using Janus-Pro-7B. We compare the baseline VQ-VAE decoder with our diffusion-based decoder. Prompts in the upper two rows are drawn from MJHQ-30K, and those in the lower two rows from GenEval.

Figure 10: Qualitative image reconstruction results (256 $\times$ 256 resolution) using LlamaGen-XL. We compare the baseline VQ-VAE decoder with our diffusion-based decoder. The top two rows show samples from ImageNet-1K, and the bottom two rows show samples from MJHQ-30K.

Figure 11: Qualitative text-to-image results (256 $\times$ 256 resolution) using LlamaGen-XL. We compare the baseline VQ-VAE decoder with our diffusion-based decoder. Prompts in the upper two rows are drawn from MJHQ-30K, and those in the lower two rows from GenEval.

T2I Generation

Text-to-image generation creates images from natural-language prompts using generative models, with research focused on fidelity, control, efficiency, and safety.

See all 299 papers

TextLDM: Language Modeling with Continuous Latent Diffusion
May 8, 2026Jiaxiu Jiang, Jingjing Ren, Wenbo Li +10Diffusion Transformer Latent Diffusion Models
Joy Future Academy · HIT · HKUST(GZ)
Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
May 24, 2026Agata Żywot, Iason Skylitsis, Thijmen Nijdam +4Diffusion Model Guidance Test-Time Optimization
University of Amsterdam, Netherlands
Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM
Apr 8, 2026Chengyue Wu, Shiyi Lan, Yonggan Fu +9Vision-Language Models Diffusion VLMs
The University of Hong Kong · NVIDIA · MBZUAI +1

May 8, 2026J/K move · Enter open · S save

TextLDM: Language Modeling with Continuous Latent Diffusion

Jiaxiu Jiang, Jingjing Ren, Wenbo Li, Bo Wang+9

Joy Future Academy · HIT · HKUST(GZ)

Diffusion Transformer Latent Diffusion Models Diffusion Language Models Continuous Diffusion Language Models Diffusion Language Model Training

Open paper AskPDF arXiv

Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos. A natural next step toward a single architecture for both generation (visual synthesis) and understanding (text generation) is to apply this framework to language modeling. We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification. A Transformer-based VAE maps discrete tokens to continuous latents, enhanced by Representation Alignment (REPA) with a frozen pretrained language model to produce representations effective for conditional denoising. A standard DiT then performs flow matching in this latent space, identical in architecture to its visual counterpart. The central challenge we address is obtaining high-quality continuous text representations: we find that reconstruction fidelity alone is insufficient, and that aligning latent features with a pretrained language model via REPA is critical for downstream generation quality. Trained from scratch on OpenWebText2, TextLDM substantially outperforms prior diffusion language models and matches GPT-2 under the same settings. Our results establish that the visual DiT recipe transfers effectively to language, taking a concrete step toward unified diffusion architectures for multimodal generation and understanding.

Abstract

T2I Generation

Explore similar work

TextLDM: Language Modeling with Continuous Latent Diffusion

Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference

Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM