Representation Conditioning for Generation

Saeed Razavi; Xingjian Leng; Liang Zheng

cs.CVOct 14, 2026

Representation Conditioning for Generation

Authors: Saeed Razavi, Xingjian Leng, Liang Zheng

Organizations: Australian National University

Abstract

Representation-alignment methods effectively accelerate the diffusion transformer training. REPA, for example, uses a pretrained vision encoder to supervise an intermediate layer through an MLP projection head which is discarded after training. It aims to import semantics from the vision encoder to the diffusion backbone. While this objective is achieved at early layers by back-propagating the alignment loss, there is no guarantee that later layers, supervised by the velocity prediction loss alone, can well access such semantics. We propose a simple solution, representation conditioning (RECON), to improve semantics in later layers. During training, we feed the pretrained vision encoder tokens into later layers; during inference, because we no longer have the clean image, we reuse the MLP head (as a proxy to the vision encoder) and project its output tokens to the later layers. Our method adds no parameters and leads to much faster convergence: RECON is $63\times$ faster than SiT and $25\times$ faster than REPA. Code is available at https://github.com/saeidrazavi/RECON

Figures & tables

Panel of Figure 2 — Figure 2: Alignment with the vision encoder, measured by CKNNA against DINOv2-B. (a) gFID against CKNNA over training checkpoints. (b) CKNNA of backbone features at each layer. Dashed line is CKNNA between the projector output $g_{\phi}(h)$ and the vision encoder, which is stronger than any layer. (c) CKNNA of every layer for REPA and RECON at $t=0.25$ .

Figure 3: Representation conditioning improves visual scaling. RECON produces higher-quality images at different steps compared with vanilla REPA. Results for both methods are sampled using the same seed, noise and class label. We use no classifier-free guidance.

Figure 4: Qualitative results on ImageNet $256\times 256$ from SiT-XL/2 + RECON, using classifier-free guidance with $w=4$ .

Table 5

Table 6

Figure 5: Qualitative comparison on text-to-image generation (BLIP3-o). Curated qualitative samples from REPA and RECON trained for 100K iterations. Prompts are shown on top.

Figure 6: Linear probing vs. layers. ImageNet-1K validation top-1, at

t=0.25

Panel of Figure 7 — Figure 7: RECON settles the structure of a sample earlier in generation. (a) measures how close the running estimate $\hat{x}_{0}(t)$ is, in semantic space, to the image that the same run finally produces. It is measured for a single sample, where RECON is above REPA at every step (both methods start from the same noise). In (b), we generate 10K images. At each diffusion timestep, we measure the percentage of samples whose running estimate $\hat{x}_{0}(t)$ similarity with the final image is greater than 0.8. RECON is above REPA.

NFE	50	100	250
REPA	10.8	8.4	7.9
RECON	3.3	3.2	3.2

Table 5: Sampling steps. gFID-50K; SiT-XL/2, 400K iter., no CFG.

Panel of Figure 8 — Figure 8: RECON vs . REPA across model scale, tokenizer and target encoder. gFID-50K of REPA and RECON. (a) SiT-B/2, SiT-L/2 and SiT-XL/2 at 20 epochs. (b) Three tokenizers at 80 epochs. (c) Five target vision encoders, SiT-B/2 at 20 epochs.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Table 12

Model scale : 20 epochs, SD-VAE latents
	REPA				RECON
	FID $\downarrow$	IS $\uparrow$	Pre. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	IS $\uparrow$	Pre. $\uparrow$	Rec. $\uparrow$
SiT-B/2	46.35	30.53	0.477	0.608	26.08	54.63	0.620	0.577
SiT-L/2	22.95	59.17	0.618	0.610	11.24	104.44	0.742	0.550
SiT-XL/2	18.88	69.34	0.647	0.608	9.25	118.80	0.760	0.543
Tokenizer : SiT-XL/2, 80 epochs

Appendix

Table 11: Full metrics for Fig. 8 . ImageNet

256\times 256

, no CFG.

Figure 9: Denoising heavily corrupted images. We take three clean images (left) and corrupt each with heavy noise (80% noise and 20% image). Then let REPA and RECON denoise them; each frame shows the clean-image estimate $\hat{x}_{0}(t)$ . RECON recovers the identity and layout of the original image better than REPA, which shows its strength in retaining semantics under heavy noise.

Panel of Figure 10 — Figure 10: Uncurated generation results of SiT-XL/2 + RECON. We use classifier-free guidance with $w=4.0$ .

Panel of Figure 11 — Figure 11: Uncurated generation results of SiT-XL/2 + RECON. We use classifier-free guidance with $w=4.0$ .

Panel of Figure 12 — Figure 12: Uncurated generation results of SiT-XL/2 + RECON. We use classifier-free guidance with $w=4.0$ .

Panel of Figure 13 — Figure 13: Uncurated generation results of SiT-XL/2 + RECON. We use classifier-free guidance with $w=4.0$ .

Explore similar work

Scaffold Then Internalize: Representation Injection for Diffusion Transformers
Sep 28, 2026Han Fu, Jiacheng Chen, Baoquan Zhao +4Diffusion Transformer Diffusion Model Acceleration
Sun Yat-sen University · Video Rebirth · The Hong Kong Polytechnic University
Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers
May 16, 2026Shaodong Xu, Zhendong Wang, Litong Gong +4Diffusion Model Alignment Diffusion Transformer
University of Science and Technology of China
REPA-G: Test-Time Conditioning with Representation-Aligned Visual Features
Feb 3, 2026Nicolas Sereyjol-Garros, Ellington Kirby, Victor Letzelter +2Diffusion Model Guidance Conditional Image Generation
Valeo.ai, Paris, France · LTCI, Télécom Paris, Institut Polytechnique de Paris, France

Sep 28, 2026J/K move · Enter open · S save

Scaffold Then Internalize: Representation Injection for Diffusion Transformers

Han Fu, Jiacheng Chen, Baoquan Zhao, Weidong Chen+3

Sun Yat-sen University · Video Rebirth · The Hong Kong Polytechnic University

Diffusion Transformer Diffusion Model Acceleration Diffusion Models

Open paper AskPDF arXiv

Recent representation alignment (REPA) methods accelerate diffusion transformer training by aligning projections of the transformer's hidden states with representations from pretrained visual encoders. In this work, we explore a reverse and complementary direction to REPA: rather than projecting diffusion representations into the encoder's space, we inject encoder representations into the diffusion transformer, allowing them to actively participate in the denoising process. To this end, we introduce REPresentation Injection (REPI), a training framework based on a scaffold-to-internalization strategy, in which projected encoder representations initially serve as a temporary scaffold and are then progressively internalized by the diffusion transformer. REPI outperforms REPA across a wide range of backbones and is highly complementary to it: combining the two yields substantial gains over either alone. Notably, with only 160K training steps, REPI + REPA matches vanilla SiT trained for 7M steps, a speedup of over

43.5\times

. Code will be available at https://jeneveuxpas.github.io/REPI

Abstract

Explore similar work

Scaffold Then Internalize: Representation Injection for Diffusion Transformers

Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers

REPA-G: Test-Time Conditioning with Representation-Aligned Visual Features