REPA-G: Test-Time Conditioning with Representation-Aligned Visual Features

Nicolas Sereyjol-Garros; Ellington Kirby; Victor Letzelter; Victor Besnier; Nermin Samet

cs.CVFeb 3, 2026

REPA-G: Test-Time Conditioning with Representation-Aligned Visual Features

Authors: Nicolas Sereyjol-Garros, Ellington Kirby, Victor Letzelter, Victor Besnier, Nermin Samet

Organizations: Valeo.ai, Paris, France · LTCI, Télécom Paris, Institut Polytechnique de Paris, France

Abstract

While representation alignment with self-supervised models has been shown to improve diffusion model training, its potential for enhancing inference-time conditioning remains largely unexplored. We introduce Representation-Aligned Guidance (REPA-G), a framework that leverages these aligned representations, with rich semantic properties, to enable test-time conditioning from features, in generation. By optimizing a similarity objective (the potential) at inference, we steer the denoising process toward a conditioned representation extracted from a pre-trained feature extractor. Our method provides versatile control at multiple levels of granularity, ranging from patch level matching via single patches to broad semantic guidance using global image feature tokens. We further extend this to multi-concept composition, allowing for the faithful combination of distinct concepts. REPA-G operates entirely at inference time with no additional training required, offering a flexible and precise alternative to often ambiguous text prompts or coarse class labels. Our approach achieves high-quality, diverse generations on ImageNet and COCO. Code is available at https://github.com/valeoai/REPA-G

Figures & tables

Panel of Figure 1 — Figure 1: Comparing class-label, text-prompt, and REPA-G conditioning. All models are trained on ImageNet. (Top) We average extracted features ( ) from a anchor image to generate a generic “rabbit” image. (Bottom) We combine a masked feature map with a specific “lava” patch ( ) to synthesize a rabbit on a volcano. While text prompts require lengthy descriptions and often lack precision, our feature-based conditioning offers better compositional control and provides more precise generation.

Panel of Figure 2 — Figure 2: Impact of representation alignment on feature space. We perform $k$ -means clustering ( $k=1{,}000$ ) on four feature spaces across ImageNet (see App. C.1 ). For a reference image (with red frame), we visualize others from its assigned cluster. Without alignment, SiT fails to form semantic groupings, making its latent space unsuitable for conditioning. In contrast, the aligned model successfully replicates the teacher’s semantic structure both before and after the projection layer, resulting in semantically consistent clusters.

		Full Feature Map					Masked Feature Map					Average Feature Map
Model	Cond.	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$
SiT	-	45.02	9.02	22.33	0.50	0.63	45.02	9.02	22.33	0.50	0.63	45.02	9.02	22.33	0.50	0.63
SiT	$\text{F}_{\text{SiT}}$	71.35	72.08	14.73	0.33	0.54	45.85	19.23	29.36	0.40	0.62	65.99	22.52	16.69	0.36	0.52
REPA	-	26.58	6.85	41.31	0.56	0.70	26.58	6.85	41.31	0.56	0.70	26.58	6.85	41.31	0.56	0.70
	$\text{F}_{\text{DINO}}$	7.23	8.86	199.69	0.65	0.69	14.17	10.59	135.08	0.58	0.64	29.06	16.79	83.85	0.46	0.69
	$\text{F}_{\text{SiT}}$	2.09	6.17	260.97	0.74	0.70	2.67	4.86	222.71	0.74	0.69	6.26	6.14	159.14	0.68	0.70

Table 1: Distribution-level comparison of REPA-G generations on ImageNet [ 33 ] . We compare the standard SiT backbone [ 6 ] (no representation alignment) against REPA [ 7 ] and REPA-E [ 8 ] variants. For each method block, the first row denotes results for unconditional generation, serving as a baseline, and

\text{F}_{\text{SiT}}

and

\text{F}_{\text{DINO}}

for REPA-G.

		Full Feature Map				Masked Feature Map				Average Feature Map
Model	Cond.	DINOv2	JEPA	CLIP	PSNR	DINOv2	JEPA	CLIP	PSNR	DINOv2	JEPA	CLIP	PSNR
SiT	$\text{F}_{\text{SiT}}$	0.27	0.36	0.43	15.35	0.44	0.48	0.46	17.98	0.12	0.35	0.83	7.74
REPA	$\text{F}_{\text{DINO}}$	0.75	0.61	0.58	11.02	0.77	0.60	0.57	10.63	0.76	0.69	0.92	8.00
REPA	$\text{F}_{\text{SiT}}$	0.85	0.78	0.71	20.46	0.87	0.79	0.71	20.72	0.84	0.84	0.95	10.82
REPA-E	$\text{F}_{\text{DINO}}$	0.83	0.69	0.64	15.23	0.85	0.68	0.63	14.36	0.91	0.83	0.96	10.97
REPA-E	$\text{F}_{\text{SiT}}$	0.86	0.76	0.71	18.68	0.88	0.78	0.71	19.07	0.91	0.88	0.96	11.96

Table 2: Instance-level comparison of REPA-G generations on ImageNet [ 33 ] . Setup is the same as in Table 1 . Alignment is measured with DINOv2 [ 12 ] , JEPA [ 36 ] , and CLIP [ 37 ] feature spaces, supplemented by PSNR for pixel-level fidelity. We highlight ours conditioning methods with

\text{F}_{\text{SiT}}

and

\text{F}_{\text{DINO}}

		Full Feature Map					Masked Feature Map					Average Feature Map
Model	Cond.	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$
SiT	-	45.13	30.15	22.33	0.42	0.54	45.13	30.15	22.33	0.42	0.54	45.13	30.15	22.33	0.42	0.54
SiT	$\text{F}_{\text{SiT}}$	77.26	108.79	9.86	0.25	0.37	51.88	42.36	15.76	0.27	0.52	66.09	46.88	16.67	0.28	0.45
REPA	-	37.85	29.11	41.31	0.45	0.58	37.85	29.11	41.31	0.45	0.58	37.85	29.11	41.31	0.45	0.58
	$\text{F}_{\text{DINO}}$	12.96	29.05	31.11	0.55	0.57	20.51	31.04	26.15	0.45	0.53	32.04	39.05	20.59	0.35	0.53
	$\text{F}_{\text{SiT}}$	6.18	25.35	33.08	0.65	0.62	6.63	24.37	32.05	0.65	0.61	9.46	25.89	29.43	0.60	0.60

Table 3: Distribution-level zero-shot evaluation on the COCO [ 34 ] dataset. The setup is the same as in Table 1 . Within each block, the first row reports unconditional generation results as a baseline, while

\text{F}_{\text{SiT}}

and

\text{F}_{\text{DINO}}

denote our method.

Panel of Figure 5 — Figure 5: Qualitative comparison on ImageNet [ 33 ] using REPA-E [ 8 ] with DINOv2 features. (Left) We show single-source conditioning across three levels of granularity: full, masked, and averaged feature maps given anchor image and input mask . (Right) We illustrate multi-source composition. Within each group, the anchor object and target background are shown at a small scale alongside the enlarged generation. We use full features from the anchor and a single feature patch sampled from the target’s background.

Table 8

Table 9

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Representation alignment with diffusion in 2D space. Ground-truth samples are shown in grey, with corresponding samples generated via diffusion in red. The conditional results in the second and third columns are guided by target feature vectors $[-1,0]$ and $[0,-1]$ , respectively. The last column shows the feature space mapping, mapping each data points to its corresponding 2D feature.

Panel of Figure 7 — Figure 7: Impact of representation alignment on SiT’s feature space. We compare clusters across ImageNet [ 33 ] using four feature spaces. For each space, we extract features for the entire ImageNet dataset and perform $k$ -means clustering into 1,000 discrete clusters. Then, we identify the specific cluster associated with a given reference image (indicated by the red frame) and randomly sample other images from that same cluster. Without alignment, the standard SiT model fails to form semantic clusters; features that are close in latent space represent unrelated concepts, which makes these internal representations unsuitable for conditioning. The aligned model successfully mimics the feature space of the visual teacher backbone both before and after the projection layer. This results in semantically consistent clusters. SiT (Align) refers to REPA [ 17 ] features extracted before the projection layer, whereas SiT (Align + Proj) denotes features extracted after the projection layer.

$\lambda$	Full				Masked				Average
$\lambda$	FID ( $\downarrow$ )	Prec. ( $\uparrow$ )	Rec. ( $\uparrow$ )	Align. ( $\uparrow$ )	FID ( $\downarrow$ )	Prec. ( $\uparrow$ )	Rec. ( $\uparrow$ )	Align. ( $\uparrow$ )	FID ( $\downarrow$ )	Prec. ( $\uparrow$ )	Rec. ( $\uparrow$ )	Align. ( $\uparrow$ )
5000	3.3	0.72	0.67	0.6	3.7	0.72	0.66	0.7	3.2	0.76	0.64	0.9
10000	2.8	0.75	0.67	0.7	2.8	0.74	0.66	0.8	2.8	0.76	0.65	0.9
20000	1.6	0.76	0.69	0.8	2.4	0.75	0.66	0.8	2.7	0.75	0.67	0.9
50000	1.4	0.76	0.69	0.8	2.3	0.75	0.67	0.9	3.3	0.73	0.68	0.9
100000	1.5	0.75	0.70	0.9	2.6	0.74	0.68	0.9	4.5	0.71	0.69	0.9

Appendix

Table 8: Guidance scale (

\lambda

) robustness. FID and DINOv2 Alignment scores for single-source conditioning under various values of the guidance scale

\lambda

Panel of Figure 8 — Figure 8: Sensitivity plots illustrating the relationship between $\lambda$ , temperature, and output quality.

		Full Feature Map					Masked Feature Map					Average Feature Map
Model	Cond.	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$
SiT	-	45.02	9.02	22.33	0.50	0.63	45.02	9.02	22.33	0.50	0.63	45.02	9.02	22.33	0.50	0.63
SiT	$\text{F}_{\text{SiT}}$	71.35	72.08	14.73	0.33	0.54	51.88	42.36	15.76	0.27	0.52	65.99	22.52	16.69	0.36	0.52
REPA	-	26.58	6.85	41.31	0.56	0.70	26.58	6.85	41.31	0.56	0.70	26.58	6.85	41.31	0.56	0.70
	$\text{F}_{\text{DINO}}$	7.23	8.86	199.69	0.65	0.69	14.17	10.59	135.08	0.58	0.64	29.06	16.79	83.85	0.46	0.69
	$\text{F}_{\text{SiT}}$	2.09	6.17	260.97	0.74	0.70	2.67	4.86	222.71	0.74	0.69	6.26	6.14	159.14	0.68	0.70

Appendix

Table 9: Distribution-level comparison of unconditional REPA-G generations on ImageNet [ 33 ] . We compare the standard SiT backbone [ 6 ] (trained without representation alignment) against REPA [ 7 ] and REPA-E [ 8 ] variants. All models are trained on ImageNet. For each method block, the first row denotes metrics for unconditional generation, serving as a baseline for the REPA-G results.

\text{F}_{\text{SiT}}^{*}

denotes features extracted after the projection layer in REPA-based models. Gray results are from Table 1 in the main text for ease of comparison.

		Full Feature Map						Masked Feature Map						Average Feature Map
Model	Feat .	DINOv2	JEPA	CLIP	MAE	MoCo	PSNR	DINOv2	JEPA	CLIP	MAE	MoCo	PSNR	DINOv2	JEPA	CLIP	MAE	MoCo	PSNR
SiT	$\text{F}_{\text{SiT}}$	0.27	0.36	0.43	0.93	0.78	15.35	0.44	0.48	0.46	0.95	0.82	17.98	0.12	0.35	0.83	0.99	0.95	7.74
REPA	$\text{F}_{\text{DINO}}$	0.75	0.61	0.58	0.94	0.86	11.02	0.77	0.60	0.57	0.94	0.84	10.63	0.76	0.69	0.92	0.99	0.98	8.00
	$\text{F}_{\text{SiT}}$	0.85	0.78	0.71	0.98	0.95	20.46	0.87	0.79	0.71	0.98	0.94	20.72	0.84	0.84	0.95	0.99	0.990	10.82
	$\text{F}_{\text{SiT}}^{*}$	0.72	0.63	0.59	0.94	0.87	11.32	0.74	0.63	0.58	0.94	0.86	11.02	0.77	0.72	0.92	0.99	0.98	8.01
REPA-E	$\text{F}_{\text{DINO}}$	0.83	0.69	0.64	0.96	0.90	15.23	0.85	0.68	0.63	0.95	0.88	14.36	0.91	0.83	0.96	0.99	0.99	10.97

Appendix

Table 10: Instance-level comparison of unconditional REPA-G generations on ImageNet [ 33 ] . Setup is the same as in Table 1 . Alignment is measured with DINOv2 [ 12 ] , JEPA [ 36 ] , CLIP [ 37 ] , MAE [ 56 ] and MoCov3 [ 57 ] feature spaces, supplemented by PSNR for pixel-level fidelity.

\text{F}_{\text{SiT}}^{*}

denotes features extracted after the projection layer in REPA-based models. Gray results are from Table 1 in the main text for ease of comparison.

		Full Feature Map						Masked Feature Map						Average Feature Map
Model	Cond.	DINOv2	JEPA	CLIP	MAE	MoCo	PSNR	DINOv2	JEPA	CLIP	MAE	MoCo	PSNR	DINOv2	JEPA	CLIP	MAE	MoCo	PSNR
SiT	$\text{F}_{\text{SiT}}$	0.25	0.35	0.4	0.93	0.76	15.18	0.4	0.45	0.44	0.95	0.82	18.56	0.11	0.38	0.81	0.99	0.95	7.66
REPA	$\text{F}_{\text{DINO}}$	$0.737$	$0.603$	$0.564$	$0.935$	$0.847$	$10.718$	$0.746$	$0.561$	$0.553$	$0.933$	$0.835$	$10.846$	$0.736$	$0.679$	$0.900$	$0.992$	$0.979$	$7.724$
	$\text{F}_{\text{SiT}}$	0.84	0.78	0.7	0.98	0.95	20.61	0.84	0.76	0.69	0.98	0.94	21.38	0.8	0.82	0.92	0.99	0.99	10.47
	$\text{F}_{\text{SiT}}^{*}$	$0.697$	$0.627$	$0.564$	$0.939$	$0.866$	$11.007$	$0.705$	$0.593$	$0.556$	$0.938$	$0.857$	$11.275$	$0.732$	$0.701$	$0.902$	$0.993$	$0.982$	$7.727$
REPA-E	$\text{F}_{\text{DINO}}$	$0.821$	$0.677$	$0.626$	$0.951$	$0.892$	$14.776$	$0.838$	$0.642$	$0.616$	$0.949$	$0.882$	$14.715$	0.9	$0.821$	$0.943$	$0.996$	$0.989$	$10.663$

Appendix

Table 11: Zero-shot evaluation on COCO [ 34 ] dataset using instance-level metrics. Alignment is measured with DINOv2 [ 12 ] , JEPA [ 36 ] , CLIP [ 37 ] , MAE [ 56 ] and MoCov3 [ 57 ] feature spaces, supplemented by PSNR for pixel-level fidelity.

\text{F}_{\text{SiT}}^{*}

denotes features extracted after the projection layer in REPA-based models.

		Full Feature Map					Masked Feature Map					Average Feature Map
Model	Cond.	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$
SiT	-	10.17	8.53	124.93	0.67	0.67	10.17	8.53	124.93	0.67	0.67	10.17	8.53	124.93	0.67	0.67
SiT	$\text{F}_{\text{SiT}}$	71.05	49.21	17.42	0.29	0.58	62.65	13.98	19.82	0.34	0.65	130.14	53.16	2.08	0.50	0.04
REPA	-	5.92	5.92	157.10	0.70	0.68	5.92	5.92	157.10	0.70	0.68	5.92	5.92	157.10	0.70	0.68
	$\text{F}_{\text{DINO}}$	6.24	8.43	197.64	0.69	0.64	13.68	10.21	127.72	0.61	0.60	27.94	15.32	84.04	0.49	0.66
	$\text{F}_{\text{SiT}}$	2.14	6.71	254.99	0.77	0.65	3.34	5.68	199.55	0.77	0.63	11.24	8.59	122.54	0.63	0.67

Appendix

Table 12: Distribution-level comparison of class-conditional REPA-G generations on ImageNet [ 33 ] . We compare the standard SiT backbone [ 6 ] (trained without representation alignment) against REPA [ 7 ] and REPA-E [ 8 ] variants. All models are trained on ImageNet. For each method block, the first row denotes metrics for unconditional generation, serving as a baseline for the REPA-G results.

\text{F}_{\text{SiT}}^{*}

denotes features extracted after the projection layer in REPA-based models.

		Full Feature Map						Masked Feature Map						Average Feature Map
Model	Cond.	DINO	JEPA	CLIP	MAE	MoCo	PSNR	DINO	JEPA	CLIP	MAE	MoCo	PSNR	DINO	JEPA	CLIP	MAE	MoCo	PSNR
SiT	$\text{F}_{\text{SiT}}$	0.3	0.39	0.43	0.94	0.78	14.79	0.4	0.44	0.44	0.95	0.81	17.66	0.13	0.37	0.83	0.99	0.95	8.64
REPA	$\text{F}_{\text{DINO}}$	$0.737$	$0.593$	$0.574$	$0.935$	$0.848$	$10.467$	$0.757$	$0.583$	$0.564$	$0.931$	$0.832$	$10.303$	$0.753$	$0.678$	$0.917$	0.99	$0.977$	$8.047$
	$\text{F}_{\text{SiT}}$	0.84	0.77	0.7	0.98	0.94	19.6	0.86	0.77	0.7	0.98	0.94	20	0.79	0.8	0.93	0.99	0.99	10.36
	$\text{F}_{\text{SiT}}^{*}$	$0.706$	$0.613$	$0.578$	$0.938$	$0.862$	$10.714$	$0.727$	$0.608$	$0.570$	$0.935$	$0.849$	$10.595$	$0.758$	$0.696$	$0.922$	0.99	$0.980$	$8.024$
REPA-E	$\text{F}_{\text{DINO}}$	$0.821$	$0.675$	$0.634$	$0.952$	$0.892$	$14.785$	$0.839$	$0.656$	$0.620$	$0.947$	$0.873$	$13.835$	$0.882$	$0.788$	$0.949$	0.99	$0.985$	$10.340$

Appendix

Table 13: Distribution-level comparison of class-conditional REPA-G generations on ImageNet [ 33 ] . We compare the standard SiT backbone [ 6 ] (trained without representation alignment) against REPA [ 7 ] and REPA-E [ 8 ] variants. All models are trained on ImageNet. For each method block, the first row denotes metrics for unconditional generation, serving as a baseline for the REPA-G results.

\text{F}_{\text{SiT}}^{*}

denotes features extracted after the projection layer in REPA-based models.

Panel of Figure 9 — Figure 9: Effect of guidance strength on generation diversity. Impact of $\lambda$ on LPIPS distance, DINOv2 cosine distance, and DINOv2-based Vendi score for samples generated from ImageNet and COCO anchors.

Panel of Figure 10 — Figure 10: Qualitative results of generation diversity. ImageNet examples are shown in the top row and COCO examples in the bottom row. For each example, the anchor image is shown on the left, followed by three samples generated with average feature conditioning and three with full feature map conditioning. We use a small guidance scale of $\lambda=2{,}000$ .

Method	Features	FID	sFID	IS	Prec.	Rec.	DINOv2	JEPA	CLIP	PSNR
REPA-G	Full	1.4	4.1	264.7	0.76	0.69	0.83	0.69	0.64	15.2
	Masked	2.3	4.8	212.2	0.75	0.67	0.85	0.68	0.63	14.4
	Average	3.2	5.2	188.9	0.73	0.67	0.91	0.83	0.96	11.0
TFG	Full	74.6	53.6	47.4	0.26	0.40	0.91	0.43	0.54	10.0
	Masked	85.8	47.2	36.0	0.24	0.38	0.94	0.45	0.55	10.0
	Average	81.1	37.0	35.8	0.23	0.41	0.99	0.51	0.91	8.5

Appendix

Table 14: TFG performance. Evaluation of TFG with different guidance granularities.

	ImageNet						COCO
Condition	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	DINOv2 $\uparrow$	FID $\downarrow$	sFID $\downarrow$	IS $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	DINOv2 $\uparrow$
-	15.07	4.46	55.14	0.65	0.69	-	36.23	29.98	55.13	0.51	0.59	-
IP-Adapter	7.52	10.65	145.36	0.71	0.63	0.62	9.90	32.23	30.45	0.62	0.56	0.61
REPA-G	1.45	4.07	264.69	0.76	0.69	0.83	4.61	23.27	35.97	0.67	0.64	0.82

Appendix

Table 15: Comparison with trained adapter-based conditioning. We compare full-feature-map conditional generation using IP-Adapter and REPA-G on ImageNet and COCO, with REPA-E as the shared base model.

Panel of Figure 11 — Figure 11: Visual comparison with the training-free baseline TFG [ 31 ] and the trained IP-Adapter [ 27 ] . While the images generated with TFG respect the conditioning to the anchors, perceptual artifacts appear and degrade image quality. On the other hand, IP-adapter manages to generate images faithful to the anchor but requires additional training and parameters.

Panel of Figure 12 — Figure 12: Qualitative results for resolution scalability. Using the fine-tuned 512 $\times$ 512 model, REPA-G successfully stochastically reconstruct the conditioning image at a 512 $\times$ 512 resolution. The top row shows the anchor images, and the bottom row shows the corresponding results for REPA-G (512x512).

	FID	sFID	IS	Prec.	Rec.	DINOv2 Align.
Uncond	33.1	10.1	39.1	0.61	0.6	-
Full	5.4	6.0	178.8	0.75	0.67	0.61
Masked	6.9	5.9	149.1	0.73	0.65	0.68
Average	8.9	5.6	139.9	0.73	0.66	0.85

Appendix

Table 16: Resolution scalability. We evaluate our method’s ability to scale to higher resolutions (512

\times

512) and compare the performance across unconditional generation with different granularity settings for our feature conditioning.

Image Resolution	Method	Throughput (img/s)	Peak GPU memory (GB)
256	Uncond.	$0.50$	8
	Ours (Single source)	$0.37$	10
	Ours (Multi source)	$0.37$	10
	TFG	$0.21$	52
	CFG	$0.26$	8
	IP-Adapter	$0.19$	9

Appendix

Table 17: Inference throughput. Feature conditioning requires backpropagating gradients through the initial blocks of the transformer. While this results in a slight reduction in throughput, the computational overhead remains minimal. The baseline is the standard class-conditioned REPA-E model [ 8 ] , whereas our approach incorporates feature guidance into this framework.

Super-category	Fine-grained Class Names
Animals	Golden retriever , Tabby cat , Zebra , Bullfrog , Tarantula , Goldfish , Monarch butterfly , Crane , Bald eagle , Tiger
Movables	Bicycle , Motor scooter , Mountain bike , Baby stroller , Lawnmower , Shopping cart , Tricycle , Car wheel , Fire engine , Sports car
Food Flora	Strawberry , Banana , Granny Smith , Plate , Pizza , Cheeseburger , Agaric , Acorn , Burrito , Bee
Tools Tech	Acoustic guitar , Broom , Hammer , Screwdriver , Soccer ball , Tennis racket , Crash helmet , Wall clock , Laptop , Ping-pong ball
Household	Teapot , Coffeepot , Toaster , Vase , Cowboy hat , Running shoe , Backpack , Umbrella , Teddy bear , Balloon
Our hand-selected target backgrounds

Appendix

Table 18: A diverse set of 50 ImageNet [ 33 ] classes and 8 hand-selected target backgrounds used to evaluate multi-source compositional generation.

Panel of Figure 13 — Figure 13: Failure cases of multiple-source compositional generation. Top row: Inconsistent aspect ratio. Middle row: Failed blending. Bottom row: Inconsistent view point.

Panel of Figure 14 — Figure 14: Qualitative results on ImageNet dataset [ 33 ] . We present randomly selected samples from various conditioning modes. The first row shows the ground-truth (anchor) images, followed by their unsupervised foreground masks. Subsequent rows show generations using the full feature map, the masked feature map (using the corresponding mask), and the averaged feature map. All visualizations are from the following configuration: an unconditioned model guided by features from the diffusion model before the projection layer. The following block displays the corresponding results for standart SiT [ 6 ] model trained without representation alignment and REPA-E [ 8 ] . We observe that SiT models trained without representation alignment struggle to produce coherent results under feature guidance.

Panel of Figure 15 — Figure 15: Qualitative results on COCO dataset [ 34 ] . We present randomly selected samples from various conditioning modes. The first row shows the ground-truth (anchor) images, followed by their unsupervised foreground masks. Subsequent rows show generations using the full feature map, the masked feature map (using the corresponding mask), and the averaged feature map. All visualizations are from the following configuration: an unconditioned model guided by features from the diffusion model before the projection layer. The following block displays the corresponding results for standart SiT [ 6 ] model trained without representation alignment and REPA-E [ 8 ] . We observe that SiT models trained without representation alignment struggle to produce coherent results under feature guidance.

Explore similar work

Embedding Prediction Helps Image Generation
Oct 1, 2026Sihan Xu, Ji Xie, Zilin Wang +2Diffusion Transformers Image Generation
University of Michigan · Carnegie Mellon University
Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds
Aug 4, 2026Ning Zhu, An Chen, Mengfei Zhao +4Text-To-Image Diffusion Models Denoising Trajectory
Glasgow College, University of Electronic Science and Technology of China · School of Mathematical Sciences, University of Electronic Science and Technology of China
Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
May 24, 2026Agata Żywot, Iason Skylitsis, Thijmen Nijdam +4Text-To-Image Diffusion Models Text-Conditioned Diffusion Model
University of Amsterdam, Netherlands

Oct 1, 2026, cs.CVJ/K move · Enter open · S save

Embedding Prediction Helps Image Generation

Sihan Xu, Ji Xie, Zilin Wang, Hui Shen+1

University of Michigan · Carnegie Mellon University

Diffusion Transformers Image Generation Autoregressive Model

Open paper AskPDF arXiv

In diffusion transformers, a class label or a text prompt is embedded once, and the same condition is reused at every denoising step. We ask whether predicted embeddings can serve as this condition instead. Next-Embedding Predictive Autoregression (NEPA) trains a Transformer to predict the next continuous embedding in a sequence. In generation, the clean image follows the noisy image, so its embeddings are the next embeddings after the condition and the noisy image. We train a NEPA model to predict them all at once with Multi-Embedding Prediction, and in Embedding Conditioned Generation, a DiT generator is conditioned on these predictions, recomputed at every denoising step, so the conditioning signal adapts to the current noisy state. Experiments on class-conditional ImageNet

256\times256

study the condition of the generator, the design of Multi-Embedding Prediction, and the scaling of both models. The NEPA model adds a second network to every sampling step; with it, and combined with REPA, our final model, NEPA-DiT-XL, reaches an FID of 1.32 using about a third of the training compute of REPA.

Abstract

Explore similar work

Embedding Prediction Helps Image Generation

Rectify Then Diffuse: Disentangling Concepts Before Denoising Trajectory Unfolds

Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference