A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers

Trung X. Pham; Kang Zhang; Ji Woo Hong; Chang D. Yoo

cs.CVFeb 25, 2026

A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers

Authors: Trung X. Pham, Kang Zhang, Ji Woo Hong, Chang D. Yoo

Organizations: Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Diffusion Transformers have achieved state-of-the-art performance in class-conditional and multimodal generation, yet the structure of their learned conditional embeddings remains poorly understood. In this work, we present the first systematic study of these embeddings and uncover a notable redundancy: class-conditioned embeddings exhibit extreme angular similarity, exceeding 99% on ImageNet-1K, while continuous-condition tasks such as pose-guided image generation and video-to-audio generation reach over 99.9%. We further find that semantic information is concentrated in a small subset of dimensions, with head dimensions carrying the dominant signal and tail dimensions contributing minimally. By pruning low-magnitude dimensions--removing up to two-thirds of the embedding space--we show that generation quality and fidelity remain largely unaffected, and in some cases improve. These results reveal a semantic bottleneck in Transformer-based diffusion models, providing new insights into how semantics are encoded and suggesting opportunities for more efficient conditioning mechanisms.

Figures & tables

Figure 1: Hidden Semantic Bottleneck: Extreme Alignment and Dimensional Sparsity. Conditional vectors

\vec{c}

in state-of-the-art Transformer diffusion models on ImageNet-1K exhibit very high pairwise cosine similarity (mostly 90–99%) while concentrating semantic information in only a few of 1,152 dimensions.

Figure 2: Transformer-based diffusion models inject conditions as a globally compact vectors $\vec{v}_{i}$ via AdaLN for outputs such as images or mel-spectrograms.

Figure 3: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using REPA-XL ( Yu et al., 2025 ) . Despite distinct semantics, embeddings show over 99% similarity for nearly all class pairs. Left : full $1000\!\times\!1000$ matrix showing global alignment. Right : zoomed $10\!\times\!10$ subset for randomly chosen classes. Additional results for other SOTA methods appear in the Appendix.

Figure 4: Sparsity and alignment of conditional embeddings in X-MDPT ( Pham et al., 2024 ) . (a) and (b): With $\tau=0.1$ , over 51% of components in the conditional vectors have magnitudes below the threshold, highlighting significant sparsity. Remarkably, pruning these dimensions has minimal effect on generation quality. (c) Cosine similarity between random test samples in DeepFashion exceeds 99.9%, confirming extreme alignment across conditional embeddings.

Figure 5: Magnitude histogram distribution of learned conditional vector embedding

\vec{c}\in\mathbb{R}^{\times 1152}

. Most dimensions have near-zero values (

<0.01

), with only

\sim 5-20

dimensions showing dominant magnitudes. This sparsity holds across multiple models, including DiT, MDT, LightningDiT, MG, SiT, and REPA. It is best viewed with 300% zoomed in.

Figure 6: Original distribution of learned conditional vector embedding

\vec{c}\in\mathbb{R}^{1\times 1152}

. Most dimensions have near-zero values (

<0.01

), with only

\sim 5-20

dimensions showing dominant magnitudes. This sparsity holds across multiple models, including DiT, MDT, LightningDiT, MG, SiT, and REPA. It is best viewed with 200% zoomed in.

Embedding Metrics	DiT	SiT	MDT	LightningDiT	MG	REPA	X-MDPT	MDSGen
Condition Dim ( $d$ )	1152	1152	1152	1152	1152	1152	1024	768
PR ( $\alpha$ )	120.69	26.25	18.45	23.70	19.98	17.67	495.75	104.22
nPR ( $\alpha_{\text{norm}}$ )	10.47%	2.28%	1.60%	2.05%	1.73%	1.53%	48.42%	13.57%
Cosine Sim. ( $cs$ )	0.9001	0.9852	0.9905	0.9779	0.9934	0.9946	0.9998	0.9999

Table 1: Participation Ratio (PR) in learned conditional embeddings of state-of-the-art models on Imagenet-1K generation (discrete) and DeepFashion/VGGSound (continuous).

Figure 7: Class-conditioned image generation with head removal. ImageNet samples after pruning top-magnitude dimensions of $\vec{c}$ (threshold $\tau$ ); removing only a few head dimensions markedly degrades quality.

Prune	Threshold $\tau$	# Removed Dims	FID ↓	IS ↑	CLIP ↑
Prune	Baseline (REPA)	0/1152 (0%)	7.1694	176.02	29.746
	$\tau=0.01$ ( $t_{i}$ )	448/1152 (38.94%)	7.2143	171.99	29.737
	$\tau=0.01$ ( $t_{0}$ )	448/1152 (38.94%)	7.1690	175.97	29.807
Tail	$\tau=0.01$ ( $t_{n-k,n}$ )	448/1152 (38.94%)	7.1598	175.49	29.805
	$\tau=0.02$ ( $t_{i}$ )	762/1152 (66.21%)	9.2202	125.15	29.221
	$\tau=0.05$ ( $t_{i}$ )	1110/1152 (96.41%)	56.2308	20.47	22.177

Table 2: Performance and semantic metrics under sparsification.

t_{i}

: prune every step,

t_{0}

: prune only at start,

t_{n-k,n}

: prune during last

k

steps.

Figure 8: Class-conditioned image generation (tail removal). ImageNet samples with progressive removal of low-magnitude dimensions in $\vec{c}$ (threshold $\tau$ on absolute value). Image quality remains high or better baseline REPA even when 38–¿80% of dimensions are pruned (generated images in the second column), as long as key head dimensions are retained.

Panel of Figure 9 — Figure 9: Variance concentration in conditional vectors $\vec{c}$ . (a) Mean variance across models stays non-zero, showing no collapse. (b) Variance is concentrated in only 15–20 head dimensions (¡2%), while the remaining 98% of tail dimensions show minimal variation, indicating that semantic information is confined to a small subspace.

Figure 10: Pose-conditioned person image generation. DeepFashion samples under different sparsification of $\vec{c}$ . $\tau$ is the magnitude threshold for pruning. Person-consistent images remain high quality even when 50–75% of dimensions are zeroed, as long as key head dimensions are preserved. Best viewed at 200% zoom; more samples appear in the Appendix.

Panel of Figure 11 — Figure 11: Sparsity of conditional embeddings. With $\tau=0.01$ , over 80% of components in the mean conditional vector fall below the threshold, revealing a highly sparse representation that starts to saturate near $\tau=0.02$ .

Figure 12: Training dynamics of conditional embeddings. Top: batchwise cosine similarity of

\vec{c}

during training. Bottom: participation ratio (nPR) over training steps, showing progressive sparsification.

Figure 13: t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from SiT-XL on ImageNet-1K; similar trends appear in other models.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 14: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using DiT-XL ( Peebles and Xie, 2023 ) .

Figure 15: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using LightningDiT-XL ( Yao et al., 2025 ) .

Figure 16: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using MDT-XL ( Gao et al., 2023 ) .

Figure 17: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using MG-XL ( Tang et al., 2025 ) .

Figure 18: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using REPA-XL ( Yu et al., 2025 ) .

Figure 19: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using SiT-XL ( Peebles and Xie, 2023 ) .

Figure 20: (DiT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from DiT-XL on ImageNet-1K; similar trends appear in other models.

Figure 21: (LightningDiT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from LightningDiT-XL on ImageNet-1K; similar trends appear in other models.

Figure 22: (MDT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from MDT-XL on ImageNet-1K; similar trends appear in other models.

Figure 23: (MG) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from MG-XL on ImageNet-1K; similar trends appear in other models.

Figure 24: (REPA) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from REPA-XL on ImageNet-1K; similar trends appear in other models.

Figure 25: (SiT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from SiT-XL on ImageNet-1K; similar trends appear in other models.

Prune	Threshold $\tau$	# Removed Dims	FID ↓	IS ↑	CLIP ↑
Prune	Baseline MG ( Tang et al., 2025 )	0/1152 (0%)	7.2478	174.5151	30.199
Tail	$\tau=0.01$ ( $t_{i}$ )	448/1152 (38.94%)	7.2791	170.55	30.140
	$\tau=0.01$ ( $t_{0}$ )	448/1152 (38.94%)	7.2466	174.5537	30.199
	$\tau=0.01$ ( $t_{n-k,n}$ )	448/1152 (38.94%)	7.2455	174.3103	30.198
.	Baseline LightningDiT ( Yao et al., 2025 )	0/1152 (0%)	7.0802	169.8574	30.720
Tail	$\tau=0.01$ ( $t_{i}$ )	448/1152 (38.94%)	7.0130	166.0569	30.7045

Appendix

Table 3: More baselines. Performance and semantic metrics under sparsification.

t_{i}

: prune every step,

t_{0}

: prune only at start,

t_{n-k,n}

: prune during last

k

steps.

Figure 26: Variance per dimension of the conditional vector learned by MDT-XL.

Figure 27: Variance per dimension of the conditional vector learned by LightningDiT-XL.

Figure 28: Variance per dimension of the conditional vector learned by MG-XL.

Figure 29: Variance per dimension of the conditional vector learned by REPA-XL.

Figure 30: Variance per dimension of the conditional vector learned by SiT-XL.

Figure 31: Variance per dimension of the conditional vector learned by DiT-XL.

Figure 32: Variance per dimension of the conditional vector learned by MDSGen.

Figure 33: Variance per dimension of the conditional vector learned by X-MDPT.

Figure 34: Class-conditional ImageNet generation with pruned embeddings (1). Removing low-magnitude dimensions from $\vec{c}$ preserves or slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.

Figure 35: Class-conditional ImageNet generation with pruned embeddings (2). Removing low-magnitude dimensions from $\vec{c}$ preserves or slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.

Figure 36: Class-conditional ImageNet generation with pruned embeddings (3). Removing low-magnitude dimensions from $\vec{c}$ preserves or slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.

Figure 37: Class-conditional ImageNet generation with pruned embeddings (4). Removing low-magnitude dimensions from $\vec{c}$ preserves or slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.

Figure 38: Pose-guided person image synthesis with pruned embeddings (1). Pruning tail dimensions in $\vec{c}$ maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.

Figure 39: Pose-guided person image synthesis with pruned embeddings (2). Pruning tail dimensions in $\vec{c}$ maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.

Figure 40: Pose-guided person image synthesis with pruned embeddings (3). Pruning tail dimensions in $\vec{c}$ maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.

Figure 41: Pose-guided person image synthesis with pruned embeddings (4). Pruning tail dimensions in $\vec{c}$ maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.

Metrics	DiT 1	SiT 1	MDT 1	LightningDiT 1	MG 1	REPA 1	UViT 2	Embed.
Cosine Sim.	0.9001	0.9852	0.9905	0.9779	0.9934	0.9946	0.97917	y + t
nPR ( $\alpha_{\text{norm}}$ ) %	10.47	2.28	1.60	2.05	1.73	1.43	50.06	y + t
Cosine Sim.	0.7774	0.5436	0.8540	0.7166	0.6853	0.5194	0.00165	y
nPR ( $\alpha_{\text{norm}}$ ) %	70.14	37.43	36.75	36.42	43.67	41.60	63.52	y

Appendix

Table 4: Separate timestep (t) and conditions (y) . Participation Ratio (PR) in learned conditional embeddings of state-of-the-art models on Imagenet-1K class-conditioned generation. With 1 denotes the methods used: AdaLN , and 2 denotes the method used: concatenation .

Metrics	X-MDPT-L 1	X-MDPT-B 1	X-MDPT-S 1	SD3.0 (2B) 1	SD3.0 (8B) 1	MDSGen 1	AudioLDM 3	Embed.
Cosine Sim.	0.9998	0.99992	0.9995	0.9962	0.9995	0.9999	0.9828	y + t
nPR ( $\alpha_{\text{norm}}$ ) %	48.42	37.59	53.41	54.79	26.67	13.57	8.62	y + t
Cosine Sim.	0.9862	0.9909	0.9492	0.9949	0.9937	0.9918	0.1406	y
nPR ( $\alpha_{\text{norm}}$ ) %	20.68	29.75	37.51	52.39	24.25	9.98	63.09	y

Appendix

Table 5: Separate timestep (t) and conditions (y) . Participation Ratio (PR) in learned conditional embeddings of state-of-the-art models on text or video-conditioned generation. With 1 denotes the methods used: AdaLN , and 3 denotes the method used: cross-attention .

Method	FID $\downarrow$	IS $\uparrow$	CLIP $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Remark
REPA ( Yu et al., 2025 )	7.1694	176.02	29.746	0.8032	0.6236	Baseline
Pruned ( $\tau=0.01$ ) $t_{0}$	7.1690	175.97	29.807	0.7878	0.6252	Ours
Pruned ( $\tau=0.01$ ) $t_{n-k,n}$	7.1598	175.49	29.805	0.8045	0.6381	Ours
Model-Guide ( Tang et al., 2025 )	7.2478	174.5151	30.199	0.7842	0.6633	Baseline
Pruned ( $\tau=0.01$ ) $t_{0}$	7.2466	174.5537	30.199	0.7854	0.6625	Ours
Pruned ( $\tau=0.01$ ) $t_{n-k,n}$	7.2455	174.3103	30.198	0.7898	0.6644	Ours

Appendix

Table 6: Precision and Recall with previous metrics: FID, IS, and CLIP.

Method	FID $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	Remark
X-MDPT ( Pham et al., 2024 )	18.6372	0.6798	0.1672	17.336	Baseline
Pruned ( $\tau=0.1$ ) 40%	18.6692	0.6792	0.1675	17.328	Ours

Appendix

Table 7: Quantitative metrics on the DeepFashion dataset of pose-guide person image generation task with masked diffusion transformers.

Figure 42: Dense vs. Sparse vectors. Compared the computation overhead. It shows that a sparse vector is more efficient in computation and has faster runtime than a dense vector (baseline).

Figure 43: Training dynamics of conditional embeddings. Top: batchwise cosine similarity of $\vec{c}$ during training. Bottom: participation ratio (nPR) over training steps, showing progressive sparsification. With limited timesteps: 50, 100, 200, and 500.

Diffusion Transformer

Diffusion Transformers use attention-based networks to generate and transform images, video, audio, and other structured data through iterative denoising.

See all 468 papers

Embedding Prediction Helps Image Generation
Oct 1, 2026Sihan Xu, Ji Xie, Zilin Wang +2Diffusion Transformer Conditional Image Generation
University of Michigan · Carnegie Mellon University
Conditioning Residuals for Diffusion Models via Representation Feedback
May 16, 2025Weilai Xiang, Hongyu Yang, Di Huang +1Representation Learning Diffusion Transformer
State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Institute of Artificial Intelligence, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China
Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformers
May 13, 2026Evelyn Turri, Davide Bucciarelli, Sara Sarto +2Diffusion Transformer Massive Activations
University of Modena and Reggio Emilia, Italy · University of Pisa, Italy

Oct 1, 2026J/K move · Enter open · S save

Embedding Prediction Helps Image Generation

Sihan Xu, Ji Xie, Zilin Wang, Hui Shen+1

University of Michigan · Carnegie Mellon University

Diffusion Transformer Conditional Image Generation

Open paper AskPDF arXiv

In diffusion transformers, a class label or a text prompt is embedded once, and the same condition is reused at every denoising step. We ask whether predicted embeddings can serve as this condition instead. Next-Embedding Predictive Autoregression (NEPA) trains a Transformer to predict the next continuous embedding in a sequence. In generation, the clean image follows the noisy image, so its embeddings are the next embeddings after the condition and the noisy image. We train a NEPA model to predict them all at once with Multi-Embedding Prediction, and in Embedding Conditioned Generation, a DiT generator is conditioned on these predictions, recomputed at every denoising step, so the conditioning signal adapts to the current noisy state. Experiments on class-conditional ImageNet

256\times256

study the condition of the generator, the design of Multi-Embedding Prediction, and the scaling of both models. The NEPA model adds a second network to every sampling step; with it, and combined with REPA, our final model, NEPA-DiT-XL, reaches an FID of 1.32 using about a third of the training compute of REPA.

Abstract

Diffusion Transformer

Explore similar work

Embedding Prediction Helps Image Generation

Conditioning Residuals for Diffusion Models via Representation Feedback

Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformers