A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers

Trung X. Pham; Kang Zhang; Ji Woo Hong; Chang D. Yoo

cs.CVFeb 25, 2026

A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers

Authors: Trung X. Pham, Kang Zhang, Ji Woo Hong, Chang D. Yoo

Organizations: Korea Advanced Institute of Science and Technology (KAIST)

Abstract

Diffusion Transformers have achieved state-of-the-art performance in class-conditional and multimodal generation, yet the structure of their learned conditional embeddings remains poorly understood. In this work, we present the first systematic study of these embeddings and uncover a notable redundancy: class-conditioned embeddings exhibit extreme angular similarity, exceeding 99% on ImageNet-1K, while continuous-condition tasks such as pose-guided image generation and video-to-audio generation reach over 99.9%. We further find that semantic information is concentrated in a small subset of dimensions, with head dimensions carrying the dominant signal and tail dimensions contributing minimally. By pruning low-magnitude dimensions--removing up to two-thirds of the embedding space--we show that generation quality and fidelity remain largely unaffected, and in some cases improve. These results reveal a semantic bottleneck in Transformer-based diffusion models, providing new insights into how semantics are encoded and suggesting opportunities for more efficient conditioning mechanisms.

Figures & tables

Figure 1: Hidden Semantic Bottleneck: Extreme Alignment and Dimensional Sparsity. Conditional vectors

\vec{c}

in state-of-the-art Transformer diffusion models on ImageNet-1K exhibit very high pairwise cosine similarity (mostly 90–99%) while concentrating semantic information in only a few of 1,152 dimensions.

Figure 2: Transformer-based diffusion models inject conditions as a globally compact vectors $\vec{v}_{i}$ via AdaLN for outputs such as images or mel-spectrograms.

Figure 3: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using REPA-XL ( Yu et al., 2025 ) . Despite distinct semantics, embeddings show over 99% similarity for nearly all class pairs. Left : full $1000\!\times\!1000$ matrix showing global alignment. Right : zoomed $10\!\times\!10$ subset for randomly chosen classes. Additional results for other SOTA methods appear in the Appendix.

Figure 4: Sparsity and alignment of conditional embeddings in X-MDPT ( Pham et al., 2024 ) . (a) and (b): With $\tau=0.1$ , over 51% of components in the conditional vectors have magnitudes below the threshold, highlighting significant sparsity. Remarkably, pruning these dimensions has minimal effect on generation quality. (c) Cosine similarity between random test samples in DeepFashion exceeds 99.9%, confirming extreme alignment across conditional embeddings.

Figure 5: Magnitude histogram distribution of learned conditional vector embedding

\vec{c}\in\mathbb{R}^{\times 1152}

. Most dimensions have near-zero values (

<0.01

), with only

\sim 5-20

dimensions showing dominant magnitudes. This sparsity holds across multiple models, including DiT, MDT, LightningDiT, MG, SiT, and REPA. It is best viewed with 300% zoomed in.

Figure 6: Original distribution of learned conditional vector embedding

\vec{c}\in\mathbb{R}^{1\times 1152}

. Most dimensions have near-zero values (

<0.01

), with only

\sim 5-20

dimensions showing dominant magnitudes. This sparsity holds across multiple models, including DiT, MDT, LightningDiT, MG, SiT, and REPA. It is best viewed with 200% zoomed in.

Embedding Metrics	DiT	SiT	MDT	LightningDiT	MG	REPA	X-MDPT	MDSGen
Condition Dim ( $d$ )	1152	1152	1152	1152	1152	1152	1024	768
PR ( $\alpha$ )	120.69	26.25	18.45	23.70	19.98	17.67	495.75	104.22
nPR ( $\alpha_{\text{norm}}$ )	10.47%	2.28%	1.60%	2.05%	1.73%	1.53%	48.42%	13.57%
Cosine Sim. ( $cs$ )	0.9001	0.9852	0.9905	0.9779	0.9934	0.9946	0.9998	0.9999

Table 1: Participation Ratio (PR) in learned conditional embeddings of state-of-the-art models on Imagenet-1K generation (discrete) and DeepFashion/VGGSound (continuous).

Figure 7: Class-conditioned image generation with head removal. ImageNet samples after pruning top-magnitude dimensions of $\vec{c}$ (threshold $\tau$ ); removing only a few head dimensions markedly degrades quality.

Prune	Threshold $\tau$	# Removed Dims	FID ↓	IS ↑	CLIP ↑
Prune	Baseline (REPA)	0/1152 (0%)	7.1694	176.02	29.746
	$\tau=0.01$ ( $t_{i}$ )	448/1152 (38.94%)	7.2143	171.99	29.737
	$\tau=0.01$ ( $t_{0}$ )	448/1152 (38.94%)	7.1690	175.97	29.807
Tail	$\tau=0.01$ ( $t_{n-k,n}$ )	448/1152 (38.94%)	7.1598	175.49	29.805
	$\tau=0.02$ ( $t_{i}$ )	762/1152 (66.21%)	9.2202	125.15	29.221
	$\tau=0.05$ ( $t_{i}$ )	1110/1152 (96.41%)	56.2308	20.47	22.177

Table 2: Performance and semantic metrics under sparsification.

t_{i}

: prune every step,

t_{0}

: prune only at start,

t_{n-k,n}

: prune during last

k

steps.

Figure 8: Class-conditioned image generation (tail removal). ImageNet samples with progressive removal of low-magnitude dimensions in $\vec{c}$ (threshold $\tau$ on absolute value). Image quality remains high or better baseline REPA even when 38–¿80% of dimensions are pruned (generated images in the second column), as long as key head dimensions are retained.

Panel of Figure 9 — Figure 9: Variance concentration in conditional vectors $\vec{c}$ . (a) Mean variance across models stays non-zero, showing no collapse. (b) Variance is concentrated in only 15–20 head dimensions (¡2%), while the remaining 98% of tail dimensions show minimal variation, indicating that semantic information is confined to a small subspace.

Figure 10: Pose-conditioned person image generation. DeepFashion samples under different sparsification of $\vec{c}$ . $\tau$ is the magnitude threshold for pruning. Person-consistent images remain high quality even when 50–75% of dimensions are zeroed, as long as key head dimensions are preserved. Best viewed at 200% zoom; more samples appear in the Appendix.

Panel of Figure 11 — Figure 11: Sparsity of conditional embeddings. With $\tau=0.01$ , over 80% of components in the mean conditional vector fall below the threshold, revealing a highly sparse representation that starts to saturate near $\tau=0.02$ .

Figure 12: Training dynamics of conditional embeddings. Top: batchwise cosine similarity of

\vec{c}

during training. Bottom: participation ratio (nPR) over training steps, showing progressive sparsification.

Figure 13: t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from SiT-XL on ImageNet-1K; similar trends appear in other models.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 14: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using DiT-XL ( Peebles and Xie, 2023 ) .

Figure 15: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using LightningDiT-XL ( Yao et al., 2025 ) .

Figure 16: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using MDT-XL ( Gao et al., 2023 ) .

Figure 17: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using MG-XL ( Tang et al., 2025 ) .

Figure 18: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using REPA-XL ( Yu et al., 2025 ) .

Figure 19: Cosine similarity of conditional vectors $\vec{c}=y+t$ across 1000 ImageNet classes using SiT-XL ( Peebles and Xie, 2023 ) .

Figure 20: (DiT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from DiT-XL on ImageNet-1K; similar trends appear in other models.

Figure 21: (LightningDiT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from LightningDiT-XL on ImageNet-1K; similar trends appear in other models.

Figure 22: (MDT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from MDT-XL on ImageNet-1K; similar trends appear in other models.

Figure 23: (MG) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from MG-XL on ImageNet-1K; similar trends appear in other models.

Figure 24: (REPA) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from REPA-XL on ImageNet-1K; similar trends appear in other models.

Figure 25: (SiT) t-SNE of class embeddings by head vs. tail dimensions. Keeping only head dimensions (b) preserves clear class clusters similar to the full embedding (a), while tail-only embeddings (c,d) collapse into entangled points, revealing weak semantic structure. Results are from SiT-XL on ImageNet-1K; similar trends appear in other models.

Prune	Threshold $\tau$	# Removed Dims	FID ↓	IS ↑	CLIP ↑
Prune	Baseline MG ( Tang et al., 2025 )	0/1152 (0%)	7.2478	174.5151	30.199
Tail	$\tau=0.01$ ( $t_{i}$ )	448/1152 (38.94%)	7.2791	170.55	30.140
	$\tau=0.01$ ( $t_{0}$ )	448/1152 (38.94%)	7.2466	174.5537	30.199
	$\tau=0.01$ ( $t_{n-k,n}$ )	448/1152 (38.94%)	7.2455	174.3103	30.198
.	Baseline LightningDiT ( Yao et al., 2025 )	0/1152 (0%)	7.0802	169.8574	30.720
Tail	$\tau=0.01$ ( $t_{i}$ )	448/1152 (38.94%)	7.0130	166.0569	30.7045

Appendix

Table 3: More baselines. Performance and semantic metrics under sparsification.

t_{i}

: prune every step,

t_{0}

: prune only at start,

t_{n-k,n}

: prune during last

k

steps.

Figure 26: Variance per dimension of the conditional vector learned by MDT-XL.

Figure 27: Variance per dimension of the conditional vector learned by LightningDiT-XL.

Figure 28: Variance per dimension of the conditional vector learned by MG-XL.

Figure 29: Variance per dimension of the conditional vector learned by REPA-XL.

Figure 30: Variance per dimension of the conditional vector learned by SiT-XL.

Figure 31: Variance per dimension of the conditional vector learned by DiT-XL.

Figure 32: Variance per dimension of the conditional vector learned by MDSGen.

Figure 33: Variance per dimension of the conditional vector learned by X-MDPT.

Figure 34: Class-conditional ImageNet generation with pruned embeddings (1). Removing low-magnitude dimensions from $\vec{c}$ preserves or slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.

Figure 35: Class-conditional ImageNet generation with pruned embeddings (2). Removing low-magnitude dimensions from $\vec{c}$ preserves or slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.

Figure 36: Class-conditional ImageNet generation with pruned embeddings (3). Removing low-magnitude dimensions from $\vec{c}$ preserves or slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.

Figure 37: Class-conditional ImageNet generation with pruned embeddings (4). Removing low-magnitude dimensions from $\vec{c}$ preserves or slightly improves image quality, confirming that semantic information is concentrated in a few head dimensions.

Figure 38: Pose-guided person image synthesis with pruned embeddings (1). Pruning tail dimensions in $\vec{c}$ maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.

Figure 39: Pose-guided person image synthesis with pruned embeddings (2). Pruning tail dimensions in $\vec{c}$ maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.

Figure 40: Pose-guided person image synthesis with pruned embeddings (3). Pruning tail dimensions in $\vec{c}$ maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.

Figure 41: Pose-guided person image synthesis with pruned embeddings (4). Pruning tail dimensions in $\vec{c}$ maintains pose fidelity and visual quality, highlighting the redundancy of low-magnitude embedding dimensions.

Metrics	DiT 1	SiT 1	MDT 1	LightningDiT 1	MG 1	REPA 1	UViT 2	Embed.
Cosine Sim.	0.9001	0.9852	0.9905	0.9779	0.9934	0.9946	0.97917	y + t
nPR ( $\alpha_{\text{norm}}$ ) %	10.47	2.28	1.60	2.05	1.73	1.43	50.06	y + t
Cosine Sim.	0.7774	0.5436	0.8540	0.7166	0.6853	0.5194	0.00165	y
nPR ( $\alpha_{\text{norm}}$ ) %	70.14	37.43	36.75	36.42	43.67	41.60	63.52	y

Appendix

Table 4: Separate timestep (t) and conditions (y) . Participation Ratio (PR) in learned conditional embeddings of state-of-the-art models on Imagenet-1K class-conditioned generation. With 1 denotes the methods used: AdaLN , and 2 denotes the method used: concatenation .

Metrics	X-MDPT-L 1	X-MDPT-B 1	X-MDPT-S 1	SD3.0 (2B) 1	SD3.0 (8B) 1	MDSGen 1	AudioLDM 3	Embed.
Cosine Sim.	0.9998	0.99992	0.9995	0.9962	0.9995	0.9999	0.9828	y + t
nPR ( $\alpha_{\text{norm}}$ ) %	48.42	37.59	53.41	54.79	26.67	13.57	8.62	y + t
Cosine Sim.	0.9862	0.9909	0.9492	0.9949	0.9937	0.9918	0.1406	y
nPR ( $\alpha_{\text{norm}}$ ) %	20.68	29.75	37.51	52.39	24.25	9.98	63.09	y

Appendix

Table 5: Separate timestep (t) and conditions (y) . Participation Ratio (PR) in learned conditional embeddings of state-of-the-art models on text or video-conditioned generation. With 1 denotes the methods used: AdaLN , and 3 denotes the method used: cross-attention .

Method	FID $\downarrow$	IS $\uparrow$	CLIP $\uparrow$	Precision $\uparrow$	Recall $\uparrow$	Remark
REPA ( Yu et al., 2025 )	7.1694	176.02	29.746	0.8032	0.6236	Baseline
Pruned ( $\tau=0.01$ ) $t_{0}$	7.1690	175.97	29.807	0.7878	0.6252	Ours
Pruned ( $\tau=0.01$ ) $t_{n-k,n}$	7.1598	175.49	29.805	0.8045	0.6381	Ours
Model-Guide ( Tang et al., 2025 )	7.2478	174.5151	30.199	0.7842	0.6633	Baseline
Pruned ( $\tau=0.01$ ) $t_{0}$	7.2466	174.5537	30.199	0.7854	0.6625	Ours
Pruned ( $\tau=0.01$ ) $t_{n-k,n}$	7.2455	174.3103	30.198	0.7898	0.6644	Ours

Appendix

Table 6: Precision and Recall with previous metrics: FID, IS, and CLIP.

Method	FID $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	Remark
X-MDPT ( Pham et al., 2024 )	18.6372	0.6798	0.1672	17.336	Baseline
Pruned ( $\tau=0.1$ ) 40%	18.6692	0.6792	0.1675	17.328	Ours

Appendix

Table 7: Quantitative metrics on the DeepFashion dataset of pose-guide person image generation task with masked diffusion transformers.

Figure 42: Dense vs. Sparse vectors. Compared the computation overhead. It shows that a sparse vector is more efficient in computation and has faster runtime than a dense vector (baseline).

Figure 43: Training dynamics of conditional embeddings. Top: batchwise cosine similarity of $\vec{c}$ during training. Bottom: participation ratio (nPR) over training steps, showing progressive sparsification. With limited timesteps: 50, 100, 200, and 500.

Diffusion Transformer

Diffusion Transformers use attention-based networks to generate and transform images, video, audio, and other structured data through iterative denoising.

See all 468 papers

Oct 1, 2026

Embedding Prediction Helps Image Generation

In diffusion transformers, a class label or a text prompt is embedded once, and the same condition is reused at every denoising step. We ask whether predicted embeddings can serve as this condition instead. Next-Embedding Predictive Autoregression (NEPA) trains a Transformer to predict the next continuous embedding in a sequence. In generation, the clean image follows the noisy image, so its embeddings are the next embeddings after the condition and the noisy image. We train a NEPA model to predict them all at once with Multi-Embedding Prediction, and in Embedding Conditioned Generation, a DiT generator is conditioned on these predictions, recomputed at every denoising step, so the conditioning signal adapts to the current noisy state. Experiments on class-conditional ImageNet

256\times256

study the condition of the generator, the design of Multi-Embedding Prediction, and the scaling of both models. The NEPA model adds a second network to every sampling step; with it, and combined with REPA, our final model, NEPA-DiT-XL, reaches an FID of 1.32 using about a third of the training compute of REPA.

Sihan Xu, Ji Xie, Zilin Wang +2

University of Michigan · Carnegie Mellon University

Diffusion Transformer Conditional Image Generation

Ask PDF

May 16, 2025

Conditioning Residuals for Diffusion Models via Representation Feedback

Diffusion models now serve as a common foundation for multimedia generation, and useful intermediate representations emerge during their generative training. Standard architectures, however, propagate these representations through the main feature stream, without explicitly reintroducing their encoded semantics to later denoising layers. Meanwhile, such backbones already provide a conditioning pathway for global modulation by predefined inputs. This work examines whether this native pathway can also route internally inferred semantics as evolving, sample-dependent cues. We propose Conditioning Residuals, a lightweight feedback mechanism that converts aggregated features into residuals added to condition embeddings. By feeding back compact feature summaries, it provides adaptive generative guidance and encourages a tighter semantic bottleneck, without external encoders, auxiliary objectives, or sampling-time changes. It supports feedback at one or multiple depths in UNet and DiT backbones, with negligible overhead. Across diffusion formulations, backbone configurations, and datasets, experiments show consistent gains in generative performance, along with stronger representations in downstream linear probing and segmentation. Mechanistic analyses reveal improved generative training dynamics and reshaped feature structure, suggesting a grounded, generalizable way to enhance diffusion backbones from within.

Weilai Xiang, Hongyu Yang, Di Huang +1

State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Institute of Artificial Intelligence, Beihang University, Beijing 100191, China · School of Computer Science and Engineering, Beihang University, Beijing 100191, China

Representation Learning Diffusion Transformer Conditional Diffusion Models

Ask PDF

May 13, 2026

Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformers

Diffusion Transformers (DiTs) and related flow-based architectures are now among the strongest text-to-image generators, yet the internal mechanisms through which prompts shape image semantics remain poorly understood. In this work, we study massive activations: a small subset of hidden-state channels whose responses are consistently much larger than the rest. We show that, despite their sparsity, these few channels effectively draw the whole picture, in three complementary senses. First, they are functionally critical: a controlled disruption probe that zeroes the massive channels causes a sharp collapse in generation quality, while disrupting an equally-sized set of low-statistic channels has marginal effect. Second, they are spatially organized: restricting image-stream tokens to massive channels and clustering them yields coherent partitions that closely align with the main subject and salient regions, exposing a structured spatial code hidden inside an apparently outlier-like subspace. Third, they are transferable: transporting massive activations from one prompt-conditioned trajectory into another, shifts the final image toward the source prompt while preserving substantial content from the target, producing localized semantic interpolation rather than unstructured pixel blending. We exploit this property in two use cases: text-conditioned and image-conditioned semantic transport, where massive activations transport enables prompt interpolation and subject-driven generation without any additional training. Together, these results recast massive activations not as activation anomalies, but as a sparse prompt-conditioned carrier subspace that organizes and controls semantic information in modern DiT models.

Evelyn Turri, Davide Bucciarelli, Sara Sarto +2

University of Modena and Reggio Emilia, Italy · University of Pisa, Italy

Diffusion Transformer Massive Activations Activation Steering Controllable Image Generation Diffusion Models

Ask PDF

Abstract

Diffusion Transformer

Explore similar work

Embedding Prediction Helps Image Generation

Conditioning Residuals for Diffusion Models via Representation Feedback

Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformers