A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs

Raghavv Goel; Risheek Garrepalli; Sudhanshu Agrawal; Chris Lott; Mingu Lee; Fatih Porikli

cs.CLMar 8, 2026

A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs

Authors: Raghavv Goel, Risheek Garrepalli, Sudhanshu Agrawal, Chris Lott, Mingu Lee, Fatih Porikli

Organizations: Qualcomm AI Research, Qualcomm Technologies Inc.

Abstract

Autoregressive (AR) language models build representations incrementally via left-to-right prediction, while diffusion language models (dLLMs) are trained through full-sequence denoising. Although recent dLLMs match AR performance, whether diffusion objectives fundamentally reshape internal representations remains unclear. We perform the first layer- and token-wise representational analysis comparing native dLLMs (LLaDA), native AR models (Qwen2.5), and AR-initialized dLLMs (Dream-7B), using cosine similarity across layers and tokens alongside static inference-time layer-skipping as an analytical probe of redundancy. We find that diffusion objectives produce more global representations with substantial early-layer redundancy and reduced recency bias, while AR objectives yield tightly coupled, locally structured representations. AR-initialized dLLMs retain AR-like dynamics despite diffusion training, revealing persistent initialization bias. Leveraging this redundancy, native dLLMs absorb up to 18.75% FLOPs reduction while retaining over 90% performance on math-reasoning and coding benchmarks, whereas AR models collapse under identical skipping, revealing that diffusion objectives, rather than architecture alone, induce depth redundancy that enables principled compression.

Figures & tables

Panel of Figure 1 — Figure 1: Representational redundancy in native dLLMs enables efficient inference-time layer skipping. Left: Layer-wise cosine similarity reveals that LLaDA develops a high-similarity plateau in early layers—objective-induced redundancy absent in AR models (Qwen2.5) and AR-initialized dLLMs (Dream-7B). Right: This structural redundancy directly enables skip tolerance: native dLLMs absorb aggressive layer skipping with minimal quality loss, while AR models degrade sharply under identical conditions.

Figure 2: Layer-wise cosine similarity across models 32 tokens decoded. Each row shows similarity between consecutive layers for (top) LLaDA, (middle) Qwen2.5, and (bottom) Dream-7B. High-similarity regions (yellow) indicate representational redundancy. Dream-7B’s pattern closely resembles Qwen2.5 despite diffusion training, revealing strong initialization bias.

Figure 3: Layer-wise cosine similarity across models (128 tokens decoded). Panels show similarity between consecutive layers for (left) LLaDA, (middle) Dream-7B, and (right) Qwen2.5. High-similarity regions (yellow) indicate representational redundancy. LLaDA exhibits a sharp two-regime structure : layers 1 to 13 form a near uniform high similarity plateau ( $>0.95$ ), while layers 14 to 31 show progressively more texture, with the lowest similarity concentrated in the final layers (24 to 31) where decoder-like refinement occurs. Dream-7B and Qwen2.5 lack this plateau entirely, and lower similarity bands emerge as early as layers 2 to 6 and persist through the last layer, closely tracking each other and confirming Dream-7B’s strong initialization bias toward Qwen2.5’s AR-like structure.

Panel of Figure 4 — Figure 4: Token-wise cosine similarity across layers and models. Rows correspond to layers ( $1,8,16,24$ ); columns show (left) LLaDA, (middle) Dream-7B, and (right) Qwen. Left: decoding limited to 32 tokens highlights early representational stabilization in native diffusion models. Right: full-sequence decoding emphasizes global context integration and architectural differences across objectives.

Figure 5: Hidden-state magnitude across depth. Layer-wise evolution of the $\ell_{2}$ norm of token hidden states for LLaDA , Dream, and Qwen. Norms remain relatively stable through the first $\sim$ 60–70% of layers and increase sharply near the top of the network. The maximum norm is dominated by rare sink tokens (spikes; often $\geq 10^{3}$ ), so max values should be interpreted as outliers rather than typical token magnitudes.

Task	Layers Skipped	Models
Task	Layers Skipped	LLada-8B-Instruct	Dream-7B-base	Dream-7B-Instruct	Qwen2.5-7B-Instruct
GSM8K	0 (baseline)	100% (0.79)	100% (0.74)	100% (0.80)	100% (0.62)
	2	95% (0.75)	101% (0.75)	91% (0.73)	82% (0.51)
	4	94% (0.74)	68% (0.50)	18% (0.14)	42% (0.26)
	6	94% (0.74)	22% (0.16)	4% (0.03)	6% (0.04)
MATH500	0 (baseline)	100% (0.38)	100% (0.36)	100% (0.48)	100% (0.41)

Table 1: Performance comparison across models and layer skipping configurations. Values represent retention percentages relative to the 0-layer baseline, with absolute accuracy shown in parentheses for baseline rows. Retention values should be interpreted alongside absolute accuracies: large percentage swings on tasks where baseline accuracy is near floor (e.g.,

\leq

0.10) reflect sampling noise rather than meaningful robustness, as small absolute fluctuations produce arbitrarily large percentage changes.

Skip	GSM8K				HumanEval
Skip	Ours	Top-k	Unif	Rand	Ours	Top-k	Unif	Rand
6	94%	95%	87%	77%	83%	70%	55%	36%
8	94%	40%	71%	62%	60%	8%	30%	21%

Table 2: Comparison of Layer Skipping Strategies (Accuracy Retention) for LLaDA-8B-Instruct where Top-K is consecutive layers, Unif skips layers uniformly, and Rand skips random layers

Figure 6: Which layers are skipped? (6-layer skip) Distribution over layer indices on LLaDA (HumanEval). Disallowing consecutive skips concentrates selections in early layers (first 40–60%).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Layers Skipped	Models
Task	Layers Skipped	LLada-8B-Instruct	Dream-7B-base	Dream-7B-Instruct	Qwen2.5-7B-Instruct
GSM8K	0 (baseline)	100% (0.79)	100% (0.74)	100% (0.80)	100% (0.62)
	1	101% (0.80)	99% (0.73)	99% (0.79)	106% (0.66)
	2	95% (0.75)	101% (0.75)	91% (0.73)	82% (0.51)
	3	100% (0.79)	77% (0.57)	71% (0.57)	79% (0.49)
	4	94% (0.74)	68% (0.50)	18% (0.14)	42% (0.26)

Appendix

Table 3: Model Performance Retention Rate Across Layer Skipping. Values represent retention percentages relative to baseline (0 layers skipped), with absolute accuracy shown in parentheses for baseline rows.

Task	Metric	Layers Skipped
Task	Metric	0 (baseline)	2	4	6	8
GSM8K	Accuracy retention	100%	95%	94%	94%	94%
GSM8K	Wall time reduction	0%	-8%	2%	4%	10%
MATH500	Accuracy retention	100%	108%	87%	97%	66%
MATH500	Wall time reduction	0%	-3%	2%	10%	29%
HumanEval	Accuracy retention	100%	96%	91%	83%	60%

Appendix

Table 4: LLada-8B-Instruct Performance and Computation Savings (wall-time reduction) on various downstream tasks

Layers Skipped	Retention Score
0 (baseline)	100 (0.78)
2	104 (0.81)
3	101 (0.79)
4	88 (0.69)
5	77 (0.60)
6	68 (0.53)

Appendix

Table 5: Llada-8B-Instruct evaluated on GSM8K with Layer Skipping and Dual Cache Method

Panel of Figure 7 — Figure 7: Token-wise similarity across early and early-middle layers. (a) Layers 0–7 and (b) layers 8–15.

Panel of Figure 8 — Figure 8: Token-wise similarity in later layers. (a) Late-middle (16–23) and (b) late (24–31).

Figure 9: Token-wise cosine similarity across all layers for LLaDA. Each subplot shows the cosine similarity between consecutive token representations ( $\mathbf{h}_{\ell,i}$ and $\mathbf{h}_{\ell,i+1}$ ) within a specific layer $\ell$ . High similarity indicates smooth representational transitions, while low similarity indicates significant representational changes between tokens. LLaDA exhibits consistently high token-wise similarity across early layers, demonstrating minimal recency bias and global representational abstraction. Later layers show increased variability, indicating task-specific refinement and decoder-like behavior. This pattern validates our hypothesis that native dLLMs develop coarse-to-fine abstraction hierarchies, with early layers establishing stable global context and later layers performing iterative refinement.

Figure 10: Token-wise cosine similarity across all layers for Dream-7B. Each subplot shows the cosine similarity between consecutive token representations within a specific layer. In stark contrast to LLaDA (Figure 9 ), Dream-7B exhibits significant recency bias across all layers, with substantial representational changes for each new token often. The lack of hierarchical abstraction—with similar token-by-token update patterns across all depths—confirms that Dream-7B retains AR-like incremental refinement and retain different representational abstraction compared to native dLLMs. This provides mechanistic evidence for the initialization bias observed in our layer-skip experiments (Table 1 ), where Dream-7B’s brittleness mirrors Qwen2.5 despite diffusion training.

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 445 papers

Layer Collapse in Diffusion Language Models
May 7, 2026Alexander Conzelmann, Albert Catalan-Tatjer, Shiwei LiuLLM Compression Diffusion Language Models
Tübingen AI Center · Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen
Enhancing Diffusion Language Models with Autoregressive Post-Training Weights
Oct 6, 2026Yiming Qin, Ke Wang, Amel Abdelraheem +2Diffusion Language Model Training Model Merging
EPFL, Lausanne, Switzerland
DARE: Diffusion Language Model Activation Reuse for Efficient Inference
May 1, 2026Natalia Frumkin, Bokun Wang, Hung-Yueh Chiang +3Self-Attention Diffusion Language Model Inference
Chandra Family Department of Electrical and Computer Engineering The University of Texas at Austin · Cornell University

May 7, 2026J/K move · Enter open · S save

Layer Collapse in Diffusion Language Models

Alexander Conzelmann, Albert Catalan-Tatjer, Shiwei Liu

Tübingen AI Center · Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen

LLM Compression Diffusion Language Models

Open paper AskPDF arXiv

Diffusion language models (DLMs) have recently emerged as competitive alternatives to autoregressive (AR) language models, yet differences in their activation dynamics remain poorly understood. We characterize these dynamics in LLaDA-8B and identify a striking layer-collapse property: a few early layers exhibit highly similar, collapsed activation patterns dominated by a single large super-outlier persisting over a long token range. Despite its apparent redundancy, this outlier is critical: pruning it causes outputs to degrade into repetitive random token loops. Paradoxically, layers in LLaDA contain more redundant representations overall, with redundancy most pronounced in earlier layers -- the reverse of AR models, where deeper layers grow redundant due to undertraining. Our analysis indicates that layer collapse in DLMs is not driven by undertraining but by overtraining: a dominant outlier becomes an indispensable information carrier while remaining representations collapse into redundant structure. These findings have strong practical implications, verified through controlled pre-training experiments. DLMs are surprisingly robust to compression: LLaDA under 3-bit GPTQ quantization drops only -1.8% on GSM8K, whereas Llama-3.1-8B drops -64.7%. Optimal sparsity allocation also reverses between families: at 50% average sparsity, allocating more to early layers in LLaDA yields +8.4% over the reverse strategy, while the same allocation costs Llama -8.4%. Our findings reveal that the DLM training objective fundamentally reshapes layer dynamics relative to AR models, with direct consequences for compression and deployment. Code: github.com/Conzel/super-outlier-dlm.

Abstract

Diffusion Language Models

Explore similar work

Layer Collapse in Diffusion Language Models

Enhancing Diffusion Language Models with Autoregressive Post-Training Weights

DARE: Diffusion Language Model Activation Reuse for Efficient Inference