Steering Video Diffusion Transformers with Massive Activations

Xianhang Cheng; Yujian Zheng; Zhenyu Xie; Tingting Liao; Hao Li

cs.CVMar 18, 2026

Steering Video Diffusion Transformers with Massive Activations

Authors: Xianhang Cheng, Yujian Zheng, Zhenyu Xie, Tingting Liao, Hao Li

Organizations: MBZUAI · Pinscreen

Abstract

In this work, we study the role of Massive Activations (MAs), which are rare, high-magnitude spikes confined to a few fixed hidden dimensions in video diffusion transformers (DiTs). We uncover a structured positional hierarchy: MA magnitudes peak at first-frame tokens and recur at the spatial boundary tokens of latent frames, with this pattern being most pronounced during early denoising. We trace this organization to an encoding asymmetry of the video VAEs, whose causal temporal padding and zero spatial padding cause the first latent frame and frame borders to carry reduced content load. Elevated MAs consistently align with these lower-content structural positions. To understand their function, we analyze intermediate representations and find that MAs act as implicit rescalers of residual computation: enlarging MAs suppresses the corresponding self-attention and feed-forward updates, while erasing them amplifies these updates. Together, these observations suggest that MAs serves as a token-level rescaler of residual computation, which video DiTs deploy unevenly, placing the strongest damping at the encoding-asymmetric structural positions. Motivated by this native rescaling behavior, we propose Structured Activation Steering (STAS), a training-free technique that steers MAs at the observed structural positions toward a scaled, model-derived reference during early denoising. STAS requires no additional forward passes and consistently improves video quality and temporal coherence across text-to-video models with negligible overhead.

Figures & tables

Panel of Figure 1 — Figure 1: Video DiTs exhibit a structured positional hierarchy of Massive Activations. Activation magnitudes across four video DiTs, averaged over 100 prompts and aggregated in bins of 60 consecutive tokens for visualization. Across all models, MA magnitudes peak at tokens of the first latent frame and recur at the spatial boundaries of subsequent latent frames.

Panel of Figure 2 — Figure 2: The MA hierarchy aligns with video-VAE encoding asymmetries. (a) Latent-frame ablation in Wan2.1 VAE shows that causal temporal padding leaves the first latent frame carrying less content than later frames, precisely where MA magnitudes peak. (b) A uniform video isolates spatial padding: the VAE latent has attenuated boundaries (left), and the DiT MA map (dimension 1188) is elevated at the corresponding boundary tokens (right; shown at token resolution, where each token aggregates a $2{\times}2$ patch of latent cells).

Figure 3: The MA hierarchy is most pronounced during early denoising.

Figure 4: MA magnitude is a token-level rescaler of residual updates.

Figure 5: Steering direction and calibration rule both matter.

Figure 6: User study.

Backbone	Method	FVMD $\downarrow$ [95% CI]
Wan2.1-1.3B	Vanilla	11111.8 [10578.6, 11645.0]
Wan2.1-1.3B	+Ours	10775.8 [10293.5, 11258.1]
Wan2.2-5B	Vanilla	11524.0 [11083.8, 11964.2]
Wan2.2-5B	+Ours	11366.6 [10865.9, 11867.3]
CogVideoX-5B	Vanilla	14647.2 [14027.3, 15267.1]
CogVideoX-5B	+Ours	13291.3 [12873.9, 13708.7]

Table 2: FVMD metric comparisons.

Method	Subject Cons. $\uparrow$	Background Cons. $\uparrow$	Temporal Flicker $\uparrow$	Motion Smooth. $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$	Quality Score $\uparrow$
CFG-Zero ∗	93.43	94.88	97.23	98.43	58.41	62.17	67.03	81.81
+ STAS	93.93	95.09	97.42	98.54	57.69	62.35	68.02	82.21
FlowMo	93.96	96.40	97.38	98.50	56.57	65.51	67.90	82.83
+ STAS	94.12	96.42	97.51	98.55	55.78	65.53	68.68	83.00

Table 4: Compatibility with CFG-Zero ∗ and FlowMo.

Figure 7: Qualitative comparison with and without STAS on different backbones.

Variant	Dim	Target Tokens	$K$	Rule	Sub. Consist.	BG. Consist.	Aesthetic Quality	Imaging Quality	Quality Score
Vanilla	–	–	–	–	94.36	95.68	63.28	67.21	82.01
What (Dimension)
Non-MA	42	first+boundary	20	max	94.33	95.70	63.24	67.23	81.97
Weak-MA	71	first+boundary	20	max	94.24	95.61	63.15	67.22	81.99
Where (Token Position)
First	1188	first	20	max	94.25	95.52	63.24	67.58	82.04

Table 5: Ablation study of STAS on Wan2.1-1.3B.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Block	$k$	Type	Dim	Max Value	Peak-to-Mean	Peak-to-Median
Wan2.1–1.3B 49	15	5	MA	1188	45.4400	58.3	59.6
Wan2.1–1.3B 49	15	5	weak-MA	71	4.8645	6.2	7.5
CogVideoX–5B 56	15	5	MA	1982	412.5200	100.1	127.3
Wan2.2–5B 49	15	5	MA	1938	63.3419	83.8	97.1
Wan2.2–5B 49	15	5	weak-MA	1389	5.0704	6.7	7.8
Wan2.2–5B 49	15	5	weak-MA	2357	4.4530	5.9	6.8

Appendix

Table 6: Model-specific massive activation dimensions across video DiTs.

Model	Blocks	Hidden size	MA dim	Applied layer
Wan2.1-1.3B	30	1536	1188	9
Wan2.2-5B	30	3072	1938	9
CogVideoX-5B	42	3072	1982	8
LTX2 (base)	48	4096	3999	7

Appendix

Table 7: Model configurations used for implementing STAS across backbones.

Model	$\lambda$	$\alpha$	$p$	$K$
	(CFG scale)	(STAS scale)	(boundary %)	(timestep cut-off)
Wan2.1-1.3B	5.0	2.5	8	20
Wan2.2-5B	5.0	2.0	12	20
CogVideoX-5B	6.0	1.2	8	20
LTX2 (base)	3.0	1.2	8	30

Appendix

Table 8: Hyperparameter setup across models.

Figure 8: VAE boundary responses on nonuniform inputs. Natural videos are shown on the left and model generated videos on the right.

Panel of Figure 9 — Figure 9: The MA hierarchy aligns with video-VAE encoding asymmetries on Wan2.2 VAE.

Panel of Figure 10 — Figure 10: The MA hierarchy aligns with video-VAE encoding asymmetries on LTX-2 VAE.

Panel of Figure 11 — Figure 11: MA peaks follow the RoPE coordinates. (a) Native RoPE. (b) Rolling the height RoPE indices by ${+}15$ moves the boundary coordinates to the center rows, and the MA peaks move with them. (c–d) Sharing each temporal RoPE coordinate across groups of $G{=}2$ or $G{=}4$ consecutive latent frames makes the first $G$ frames share the first-frame coordinate and exhibit a common elevated MA plateau. (e) The aggregated spatial profiles show the MA peak moving from the boundary rows to the center rows under the height roll. (f) The normalized temporal profiles show elevated plateaus spanning one, two, and four latent frames under native RoPE, $G{=}2$ , and $G{=}4$ , respectively.

Figure 12: Structural MAs are not uniformly elevated.

Type	Formulation	Applied to	Reference / Target
CFG	$\widehat{D}=D+w\!\left(D-D_{\hat{c}}\right)$	Denoiser output	Unconditional $D_{\hat{c}}$
Autoguidance	$\widehat{D}=D+w\!\left(D-D_{\theta^{*}}\right)$	Denoiser output	Under-capacity $D_{\theta^{*}}$
DG	$\widehat{D}=D+w\!\left(D-D_{\hat{z}}\right)$	Denoiser output	MA-disrupted $D_{\hat{z}}$
STAS	$\widehat{H}=H+\chi\!\left(g-H\right)$	Hidden activation	MA-related adaptive target $g$

Appendix

Table 9: Unified formulations of output guidance and STAS activation steering.

Dimension	Wan2.1-1.3B		CogVideoX-5B		Wan2.2-5B		LTX-2 (base)
Dimension	Vanilla	+Ours	Vanilla	+Ours	Vanilla	+Ours	Vanilla	+Ours
Subject Consistency	94.63	95.00	93.40	93.80	95.13	95.37	92.03	92.11
BG. Consistency	95.81	95.93	95.29	95.47	96.63	96.70	94.73	94.77
Temporal Flickering	97.93	98.09	95.97	96.21	97.81	97.87	97.80	97.80
Motion Smoothness	98.78	98.85	98.18	98.28	98.94	98.94	98.90	98.90
Dynamic Degree	45.15	43.64	48.71	45.85	39.32	38.24	51.74	51.63

Appendix

Table 10: Quantitative comparison on VBench.

Panel of Figure 13 — Figure 13: More ablations. Impact on boundary percentage $p$ , STAS weight $\alpha$ and applied layer depth $m$ . Moderate values achieve the best trade-off across metrics.

Figure 14: A failure case of Dynamic Degree evaluation. Matched frames for “a surfboard and a tennis racket”. In the baseline (top), the tennis racket disappears in the early frames and later reappears together with an additional green racket. Dynamic Degree counts these identity changes as motion and predicts True . With STAS (bottom), one racket persists across all frames and the prediction changes to False .

Figure 15: More qualitative comparisons with and without STAS on Wan2.1-1.3B.

Figure 16: More qualitative comparisons with and without STAS on Wan2.1-1.3B.

Figure 17: More qualitative comparisons with and without STAS on Wan2.2-5B.

Figure 18: More qualitative comparisons with and without STAS on Wan2.2-5B.

Figure 19: More qualitative comparisons with and without STAS on CogvideoX-5B.

Figure 20: More qualitative comparisons with and without STAS on CogvideoX-5B.

Figure 21: More qualitative comparisons with and without STAS on LTX2 (base).

Figure 22: More qualitative comparisons with and without STAS on LTX2 (base).

Massive Activations

See all 14 papers

Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulation
Jul 3, 2026Chaofan Gan, Zicheng Zhao, Yuanpeng Tu +6Visual Representation Learning Diffusion Transformer
Shanghai Jiao Tong University, Shanghai, China · The University of Hong Kong, Hong Kong, China · Monash University, Melbourne, Australia +2
Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation
May 7, 2026Ernie Chu, Vishal M. PatelVideo Diffusion Models Diffusion Transformer
Johns Hopkins University Baltimore, MD 21218
ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers
Jun 22, 2026Ruiliang Zhou, Xuecheng Wu, Kang He +6Video Diffusion Models Diffusion Transformer
KlingAI Research · Beijing Institute of Technology · NVIDIA +1

Jul 3, 2026J/K move · Enter open · S save

Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulation

Chaofan Gan, Zicheng Zhao, Yuanpeng Tu, Xi Chen+5

Shanghai Jiao Tong University, Shanghai, China · The University of Hong Kong, Hong Kong, China · Monash University, Melbourne, Australia+2

Visual Representation Learning Diffusion Transformer Diffusion Guidance Diffusion-Based Image Generation Massive Activations

Open paper AskPDF arXiv

Massive Activations (MAs) have been widely observed in Transformer-based models, yet their structure and functional roles in Diffusion Transformers (DiTs) remain insufficiently understood. In this work, we systematically analyze MAs in representative DiTs and find that they are spatially distributed across image tokens while concentrated in a small set of fixed feature dimensions. We further show that these dimensions are closely aligned with AdaLN residual scaling factors and are primarily modulated by the denoising timestep rather than text conditions. This structure leads to two task-dependent effects: for generation, MAs are critical for fine-grained detail synthesis while having limited influence on global semantics; for understanding, their shared high-magnitude directions make raw DiT features overly similar across spatial tokens and weaken dense feature discrimination. Based on these findings, we introduce Eliciting Massive Activation (EMA), a training-free framework that leverages Massive Activations (MAs) as a unified modulation signal to improve both generative and representational capabilities of DiTs. For generation, EMA proposes MA-driven Detail G}uidance (DG), which suppresses MA dimensions to construct a detail-deficient counterfactual prediction and guides sampling toward finer visual details. DG further supports efficient partial-forward inference, integration with classifier-free guidance, and token-level Local DG for refining selected image regions. For understanding, EMA introduces MA-modulated REPresentation extraction (MREP), which uses pretrained AdaLN channel-wise modulation to reduce MA directional dominance and concatenates spatially normalized MA maps to preserve useful spatial structure. Extensive experiments demonstrate that EMA consistently improves both the generation quality and representation capability of DiTs.

Abstract

Massive Activations

Explore similar work

Awakening Diffusion Transformers: Eliciting Stronger Generation and Understanding via Massive Activation Modulation

Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers