Diffusion transformers have demonstrated remarkable capabilities in generating videos. However, their practical deployment is severely constrained by high memory usage and computational cost. Post-Training Quantization provides a practical way to reduce memory usage and boost computation speed. Existing quantization methods typically apply a static bit-width allocation, overlooking the quantization difficulty of activations across diffusion timesteps, leading to a suboptimal trade-off between efficiency and quality. In this paper, we propose a inference time NVFP4/INT8 Mixed-Precision Quantization framework. We find a strong linear correlation between a block's input-output difference and the quantization sensitivity of its internal linear layers. Based on this insight, we design a lightweight predictor that dynamically allocates NVFP4 to temporally stable layers to maximize memory compression, while selectively preserving INT8 for volatile layers to ensure robustness. This adaptive precision strategy enables aggressive quantization without compromising generation quality. Beside this, we observe that the residual between the input and output of a Transformer block exhibits high temporal consistency across timesteps. Leveraging this temporal redundancy, we introduce Temporal Delta Cache (TDC) to skip computations for these invariant blocks, further reducing the computational cost. Extensive experiments demonstrate that our method achieves 1.92× end-to-end acceleration and 3.32× memory reduction, setting a new baseline for efficient inference in Video DiTs.
Figures & tables
Figure 1: Relative L2 quantization error(defined in Eq. 4 ) of individual linear layers in CogVideoX [ 38 ] across denoising timesteps. The severe temporal fluctuations demonstrate that activation sensitivity to quantization is highly dynamic, highlighting the limitations of static mixed-precision policies.
Figure 2: Overview of our proposed methods. We using DMPQ to decide the quantization bits of each linear layer activation based on the block preceding timestep input output relative L1 loss Γ . And using TDC to decide whether to skip block computation based on preceding two timesteps block updates Δ .
Figure 3: The linear relationship between the block-level input-output relative L1 distance at the previous timestep ( Γt−1 ) and the layer-wise relative quantization error ( Erel ) at the current timestep. The data points are collected by executing the diffusion model over a calibration set. For each block, we extract the time-shifted valid data pairs across all timesteps to perform linear regression ( y=αx+β ) for each internal linear layer.
Figure 4: Temporal redundancy of Transformer block updates in Video DiTs. We visualize the Cosine Similarity (left) and Relative L2 Difference (right) of the residual updates ( Δtl and Δt−1l ) between adjacent timesteps. The inherently high similarity across most timesteps and layers motivates our adaptive Temporal Delta Cache.
Figure 5: Visual comparisons between ours and FP16 baseline [ 38 ] ), together with quantization methods [ 1 ] , SmoothQuant [ 37 ] and ViDiT-Q [ 51 ]
Method
Bit-width
Aesthetic
BG.
Overall
Dynamic
Subject
Imaging
Scene
Motion
(W/A)
Quality
Consist.
Consist.
Degree
Consist.
Quality
Consist.
Smooth.
CogVideo-2B
Original
W16/A16
0.5464
0.9500
0.2423
0.7639
0.9491
0.5713
0.3590
0.9714
SmoothQuant
W4A8
0.5332
0.9441
0.2473
0.6806
0.9319
0.5460
0.3343
0.9680
QuaRot
W4A8
0.5158
0.9427
0.2383
0.5139
0.9202
0.5258
0.2682
0.9715
ViDiT-Q
W4A8
0.5332
0.9449
0.2464
0.6528
0.9272
0.5534
0.3372
0.9692
Table 1: Performance comparison of various quantization methods on VBench using CogVideo-2B and CogVideo-5B.
Method
Bit-width
Speedup
Aesthetic
BG.
Overall
Dynamic
Subject
Imaging
Scene
Motion
(W/A)
Quality
Consist.
Consist.
Degree
Consist.
Quality
Consist.
Smooth.
Original
W16A16
1.00×
0.5464
0.9500
0.2423
0.7639
0.9491
0.5713
0.3590
0.9714
Ours(DMPQ)
W4A6
1.36×
0.5437
0.9455
0.2474
0.6806
0.9282
0.5511
0.3307
0.9691
Ours(Full)
W4A6 ∗
1.92×
0.5366
0.9421
0.2492
0.7083
0.9255
0.5369
0.3307
0.9691
Table 2: Performance evaluation of integrating TDC into DMPQ on VBench [ 9 ] .
Method
DMPQ
TDC
PDR
VQA-A ↑
VQA-T ↑
CLIP ↑
CLIP-T ↑
Flow Score ↑
Original (FP16)
-
-
-
74.6175
77.7548
0.1824
0.9973
5.8389
Uniform W4A4
-
-
-
68.9105
73.6116
0.1818
0.9974
4.2518
Uniform W4A8
-
-
-
71.2093
78.6070
0.1859
0.9972
5.3140
Ablation 1
✓
75.5332
77.8404
0.1814
0.9976
5.3266
Ablation 2
✓
72.2671
78.1517
0.1838
0.9975
5.7602
Ablation 3
✓
✓
73.7243
77.2833
0.1835
0.9978
4.7271
Table 3: Ablation Study of Proposed Components. Evaluation of Dynamic Mixed Precision Quantization (DMPQ), Temporal Delta Cache (TDC), and Purified Delta Refresh (PDR).
Diffusion model quantization compresses model weights and activations to reduce inference memory and latency while preserving generated image, video, and speech quality.
Diffusion Transformers (DiTs) achieve state-of-the-art video generation quality, but their substantial memory and computational footprints hinder edge deployment. Quantization can reduce these costs, yet existing methods often degrade video quality due to high activation variation and the difficulty of preserving semantic and temporal coherence. We propose SemanticDialect, which advances block-wise mixed-format quantization. In this framework, each block selects an optimal format (dialect) from a candidate set (formatbook), which is augmented with lookup tables that store quantization errors and quantized indices, enabling efficient per-block format selection and quantization with minimal online overhead. We further introduce attention-guided activation decomposition, which reduces quantization error via residual quantization, and semantic-aware dialect assignment (SeDA), which reduces cross-token quantization inconsistency by enforcing format uniformity among semantically correlated tokens. Experiments demonstrate that SemanticDialect outperforms prior quantization methods and block-wise formats (MXFP4, NVFP4) while approaching FP16 quality on Open-Sora 2.0. We also validate hardware deployability through RTL design and GPU kernel implementation.
Attention over tens of thousands of visual tokens dominates the inference of video diffusion transformers (DiTs). Existing methods reduce this cost through sparsity, low precision, or both, but use a single precision for all kept tiles. We observe that the output error introduced by approximation varies widely across tiles. Many tiles can be skipped with little error, others must be computed but gain little beyond low precision, and a few retain substantial error even at low precision. We present Triage, a training-free attention method that assigns each tile one of three approximation tiers: skipping it by replacing it with its block means, skimming it in NVFP4, or refining it with NVFP4 residual correction. A lightweight router predicts each tile's error under each tier from block statistics and moves a tile to a more accurate tier only when the predicted error reduction justifies the extra compute, with a single price setting this trade-off. A fused kernel executes all three tiers in a single pass on the FP4 tensor cores of RTX Blackwell GPUs. Across Wan2.2, HunyuanVideo, and LTX-2.5, Triage accelerates end-to-end generation by up to 2.32x over dense FlashAttention, and in each of five settings it runs faster than every baseline while reaching 0.3-3.0 dB higher PSNR than the highest-PSNR baseline.
Yeonjae Kim, Omin Kwon, Seong Hoon Seo +1
Department of Computer Science and Engineering, Seoul National University, Seoul, Republic of Korea.
Video generation has broad applications in content creation and entertainment. Diffusion transformers have advanced the quality of generated videos, but attention over spatiotemporal tokens becomes increasingly expensive as video resolution and duration increase. We present DraftAttention2, a training-free framework that uses the low-resolution draft attention map to jointly select attention blocks and assign their numerical precision. Specifically, spatial 2D average- and max-pooled queries and keys capture complementary regional statistics to estimate block importance, and a shared ranking assigns higher precision to important blocks, lower precision to less important retained blocks, and skips the rest under configurable budgets. Our analysis separates sparsification error from attention-weighted quantization error, establishing when recovering skipped interactions with low-bit computation tightens the output-error bound. This analysis motivates retaining more interactions at low precision while reserving higher precision for blocks with larger attention mass. To translate these fine-grained assignments into practical speedups, we further develop fused operand preparation and a single attention kernel with precision-specific phases, sharing data movement, softmax statistics, and output accumulation across precisions. Experiments demonstrate that our method achieves a superior quality-efficiency trade-off over existing efficient video generation methods. Notably, its advantage is particularly pronounced for few-step video diffusion, where jointly combining sparsity with 4- and 8-bit mixed-precision computation substantially improves generation quality while retaining significant acceleration. Code is available at https://github.com/anemoi-project/anemoi
Rui Ding, Haopeng Li, Weize Ma +10
Nanjing University · HKUST(GZ) · Duke University +3