RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

Zekun Zhang; Yixiang Cai; Yuxi Liu; Tengxu Sun; Tianle Liu; Zhoutong Wu; Haoyu Li; Baole Ai

cs.CVSep 6, 2026

RoLA: Rotary-Positioned Low-Rank Linear Attention for Efficient Diffusion Transformers

Authors: Zekun Zhang, Yixiang Cai, Yuxi Liu, Tengxu Sun, Tianle Liu, Zhoutong Wu, Haoyu Li, Baole Ai, +4 more

Abstract

Diffusion Transformers (DiTs) achieve strong video generation quality, but their dense spatiotemporal self-attention scales quadratically with sequence length and quickly becomes the dominant inference bottleneck. Sparse low-rank hybrids alleviate this cost by combining a local sparse branch with a global compressed branch. In video DiTs equipped with 3D Rotary Position Embeddings (RoPE), the global branch faces a structural compatibility issue: when RoPE is applied before a nonlinear feature map, the rotation and nonlinearity generally do not commute, making it difficult to keep a query-independent linear summary while preserving relative rotary geometry. Existing work often sidesteps this issue by replacing genuine cross-token global aggregation with coordinate-conditioned surrogates or learnable absolute positional modules. These compromises can be effective, but they approximate relative decay from absolute coordinates and introduce extra positional parameters. We propose \textbf{RoLA}, a rotary-positioned low-rank linear-attention branch that keeps genuine cross-token aggregation while remaining compatible with a reusable linear summary. The design applies RoPE \emph{outside} the nonlinear low-rank feature map and reuses a truncated subset of the pre-trained rotary schedule matched to the low-rank bottleneck. This yields a linear-time low-rank global branch with relative positional behavior by design and no additional positional parameters; the full sparse--low-rank module still includes the fixed-sparsity sparse branch. Experiments on open-source video DiTs show that the resulting method remains competitive in generation quality at 90% sparsity while achieving 2.63 $\times$ end-to-end inference speedup on Wan2.1-14B (720p, 81 frames, measured on an NVIDIA H100 GPU).

Explore similar work

May 20, 2026cs.CV

RoPeSLR: 3D RoPE-driven Sparse-LowRank Attention for Efficient Diffusion Transformers

Diffusion Transformers (DiTs) have revolutionized high-fidelity video generation, yet their

\mathcal{O}(L^2)

attention complexity poses a formidable bottleneck for long-sequence synthesis. While recent sparse-linear attention hybrids aim to mitigate this, their performance severely degrades at extreme sparsity due to the "RoPE Dilemma": standard linear attention fails to preserve the orthogonal relative-position structure of 3D Rotary Position Embeddings (RoPE), neutralizing vital distance awareness. To address this, we propose \textbf{RoPeSLR}, a 3D RoPE-guided Sparse-LowRank attention framework. We establish that under empirically validated assumptions, the DiT attention manifold admits a decoupling into a high-frequency semantic spike set (bounded by

\mathcal{O}(L^{3/2})

sparsity) and an extreme low-rank (

\mathcal{O}(d_h \log L)

) background continuum. Guided by this structural prior, RoPeSLR eschews standard linear attention for a head-wise low-rank parameterization equipped with a learnable 3D Absolute Positional Embedding (PE) injection, seamlessly synthesizing long-range relative distance decay. By guaranteeing sub-quadratic sparsity and sub-linear rank growth, RoPeSLR is exceptionally suited for scaling to ultra-long video inference. Extensive evaluations validate this scalable superiority: at 90% sparsity, RoPeSLR achieves up to

10\times

fewer FLOPs on Wan2.1-1.3B and delivers a

2.26\times

end-to-end inference speedup on the ultra-long 100K+ token sequences of HunyuanVideo-13B, all while maintaining near-lossless generation fidelity (less than 1.3% average VBench degradation).

Yuxi Liu, Zekun Zhang, Yixiang Cai +3

Dynamic Sparse Attention Rotary Position Embedding Transformer Architectures

Ask PDF

Jul 1, 2026cs.CV

RotateAttention: RoPE-Aware Rotation and Range Rectification for INT4 Quantized Attention in Video Generation

\textbf{DiT-based video generation models equipped with 3D Rotary Position Embeddings (3D RoPE)}

, the attention mechanism remains a primary computational bottleneck due to its quadratic complexity with respect to sequence length. While quantized

\textbf{FlashAttention}

offers a promising path toward hardware acceleration, existing low-bit quantization methods overlook two critical challenges in this setting:

\textbf{1)}

applying online rotation matrices -- a widely used technique for mitigating outliers in Queries (

Q

) and Keys (

K

) -- is difficult to reconcile with

\textbf{RoPE}

; and

\textbf{2)}

the non-negative attention matrix

P = \exp(QK - \max(QK))

makes symmetric quantization waste half of the 4-bit dynamic range. In this work, we observe that the outlier distributions of

Q

and

K

are strongly affected by the dimensional partitioning of

\textbf{3D RoPE}

. Based on this finding, we propose

\textbf{RotateAttention}

, an efficient

\textbf{mixed-precision INT4 FlashAttention}

framework tailored for

\textbf{DiT-based video generation models with 3D RoPE}

, using selective

\textbf{FP16 fallback}

for accuracy-sensitive attention blocks and denoising steps. RotateAttention introduces two core techniques:

\textbf{1) RoPE-aware Rotation}

, which employs either mergeable rotation matrices that can be fused into RoPE or negligible-overhead matrices to mitigate RoPE-induced outliers in

Q

and

K

; and

\textbf{2) Range-optimized

Quantization}

, which uses fixed scales and zero-points to fully exploit the

\textbf{INT4 numerical range}

with minimal computational overhead. Experiments show that

\textbf{RotateAttention}

preserves video generation quality nearly identical to full-precision baselines while achieving up to 1.68

\times

end-to-end speedup and 2.2

\times

kernel-level acceleration.

Yaofu Liu, Wanli Lan, Jinxi Li +2

Quadratic Attention Rectification

Ask PDF

Aug 3, 2026cs.CV

Token Radius Attention for Efficient Video Generation

Video Diffusion Transformers (VDiTs) enable high-fidelity generation but incur quadratic cost from dense 3D self-attention. Existing head- and block-level sparse methods share computation budgets across queries, overlooking token-specific attention demand. We observe that retained density varies across queries yet correlates log-linearly with attention entropy, while dominant interactions form query-centered neighborhoods with token-dependent radii. Based on these findings, we propose Token Radius Attention (TRA), a training-free framework that maps query entropy to an analytic token budget and converts it into a temporally decayed radius without explicit key ranking. Fused entropy extraction, warm-up reuse, and block-sparse mask construction further reduce overhead. Across seven Wan2.1, Wan2.2, and HunyuanVideo T2V/I2V configurations, TRA retains only 9-19% of attention interactions and achieves 1.56x-2.05x speedup with competitive generation quality. Code is available at https://github.com/IF-LAB-PKU/Token-Radius-Attention.

Jiayu Chen, Zhikun Jiang, Maoliang Li +6

Dynamic Sparse Attention Neighbors

Ask PDF