cs.CLNov 17, 2026

RISA: Recoverable Sparse Attention for Block-Diffusion Language Models

Authors: Xinrui Zhong, Fanjiang Ye, Yuke Wang

Organizations: Rice University

Abstract

Block-wise diffusion language models (DLMs) refine multiple tokens in parallel, offering an alternative to token-by-token autoregressive decoding. Their long-context inference, however, remains bottlenecked by memory-bound attention: every denoising step repeatedly reads a growing historical key--value (KV) cache. Directly applying sparse attention does not fully solve this problem, because different queries can select different pages, inflating the block's collective read set, while repeated support construction adds overhead and permanent eviction can discard information needed later. We present RISA, a recoverable sparse-attention method with shared read plans for block-wise DLMs. RISA exploits the fact that historical KV remains fixed within a block while queries evolve across denoising steps. Newly generated history is organized into exact priority and recovery tiers; seed queries then build a protected, bounded read plan whose indices are reused across the block while attention is recomputed with fresh queries and current-block KV. This separates history preservation from repeated reading and amortizes sparse-support construction without changing the backbone or decoder. On DreamReasoner-8B, RISA retains comparable task performance. At 32K, the default configuration speeds up attention by 2.67×2.67\times and fixed-trajectory requests by 1.36×1.36\times over dense, and attention speedup reaches 5.53×5.53\times at 128K. Across eight settings on three models, RISA decodes faster than the tested MAGE, QUEST, and LoSA budgets at matched KL.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models

    Apr 13, 2026Haocheng Xi, Harman Singh, Yuezhou Hu +9Diffusion Language Model InferenceDiffusion Model Inference Acceleration

  2. Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention

    May 19, 2026Wenhu Zhang, Yiming Wu, Huanyu Wang +6Self-AttentionLong-Context Language Modeling

  3. Step-dLLM: Adaptive Step-aware Sparse Attention for Efficient Diffusion LLM Inference

    Oct 31, 2026Zhichen Zeng, Xichong Zhang, Junpan Wu +7Diffusion Language Model InferenceDiffusion Language Models