cs.LGOct 30, 2026

SparseGDN: Chunk Sparse in Linear Attention

Authors: Jiedong Zhuang, Peng Zhao, Jialin Tang, Haoji Hu

Organizations: Zhejiang University · ByteDance

Abstract

Gated Delta Networks (GDNs) offer linear-time sequence modeling, yet long-context prefill remains costly due to dense intra-chunk computation, recurrent-state updates, and intermediate state storage. We introduce SparseGDN, a hardware-aware framework that combines structured intra-chunk approximation with input-dependent recurrent-event selection while producing outputs for every token. A block-adjacent solver reduces local Delta-solve computation, while a dynamic selector allocates recurrent-update budgets across head pairs. The GPU implementation materializes updates only for selected events and reconstructs token outputs using compact state snapshots and group-local interactions, reducing computation and memory traffic. We evaluate SparseGDN on Qwen3.5-4B and Qwen3.5-9B across RULER tasks and context lengths from 4K to 128K. Retaining approximately 29% of recurrent events on average, SparseGDN achieves 1.50-1.94x GDN-core prefill speedups over native FLA kernels, including online selection overhead.

Figures & tables

Explore similar work

CardsList
  1. A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA

    Mar 6, 2026Neelesh Gupta, Peter Wang, Rajgopal Kannan +1LLM Inference AccelerationFPGA-Based Neural Network Acceleration

  2. SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

    Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2KV-Cache OffloadingLong-Context Language Model Inference

  3. HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

    Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language ModelingLinear Attention