Gated Delta Networks (GDNs) offer linear-time sequence modeling, yet long-context prefill remains costly due to dense intra-chunk computation, recurrent-state updates, and intermediate state storage. We introduce SparseGDN, a hardware-aware framework that combines structured intra-chunk approximation with input-dependent recurrent-event selection while producing outputs for every token. A block-adjacent solver reduces local Delta-solve computation, while a dynamic selector allocates recurrent-update budgets across head pairs. The GPU implementation materializes updates only for selected events and reconstructs token outputs using compact state snapshots and group-local interactions, reducing computation and memory traffic. We evaluate SparseGDN on Qwen3.5-4B and Qwen3.5-9B across RULER tasks and context lengths from 4K to 128K. Retaining approximately 29% of recurrent events on average, SparseGDN achieves 1.50-1.94x GDN-core prefill speedups over native FLA kernels, including online selection overhead.
Figures & tables
Figure 1: GDN-core prefill latency across context lengths for Qwen3.5-9B and Qwen3.5-4B. Compared with FLA GDN, SparseGDN additionally includes the online selector overhead. Latency is aggregated over all 24 GDN layers. SparseGDN reduces latency at every context length, reaching 1.93× and 1.94× speedup at 128K for the 9B and 4B models, respectively. Lower is better.
Figure 2: Two empirical structures observed in layer 16 of native-dense Qwen3.5-4B on an 8K RULER input ( niah_multikey_3 , row 0). (a) Adjacent chunk-boundary states are frequently nearly stationary, while the magnitudes of their changes vary substantially across value heads. (b) The left panel shows ∣M−1−I∣ , averaged over 123 complete 64-token chunks and all 32 value heads. Here M is the native Delta-solve matrix defined in Section 2.3 ; subtracting I removes the unit diagonal, and absolute values are taken elementwise before averaging. The coefficients concentrate near the causal diagonal and weaken across more distant token blocks. This observed locality motivates the block-adjacent approximation, whose computation support is shown in the right panel.
Figure 3: End-to-end SparseGDN execution pipeline. The block-adjacent intra-chunk kernel computes transformed values U , compact update coefficients, and selector scores. The dynamic selector converts these scores into a structured event mask. Only selected events trigger deferred W materialization and recurrent-state updates, producing compact event states. During output reconstruction, the mapped output kernel retrieves the appropriate preceding event state from HBM and applies the accumulated gap decay, while computing group-local interactions and producing outputs for every token.
Figure 4: Dense versus SparseGDN execution. The dense chunkwise implementation materializes W , updates the recurrent state, and stores a chunk-boundary state for every chunk. SparseGDN materializes W and performs recurrent read/write only at selected events; accumulated decay bridges skipped chunks (red arrows), and only intermediate state snapshots required by downstream output groups are stored in HBM. The final recurrent state is returned for decoding. Mapped group-2 reconstruction reuses one group-start state for two adjacent chunks while still producing outputs for every token. Dashed boxes denote skipped computation or elided state storage.
Figure 5: Dynamic event selection and packing. Here, ZBi denotes the i -th 16-token block of the pre-activation output-gate tensor Z within a chunk. Chunk scores determine the retention order within each head pair. Recurrent lifetime and output-visibility proxies weight the estimated gains from larger budgets, which determine the low-, medium-, and high-retention tiers. The example assigns 9, 4, and 3 pairs to these tiers. Selected events are packed in temporal order into CSR chains for recurrent execution.
Figure 6: Tier-aware scheduling of heterogeneous recurrent event chains. High-, medium-, and low-retention head pairs induce event chains with different workloads. Scheduling them in physical-head order can place a long high-tier chain behind shorter tasks, creating an SM straggler and extending kernel completion time (left). SparseGDN orders chains by expected workload and uses event-count-adaptive execution to distribute them across SMs (right), reducing load imbalance and tail latency. The red dashed line denotes kernel completion.
Qwen3.5-4B
Qwen3.5-9B
Length
Base
Sparse
Δ (%)
Event sp. (%)
Speedup
Base
Sparse
Δ (%)
Event sp. (%)
Speedup
4K
96.641
96.377
−0.264
70.937
1.498×
96.821
96.846
+0.026
70.921
1.500×
8K
96.282
96.333
+0.051
70.955
1.504×
96.872
96.179
−0.692
71.005
1.506×
16K
96.121
96.462
+0.341
70.915
1.696×
96.369
96.308
−0.062
70.971
1.702×
32K
96.344
96.097
−0.246
70.934
1.830×
96.744
96.831
+0.087
70.917
1.825×
64K
95.985
96.274
+0.290
70.998
1.891×
96.641
96.538
−0.103
70.976
1.892×
Table 1: RULER scores, recurrent-event sparsity, and GDN-core speedup for Qwen3.5-4B and Qwen3.5-9B. Scores follow the official RULER scoring rules and are reported as percentages. Δ is the arithmetic difference between the Sparse and Base columns, computed from unrounded scores. Both scores and their differences are expressed on the percentage scale. Speedup compares native Intra+ H + O against SparseGDN Selector+Intra+ H + O , using the 24-layer GDN-core timing protocol in Section 3.1 . Event sparsity is the mean fraction of valid recurrent pair–chunk events omitted across evaluation requests; padding chunks are excluded. The All row averages across the six context lengths.
Figure 7: Per-stage GDN-core prefill latency. The top and bottom rows show Qwen3.5-4B and Qwen3.5-9B, respectively. The left column compares native Intra with SparseGDN Intra plus online Selector; the middle and right columns compare H and O . For SparseGDN , H includes deferred W materialization. Each component reports the median of per-iteration CUDA-event times summed over all 24 GDN layers. The stacked Intra-plus-Selector bars sum the two component medians. Boxed values denote the corresponding native-to-sparse latency ratios. Lower latency is better.
Figure 8: GDN-core speedup across batch sizes and context lengths. Speedup is Native/Sparse, includes online selection, and uses batch latencies aggregated over all 24 GDN layers. Crosses mark configurations without a paired speedup because Native runs out of memory. Crosses are placed at a fixed height for display and do not represent measured speedups. The corresponding sparse-only outcomes are detailed in Table 2 . Every completed paired configuration favors SparseGDN . Higher is better.
Model
Batch
4K
8K
16K
32K
64K
128K
4B
2
1.649×
1.723×
1.826×
1.933×
1.945×
1.988×
4B
4
1.582×
1.659×
1.738×
1.799×
1.833×
OOM†
4B
8
1.560×
1.615×
1.675×
1.750×
OOM†
OOM
9B
2
1.651×
1.736×
1.839×
1.928×
1.931×
1.939×
9B
4
1.574×
1.653×
1.737×
1.807×
1.830×
OOM
9B
8
1.560×
1.611×
1.673×
1.748×
OOM
OOM
Table 2: Batched GDN-core speedup (Native/Sparse). OOM† indicates that Native runs out of memory in the benchmark harness, while SparseGDN completes in a separate process. Plain OOM indicates that Native runs out of memory and sparse completion has not been independently verified. Speedups include online selection and cover only the GDN core.
Qwen3.5-4B
Qwen3.5-9B
Operating point
Keep (%)
Δ (%)
Speedup
Keep (%)
Δ (%)
Speedup
10%-budget
9.760
−2.063
2.089×
9.755
−2.281
2.099×
20%-budget
19.493
−0.821
1.867×
19.502
−0.727
1.868×
Default
29.044
+0.017
1.718×
29.043
−0.115
1.717×
40%-budget
39.253
+0.075
1.618×
39.261
+0.051
1.618×
50%-budget
49.325
+0.053
1.425×
49.337
+0.033
1.425×
Table 3: Retention-rate trade-off over all six context lengths. Keep is the mean fraction of valid recurrent pair–chunk events retained across evaluation requests; padding chunks are excluded. Δ=Ssparse−Sbase is the arithmetic difference in aggregate RULER scores, computed from unrounded scores over 7,800 examples per model using the official scoring rules. Scores and their differences are expressed on the percentage scale. Speedup is the geometric mean of the six GDN-core speedups at batch size one under the timing protocol in Section 3.1 , including online selection. Higher speedup and Δ are better.
Figure 9: Accuracy–efficiency trade-off across retention budgets. Each panel plots the aggregate RULER score difference Δ , defined in Table 3 , against the geometric mean GDN-core speedup over 4K–128K. Point labels show measured recurrent-event keep rates. Solid lines connect nondominated configurations among the evaluated sparse operating points. The cross marks the approximately 49% -keep configuration, which is dominated by the approximately 39% -keep configuration for both models under these aggregate metrics. Speedups include online selection. Higher and farther right is better.
Qwen3.5-4B
Qwen3.5-9B
Variant
Score
Δ (%)
Keep (%)
Score
Δ (%)
Keep (%)
Native
96.641
+0.000
100.000
96.821
+0.000
100.000
Block-adjacent + all
96.410
−0.231
100.000
96.846
+0.026
100.000
Full64 + adaptive
96.669
+0.028
28.991
96.787
−0.033
29.017
Block-adjacent + random
95.067
−1.574
29.029
95.621
−1.200
29.029
SparseGDN
96.377
−0.264
29.063
96.846
+0.026
29.079
Table 4: Algorithmic component ablation on all 13 RULER tasks at 4K (1,300 examples per model). Scores and their arithmetic differences are expressed on the percentage scale. Δ is the score difference from Native, computed from unrounded scores. Full64 + adaptive restores the full lower-triangular 64-token intra-chunk solve while retaining adaptive event selection; Block-adjacent + all retains every recurrent event. Adaptive and random variants use matched approximately 29% event budgets. All custom variants retain the same group-local output rule; Native uses the dense operator. Adaptive masks are recomputed for each Intra variant. Bold highlights the proposed method.
Allocation
4K
8K
16K
32K
64K
128K
All
Keep (%)
Fixed counts 9/4/3
96.691
96.436
95.815
96.505
96.385
95.267
96.183
29.816
Dynamic counts
96.846
96.179
96.308
96.831
96.538
96.310
96.502
29.043
Table 5: Effect of adaptive tier counts on Qwen3.5-9B. Scores are official RULER percentages over 100 examples per task at each context length (13 tasks; 7,800 examples in total). All is the mean over the six lengths. Score differences reported in the text are computed from unrounded values. Keep is the mean per-request fraction of valid recurrent pair–chunk events retained, expressed as a percentage. Both variants use the same retention tiers; realized keep rates are not forced to match. Bold indicates the higher score at each length and in aggregate.
Figure 10: Tail-window ranking quality, scan cost, and illustrative task scores. Left: change in retained non-anchor dense-decode importance mass relative to Tail-16 on 130 stratified 8K requests per model, with task-stratified paired 95% bootstrap confidence intervals. Pair-tier assignments, exact per-pair keep counts, and anchors are fixed to Tail-16. The gray curve reports standalone score-kernel latency per layer at 8K. Right: illustrative 128K score curves. The shaded bands highlight the default Tail-16 window in both panels.
Qwen3.5-4B
Qwen3.5-9B
Optimization
Scope
Geomean
128K
Geomean
128K
Deferred- W
Intra+ H + O
1.336×
1.390×
1.338×
1.391×
CSR event scan
H
1.149×
1.226×
1.153×
1.230×
Compact states
H
1.398×
1.406×
1.402×
1.405×
Tier scheduling
H
1.029×
1.044×
1.030×
1.045×
Mapped O
O
1.239×
1.374×
1.226×
1.368×
Table 6: Kernel-pipeline ablations at batch size one on a GPU. Speedup is the ratio of reference to optimized median latency over the stages listed in Scope. Each variant uses five warmup forwards and 30 interleaved CUDA-event measurements. Within each measurement, the indicated stage times are summed across all 24 GDN layers before computing the median across measurements. Separately timed Selector work is excluded; statistics fused into Intra remain included. Geomean is the geometric mean of speedups over the six lengths from 4K to 128K; the 128K column reports the longest-context result. Each row uses its own reference, so the isolated speedups should not be multiplied. Compact+mapped reports the joint effect of compact state storage and mapped output reads.
Figure 11: Complete-model prefill peak allocated memory across context lengths. Measurements use BF16, batch size one, use_cache=True . The shaded region shows the absolute memory saved by SparseGDN wherever both paths complete. The dashed line marks the nominal 80 GB device capacity as a reference. Crosses indicate Native OOM at 256K and are placed on this reference line for display; they do not represent measured allocation peaks. SparseGDN completes at that length for both models. Lower is better.
Model
Length
Native peak
Sparse peak
Native incr.
Sparse incr.
Incr. reduction
4B
128K
53.116
26.894
44.027
17.804
59.56%
9B
128K
64.066
37.844
45.235
19.012
57.97%
4B
256K
OOM
44.637
OOM
35.544
–
9B
256K
OOM
56.794
OOM
37.960
–
Table 7: Complete-model prefill peak allocated memory. “Incr.” denotes the peak allocation above memory already live at forward entry. “Incr. reduction” is the relative decrease in this increment, computed from unrounded values. Native OOMs during its first 256K warmup, so no completed-forward peak or increment is available. Lower is better.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.