SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving
Authors: Haolin Jiang, Weihang Li, Zhongzhu Zhou, Jinwen Pan, Santosh Pandey, Hang Liu
Organizations: Rutgers University, New Brunswick, New Jersey, USA · University of Sydney, Sydney, New South Wales, Australia · University of South Florida, Tampa, Florida, USA
Hybrid models combine softmax attention with linear attention, which compresses history into fixed-size recurrent states to balance model quality and inference efficiency. However, linear attention has not pushed this compression to its limit: we find that the queries, keys, and values of linear layers concentrate in a stable, low-dimensional subspace, a phenomenon we call the linear attention sink, so each state is effectively low-rank, with a rank that varies across states. Yet existing serving systems keep these redundant states dense, reading and rewriting them in full at every decoding step, so their memory traffic grows with the batch size and dominates the decoding cost. Thus SinkState stores each recurrent state in an adaptive low-rank form throughout inference with buffered updates and a local SVD restricted to the top subspace. SinkState further consolidates the remaining components asynchronously on the CPU. On four hybrid models with Mamba-2 and Gated DeltaNet (GDN) layers, from 4B to 120B parameters, SinkState reduces the peak memory of recurrent states by up to 2.6x (1.9x on average) and speeds up decoding by up to 1.58x (1.25x on average) over Mamba-SSM and FLA, while maintaining accuracy.
Figures & tables
Figure 1: The qt , kt , and vt vectors of three linear-attention models on PG-19 (top rows) and uniformly random tokens (bottom rows), profiled on 32 sequences of 64k tokens for each state Ht . (a) Energy spectrum: the fraction of energy captured by the top- r directions of the stacked qt , kt , and vt . (b) Stability: energy of each segment [nk,(n+1)k) captured by the top-8 subspace computed on [0,1k) , versus its own optimal top-8 subspace. Gray dashed lines show the results for uniformly random (isotropic) directions.
Figure 2: Fractions of Ht ’s energy and output Htqt retained when only the top- r components of Ht are kept. Measurements are taken every 0.5k tokens across 32 sequences of 64k tokens each.
Figure 3: Share of an incoming vt , kt on each component of Ht−1 .
Figure 4: Rank stored per head under local SVD and exact SVD after 128 decoding steps on GSM8K.
Figure 5: Overview of SinkState .
Figure 6: Prefill and decoding with local SVD updates.
Figure 9: State-kernel time per decoding step (top) and speedup (bottom) under different batch sizes of SinkState vs. Mamba-SSM/FLA. Shaded regions: the baselines run out of memory.
Figure 10: Accuracy on long-context retrieval (RULER) and long reasoning in thinking mode (MATH-500, GPQA Diamond, up to 32K generated tokens). Numbers show the accuracy change of SinkState over the baseline.
Figure 11: Distribution of the number of generated tokens on MATH-500 under SinkState and the baseline.
Figure 12: Ablation of (a) staggered compaction and (b) CPU consolidation. Numbers show the step-time change (ms) in (a) and the state-memory ratio of w/ to w/o the CPU pipeline in (b).
Figure 13: Stored rank per head of Nemo-4B on RULER after prefill and after 8K decoded tokens (top; gray: dense), and the share of heads per rank range (bottom).
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
State-space models (SSMs) maintain a fixed-size recurrent state, but reading and writing it at every decoding step can bottleneck hybrid models. This eager state management causes large state I/O with low arithmetic intensity, leaves speculative decoding no cheap rollback, and feeds state quantization error into subsequent recurrent updates. We introduce ReplaySSM, which unrolls the recurrence to represent the state as a checkpoint and a buffer of recent SSM inputs. This separates output computation from checkpoint updates. By updating the checkpoint periodically, ReplaySSM avoids state writes on most steps and roughly halves the dominant state memory traffic. Keeping recent inputs also makes speculative rollback a buffer-pointer operation and enables output-only verification without materializing intermediate states. The same separation allows a low-precision checkpoint to serve output computation on most steps while a higher-precision checkpoint serves recurrent updates, so rounding error in the low-precision copy never enters later updates. We implement ReplaySSM in vLLM for Mamba-2 and Gated DeltaNet. At serving batch size 256, it accelerates standard decoding by 1.20--1.47× over a tuned vLLM baseline across four hybrid models, and delivers about 1.8× the throughput of vLLM's speculative decoding on the two 120B-class models. Separate state precisions also reduce output KL divergence and downstream accuracy losses compared with a shared low-precision checkpoint.
Hybrid-attention models replace most softmax attention layers with linear attention, reducing KV-cache growth and enabling larger decode batches where recurrent-state access becomes a major bottleneck. ReplaySSM amortizes state updates by buffering keys and values, but each new query still requires a full-state read even though the state remains unchanged between state updates. We observe that low-rank state-weighted query approximation accurately preserves state-read outputs. Although future queries are unknown, the basis vectors used to approximate them can be fixed offline. Based on this observation, we introduce SketchSSM, which preserves full-state updates while approximating reads. At each state update, SketchSSM reads the full state once to precompute outputs for these basis vectors, storing them in a compact sketch. Each subsequent decode step combines the sketch vectors with query-dependent coefficients to reconstruct the output without a full-state read. Across four Mamba-2-, GDN-, and KDA-based models, SketchSSM at a mean sketch rank of 8 reduces state-access traffic by approximately 10× while matching the average accuracy of the FP32 full-state baseline across four decode benchmarks, and preserves recall on four RULER retrieval tasks. At this rank on one NVIDIA B300, linear-attention kernel speedups over the Standard vLLM baseline reach 7.30×, 5.02×, and 5.24× for Mamba-2, GDN, and KDA, respectively, with up to 2.77× higher decode throughput on Nemotron 3 Super.
Linear attention models eliminate the quadratic prefix computation and context-growing KV cache of softmax attention by replacing pairwise token interactions with recurrent state updates. However, existing decoding implementations often materialize and write back the full recurrent state after every generated token, making state maintenance a major source of memory traffic, especially for models with large states and many heads. This paper presents DeltaLog, a recurrent-state decoding scheme that reduces this overhead without changing the model semantics. Specifically, DeltaLog represents the recurrent state as a dense base state together with a bounded log of recent compact updates. Most decode steps append only compact update factors to this log, while periodic merge steps fold the accumulated updates back into the dense base state. Thus, the model observes the same dense state as in eager decoding, but most full-state write-backs are replaced by lightweight append operations. We implement DeltaLog for GDN, KDA, and RWKV6 and integrate it into a prototype serving stack. Across these models, DeltaLog accelerates the recurrent-state update kernel by up to 1.86×, reduces profiled recurrent-state write traffic by up to 7.83×, and achieves 1.05--1.20× end-to-end serving speedups over dense recurrent baselines.