SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving
Authors: Haolin Jiang, Weihang Li, Zhongzhu Zhou, Jinwen Pan, Santosh Pandey, Hang Liu
Organizations: Rutgers University, New Brunswick, New Jersey, USA · University of Sydney, Sydney, New South Wales, Australia · University of South Florida, Tampa, Florida, USA
Hybrid models combine softmax attention with linear attention, which compresses history into fixed-size recurrent states to balance model quality and inference efficiency. However, linear attention has not pushed this compression to its limit: we find that the queries, keys, and values of linear layers concentrate in a stable, low-dimensional subspace, a phenomenon we call the linear attention sink, so each state is effectively low-rank, with a rank that varies across states. Yet existing serving systems keep these redundant states dense, reading and rewriting them in full at every decoding step, so their memory traffic grows with the batch size and dominates the decoding cost. Thus SinkState stores each recurrent state in an adaptive low-rank form throughout inference with buffered updates and a local SVD restricted to the top subspace. SinkState further consolidates the remaining components asynchronously on the CPU. On four hybrid models with Mamba-2 and Gated DeltaNet (GDN) layers, from 4B to 120B parameters, SinkState reduces the peak memory of recurrent states by up to 2.6x (1.9x on average) and speeds up decoding by up to 1.58x (1.25x on average) over Mamba-SSM and FLA, while maintaining accuracy.
Figures & tables
Figure 1: The qt , kt , and vt vectors of three linear-attention models on PG-19 (top rows) and uniformly random tokens (bottom rows), profiled on 32 sequences of 64k tokens for each state Ht . (a) Energy spectrum: the fraction of energy captured by the top- r directions of the stacked qt , kt , and vt . (b) Stability: energy of each segment [nk,(n+1)k) captured by the top-8 subspace computed on [0,1k) , versus its own optimal top-8 subspace. Gray dashed lines show the results for uniformly random (isotropic) directions.
Figure 2: Fractions of Ht ’s energy and output Htqt retained when only the top- r components of Ht are kept. Measurements are taken every 0.5k tokens across 32 sequences of 64k tokens each.
Figure 3: Share of an incoming vt , kt on each component of Ht−1 .
Figure 4: Rank stored per head under local SVD and exact SVD after 128 decoding steps on GSM8K.
Figure 5: Overview of SinkState .
Figure 6: Prefill and decoding with local SVD updates.
Figure 9: State-kernel time per decoding step (top) and speedup (bottom) under different batch sizes of SinkState vs. Mamba-SSM/FLA. Shaded regions: the baselines run out of memory.
Figure 10: Accuracy on long-context retrieval (RULER) and long reasoning in thinking mode (MATH-500, GPQA Diamond, up to 32K generated tokens). Numbers show the accuracy change of SinkState over the baseline.
Figure 11: Distribution of the number of generated tokens on MATH-500 under SinkState and the baseline.
Figure 12: Ablation of (a) staggered compaction and (b) CPU consolidation. Numbers show the step-time change (ms) in (a) and the state-memory ratio of w/ to w/o the CPU pipeline in (b).
Figure 13: Stored rank per head of Nemo-4B on RULER after prefill and after 8K decoded tokens (top; gray: dense), and the share of heads per rank range (bottom).
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.