cs.LGNov 10, 2026

SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving

Authors: Haolin Jiang, Weihang Li, Zhongzhu Zhou, Jinwen Pan, Santosh Pandey, Hang Liu

Organizations: Rutgers University, New Brunswick, New Jersey, USA · University of Sydney, Sydney, New South Wales, Australia · University of South Florida, Tampa, Florida, USA

Abstract

Hybrid models combine softmax attention with linear attention, which compresses history into fixed-size recurrent states to balance model quality and inference efficiency. However, linear attention has not pushed this compression to its limit: we find that the queries, keys, and values of linear layers concentrate in a stable, low-dimensional subspace, a phenomenon we call the linear attention sink, so each state is effectively low-rank, with a rank that varies across states. Yet existing serving systems keep these redundant states dense, reading and rewriting them in full at every decoding step, so their memory traffic grows with the batch size and dominates the decoding cost. Thus SinkState stores each recurrent state in an adaptive low-rank form throughout inference with buffered updates and a local SVD restricted to the top subspace. SinkState further consolidates the remaining components asynchronously on the CPU. On four hybrid models with Mamba-2 and Gated DeltaNet (GDN) layers, from 4B to 120B parameters, SinkState reduces the peak memory of recurrent states by up to 2.6x (1.9x on average) and speeds up decoding by up to 1.58x (1.25x on average) over Mamba-SSM and FLA, while maintaining accuracy.

Figures & tables

Explore similar work

CardsList
  1. ReplaySSM: Cache SSM Inputs, Not State

    Oct 9, 2026Ze-Wei Liou, Harris Nover, Amey Naik +4Inference AccelerationLLM Inference Acceleration

  2. SketchSSM: Write to the Full State, Read from a Compact Sketch

    Sep 27, 2026Omin Kwon, JoongWon Shin, Minseo Kim +3LLM Inference AccelerationLinear Attention

  3. DeltaLog: Deferred Materialization of Recurrent States for Linear Attention Decoding

    Aug 16, 2026Junqing Lin, Jingwei Sun, Guangzhong SunLLM Inference AccelerationLinear Attention