Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall

Yasong Fan

cs.LGApr 9, 2026

Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall

Authors: Yasong Fan

Organizations: Independent Researcher

Abstract

We present FDM (Fan Duality Model), a linear sequence architecture that resolves the fundamental tension between memory efficiency and associative recall in sequence modeling. FDM separates sequence processing into two components: a wave component (recurrent scan via phase-preserving Givens rotations) that compresses long-range patterns into a fixed-size complex hidden state, and a particle component (local-global cache) that retrieves specific tokens via learned associative addressing with W+K=272 slots independent of sequence length N. This yields strictly O(1) decode memory: 867 MB fixed across all prompt lengths 128-8,192 tokens, versus Transformer's 853-4,247 MB (4.9x reduction at N=8,192). Beyond the architecture, we discover that jointly training the wave and particle components leads to suboptimal convergence. We propose Freeze-Scan, a two-phase training strategy that freezes the recurrent scan and optimizes the cache jointly with embeddings, achieving PPL=64.9 on WikiText-103 in 44K steps -- a 7.5x improvement over full fine-tuning (PPL=487). On Multi-Query Associative Recall (MQAR), FDM achieves 0.966 accuracy, surpassing Transformer (0.606) by 59.5%, while pure scan without cache scores only 0.011, confirming the necessity of the particle component. Finally, we introduce Holographic Reference Beam Decoding, interpreting the complex hidden state h_t as a holographic plate encoding the entire temporal history. Using the current input x_t as a reference beam to modulate h_t reduces PPL by up to 2.13 points (PPL=62.79) with a 4-head orthogonal reference beam using only 1.3M additional parameters, providing empirical support for the holographic interpretation. Code and pretrained weights: https://github.com/YasongFan/FDM

Figures & tables

Model	Params	Steps	Val PPL $\downarrow$
Transformer	122M	44K	36.3
FDM — Full fine-tuning	137M	44K	487
FDM — Freeze-Scan	137M	10K	148.5
FDM — Freeze-Scan	137M	44K	64.9
FDM + Holographic Decoding (4-head)	138.3M	44K+5K	62.79

Table 1: WikiText-103 validation perplexity.

Configuration	Params	Steps	Val PPL	$\Delta$ PPL
FDM baseline	—	—	64.92	—
Single-head (layer 0 only)	+0.3M	5K	63.10	$-1.82$
Sequential layerwise (layers 0–11)	+4M	24K	63.34	$-1.58$
4-head orthogonal (layer 0)	+1.3M	5K	62.79	$\mathbf{-2.13}$
8-head orthogonal (layer 0)	+2.7M	5K	$\approx$ 62.8	$\approx$ $-2.1$

Table 2: Holographic reference beam ablation (layer-wise and multi-head).

Model	Easy	Medium (seq=128)
Transformer	0.606	0.238
FDM ( $K{=}0$ , scan only)	0.011	0.011
FDM ( $K{=}16$ )	0.966	0.064

Table 3: MQAR accuracy (Easy: seq=64, 8 KV pairs).

Prompt	Decode Memory (MB)		Decode Speed (tok/s)
Prompt	TF	FDM	TF	FDM
128	853	867	149.5	22.1
1,024	1,210	867	133.1	21.4
4,096	2,462	867	55.1	20.2
8,192	4,247	867	25.7	18.7

Table 4: Decode memory (MB) and speed (tok/s) vs. prompt length.

Model	LAMBADA	HellaSwag	WinoGrande	Random
Transformer (36.3 PPL)	6.0%	30.3%	49.3%	25%/50%
FDM (64.9 PPL)	1.0%	28.3%	50.8%	—

Table 5: Zero-shot downstream benchmark accuracy.

Property	Transformer	FDM
Language modeling PPL	36.3	64.9
MQAR accuracy ( $K{=}16$ )	0.606	0.966
Decode memory @ $N{=}8$ K	4,247 MB	867 MB
Decode speed consistency	Poor ( $6\times$ degr.)	Good ( $15\%$ )
Training efficiency	Standard	$14\times$ faster
WinoGrande	49.3%	50.8%

Table 6: FDM and Transformer occupy complementary operating regimes.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

Aug 3, 2026

DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling

Modern language models are built primarily from Transformers, recurrent models, and their hybrid architectures. Transformers rely on token-level attention memories, while recurrent models such as state space models (SSMs) and linear attention maintain compact recurrent states. These architectures are typically instantiated separately or interleaved at the layer level, leaving open whether a shared memory representation can support both recurrent compression and attention-style retrieval. We study this question through the state space duality (SSD) view of Mamba-2, where the SSM state can be interpreted as a compressed associative key--value (KV) cache. We observe that Mamba-2 decodes token-conditioned values from this state but does not decode token-conditioned keys. Based on this observation, we propose DART (Decoded Attention over Recurrent sTates), which retains the chunk state contributions produced by the Mamba-2 chunked scan as chunk state memories, decodes token-conditioned keys and values from these memories, and performs state-memory attention (SMA) over the resulting KV pairs. The retrieved output is then combined with the native Mamba-2 output through a gated residual connection. DART supports practical training by reusing the Mamba-2 chunked scan and implementing SMA as a FlashAttention-style computation. Our analysis and experiments show that DART substantially reduces the length-dependent inference cache compared with a matched attention baseline (e.g.,

75\%

savings when the chunk size is

S=256

and the state size is

N=128

). Compared with Mamba-2, DART substantially improves associative recall and retrieval while preserving general language-modeling quality.

Yixiao Qian, Song Chen, Pengkai Wang +3

College of Control Science and Engineering, Zhejiang University · Department of Mathematics, National University of Singapore · Hong Kong Polytechnic University +1

Memory-Augmented Language Models KV Caching Long-Context Language Modeling Attention Mechanisms Associative Memory State Space Models

Ask PDF

May 7, 2026

Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators

Long chain-of-thought reasoning and agentic tool-calling produce traces spanning tens of thousands of tokens, yet Transformer KV caches grow linearly with sequence length, creating a memory bottleneck on commodity hardware. State-space models offer constant-memory recurrence but suffer a memory cliff: retrieval accuracy collapses once the gap between a stored fact and its query exceeds the effective horizon of the recurrent state. We introduce Echo, a KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA); a drop-in replacement for attention layers that augments SSM blocks with a closed-form dynamical operator whose sufficient statistics are accumulated in constant memory with no KV cache. Echo fits a spectral linear system to the key and value history via kernel ridge regression and retrieves through a learned power-iterated filter, all from

O(r^{2})

streaming state where

r

is a small projection rank. On the Multi-Query Associative Recall benchmark, a pure Mamba-2 SSM fails to exceed chance accuracy (

{\sim}3\%

) across all gap lengths and KV-pair counts, while at the 50M parameter scale SKA-augmented models achieve

100\%

retrieval accuracy on every configuration tested, including distractor gaps of

4{,}096

tokens with

32

KV pairs. Across five additional transfer benchmarks including needle-in-a-haystack, tool-trace, and multi-hop retrieval, SKA consistently outperforms both pure SSM and SSM+Attention hybrids while maintaining constant inference memory. Ablations confirm that the spectral operator, not the prefix masking strategy, drives the retrieval gain.

Anupama Sridhar, Alexander Johansen

California, USA · Stanford University Computer Science California, USA

State Space Language Models Koopman Operator Learning Memory-Efficient Inference Language Modeling Associative Memory

Ask PDF

Apr 23, 2026

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

Transformers process tokens in parallel but are temporally shallow: at position

t

, each layer attends to key-value pairs computed based on the previous layer, yielding a depth capped by the number of layers. Recurrent models offer unbounded temporal depth but suffer from optimization instability and historically underutilize modern accelerators. We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations, yielding layerwise recurrent memory while preserving standard autoregressive decoding cost. We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-token recurrent updates under mild assumptions, while avoiding optimization instability. Naively, prefill/training appears bandwidth-bound with effective arithmetic intensity near

1

because keys and values are revealed sequentially; we give an exact tiling-based algorithm that preserves the mathematical computation while reducing HBM traffic from

Θ(N^2)

Θ(N\log N)

, increasing effective arithmetic intensity to

Θ(N/\log N)

for sequence length

N

. On 150M and 300M parameter C4 pretraining, Recurrent Transformers improve cross-entropy over a parameter-matched Transformer baseline and achieve the improvement with fewer layers (fixed parameters), suggesting that recurrence can trade depth for width, thus reducing KV cache memory footprint and inference latency.

Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3

Harvard University

Autoregressive Language Modeling Self-Attention Attention Mechanisms Efficient Transformer Inference Recurrent Transformers Efficient Attention+1

Ask PDF

Abstract

LLM Inference Acceleration

Explore similar work

DART: Decoded Attention over Recurrent States for Efficient Long-Context Sequence Modeling

Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding