Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Oliver Sieberling; Bharat Runwal; David Jin; Ryan Chin; Rameswar Panda; Yoon Kim

cs.LGSep 29, 2026

Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Authors: Oliver Sieberling, Bharat Runwal, David Jin, Ryan Chin, Rameswar Panda, Yoon Kim

Organizations: Massachusetts Institute of Technology · MIT-IBM Computing Research Lab

Abstract

Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear attention, which extends the vector-valued hidden states of ordinary RNNs to matrix-valued hidden states. Crucially, linear attention does so in a parameter-efficient way, in particular by using an outer product of the key and value vectors to write to the matrix-valued hidden state. We generalize this construction and propose triadic linear attention, which writes the triadic outer product of a key, a second key, and a value, into a third-order (i.e., 3D) tensor state, and reads from it by contracting both key axes with two queries. An $E$ -dimensional second key thus yields an $E$ -fold increase in state size while adding only two projections. Triadic linear attention is compatible with data-dependent forgetting, the delta rule, and chunkwise-parallel training. Applied to Gated DeltaNet and scalar-gated linear attention, triadic linear attention substantially improves long-context language modeling and recall, outperforming alternatives that enlarge the state.

Figures & tables

Figure 1: MQAR with plain linear attention (state 16 $\times$ 16) and triadic linear attention with $E\!\in\!\{2,4,8,16\}$ (state 16 $\times$ E $\times$ 16).

Panel of Figure 2 — Figure 2: Triadic Gated DeltaNet with a growing second key dimension $E$ at 400M parameters (top) and 1.3B parameters (bottom). Left: Loss difference to the Transformer on PG19 by context position, with dashed lines marking where the key-value cache of the Transformer exceeds the state of each model. Right: Accuracy on six recall-intensive tasks.

Model	State	Params	0-shot $\uparrow$	Wiki. $\downarrow$	PG19 $\leq$ 4k $\downarrow$	PG19 4k–16k $\downarrow$	PG19 16k–64k $\downarrow$	Recall $\uparrow$
Transformer	12.6/1k tok	377.0	52.6	11.11	15.39	14.41	13.92	41.6
GDN (base)	6.3	380.9	53.1	11.25	15.03	14.38	14.15	26.2
Larger heads ( $d{=}256$ )	12.6	380.7	52.6	11.22	15.10	14.42	14.17	26.6
Grouped values (2 per key)	12.6	378.2	52.9	11.18	15.05	14.37	14.12	27.1
Wider values ( $d_{v}{=}256$ )	12.6	377.9	53.2	11.20	15.07	14.39	14.14	27.4
More heads (16)	12.6	381.6	53.3	11.32	15.25	14.54	14.28	27.5

Table 1: Different ways of enlarging the state size of Gated DeltaNet (GDN) and scalar-gated linear attention (sGLA) by

2\times

and

4\times

at matched parameter count. State in MB (assuming 2 bytes per entry), and parameters in millions. Results are averaged over three training seeds.

Figure 3: RULER needle-in-a-haystack accuracy by context length at 1.3B parameters.

Model	State	Params	0-shot $\uparrow$	Wiki. $\downarrow$	PG19 $\leq$ 4k $\downarrow$	PG19 4k–16k $\downarrow$	PG19 16k–64k $\downarrow$	Recall $\uparrow$
GDN (base)	6.3	380.9	53.1	11.25	15.03	14.38	14.15	26.2
Triadic ( $E{=}8$ ), from scratch	50.3	385.4	53.4	10.87	14.79	14.04	13.74	33.1
Triadic ( $E{=}8$ ), upcycled	50.3	385.4	53.2	11.04	14.91	14.18	13.86	29.7
sGLA (base)	6.3	380.9	53.5	11.41	15.19	14.55	14.32	26.4
Triadic ( $E{=}8$ ), from scratch	50.3	385.4	53.4	11.03	14.96	14.23	13.95	32.1
Triadic ( $E{=}8$ ), upcycled	50.3	385.4	53.3	11.16	15.04	14.32	14.03	30.0

Table 2: Upcycling a pretrained model from

E=1

E=8

during long-context extension, compared with the corresponding base model and with

E=8

pretrained from scratch. Results are averaged over three seeds at 400M and reported for a single seed at 1.3B.

Model	State @ 4k $\downarrow$	State @ 64k $\downarrow$	0-shot $\uparrow$	Wiki. $\downarrow$	PG19 $\leq$ 4k $\downarrow$	PG19 4k–64k $\downarrow$	Recall $\uparrow$	NIAH Avg. $\uparrow$
Transformer	50	805	52.6	11.11	15.39	14.02	41.6	48.2
GDN	6	6	53.1	11.25	15.03	14.19	26.2	21.7
3:1 GDN/GQA-8 (base)	17	206	53.0	10.69	14.86	13.50	43.7	51.5
Larger KV cache (GQA-4)	30	407	53.5	10.67	14.83	13.47	45.3	53.0
Triadic state ( $E{=}4$ )	31	220	53.8	10.63	14.76	13.39	44.5	56.3

Table 3: Enlarging either the key-value cache (GQA-4) or the linear-attention state (Triadic,

E=4

) in a 3:1 GDN/GQA-8 hybrid at 400M parameters. State is reported in MB at 4k and 64k tokens. NIAH is averaged over eight RULER tasks. Results are averaged over three seeds.

Figure 4: Forward and backward time of one block of the 1.3B models per sequence, at batch size 4 on H100 GPUs. Triadic GDN with $E=8$ adds moderate overhead over GDN and is faster than the Transformer beyond 4k tokens.

Model	State	Params	0-shot $\uparrow$	Wiki. $\downarrow$	PG19 $\leq$ 4k $\downarrow$	PG19 4k–16k $\downarrow$	PG19 16k–64k $\downarrow$	Recall $\uparrow$
Triadic GDN (scalar gate), $d_{k}=128$ , $E=8$	50.3	384.0	53.7	10.80	14.77	14.01	13.70	33.4
$d_{k}=64$ , $E=16$	50.3	380.8	52.9	10.84	14.79	14.04	13.73	32.5
$d_{k}=32$ , $E=32$	50.3	383.9	53.3	10.90	14.82	14.08	13.78	32.5
$d_{k}=32$ , $E=32$ , tied keys	50.3	380.7	52.1	10.89	14.81	14.09	13.79	31.9
Triadic GDN ( $E=8$ ), Softplus activation	50.3	385.4	53.5	10.86	14.81	14.07	13.77	32.2
Sigmoid activation	50.3	385.4	53.7	10.86	14.82	14.07	13.77	33.3

Table 4: Ablations of Triadic Gated DeltaNet at 400M parameters. Top: first key dimension

d_{k}

versus second key dimension

E

at fixed joint key dimension

d_{k}\cdot E=1024

, using a single forget gate per head. Bottom: activation function applied to the second key and query for

E=8

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: How Triadic GDN ( $E{=}8$ ) uses the larger state compared to GDN. Left: change in loss on the final 4k tokens of a book when increasing the preceding context from 4k to 64k. Middle: distribution of delta strengths $\beta$ . Dashed lines mark the means. Right: half-life of each state slice, ranked within its head.

Explore similar work

Jul 8, 2026

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.

Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6

Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen

Long-Context Retrieval Memory-Augmented Language Models Recurrent Neural Networks Delta-Rule Linear Attention In-Context Learning Gated Delta Networks+1

Ask PDF

Oct 11, 2026

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: recurrent computation need not bear the full burden of long-term storage. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This training-free, plug-and-play interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds

\leqslant

0.025% to analytical FLOPs, with persistent context storage approximately 1/2,600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to Mamba-2-8B and post-trained Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.

Li Nanbo, Firas Laakom, Yilan Zhang +10

King Abdullah University of Science and Technology (KAUST)

Long-Context Modeling Memory-Augmented Language Models Linear Attention

Ask PDF

Oct 5, 2026

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

Linear attention enables efficient long-context autoregressive decoding by compressing history into recurrent states, but this compression can make selective access to sparse and distant information difficult. Existing chunk-based extensions increase memory capacity, yet learned chunk-mixing coefficients may remain fixed with respect to input content and therefore cannot adapt historical access to each query. We introduce Hybrid Linear Attention (HLA), a query-dependent chunk-level attention mechanism for Gated DeltaNet (GDN). HLA represents each completed chunk as an exact affine state transition and computes content-dependent routing gates from compact, self-attentively pooled representatives. Each gate interpolates the corresponding historical transition with the identity map, controlling both the chunk's additive memory and its transformation of earlier states. Effective-support regularization further encourages concentrated routing for sparse inference. We evaluate HLA under both pretrained adaptation and from-scratch training. Across Qwen3.5 models from 0.8B to 9B, HLA consistently improves over native GDN and fixed chunk mixing, with gains of up to 5.57 percentage points on LongBench-V2 and 3.97 points on RULER. In a controlled from-scratch 1.3B setting trained for 100B tokens with a 4K context, HLA also improves RULER performance from 4K to 32K, with gains increasing from 0.83 points at 4K to 4.22 points at 32K. These results demonstrate that query-dependent composition of recurrent memory improves long-context modeling and remains effective beyond the training context while using compact per-chunk affine summaries. Project page: https://caesarhhh.github.io/hla/

Zhuokun Chen, Xi Lin, Xiyu Wu +3

Monash University · Zhejiang University

Long-Context Language Modeling Linear Attention Delta-Rule Linear Attention Attention Mechanisms

Ask PDF