Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Oliver Sieberling; Bharat Runwal; David Jin; Ryan Chin; Rameswar Panda; Yoon Kim

cs.LGSep 29, 2026

Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

Authors: Oliver Sieberling, Bharat Runwal, David Jin, Ryan Chin, Rameswar Panda, Yoon Kim

Organizations: Massachusetts Institute of Technology · MIT-IBM Computing Research Lab

Abstract

Recurrent neural networks (RNNs) compress the historical context into a memory state of fixed size, thus allowing for constant-time inference. The memory state size is a crucial factor in their performance, as exemplified by the strong performance and resurgence of linear attention, which extends the vector-valued hidden states of ordinary RNNs to matrix-valued hidden states. Crucially, linear attention does so in a parameter-efficient way, in particular by using an outer product of the key and value vectors to write to the matrix-valued hidden state. We generalize this construction and propose triadic linear attention, which writes the triadic outer product of a key, a second key, and a value, into a third-order (i.e., 3D) tensor state, and reads from it by contracting both key axes with two queries. An $E$ -dimensional second key thus yields an $E$ -fold increase in state size while adding only two projections. Triadic linear attention is compatible with data-dependent forgetting, the delta rule, and chunkwise-parallel training. Applied to Gated DeltaNet and scalar-gated linear attention, triadic linear attention substantially improves long-context language modeling and recall, outperforming alternatives that enlarge the state.

Figures & tables

Figure 1: MQAR with plain linear attention (state 16 $\times$ 16) and triadic linear attention with $E\!\in\!\{2,4,8,16\}$ (state 16 $\times$ E $\times$ 16).

Panel of Figure 2 — Figure 2: Triadic Gated DeltaNet with a growing second key dimension $E$ at 400M parameters (top) and 1.3B parameters (bottom). Left: Loss difference to the Transformer on PG19 by context position, with dashed lines marking where the key-value cache of the Transformer exceeds the state of each model. Right: Accuracy on six recall-intensive tasks.

Model	State	Params	0-shot $\uparrow$	Wiki. $\downarrow$	PG19 $\leq$ 4k $\downarrow$	PG19 4k–16k $\downarrow$	PG19 16k–64k $\downarrow$	Recall $\uparrow$
Transformer	12.6/1k tok	377.0	52.6	11.11	15.39	14.41	13.92	41.6
GDN (base)	6.3	380.9	53.1	11.25	15.03	14.38	14.15	26.2
Larger heads ( $d{=}256$ )	12.6	380.7	52.6	11.22	15.10	14.42	14.17	26.6
Grouped values (2 per key)	12.6	378.2	52.9	11.18	15.05	14.37	14.12	27.1
Wider values ( $d_{v}{=}256$ )	12.6	377.9	53.2	11.20	15.07	14.39	14.14	27.4
More heads (16)	12.6	381.6	53.3	11.32	15.25	14.54	14.28	27.5

Table 1: Different ways of enlarging the state size of Gated DeltaNet (GDN) and scalar-gated linear attention (sGLA) by

2\times

and

4\times

at matched parameter count. State in MB (assuming 2 bytes per entry), and parameters in millions. Results are averaged over three training seeds.

Figure 3: RULER needle-in-a-haystack accuracy by context length at 1.3B parameters.

Model	State	Params	0-shot $\uparrow$	Wiki. $\downarrow$	PG19 $\leq$ 4k $\downarrow$	PG19 4k–16k $\downarrow$	PG19 16k–64k $\downarrow$	Recall $\uparrow$
GDN (base)	6.3	380.9	53.1	11.25	15.03	14.38	14.15	26.2
Triadic ( $E{=}8$ ), from scratch	50.3	385.4	53.4	10.87	14.79	14.04	13.74	33.1
Triadic ( $E{=}8$ ), upcycled	50.3	385.4	53.2	11.04	14.91	14.18	13.86	29.7
sGLA (base)	6.3	380.9	53.5	11.41	15.19	14.55	14.32	26.4
Triadic ( $E{=}8$ ), from scratch	50.3	385.4	53.4	11.03	14.96	14.23	13.95	32.1
Triadic ( $E{=}8$ ), upcycled	50.3	385.4	53.3	11.16	15.04	14.32	14.03	30.0

Table 2: Upcycling a pretrained model from

E=1

E=8

during long-context extension, compared with the corresponding base model and with

E=8

pretrained from scratch. Results are averaged over three seeds at 400M and reported for a single seed at 1.3B.

Model	State @ 4k $\downarrow$	State @ 64k $\downarrow$	0-shot $\uparrow$	Wiki. $\downarrow$	PG19 $\leq$ 4k $\downarrow$	PG19 4k–64k $\downarrow$	Recall $\uparrow$	NIAH Avg. $\uparrow$
Transformer	50	805	52.6	11.11	15.39	14.02	41.6	48.2
GDN	6	6	53.1	11.25	15.03	14.19	26.2	21.7
3:1 GDN/GQA-8 (base)	17	206	53.0	10.69	14.86	13.50	43.7	51.5
Larger KV cache (GQA-4)	30	407	53.5	10.67	14.83	13.47	45.3	53.0
Triadic state ( $E{=}4$ )	31	220	53.8	10.63	14.76	13.39	44.5	56.3

Table 3: Enlarging either the key-value cache (GQA-4) or the linear-attention state (Triadic,

E=4

) in a 3:1 GDN/GQA-8 hybrid at 400M parameters. State is reported in MB at 4k and 64k tokens. NIAH is averaged over eight RULER tasks. Results are averaged over three seeds.

Figure 4: Forward and backward time of one block of the 1.3B models per sequence, at batch size 4 on H100 GPUs. Triadic GDN with $E=8$ adds moderate overhead over GDN and is faster than the Transformer beyond 4k tokens.

Model	State	Params	0-shot $\uparrow$	Wiki. $\downarrow$	PG19 $\leq$ 4k $\downarrow$	PG19 4k–16k $\downarrow$	PG19 16k–64k $\downarrow$	Recall $\uparrow$
Triadic GDN (scalar gate), $d_{k}=128$ , $E=8$	50.3	384.0	53.7	10.80	14.77	14.01	13.70	33.4
$d_{k}=64$ , $E=16$	50.3	380.8	52.9	10.84	14.79	14.04	13.73	32.5
$d_{k}=32$ , $E=32$	50.3	383.9	53.3	10.90	14.82	14.08	13.78	32.5
$d_{k}=32$ , $E=32$ , tied keys	50.3	380.7	52.1	10.89	14.81	14.09	13.79	31.9
Triadic GDN ( $E=8$ ), Softplus activation	50.3	385.4	53.5	10.86	14.81	14.07	13.77	32.2
Sigmoid activation	50.3	385.4	53.7	10.86	14.82	14.07	13.77	33.3

Table 4: Ablations of Triadic Gated DeltaNet at 400M parameters. Top: first key dimension

d_{k}

versus second key dimension

E

at fixed joint key dimension

d_{k}\cdot E=1024

, using a single forget gate per head. Bottom: activation function applied to the second key and query for

E=8

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 5: How Triadic GDN ( $E{=}8$ ) uses the larger state compared to GDN. Left: change in loss on the final 4k tokens of a book when increasing the preceding context from 4k to 64k. Middle: distribution of delta strengths $\beta$ . Dashed lines mark the means. Right: half-life of each state slice, ranked within its head.

Explore similar work

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Jul 8, 2026Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy +6Long-Context Retrieval Memory-Augmented Language Models
Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
Oct 11, 2026Li Nanbo, Firas Laakom, Yilan Zhang +10Long-Context Modeling Memory-Augmented Language Models
King Abdullah University of Science and Technology (KAUST)
HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing
Oct 5, 2026Zhuokun Chen, Xi Lin, Xiyu Wu +3Long-Context Language Modeling Linear Attention
Monash University · Zhejiang University

Jul 8, 2026J/K move · Enter open · S save

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

Loïc Cabannes, Pierre-Emmanuel Mazaré, Gergely Szilvasy, Matthijs Douze+5

Meta FAIR · Inria Paris & ENS-PSL University · University of Tübingen

Long-Context Retrieval Memory-Augmented Language Models Recurrent Neural Networks Delta-Rule Linear Attention In-Context Learning Gated Delta Networks+1

Open paper AskPDF arXiv

Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.

Abstract

Explore similar work

Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

History Replays Itself: Unleashing Long-Context Capability in Linear Attention

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing