Recurrent memories let language models process long sequences with a fixed-size state. Linear memories such as DeltaNet and KDA store one matrix and take one delta-rule step per token. The product of a chunk of such steps has an exact WY form, so they train fast. Deep memories such as TTT-MLP and Titans store a small MLP, which can hold maps that no matrix can, but their update has no WY form. Toaster bridges the two by building on a known fact: a gradient step on a linear layer is a delta-rule write. It builds a two-layer nonlinear memory from two linear memories with an activation between them. We use target propagation to give the hidden memory its values by projecting the output error back through the second memory. Given a chunk's cross-layer signals, approximated once per chunk, each layer can then run on an unmodified delta-rule kernel. Our default runs this kernel on the output layer and updates the hidden layer once per chunk. Running it on both layers gives a better model in our ablation. At matched backbones and training-token budgets, Toaster has lower loss than KDA at all five tested sizes (0.2B-1.6B non-embedding parameters) on both C4 and PG19. Its C4 perplexity also matches or slightly improves on the attention baseline at every size. At 1.2B it is also below Titans. In a 3:1 hybrid with global attention, the Toaster hybrid has the lowest perplexity of the four hybrids tested, and it is 0.18 (C4) and 0.44 (PG19) below the attention baseline. At the largest size, Toaster's mean three-shot accuracy on 13 tasks is 51.84%, compared with 50.52% for KDA and 50.04% for Titans. In length extrapolation, Toaster's loss stays below KDA's up to 20K tokens (5x the training length). On the 145M backbone, at a hidden width matched to Titans' memory, Toaster takes at most 1.33x KDA's training-step time and is 2.8-5.5x faster than the tested Titans configurations at 32K context.
Figures & tables
Figure 1: Toaster at a glance. (a) A linear memory applies one Householder step per token, so a chunk of updates has an exact WY form. (b) A deep memory is more expressive, but its gradient update through the nonlinearity has no such form. (c) Toaster cascades two linear memories around a pointwise activation σ . The output error et is projected back through the second memory into a target v~t for the first, so both layers are plain linear memories that the WY kernel can advance (by default the first uses the cheaper apply-then-update step of Algorithm 1 ); a chunk’s targets and hidden keys are computed once, in parallel, from the chunk-initial states.
Deep (test-time training): TTT-MLP, Titans, Atlas, LaCT, TNT
✗
✓
Toaster (this work)
✓ per layer †
✓
Table 1: Comparison between linear memory, deep memory, and Toaster . † Given the chunk’s cross-layer signals; by default the output memory uses the kernel and the hidden memory a summed chunk step (Section 3.4 ).
C4 perplexity ↓ (in-domain)
PG19 perplexity ↓
Model
145M
300M
550M
850M
1.2B
145M
300M
550M
850M
1.2B
Attention
15.92
13.52
11.94
10.78
9.96
22.41
18.73
16.40
14.52
13.19
KDA
16.00
13.57
11.97
10.83
10.00
22.64
18.98
16.56
14.66
13.37
Toaster
15.69
13.40
11.85
10.76
9.94
22.21
18.68
16.37
14.57
13.29
Table 2: Validation perplexity across the scaling ladder (lower is better; best per column in bold). Figure 2 plots the C4 columns against training compute; the corresponding losses are in Appendix H .
Figure 2: Language modeling across the scaling ladder. Left: the C4 perplexity of Table 2 against training compute ( 6ND FLOPs). Right: compute-efficiency gain relative to Attention at matched 6ND compute. Details and all numbers are in Appendix G.2 and Appendix H .
Perplexity ↓
Downstream accuracy (%) ↑
Model
C4
PG19
ARC-c
ARC-e
COPA
Hella.
OBQA
PIQA
SWAG
MultiRC
MathQA
CB
MNLI
RTE
SST-2
Avg.
All layers replaced
Attention
9.96
13.19
30.64
60.68
67.00
62.03
38.60
75.14
61.20
57.16
21.70
50.00
31.93
53.79
49.08
50.69
KDA
10.00
13.37
31.16
61.06
70.00
61.33
38.40
75.19
62.47
57.12
21.68
50.00
31.99
47.29
49.08
50.52
Titans
10.09
13.48
30.84
56.07
73.00
58.47
39.20
73.72
61.20
57.05
21.65
50.00
31.87
48.38
49.08
50.04
Toaster
9.94
13.29
32.62
61.55
72.00
61.23
38.40
76.16
62.17
57.20
21.72
53.57
33.73
54.51
49.08
51.84
Table 3: Validation perplexity and three-shot downstream accuracy (%) at 1.2B , with all layers replaced by the sequence mixer (top) and in a 3:1 hybrid with global attention (bottom), in which one softmax-attention layer follows every three mixer layers; SWA is sliding-window attention with a 128-token window. Best and second best per column, within each group. Per-category averages and task details are in Appendix G.5 and Appendix H .
Figure 3: Length extrapolation at 1.2B. Left: perplexity by position. Right: Toaster ’s loss minus KDA’s (negative favors Toaster ). Curves are running averages over positions (Appendix G.6 ).
S-NIAH-1 (passkey)
S-NIAH-2 (number)
S-NIAH-3 (UUID)
Model
512
1K
2K
4K
512
1K
2K
4K
512
1K
2K
4K
Attention
100.0
100.0
100.0
96.3
100.0
100.0
100.0
79.0
100.0
100.0
100.0
85.7
KDA
100.0
100.0
100.0
100.0
100.0
99.0
88.7
37.3
98.0
96.0
77.7
28.0
Toaster
100.0
100.0
100.0
100.0
100.0
99.3
90.3
62.7
98.0
97.0
73.7
27.3
Table 4: Single needle-in-a-haystack accuracy (%) at 1.2B , averaged over 15 needle positions, by context length (Appendix G.7 ).
Figure 8
Reference
State width
Activation σ
Hidden-layer learning
Bias
KDA
Toaster
dh=dk
dh=2dk
identity
leaky ReLU
γ=0
WY kernel
w/o
C4 ppl ( ↓ )
16.00
15.69
15.90
15.76
15.80
15.71
15.71
15.64
15.86
Δ
+ 0.31
–
+ 0.22
+ 0.07
+ 0.11
+ 0.02
+ 0.02
− 0.05
+ 0.17
Table 5: Ablations at 145M : C4 perplexity and its difference Δ to the default Toaster (red: worse, blue: better); each column changes one component. Full results in Table 13 (Appendix G.9 ).
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Size
Layers
Width D
Heads
Non-emb.
Total
Batch
Steps
Tokens
Peak LR
Weight decay
145M
12
1024
16
201M
234M
64
35,050
9.2B
1.95×10−3
2.28×10−4
300M
15
1280
20
390M
431M
96
42,708
16.8B
1.56×10−3
1.87×10−4
550M
18
1536
24
671M
721M
160
42,630
27.9B
1.30×10−3
1.88×10−4
850M
21
1792
28
1.06B
1.12B
192
55,095
43.3B
1.12×10−3
1.45×10−4
1.2B
24
2048
32
1.58B
1.65B
256
60,725
63.7B
9.77×10−4
1.32×10−4
Appendix
Table 6: Scaling ladder. Depth, width, and number of heads grow together; the head dimension is 64, the feed-forward inner width is 4D , and the context length is 4,096 tokens throughout. Parameter counts are those reported for the ladder configuration; the recurrent mixers add a small number of parameters for their gates and convolutions. The token budget is roughly proportional to model size (46 to 40 tokens per non-embedding parameter).
Compute-efficiency gain vs. Attention
Corpus
Model
α
R2
145M
300M
550M
850M
1.2B
C4 (in-domain)
Attention
0.047
0.9997
1
1
1
1
1
KDA
0.047
0.9996
0.96
0.97
0.98
0.96
0.96
Toaster
0.045
0.9998
1.12
1.07
1.07
1.02
1.01
PG19
Attention
0.047
0.9997
1
1
1
1
1
KDA
0.046
0.9998
0.93
0.91
0.93
0.92
0.89
Appendix
Table 7: Power-law fits L=(Cc/C)α of validation loss against training compute (log–log least squares over the five sizes) and the compute-efficiency gain relative to Attention of Eq. ( 21 ) at the five measured sizes ( g>1 : Attention needs g times the model’s compute to reach the model’s loss).
145M
300M
550M
850M
1.2B
Mixer share
Attention
25.0%
20.0%
16.7%
14.3%
12.5%
KDA
2.1%
1.7%
1.4%
1.2%
1.1%
Toaster
15.2%
12.2%
10.2%
8.7%
7.6%
Toaster , kernel on both layers
18.9%
15.2%
12.6%
10.8%
9.5%
C4 gain, mixer-inclusive
KDA
1.18
1.14
1.13
1.08
1.07
Toaster
1.22
1.15
1.13
1.07
1.06
Appendix
Table 8: Sequence-mixer cost and the compute-efficiency gain with it counted. Top: forward operations of the sequence mixer per token as a share of a block’s dense projections ( 16D2 ; Appendix F ), for Attention at the 4,096-token context. Middle: C4 gain relative to Attention, Eq. ( 21 ), with each model’s compute multiplied by one plus its share. Bottom: ratio of Toaster ’s gain to KDA’s at matched 6ND and at matched mixer-inclusive compute.
Model
Setting
2K
4K
8K
16K
32K
Flash Attention
639
796
817
1,027
1,138
KDA
C=32
880
905
948
990
1,085
Titans
dh=64 , C=16
1,793
2,098
2,808
4,432
7,980
Titans
dh=64 , C=32
1,202
1,215
1,641
2,399
4,097
Toaster
dh=64 , C=32 (width-matched)
990
1,023
1,233
1,303
1,444
Toaster
dh=256 , C=32 (default)
1,105
1,223
1,325
1,463
1,544
Appendix
Table 9: Training-step time (ms) at a fixed budget of 512K tokens per step, measured on the 145M configuration. The first Toaster row is the width-matched configuration of Figure 5 ; the second is the default configuration of all quality experiments. Both advance the hidden memory by apply-then-update and the output memory by the KDA kernel. The last two rows give ratios of step times.
Figure 6: Scaling ladder on C4 and PG19 (single runs). Top: validation loss against training compute ( 6ND FLOPs) with the power-law fits of Appendix G.2 . Middle: loss minus Attention’s loss at the same compute (below zero is better than Attention). Bottom: compute-efficiency gain relative to Attention, Eq. ( 21 ), at the measured sizes.
Loss (nats)
Perplexity
Δ loss of Toaster vs.
Corpus / size
Attention
KDA
Toaster
Attention
KDA
Toaster
KDA
Attention
C4 (in-domain)
145M
2.7678
2.7725
2.7530
15.92
16.00
15.69
− 0.0195
− 0.0148
300M
2.6039
2.6082
2.5955
13.52
13.57
13.40
− 0.0127
− 0.0084
550M
2.4799
2.4820
2.4726
11.94
11.97
11.85
− 0.0094
− 0.0073
850M
2.3774
2.3822
2.3755
10.78
10.83
10.76
− 0.0067
− 0.0019
Appendix
Table 10: Validation loss (nats per token) and perplexity across the scaling ladder (lower is better; best per row within each block in bold), and the difference in loss between Toaster and KDA and between Toaster and Attention (negative favors Toaster ). Single runs.
Figure 7: Per-task three-shot accuracy relative to Attention at 1.2B , for the three models trained along the ladder (points; right of the dashed line is better than Attention; single runs). Toaster is above Attention on 10 of 13 tasks and tied on SST-2; its largest margins (COPA, CB, RTE) fall on the smallest evaluation sets, and most other differences are within one binomial standard error (Appendix G.5 ).
Figure 8: Pressure test on S-NIAH-1 (passkey) at 1.2B. Accuracy (%) for every needle depth (0%: needle at the start of the context, farthest from the query) and context length (Appendix G.7 ).
Figure 9: Pressure test on S-NIAH-2 (number in essays) at 1.2B. Same layout as Figure 8 .
Figure 10: Pressure test on S-NIAH-3 (UUID in essays) at 1.2B. Same layout as Figure 8 .
All layers replaced
Hybrid (3 mixer : 1 attention)
Task
Attention
KDA
Titans
Toaster
SWA
KDA
Titans
Toaster
Science QA
ARC-Challenge
30.64
31.16
30.84
32.62
33.39
32.02
29.36
32.19
ARC-Easy
60.68
61.06
56.07
61.55
61.01
60.25
57.72
62.07
average
45.66
46.11
43.45
47.08
47.20
46.14
43.54
47.13
Commonsense reasoning
Appendix
Table 11: Three-shot accuracy (%) on all 13 downstream tasks at 1.2B (nominal), with all layers replaced by the sequence mixer (left) and in the 3:1 hybrid with global attention (right; Table 3 ), and the average within each skill category. Bold : best per row within each group; shaded : second best; rows on which a whole group ties (SST-2) are left unmarked.
Window
Attention
KDA
Toaster
Toaster − KDA
0–4K
2.579
2.589
2.582
− 0.008
4–8K
2.673
2.493
2.479
− 0.014
8–12K
3.444
2.491
2.481
− 0.010
12–16K
6.012
2.480
2.469
− 0.011
16–20K
6.756
2.497
2.488
− 0.009
4–20K
4.721
2.490
2.479
− 0.011
Appendix
Table 12: Length extrapolation at 1.2B: mean loss (nats per token) of each model in 4K-token windows of the 20K-token sequences of Figure 3 , and the difference between Toaster and KDA (negative favors Toaster ; the per-window standard error of the difference is about 0.001 nats). The five 4K windows are consecutive and do not overlap; the last row pools 4–20K. The training context is 4K tokens.
Toaster ablations (145M)
C4 ppl ( ↓ )
Δ
KDA (linear memory)
16.00
+ 0.31
Toaster (default)
15.69
–
Bias
w/o. homogeneous-coordinate bias
15.86
+ 0.17
Hidden width
dh=dk (64)
15.90
+ 0.22
Appendix
Table 13: Ablation study at 145M (9.2B tokens; Appendix G.9 ). Each row changes one component of the default Toaster (bias, dh=4dk , GELU, γ=0.01 , apply-then-update) and reports C4 perplexity and its difference Δ to the default (red: worse, blue: better). The default row is the ablation’s own run (15.687; the ladder’s 145M run gives 15.690).
Model (145M)
S(1) changes
C4 ppl ( ↓ )
KDA
–
16.00
Toaster , S(1) frozen completely
no (0%)
15.86
Toaster , γ=0 (forget gate still active)
yes
15.71
Toaster , default (full target propagation)
yes (4.7%)
15.69
Appendix
Table 14: Freezing the hidden memory (145M, C4 perplexity). All Toaster rows share the architecture and the state size; the γ=0 row is that of Table 13 . The middle column gives the relative change of S(1) over a 4K context.
Decoding chunk size C′
1
4
8
16
32
64
Default kernel on both layers
15.97
15.90
15.84
15.79
15.77
15.79
Training chunk size C
8
16
32
64
128
256
Toaster (default)
15.59
15.56
15.69
15.67
15.69
15.74
Appendix
Table 15: Effect of the chunk size (145M, C4 perplexity). Top: one model with hidden width dh=dk and the default kernel on both layers (token by token, with apply-then-update on the hidden memory and update-then-apply on the output memory; a separate run, not a row of Table 13 ), trained with C=32 and evaluated with decoding chunk size C′ ; C′=1 is the exact per-token recurrence. Bottom: the default Toaster ( dh=4dk ) trained with chunk size C .
Total state (relative to default KDA)
1×
2×
4×
8×
KDA, heads × head dimension
16×64
8×128
4×256
2×512
KDA, C4 ppl
16.00
15.96
15.84
15.79
Toaster , hidden width dh
–
64
128
256
Toaster , C4 ppl
–
15.90
15.76
15.69
Appendix
Table 16: KDA with a larger state (145M, C4 perplexity). KDA trades heads for wider heads at fixed H⋅dk=D=1024 ; Toaster keeps 16 heads of dimension 64 and widens its hidden layer.
S-NIAH-1 (passkey)
S-NIAH-2 (number)
S-NIAH-3 (UUID)
Hybrid
512
1K
2K
4K
512
1K
2K
4K
512
1K
2K
4K
SWA
100.0
100.0
100.0
89.3
99.3
100.0
100.0
78.7
98.7
93.7
91.7
56.3
KDA
100.0
99.7
99.3
98.7
100.0
99.7
100.0
73.3
93.0
96.7
81.7
44.0
Titans
100.0
100.0
100.0
84.7
100.0
99.7
93.7
81.7
99.3
91.0
96.0
59.7
Toaster
100.0
100.0
100.0
99.7
100.0
100.0
97.3
89.3
99.7
90.7
96.3
77.3
Appendix
Table 17: Single needle-in-a-haystack accuracy (%) of the 3:1 hybrids at 1.2B , averaged over 15 needle positions, by context length (protocol of Table 4 ).
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.