Toaster: Nonlinear Memory from Cascaded Linear Memories and Target Propagation
Organizations: Google Research · University of Southern California
Abstract
Recurrent memories let language models process long sequences with a fixed-size state. Linear memories such as DeltaNet and KDA store one matrix and take one delta-rule step per token. The product of a chunk of such steps has an exact WY form, so they train fast. Deep memories such as TTT-MLP and Titans store a small MLP, which can hold maps that no matrix can, but their update has no WY form. Toaster bridges the two by building on a known fact: a gradient step on a linear layer is a delta-rule write. It builds a two-layer nonlinear memory from two linear memories with an activation between them. We use target propagation to give the hidden memory its values by projecting the output error back through the second memory. Given a chunk's cross-layer signals, approximated once per chunk, each layer can then run on an unmodified delta-rule kernel. Our default runs this kernel on the output layer and updates the hidden layer once per chunk. Running it on both layers gives a better model in our ablation. At matched backbones and training-token budgets, Toaster has lower loss than KDA at all five tested sizes (0.2B-1.6B non-embedding parameters) on both C4 and PG19. Its C4 perplexity also matches or slightly improves on the attention baseline at every size. At 1.2B it is also below Titans. In a 3:1 hybrid with global attention, the Toaster hybrid has the lowest perplexity of the four hybrids tested, and it is 0.18 (C4) and 0.44 (PG19) below the attention baseline. At the largest size, Toaster's mean three-shot accuracy on 13 tasks is 51.84%, compared with 50.52% for KDA and 50.04% for Titans. In length extrapolation, Toaster's loss stays below KDA's up to 20K tokens (5x the training length). On the 145M backbone, at a hidden width matched to Titans' memory, Toaster takes at most 1.33x KDA's training-step time and is 2.8-5.5x faster than the tested Titans configurations at 32K context.
Figures & tables
| Memory | Exact chunkwise (WY) form | Nonlinear state |
|---|---|---|
| Linear (delta rule): DeltaNet, Gated DeltaNet, KDA, Gated DeltaNet-2, Comba, DeltaProduct | ✓ | ✗ |
| Deep (test-time training): TTT-MLP, Titans, Atlas, LaCT, TNT | ✗ | ✓ |
| Toaster (this work) | ✓ per layer † | ✓ |
| C4 perplexity (in-domain) | PG19 perplexity | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | 145M | 300M | 550M | 850M | 1.2B | 145M | 300M | 550M | 850M | 1.2B |
| Attention | 15.92 | 13.52 | 11.94 | 10.78 | 9.96 | 22.41 | 18.73 | 16.40 | 14.52 | 13.19 |
| KDA | 16.00 | 13.57 | 11.97 | 10.83 | 10.00 | 22.64 | 18.98 | 16.56 | 14.66 | 13.37 |
| Toaster | 15.69 | 13.40 | 11.85 | 10.76 | 9.94 | 22.21 | 18.68 | 16.37 | 14.57 | 13.29 |
| Perplexity | Downstream accuracy (%) | |||||||||||||||
| Model | C4 | PG19 | ARC-c | ARC-e | COPA | Hella. | OBQA | PIQA | SWAG | MultiRC | MathQA | CB | MNLI | RTE | SST-2 | Avg. |
| All layers replaced | ||||||||||||||||
| Attention | 9.96 | 13.19 | 30.64 | 60.68 | 67.00 | 62.03 | 38.60 | 75.14 | 61.20 | 57.16 | 21.70 | 50.00 | 31.93 | 53.79 | 49.08 | 50.69 |
| KDA | 10.00 | 13.37 | 31.16 | 61.06 | 70.00 | 61.33 | 38.40 | 75.19 | 62.47 | 57.12 | 21.68 | 50.00 | 31.99 | 47.29 | 49.08 | 50.52 |
| Titans | 10.09 | 13.48 | 30.84 | 56.07 | 73.00 | 58.47 | 39.20 | 73.72 | 61.20 | 57.05 | 21.65 | 50.00 | 31.87 | 48.38 | 49.08 | 50.04 |
| Toaster | 9.94 | 13.29 | 32.62 | 61.55 | 72.00 | 61.23 | 38.40 | 76.16 | 62.17 | 57.20 | 21.72 | 53.57 | 33.73 | 54.51 | 49.08 | 51.84 |
| S-NIAH-1 (passkey) | S-NIAH-2 (number) | S-NIAH-3 (UUID) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | 512 | 1K | 2K | 4K | 512 | 1K | 2K | 4K | 512 | 1K | 2K | 4K |
| Attention | 100.0 | 100.0 | 100.0 | 96.3 | 100.0 | 100.0 | 100.0 | 79.0 | 100.0 | 100.0 | 100.0 | 85.7 |
| KDA | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 99.0 | 88.7 | 37.3 | 98.0 | 96.0 | 77.7 | 28.0 |
| Toaster | 100.0 | 100.0 | 100.0 | 100.0 | 100.0 | 99.3 | 90.3 | 62.7 | 98.0 | 97.0 | 73.7 | 27.3 |
| Reference | State width | Activation | Hidden-layer learning | Bias | |||||
|---|---|---|---|---|---|---|---|---|---|
| KDA | Toaster | identity | leaky ReLU | WY kernel | w/o | ||||
| C4 ppl ( ) | 16.00 | 15.69 | 15.90 | 15.76 | 15.80 | 15.71 | 15.71 | 15.64 | 15.86 |
| 0.31 | – | 0.22 | 0.07 | 0.11 | 0.02 | 0.02 | 0.05 | 0.17 | |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Size | Layers | Width | Heads | Non-emb. | Total | Batch | Steps | Tokens | Peak LR | Weight decay |
|---|---|---|---|---|---|---|---|---|---|---|
| 145M | 12 | 1024 | 16 | 201M | 234M | 64 | 35,050 | 9.2B | ||
| 300M | 15 | 1280 | 20 | 390M | 431M | 96 | 42,708 | 16.8B | ||
| 550M | 18 | 1536 | 24 | 671M | 721M | 160 | 42,630 | 27.9B | ||
| 850M | 21 | 1792 | 28 | 1.06B | 1.12B | 192 | 55,095 | 43.3B | ||
| 1.2B | 24 | 2048 | 32 | 1.58B | 1.65B | 256 | 60,725 | 63.7B |
| Compute-efficiency gain vs. Attention | ||||||||
| Corpus | Model | 145M | 300M | 550M | 850M | 1.2B | ||
| C4 (in-domain) | Attention | 0.047 | 0.9997 | 1 | 1 | 1 | 1 | 1 |
| KDA | 0.047 | 0.9996 | 0.96 | 0.97 | 0.98 | 0.96 | 0.96 | |
| Toaster | 0.045 | 0.9998 | 1.12 | 1.07 | 1.07 | 1.02 | 1.01 | |
| PG19 | Attention | 0.047 | 0.9997 | 1 | 1 | 1 | 1 | 1 |
| KDA | 0.046 | 0.9998 | 0.93 | 0.91 | 0.93 | 0.92 | 0.89 | |
| 145M | 300M | 550M | 850M | 1.2B | ||
| Mixer share | Attention | 25.0% | 20.0% | 16.7% | 14.3% | 12.5% |
| KDA | 2.1% | 1.7% | 1.4% | 1.2% | 1.1% | |
| Toaster | 15.2% | 12.2% | 10.2% | 8.7% | 7.6% | |
| Toaster , kernel on both layers | 18.9% | 15.2% | 12.6% | 10.8% | 9.5% | |
| C4 gain, mixer-inclusive | KDA | 1.18 | 1.14 | 1.13 | 1.08 | 1.07 |
| Toaster | 1.22 | 1.15 | 1.13 | 1.07 | 1.06 |
| Model | Setting | 2K | 4K | 8K | 16K | 32K |
|---|---|---|---|---|---|---|
| Flash Attention | 639 | 796 | 817 | 1,027 | 1,138 | |
| KDA | 880 | 905 | 948 | 990 | 1,085 | |
| Titans | , | 1,793 | 2,098 | 2,808 | 4,432 | 7,980 |
| Titans | , | 1,202 | 1,215 | 1,641 | 2,399 | 4,097 |
| Toaster | , (width-matched) | 990 | 1,023 | 1,233 | 1,303 | 1,444 |
| Toaster | , (default) | 1,105 | 1,223 | 1,325 | 1,463 | 1,544 |
| Loss (nats) | Perplexity | loss of Toaster vs. | ||||||
| Corpus / size | Attention | KDA | Toaster | Attention | KDA | Toaster | KDA | Attention |
| C4 (in-domain) | ||||||||
| 145M | 2.7678 | 2.7725 | 2.7530 | 15.92 | 16.00 | 15.69 | 0.0195 | 0.0148 |
| 300M | 2.6039 | 2.6082 | 2.5955 | 13.52 | 13.57 | 13.40 | 0.0127 | 0.0084 |
| 550M | 2.4799 | 2.4820 | 2.4726 | 11.94 | 11.97 | 11.85 | 0.0094 | 0.0073 |
| 850M | 2.3774 | 2.3822 | 2.3755 | 10.78 | 10.83 | 10.76 | 0.0067 | 0.0019 |
| All layers replaced | Hybrid (3 mixer : 1 attention) | |||||||
| Task | Attention | KDA | Titans | Toaster | SWA | KDA | Titans | Toaster |
| Science QA | ||||||||
| ARC-Challenge | 30.64 | 31.16 | 30.84 | 32.62 | 33.39 | 32.02 | 29.36 | 32.19 |
| ARC-Easy | 60.68 | 61.06 | 56.07 | 61.55 | 61.01 | 60.25 | 57.72 | 62.07 |
| average | 45.66 | 46.11 | 43.45 | 47.08 | 47.20 | 46.14 | 43.54 | 47.13 |
| Commonsense reasoning | ||||||||
| Window | Attention | KDA | Toaster | Toaster KDA |
|---|---|---|---|---|
| 0–4K | 2.579 | 2.589 | 2.582 | 0.008 |
| 4–8K | 2.673 | 2.493 | 2.479 | 0.014 |
| 8–12K | 3.444 | 2.491 | 2.481 | 0.010 |
| 12–16K | 6.012 | 2.480 | 2.469 | 0.011 |
| 16–20K | 6.756 | 2.497 | 2.488 | 0.009 |
| 4–20K | 4.721 | 2.490 | 2.479 | 0.011 |
| Toaster ablations (145M) | C4 ppl ( ) | |
|---|---|---|
| KDA (linear memory) | 16.00 | 0.31 |
| Toaster (default) | 15.69 | – |
| Bias | ||
| w/o. homogeneous-coordinate bias | 15.86 | 0.17 |
| Hidden width | ||
| (64) | 15.90 | 0.22 |
| Model (145M) | changes | C4 ppl ( ) |
|---|---|---|
| KDA | – | 16.00 |
| Toaster , frozen completely | no (0%) | 15.86 |
| Toaster , (forget gate still active) | yes | 15.71 |
| Toaster , default (full target propagation) | yes (4.7%) | 15.69 |
| Decoding chunk size | 1 | 4 | 8 | 16 | 32 | 64 |
| Default kernel on both layers | 15.97 | 15.90 | 15.84 | 15.79 | 15.77 | 15.79 |
| Training chunk size | 8 | 16 | 32 | 64 | 128 | 256 |
| Toaster (default) | 15.59 | 15.56 | 15.69 | 15.67 | 15.69 | 15.74 |
| Total state (relative to default KDA) | ||||
|---|---|---|---|---|
| KDA, heads head dimension | ||||
| KDA, C4 ppl | 16.00 | 15.96 | 15.84 | 15.79 |
| Toaster , hidden width | – | 64 | 128 | 256 |
| Toaster , C4 ppl | – | 15.90 | 15.76 | 15.69 |
| S-NIAH-1 (passkey) | S-NIAH-2 (number) | S-NIAH-3 (UUID) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Hybrid | 512 | 1K | 2K | 4K | 512 | 1K | 2K | 4K | 512 | 1K | 2K | 4K |
| SWA | 100.0 | 100.0 | 100.0 | 89.3 | 99.3 | 100.0 | 100.0 | 78.7 | 98.7 | 93.7 | 91.7 | 56.3 |
| KDA | 100.0 | 99.7 | 99.3 | 98.7 | 100.0 | 99.7 | 100.0 | 73.3 | 93.0 | 96.7 | 81.7 | 44.0 |
| Titans | 100.0 | 100.0 | 100.0 | 84.7 | 100.0 | 99.7 | 93.7 | 81.7 | 99.3 | 91.0 | 96.0 | 59.7 |
| Toaster | 100.0 | 100.0 | 100.0 | 99.7 | 100.0 | 100.0 | 97.3 | 89.3 | 99.7 | 90.7 | 96.3 | 77.3 |
Memory-Augmented Language Models
Memory-augmented language models combine persistent or temporary memory with neural computation to retain, retrieve, and use information beyond the active context.
See all 314 papers