Delta-Rule Linear Attention

Delta-rule linear attention studies sequence models that store key–value associations in a fixed-size matrix, updating it with prediction errors as new tokens arrive. Gated DeltaNet and Kimi Delta Attention add controls over forgetting and writing, while preconditioning, momentum, and query-aware updates aim to improve learning and recall; chunkwise WY algorithms make these recurrences parallelizable during training. Work evaluates language modeling, associative-recall and needle-in-a-haystack benchmarks, long-context generalization, and increasingly the memory and speed of model serving. A central tension is how to make recurrent memory more expressive and reliable without losing stable updates, efficient parallel computation, or the compact-state advantage.