Attention at Absolute Zero
Organizations: MIT Media Lab
Abstract
Attention scores candidate supports of an answer, which we call witnesses, and reads the scores at a temperature. We build a higher-order attention layer whose state lives on pairs of items: for a pair (a, b), each third item k is a witness judged through its legs (a, k) and (k, b). It reads witnesses at zero temperature, keeping the best and the number tied with it, passes on only bits comparing such counts, rebuilds its pair state from the input and these bits at every loop, and is trained from answers alone. Each choice removes one way soft attention fails at size: dispersed weight, unresolved gaps between learned scores, and a drifting soft state. In maximum retrieval it is right on at least 99.95% of sets 1,000 times larger than in training, on all ten seeds; trained with a soft count, the same network falls to 42% at 4,096 items. On CLRS-30 without hints, a general form of the layer (half its channels read at zero temperature, numbers embedded, state carried) beats the best published model, which uses hints on some tasks; the full cold layer is exact on every test input of 9 of the 12 tasks whose answers depend on numbers only through comparisons. From yes-or-no answers it decides 2-SAT and two-colouring exactly at 500 variables and, as our theory predicts, not parity equations. Beside a frozen 7B language model it chains family relations exactly at 1,000 statements after training on 16, where the model alone falls to 44%; finite checks on the weights prove every block that learned correct at every size, on acyclic contexts and given what the model reads. On CLUTRR it is right on every story of five to ten relations given the graphs, learning the 62 rules exactly; from text, with the model frozen, no hand-written rule and only a small read-in trained, it lifts an open 7B model from 33% to 65%, above every system we found without hand-written rules except frontier reasoning models.
Figures & tables
| Run | Forecast | Odds | Outcome |
|---|---|---|---|
| General block for all 30 tasks, second seed | 30-task mean within 0.4 of the first seed’s 96.96 | 70% | hit: 96.71 (0.25 under) |
| 30-task mean above FloydNet’s 96.64 | 65% | hit: 96.71 | |
| Depth-first search between 70 and 80 after 8 loops | 80% | miss: 69.29 | |
| General-and-cold block for all 30 tasks (3 seeds) | Three-seed mean at least 97.04 (best single shared block so far) | 70% / 60% | miss: 96.55 |
| Three-seed mean at least 97.5 | 50% / 40% | miss: 96.55 | |
| Some seed below FloydNet’s 96.64 | 15% / 20% | hit: two seeds (96.41, 96.48) |
Algorithmic Reasoning
Algorithmic reasoning studies how AI systems learn, execute, and generalize procedures for solving structured problems.
See all 114 papers