Retrofitting Unbounded Context onto Stock Language Models
Organizations: School of Computer Science and Engineering, University of Washington
Abstract
A language model that never had to forget would keep every document it has read and every turn it has spoken, and answer from any of it. This paper shows that two stock models can do that on unmodified weights by manipulating their geometry. Rotary attention turns each pair of key dimensions, a plane, at its own rate. The slow planes turn once in thousands of tokens and carry the address a query matches when it reads a far key; the fast planes turn once in a few tokens, and the order of the tokens inside a span is already written into its keys and values by the time the span leaves the near window. At distance the position blurs, and what attention reads is a bag rather than a sequence. A far span is read by its words: under competition it is read only if it is visible at the query of the question's own handle word, and a read through a description rather than a word fails at distance in every model measured, a 70B included. These facts allow a trick. What sits in the far part of attention does not have to be the history as it was; it can be installed ahead of the prediction. The design keeps a near window of 8K tokens at their true positions and changes what lies beyond it with the prompt: every far key and value goes to the host exact, a word index finds the units the request's rare words name, a far window of slots inside the trained range holds them while the request is encoded, and a rare word the model generates fetches its units mid-generation with one exact re-encode. Implemented in llama.cpp on Mistral-7B and Llama-3.1-8B, the design reads planted single needles at 99 to 100 of 100 and three-key discrimination at 88 to 100 of 100 from 32K to one million tokens, scores 70.8 on RULER against 69.9 for dense attention on Llama-3.1-8B at 128K and 67.8 against 75.5 on Mistral-7B at 32K, and at 128K prefills at 1.85 times and decodes at 1.65 to 1.68 times stock speed.
Figures & tables
| family | operation | share | family | operation | share |
|---|---|---|---|---|---|
| routing | self (the query’s own token) | 7.2% | lookup | far content, 65 or more back | 18.4% |
| routing | previous token | 8.1% | anchoring | far function tokens, 65 or more back | 17.3% |
| routing | tokens 2–8 back | 18.1% | induction | token after the last occurrence | 1.7% |
| routing | tokens 9–16 back | 6.4% | induction | same, keyed on a mid-layer state | 1.7% |
| routing | tokens 17–64 back | 12.7% | copy | an earlier occurrence of the token | 1.5% |
| structure | sentence or paragraph start | 2.5% | copy | the token before that occurrence | 1.0% |
| measure | result |
|---|---|
| the heads (all 1,024 of Mistral-7B) | |
| purity, by the share of a head’s mass on its best operation | 1 near-pure, 18 mixed, 1,005 diffuse (residual excluded); 1, 91 and 932 with far function tokens counted as an operation; the median head’s best operation holds 23 percent of its non-sink mass |
| best operation, heads ranked | far content 490; the previous token 214; the band 17–64 back 208; the query’s own token 69; the other six operations 43 together |
| the long tail of mass across heads | 323 heads carry half of the non-sink mass, 619 carry 80 percent, 860 carry 95 percent |
| the twelve operations folded into four families | 191 heads near-pure local, 159 mixing two families, 674 mixing local reading with far content and far function tokens |
| heads with a lift of at least 3 on a single-token aligned match (Table 4 ) | 107; of them 101 scale with the match length by at least 1.5 times, five are flat and one falls; the duplicate alignment gives the same picture (pooled 1.72, 3.80 and 6.08 at , 2 and 3; the heads that fall there are layer-0 heads); all 13 shared reader heads scale |
| rule | form | result (direction A / B) |
| rules built from the 16 heads carrying most of each operation, scored on about 470,000 held-out positions | ||
| self, the previous token, 9–16 back | copy what followed the current token inside the span; the token after the span | at most points in either form |
| 17–64 back | the same two forms | / against / for the best control that uses no attention; “the token after the span” at precision about 0.01 for every local operation |
| far function tokens, 65 or more back | the 16 heads carrying it most (layer-0 heads L0H25, L0H23, L0H22 and L0H19 lead) | / , below every no-attention control |
| far content: the 16 reader heads | copy what followed the current token at the occurrence the heads attend to | fires at 20 / 10 percent of positions, precision 0.47 / 0.62, / points |
| the same copy with the occurrence chosen without attention (Figure 4 ) | ||
| induction alignment, lift at | duplicate alignment, lift at | mass on | |||||
| head | 1 | 2 | 1 | 2 | ind. / dup. | ||
| all 1,024 heads, mass-weighted | 1.71 | 3.80 | 4.6 | 1.72 | 3.80 | 5.6 | |
| L15H21 | 2.1 | 7.9 | 7.5 | 16.3 | 100.0 | 124.3 | 0.03 / 0.29 |
| L17H26 | 5.1 | 19.8 | 19.0 | 1.8 | 5.5 | 5.7 | 0.08 / 0.02 |
| L17H27 | 1.9 | 6.1 | 6.4 | 5.5 | 33.0 | 40.9 | 0.03 / 0.09 |
| L18H0 | 20.7 | 176.3 | 147.0 | 0.6 | 5.6 | 5.9 | 0.51 / 0.01 |
| measurement | result (Mistral-7B unless stated) |
|---|---|
| the slow planes address | |
| planes a key is coherent on, by distance (the plane’s turn under about a radian) | 39 of 64 at 200 tokens back, 26 at 3,000, 21 at 10,000, 18 at 20,000 and 15 at the edge of the 32K trained range; from the rotary schedule alone |
| share of the query-key score carried by the 15 slowest planes | 59 percent, against the 23 percent that 15 of 64 would carry if the planes were interchangeable; the same at every distance (one 8K document, far keys 1K to 8K back) |
| heads flat under a -token jitter of the far key positions | 391 of 1,024; the position-sensitive heads concentrate in layer 1 |
| keeping only the 15 slowest planes of every key older than 1,024 tokens (8K documents, dense attention at true positions) | nats ( over seven documents); when the cut starts at 4,096; and for 24 and 32 planes past 1,024; for 8; the model-card rule that keeps every plane coherent at 1,024 tokens (31 planes) ; planted needles 38/38 under the 15-plane cut against 100/100 unpatched |
| keeping the 15 fastest planes instead | nats and 0/30: the sink’s share of mass falls from 0.459 to 0.0005 |
| planes kept (of 64) | bytes per token | single needle | three-key | loss, nats |
|---|---|---|---|---|
| 64 (bf16, uncut) | 131 KB | 16/16 | 16/16 | 2.1944–2.1948 |
| 40 | 33.0 KB | 16/16 | 16/16 | 2.1944–2.1948 |
| 32 | 30.0 KB | 16/16 | 16/16 | 2.1944–2.1948 |
| 28 | 29.0 KB | 16/16 | 16/16 | 2.1944–2.1948 |
| 22 (every plane coherent across 8,192 tokens) | 27.5 KB | 16/16 | 16/16 | 2.1944–2.1948 |
| 15 | 24.8 KB | 16/16 | 14/16 | 2.1944–2.1948 |
| measurement | result |
|---|---|
| single planted needles under read-time edits of the cached positions (Mistral-7B, llama.cpp , ) | 7500/7500 under 25 different edits, including every far key moved to one position and the needle moved to 30K back; moved to 40K and 60K on the 32K-trained model 0.993 and 0.960; at 100K 0.213 (0.696 per digit); distance zero, the current token’s own phase, is the one placement that fails |
| the whole far history placed at one position just beyond the near window (Mistral-7B at 32K, ) | loss 2.1863 against 2.1857 under full attention; needles 14/16 against 14/16; the same keys scored at their true graded distance 2.1937 and 7/16 |
| a set of far units placed together at one distance of 8,192 tokens | 23/24 |
| a single shift of the whole sequence by to positions | the top-1 token changes at no more than one of 256 decoded positions; key cosine at least 0.9992 |
| dense logits at a rotary offset of against offset 0 | differ by |
| a sliding window whose sink sits at its true distance, to one million tokens | nats on Mistral-7B and on Llama-3.1-8B; with the sink placed in range the same window is flat ( and nats over six lengths from 32K) |
| the unit is visible from | no competitors | three same-format competitors |
|---|---|---|
| Mistral-7B at 32K, needle harness: without competitors, with | ||
| the handle word’s own query | 16/16 | 8/8 |
| one / two / four tokens before the first digit, handle excluded | 14/16 / 14/16 / 15/16 | 3/8 / 3/8 / 4/8 |
| the query that predicts the first digit | 8/16 | 2/8 |
| 6 / 20 / 256 tokens before the handle | 16/16 / 16/16 / 16/16 | |
| the window’s tail only, evicted at the question | 0/16 | |
| Mistral-7B | Llama-3.1-8B | ||||
| question | re-encode | 32K | 128K | 32K | 128K |
| plain copy, no worked example | no | 16/16 | 15/16 | 16/16 | |
| plain copy, no worked example | yes | 16/16 | 16/16 | 16/16 | |
| copy, one worked example in the request | no | 0/16 | 0/16 | 0/16 | 0/16 |
| copy, one worked example in the request | yes | 12/16 | 13/16 | 16/16 | 16/16 |
| chained hop, second name given | no / yes | 4/7 / 7/7 | 5/5 / 5/5 | ||
| measurement | setting | result |
|---|---|---|
| same-word cosine of values, against context-neighbor cosine | layers 2 / 16 / 31; pairs over 64 tokens apart; chance | 0.77 / 0.31 / 0.28 against 0.01 / 0.26 / 0.11 |
| loss with far values replaced, keys real | one 8K document; real loss 1.919 nats | zeroed ; token-identity code ; map fitted on anchor types ; identities permuted to |
| what the readers consume | far values reduced to the token’s own earlier value | 84% of the read’s worth kept ( against for deletion); context remainder alone 18%; wrong code nats, seven times deletion |
| synthesized keys, one vector per token | from one shared layer-8 vector through a per-layer map: cosine 0.85 per word type, 0.98 per occurrence, a shortlist within one to six points of the real keys; per-token keys at cosine 0.94 | needle read 0/16 on both models |
| synthesized values | keys real; addressing intact (0.7–0.99 of the far mass on the unit) | read 0/16; Llama-3.1-8B loss: synthesized keys , synthesized values |
| values rebuilt by a forward over the unit’s own tokens | no preceding context; key cosine 0.84 | 16/16 at 32K, 16/16 at 128K, 14/14 at 1M; bulk loss nats over all 776 eligible units of a 32K document, with 256 tokens of preceding context |
| measurement | result (Mistral-7B / Llama-3.1-8B) |
|---|---|
| human relatedness predicted from the embedding table’s rows, over 6,607 rated word pairs | Spearman 0.15 / 0.06 for pairs of multi-token words, against 0.63 / 0.62 for single-token pairs |
| the same from the layer-5 to layer-6 residual at the word’s last piece | multi-token pairs 0.59 / 0.51; single-token pairs gain at most 0.06 over their table rows |
| the assembled vector is order-coded | cosine to the intact vector 0.21 with the pieces reversed and 0.27 shuffled, against a 0.96 single-token ceiling |
| built by the feed-forward layers 1–3 | ablating them takes the best multi-token head’s discrimination from 0.92 to 0.62 |
| read through the embedding table | names a single-token partner at a median rank of 7 of 13,571, where the logit lens ranks it 96 |
| Mistral-7B 32K | Llama-3.1-8B 128K | Llama-3.1-70B 128K | |
| the description, inside the window (percent read; worked example in the document tail) | |||
| 1K / 3K / 5K back | 95.1 / 71.6 / 63.0 | 88.9 / 60.5 / 44.4 | 98.8 / 97.5 / 84.0 |
| 10K / 20K back | 28.4 / 17.3 | 97.5 / 75.3 | |
| 3K back with no worked example (control haystack) | 8.6 | 81.5 | |
| at range | |||
| the description, with / without the worked example | 6/81 / 9/81 | 7/81 / 13/81 | 0/82 |
| knob | value (Mistral-7B / Llama-3.1-8B) | rule, from the model card | the measurement behind it |
|---|---|---|---|
| trained range | 32,768 / 131,072 | the hard ceiling: refuse if | a key presented outside the range is unreadable: the sink at its true distance costs / nats at 1M, at a slot the loss is flat to 1M (Section 2.3 ) |
| near window , slot band | 8,192 / 16,384 | split the context budget ; with one given derive the other and re-check against | the measured setting; halving the band at 128K costs single needles (0.875 against 1.000, ) |
| pinned prefix | 128 | the first 128 tokens of the document, at slots | the measured setting; at true distance the prefix costs 3.9–5 nats at 1M |
| key planes kept | 28 of 64 (harness, Mistral-7B; the engine stores keys uncut; Llama-3.1-8B rungs unmeasured) | the analytic cut keeps every plane coherent across ( at the effective per-plane rate, 22 planes on Mistral-7B); the default adds one measured rung of margin, a calibration rather than a derivation | three-key discrimination against planes kept (Section 2.2 , harness, ): 16/16 down to 22 planes, 14/16 at 15; the analytic cut keeps 22, the default 28 keeps one rung of margin |
| key and value bits | 4 / 4, groups of 32 (measured; the engine stores f16) | the smallest width that is free | bits are free: four-bit keys and values read the same single and three-key rows as eight-bit and bf16, loss within 0.0025 nats across the sweep |
| key centering | per head | the per-head mean over the first tokens, then frozen | measured free in the harness; is the span the window holds anyway |
| part | what we measured | result |
| the host store and its index | ||
| store size | keys and values of every far token at f16, 32 layers | 131 KB per token; 128.5 GiB at one million tokens on Mistral-7B |
| admission at 15 nats, last 16 occurrences | tokens a gather asks for at 1M, and coverage of the planted item | 200,002 tokens per gather without the two rules, 1,700–2,800 with them; coverage 0.08 to about 1.0, the item’s unit ranking first among the hits; nothing changes at 32K |
| sentence units against / windows | needle cells, band traffic, bulk loss, RULER volume tasks | sentence units win or tie every needle cell (two-hop on Llama-3.1-8B at 128K 84 against 66–68) at about a tenth of the band traffic; windows win bulk loss by 0.009 nats on code and 0.001–0.004 on prose, and the volume subset of RULER on Mistral-7B, 55.7 against 47.6 |
| whole words against fixed chunk boundaries | single needles, harness | 14/16 when the chunk cuts the handle, 16/16 with whole words |
| compression of the key planes (harness, Mistral-7B) | three-key discrimination against planes kept; bytes per token | 16/16 down to 22 planes, 14/16 at 15; the analytic cut (wavelength at least ) keeps 22, the default 28 keeps one rung of margin; 29.0 KB per token at 28 planes and four bits, against 33.0 at 40, 27.5 at 22, 24.8 at 15, and 131 KB uncompressed; loss moves by less than 0.0005 nats across the ladder |
| check | setting | result |
| against the stock engine | ||
| logits below the trained range | 9,320, 16,448 and 24,320 tokens | 0 of 32,768 logits differ |
| provenance of every cell | band and window cells after moves | 0 bad of 786,432 band cells and 393,216 window cells |
| the pinned prefix after the moves of a 1M prefill | re-placed from the host at every move / moved in place | cosine 0.999999 / 0.992304 to its true keys |
| the re-encode at an event | both models | byte-identical in its logits to a fresh prefill |
| the batched performance path | 64 regression texts, against one block at a time | byte-identical on 62 of 64; no exact-match count changes |
| model | hand-off | pinned prefix during prefill | sink key cosine to its true value | reads |
|---|---|---|---|---|
| Mistral-7B | after the question | moved in place | 0.432 (all 16 examples) | 0/16 |
| Mistral-7B | after the question | moved in place, rewritten once before the question | 0.99997 at the question | 7/16 |
| Mistral-7B | before the question | moved in place | 0.432, then rewritten by the hand-off | 9/16 |
| Mistral-7B | after the question | re-placed from the host at every move | 0.99997 | 12/16 |
| Mistral-7B | before the question | re-placed from the host at every move | 0.99997 | 16/16 |
| Llama-3.1-8B | before the question | moved in place | not measured | 16/16 |
| task | model | context | design | fetched tokens | dense | floor | matched floor |
|---|---|---|---|---|---|---|---|
| single needle | Mistral-7B | 32K | 99 [95–100] | 1,142 | 99 [95–100] | 18 [12–27] | |
| single needle | Mistral-7B | 128K | 100 [96–100] | 1,768 | 0 [0–4] † | 0 [0–4] | 11 [6–19] |
| single needle | Llama-3.1-8B | 32K | 99 [95–100] | 1,062 | 100 [96–100] | 18 [12–27] | |
| single needle | Llama-3.1-8B | 128K | 100 [96–100] | 1,604 | 100 [96–100] | 0 [0–4] | 11 [6–19] |
| three-key | Mistral-7B | 32K | 88 [80–93] | 1,132 | 93 [86–97] | 31 [23–41] | |
| three-key | Mistral-7B | 128K | 93 [86–97] | 1,759 | 0 [0–4] † | 1 [0–5] | 9 [5–16] |
| model | context | design | dense | floor | matched floor | fetched tokens |
|---|---|---|---|---|---|---|
| code / word | code / word | code / word | code / word | per request | ||
| Mistral-7B | 32K | 66 / 73 | 91 / 95 | 0 / 75 | 1,080 | |
| Mistral-7B | 128K | 59 / 59 | 0 / 0 † | 0 / 78 | 0 / 84 | 1,759 |
| Llama-3.1-8B | 32K | 87 / 46 | 94 / 85 | 0 / 64 | 1,104 | |
| Llama-3.1-8B | 128K | 92 / 52 | 92 / 89 | 0 / 72 | 0 / 68 | 1,610 |
| Llama-3.1-8B at 128K | Mistral-7B at 32K | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| task | design | dense | floor | matched ∗ | design dense | design | dense | floor | design dense |
| single needle 1 | 100.0 | 100.0 | 2.0 | 18.0 | 100.0 | 100.0 | 22.0 | ||
| single needle 2 | 98.0 | 100.0 | 8.0 | 19.0 | 100.0 | 100.0 | 28.0 | ||
| single needle 3 | 100.0 | 90.0 | 5.0 | 15.0 | 77.0 | 72.0 | 21.0 | ||
| multi-key 1 | 99.0 | 87.0 | 8.0 | 18.0 | 96.0 | 93.0 | 28.0 | ||
| multi-key 2 | 71.0 | 59.0 | 5.0 | 19.0 | 58.0 | 76.0 | 22.0 | ||
| model | context | stock prefill / decode | design prefill | ratio | design decode | ratio | re-encodes / 128 |
| Mistral-7B | 8K | 11,017 / 234.1 | 10,324 | 0.94 | 228 | 0.97 | 0 |
| Mistral-7B | 32K | 8,850 / 180.2 | 8,084 | 0.91 | 184 | 1.02 | 0 |
| Mistral-7B | 128K | 4,819 / 91.1 | 8,920 | 1.85 | 150 | 1.65 | 4 |
| Llama-3.1-8B | 8K | 11,003 / 226.8 | 10,319 | 0.94 | 214 | 0.95 | 0 |
| Llama-3.1-8B | 32K | 8,835 / 175.4 | 8,048 | 0.91 | 161 | 0.92 | 1 |
| Llama-3.1-8B | 128K | 4,833 / 89.4 | 8,938 | 1.85 | 150 | 1.68 | 3 |
| arm | Mistral-7B 32K | Llama-3.1-8B 128K |
|---|---|---|
| naming, under dense attention | ||
| the description asked directly (harness) | 9/81 | 13/81 |
| name first, then the code (harness) | 56/81 | 61/81 |
| name first (engine) | 59/81 | 64/81 |
| the thing named at all (harness) | 58/81 | 64/81 |
| given a correct name | 59/59 | |
| model | context | misses | what the misses were |
|---|---|---|---|
| Mistral-7B | 32K | 38 | 6 units not admitted; 23 resident without the bridge word; 9 resident with the word produced and misread |
| Mistral-7B | 128K | 45 | 11 not admitted; 25 resident without the bridge word; 4 wrong after a re-encode; 5 misread |
| Llama-3.1-8B | 32K | 26 | 14 handles outside the 32-word seed; 12 no bridge word |
| Llama-3.1-8B | 128K | 16 | 9 handles outside the seed; 7 no bridge word |
Long-Context Language Model Inference
Long-context language model inference makes processing and generating over lengthy prompts faster and more memory-efficient.
See all 243 papers