RidgeFill: Training-Free Late-Layer Cache Fill for Faster Prefill
Organizations: Independent researcher
Abstract
RidgeFill shortens prefill by not computing most of a model's upper layers for the prompt. The lower layers run exactly up to a split; from the split layer's state a per-layer map predicts every later layer's input, and each late layer writes its cache from that prediction. The design follows YOCO and DeepSeek-V4.1-Flash, where the projection is trained into the model. RidgeFill retrofits it onto models that were never trained for it: the map is a ridge regression fitted once in closed form from one pass over calibration text, the layer's own projections run unchanged, and a chosen set of rows, the last few thousand tokens, and every decoding step run the full model. Nothing is trained and no token is dropped. Most of the work was in measurement, finding a repeatable methodology. Loss averaged over all tokens hides RidgeFill's error. Needle retrieval in the filled region, loss on tokens that depend on distant context, and splice tests that swap filled keys or values for exact ones show which component carries it: the keys on Llama-3.1-8B, half of it in their scale, and the values and the recurrent state on Qwen3.5-9B. On Qwen3.8-27B in a llama.cpp fork on an A100, time to first token is 1.25 to 1.31x faster at 16K to 120K tokens with the maps in VRAM; RULER changes by -1.2 and -1.4 points and LongBench v2 by -1.4; the 95 % intervals include zero but do not rule out losses of a few points. On Qwen3.8-Flash-Next in vLLM it is 1.50 to 1.60x faster than exact at 9K to 32K tokens, with RULER's needle tasks at 99.8 against 100.0 and perplexity on novel tokens 2.65 % higher [-0.02 %, +4.93 %]. On three 8 to 9B reference models, at the same split and tail as the identity fill, the fitted map retrieves 49.1 and 49.6 more filled-region RULER points on the two dense models (at that split Llama's loss is worse), and its long-document loss is +/-0.03 nats of the released SwiftKV checkpoint's.
Figures & tables
| Fill (every token keeps a cache entry) | Drop (pruned tokens leave the cache) | |
|---|---|---|
| Training-free | RidgeFill (closed-form map); POP (identity) | FastKV, GemFilter, LazyLLM, Speculative Prefill |
| Trained | SwiftKV (distilled projections); KV Prediction (trained auxiliary model); KVA projector (trained heads); YOCO and DeepSeek-V4.1-Flash (pretrained in) | TokenGate (learned gate + LoRA); SPEED (fine-tuned) |
| Symbol | Meaning |
|---|---|
| , | number of layers; number of prompt tokens |
| , | split layer; tail length in tokens |
| , | a layer; a token |
| hidden size | |
| , | true and predicted input to layer |
| , | the map’s matrix and offset for layer |
| Arm | Input to layer for prefix rows |
|---|---|
| Exact | the true ; nothing is skipped |
| No fill | none; the late layers write nothing for the prefix, so there the tail attends only to itself |
| Identity | (POP’s fill) |
| RidgeFill |
| Reference study | Qwen3.8-27B | Qwen3.8-Flash-Next | |
|---|---|---|---|
| Models | Llama-3.1-8B, Qwen3.5-9B, Qwen3-8B | Qwen3.8-27B | Qwen3.8-Flash-Next |
| Split / layers | 12/32, 12/32, 13/36 | 16/64 | 24/48 |
| Tail | 4,096 | 4,096 | 2,048 |
| Exact prefix rows | none; attention-selected 10 % in “+ rows” arms | probe, special tokens, lexical net | none; protected tokens in quality mode |
| 0.03, 0.03, 0.01 | 0.03 | 0.03 | |
| Value rescale | off | on, | off |
| Llama-3.1-8B | Qwen3.5-9B | Qwen3-8B | |
| RULER lengths | 8K–64K | 8K–64K | 8K–32K |
| Filled-region samples, all lengths | 216 | 219 | 138 |
| Filled-region samples, 8K–32K | 136 | 139 | 138 |
| Long documents | 126 | 126 | 90 |
| Scored tokens per document | 431–539 | 512 | 454–523 |
| Key tokens (documents) | 467 (80) | 605 (76) | 344 (63) |
| Arm | Definition |
|---|---|
| POP native | identity fill at POP’s own split, , with a one-token tail |
| SwiftKV as released | the released Llama-3.1-8B-Instruct SwiftKV checkpoint (Snowflake, 2024) at its split 16, with its own one-token tail |
| SwiftKV maps + exact tail | SwiftKV’s trained projections with our 4,096-token tail |
| + rows | attention-selected exact rows (§ 2.4 ) |
| + random rows | the same number of rows, chosen uniformly from the prefix with a per-sample seed; the identity and RidgeFill random arms use the same rows |
| + longer tail | the same number of rows added to the exact tail instead |
| Deployment | Measure | Interval |
|---|---|---|
| Reference study, Flash-Next | all | bias-corrected and accelerated bootstrap (Efron, 1987), 5,000 resamples within task or domain |
| Qwen3.8-27B | RULER | paired item bootstrap, 10,000 resamples within task |
| Qwen3.8-27B | LongBench v2 | paired percentile bootstrap over items, 10,000 resamples, computed by our report script (data repository); sign test on discordant items |
| Qwen3.8-27B | agreement (§ 4.7 ) | bootstrap over documents |
| Prompt tokens | Stock s | Resident s | Speedup | Streamed s | Speedup | VRAM over stock, resident / streamed |
|---|---|---|---|---|---|---|
| 16,384 | 12.97 | 10.38 | 1.249 | 10.88 | 1.192 | +1,592 / +268 MiB |
| 32,768 | 27.49 | 21.11 | 1.302 | 21.69 | 1.267 | +1,802 / +478 MiB |
| 65,536 | 60.50 | 46.28 | 1.307 | 46.55 | 1.300 | +2,270 / +944 MiB |
| 120,000 | 128.10 | 100.01 | 1.281 | 99.06 | 1.293 | +3,014 / +1,732 MiB |
| Context | n | Stock % | RidgeFill % | points [95 % CI] |
|---|---|---|---|---|
| 16,384 | 260 | 85.69 [83.15, 88.04] | 84.51 [81.87, 87.00] | 1.19 [ 3.13, +0.79] |
| 32,768 | 260 | 87.85 [85.46, 90.04] | 86.42 [83.88, 88.81] | 1.42 [ 3.21, +0.35] |
| Task (n = 20) | 16K stock | 16K RidgeFill | 32K stock | 32K RidgeFill |
|---|---|---|---|---|
| cwe | 64.00 | 54.00 | 82.00 | 68.50 |
| fwe | 100.00 | 98.33 | 100.00 | 100.00 |
| niah_multikey_1 / 2 / 3 | 100.00 | 100.00 | 100.00 | 100.00 |
| niah_multiquery | 100.00 | 98.75 | 100.00 | 98.75 |
| niah_multivalue | 100.00 | 97.50 | 100.00 | 96.25 |
| niah_single_1 / 2 / 3 | 100.00 | 100.00 | 100.00 | 100.00 |
| Group | n | Stock % | RidgeFill % | points [95 % CI] |
|---|---|---|---|---|
| Overall | 503 | 38.57 | 37.18 | 1.39 [ 4.97, +2.19] |
| Easy | 192 | 42.71 | 40.62 | 2.08 [ 8.85, +4.69] |
| Hard | 311 | 36.01 | 35.05 | 0.96 [ 5.14, +3.54] |
| Short | 180 | 46.11 | 44.44 | 1.67 [ 7.22, +4.44] |
| Medium | 215 | 38.14 | 36.74 | 1.40 [ 7.44, +4.65] |
| Long | 108 | 26.85 | 25.93 | 0.93 [ 7.41, +6.48] |
| Context | Mean KL [CI] | Median KL | p99 KL | Top-1 agreement | Mean NLL [CI] | Median NLL |
|---|---|---|---|---|---|---|
| 16,384 | 0.0894 [0.0399, 0.1556] | 0.0040 | 0.965 | 94.64 % | 0.0330 [ 0.0875, +0.0059] | +0.0044 |
| 32,768 | 0.2656 [0.0599, 0.6017] | 0.0042 | 12.77 | 93.60 % | 0.2142 [ 0.5629, 0.0025] | +0.0071 |
| Context | Copyable share | Docs | Mean KL | Mean NLL |
|---|---|---|---|---|
| 16,384 | < 0.2 | 86 | 0.0461 | 0.0055 |
| 16,384 | 0.2 to 0.5 | 7 | 0.2949 | 0.1540 |
| 16,384 | 0.5 | 7 | 0.4157 | 0.2488 |
| 32,768 | < 0.2 | 82 | 0.0352 | +0.0069 |
| 32,768 | 0.2 to 0.5 | 13 | 0.3074 | 0.0849 |
| 32,768 | 0.5 | 5 | 3.9351 | 4.1765 |
| Schedule | New request TTFT s, stock / RidgeFill | Other users tok/s, stock / RidgeFill | Others’ ITL mean ms, stock / RidgeFill | Others’ ITL p99 ms, stock / RidgeFill | Others’ ITL max ms, stock / RidgeFill |
|---|---|---|---|---|---|
| Unified C1 | 13.06 / 11.11 | n/a | n/a | n/a | n/a |
| Unified C2 | 13.53 / 11.70 | 19.92 / 21.60 | 49.33 / 45.50 | 1,605.7 / 863.0 | 1,707.8 / 1,317.1 |
| Unified C4 | 14.18 / 12.21 | 60.06 / 62.96 | 46.47 / 44.20 | 148.0 / 669.0 | 1,785.0 / 1,323.7 |
| Unified C8 | 15.32 / 13.53 | 86.90 / 89.48 | 71.58 / 69.35 | 621.7 / 774.9 | 1,919.6 / 1,876.1 |
| Separate KV C2 | 13.48 / 11.34 | 19.91 / 21.87 | 49.38 / 44.90 | 1,592.2 / 858.3 | 1,703.4 / 1,310.7 |
| Separate KV C8 | 13.93 / 11.86 | 88.88 / 91.98 | 69.84 / 67.46 | 584.7 / 654.3 | 1,821.1 / 1,809.6 |
| Arm | Perplexity gap [95 % CI] | RULER, all | RULER, filled region |
|---|---|---|---|
| Exact | n/a | 100.0 | 100.0 |
| RidgeFill | +2.65 % [ 0.02 %, +4.93 %] | 99.8 | 99.8 |
| Quality mode | +1.60 % [ 0.83 %, +4.32 %] | 99.9 | 99.9 |
| Arm | TTFT 9K | TTFT 16K | TTFT 32K | 64K request |
|---|---|---|---|---|
| Exact | 1.00 | 1.00 | 1.00 | 1.00 (82.6 s) |
| RidgeFill | 1.50 | 1.54 | 1.60 | 1.74 (47.5 s) |
| Quality mode | 1.25 | 1.22 | 1.27 | 1.49 (55.6 s) |
| Model | Depth | S | RidgeFill | Identity | No fill | 95 % CI, RidgeFill identity |
|---|---|---|---|---|---|---|
| Llama-3.1-8B | 1/4 | 8 | +0.804 | +0.409 | +1.983 | [+0.128, +0.766] |
| 3/8 | 12 | +1.006 | +0.505 | +1.591 | [+0.307, +0.748] | |
| 1/2 | 16 | +0.085 | +0.563 | +1.471 | [ 0.548, 0.430] | |
| 5/8 | 20 | +0.030 | +0.406 | +1.225 | [ 0.422, 0.343] | |
| POP native | 22 | n/a | +0.373 | n/a | n/a | |
| Qwen3.5-9B | 1/4 | 8 | +0.033 | +0.032 | +0.056 | [ 0.004, +0.005] |
| Metric | Llama-3.1-8B | Qwen3.5-9B | Qwen3-8B |
|---|---|---|---|
| RULER filled (points) | +39.7 [+32.6, +46.5] | +3.3 [ 1.3, +7.8] | +49.6 [+43.3, +56.2] |
| NLL novel (nats) | +0.501 [+0.307, +0.748] | +0.001 [ 0.003, +0.005] | +0.019 [+0.012, +0.025] |
| NLL key (nats) | +1.047 [+0.521, +1.666] | 0.347 [ 0.523, 0.196] | 0.400 [ 0.634, 0.159] |
| Decode agreement (points) | +6.3 [+4.6, +8.5] | +0.5 [+0.1, +1.0] | +1.2 [+0.6, +2.0] |
| LongBench average | +2.4 [0.0, +4.7] | n/a | n/a |
| Arm (Llama-3.1-8B, split 16) | NLL novel | NLL key |
|---|---|---|
| SwiftKV as released (one-token tail) | +0.095 | +0.568 |
| SwiftKV maps + exact tail | +0.061 | +0.115 |
| RidgeFill (tail 4,096) | +0.085 | +0.339 |
| Contrast | Novel tokens [95 % CI] | Key tokens [95 % CI] |
|---|---|---|
| SwiftKV as released RidgeFill | +0.010 [ 0.004, +0.025] | +0.229 [ 0.020, +0.485] |
| SwiftKV maps + exact tail RidgeFill | 0.024 [ 0.037, 0.015] | 0.224 [ 0.415, 0.057] |
| RULER filled, 8K–32K | Llama-3.1-8B (n = 136) | Qwen3-8B (n = 138) |
|---|---|---|
| Exact | 99.8 | 98.7 |
| POP native | 94.1 (split 22) | 98.7 (split 24) |
| Identity at 5/8 | 95.0 | 98.9 |
| RidgeFill at 5/8 | 98.7 | 99.1 |
| RidgeFill at 5/8 POP native | +4.6; lower bound 2.18 (margin 3) | +0.4; lower bound 2.15 (margin 3) |
| RidgeFill identity at 5/8 | +3.7 [+2.0, +6.2] | +0.2 [0.0, +0.9] |
| RULER filled contrast | Llama-3.1-8B | Qwen3.5-9B | Qwen3-8B |
|---|---|---|---|
| RidgeFill + rows exact | +0.2; lower bound 1.38 (n = 216) | 0.0; lower bound 1.36 (n = 219) | 8.9 [ 13.6, 5.4] (n = 138) |
| RidgeFill + rows RidgeFill + random rows | +29.6 [+23.3, +36.9] | +18.3 [+13.1, +24.5] | 4.0 [ 8.5, +0.5] |
| RidgeFill + rows RidgeFill + longer tail | +24.8 [+19.1, +31.6] | +16.5 [+11.5, +22.3] | 4.0 [ 8.5, +0.5] |
| Question-blind rows rows | 16.0 [ 21.5, 11.8] | 14.0 [ 19.2, 9.5] | n/a |
| RidgeFill + rows identity + rows | +0.8 [ 0.1, +2.9] | +0.8 [ 0.1, +2.7] | +26.4 [+19.9, +32.6] |
| Model | Splice | NLL novel | Loss removed [95 % CI] | RULER filled | Gap closed |
|---|---|---|---|---|---|
| Llama-3.1-8B | RidgeFill | +1.006 | n/a | 60.7 | n/a |
| + exact keys | +0.031 | 97.0 % [96.0, 97.7] | 99.1 | 98.6 % | |
| + exact values | +0.120 | 88.1 % [86.6, 89.5] | 79.6 | 48.6 % | |
| + exact keys and values | 0.000 | 100.0 % | n/a | n/a | |
| + key scale (tail) | +0.502 | 50.1 % [45.5, 54.5] | n/a | n/a | |
| + value scale (tail) | +1.103 | n/a | n/a | n/a |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Arm | RULER filled, 8K–32K (n = 136) | RULER filled, 8K–64K (n = 216) | NLL novel | NLL key | Decode agree | Decode KL | Turn 2 |
|---|---|---|---|---|---|---|---|
| Exact | 99.6 | 99.3 | 0.000 | 0.000 | n/a | n/a | 100.0 |
| No fill | 0.0 | n/a | +1.591 | +5.401 | 71.5 % | 1.262 | n/a |
| Identity | 11.6 | 9.7 | +0.505 | +2.223 | 85.1 % | 0.334 | n/a |
| POP native | 93.4 | n/a | +0.373 | +0.275 | n/a | n/a | n/a |
| RidgeFill | 60.7 | 49.4 | +1.006 | +3.271 | 91.4 % | 0.124 | 57.0 |
| RidgeFill + rows | 99.6 | 99.5 | +0.067 | +0.409 | n/a | n/a | 100.0 |
| Arm | RULER filled, 8K–32K (n = 139) | RULER filled, 8K–64K (n = 219) | NLL novel | NLL key | Decode agree | Decode KL | Turn 2 |
|---|---|---|---|---|---|---|---|
| Exact | 99.8 | 99.9 | 0.000 | 0.000 | n/a | n/a | 100.0 |
| No fill | 0.0 | n/a | +0.046 | +2.760 | 96.8 % | 0.045 | n/a |
| Identity | 80.4 | 78.1 | +0.023 | +1.548 | 97.7 % | 0.021 | n/a |
| POP native | 100.0 | n/a | +0.016 | +0.235 | n/a | n/a | n/a |
| RidgeFill | 83.6 | 81.4 | +0.024 | +1.201 | 98.2 % | 0.012 | 86.0 |
| RidgeFill + rows | 99.8 | 99.9 | +0.008 | +0.431 | n/a | n/a | 100.0 |
| Arm | RULER filled (n = 138) | Eight RULER tasks | NLL novel | NLL key | Decode agree | Decode KL | Turn 2 |
|---|---|---|---|---|---|---|---|
| Exact | 98.7 | 91.8 | 0.000 | 0.000 | n/a | n/a | 100.0 |
| No fill | 0.0 | 19.0 | +0.243 | +3.072 | 88.4 % | 0.172 | n/a |
| Identity | 44.0 | 39.0 | +0.026 | +1.833 | 95.7 % | 0.032 | n/a |
| POP native | 98.7 | 86.3 | +0.044 | +0.255 | n/a | n/a | n/a |
| RidgeFill | 93.7 | 76.4 | +0.044 | +1.434 | 96.9 % | 0.013 | 97.3 |
| RidgeFill + rows | 89.9 | 81.7 | +0.009 | +0.735 | n/a | n/a | 89.3 |
| Contrast | Metric | n | 95 % CI | |
|---|---|---|---|---|
| Llama: RidgeFill no fill | RULER filled | 136 | +60.7 | [+53.2, +67.6] |
| Llama: RidgeFill no fill | NLL novel | 126 | 0.586 | [ 0.844, 0.266] |
| Llama: RidgeFill POP native | NLL novel | 126 | +0.633 | [+0.401, +0.921] |
| Llama: RidgeFill identity at 1/2 | RULER filled | 136 | +12.5 | [+8.6, +17.5] |
| Llama: RidgeFill identity, 64K only | RULER filled | 80 | +23.8 | [+13.6, +32.8] |
| Llama: RidgeFill + rows exact, 64K only | RULER filled | 80 | +0.6 | [0.0, +3.1] |
| Task | Length | n | Exact | RidgeFill | Identity | No fill | RidgeFill + rows | Identity + rows |
|---|---|---|---|---|---|---|---|---|
| multikey_1 | 8K | 10 | 100.0 | 80.0 | 10.0 | 0.0 | 100.0 | 100.0 |
| multikey_1 | 16K | 19 | 100.0 | 63.2 | 5.3 | 0.0 | 100.0 | 100.0 |
| multikey_1 | 32K | 20 | 100.0 | 50.0 | 20.0 | 0.0 | 100.0 | 100.0 |
| multikey_1 | 64K | 24 | 100.0 | 20.8 | 0.0 | n/a | 100.0 | 100.0 |
| multiquery | 16K | 6 | 100.0 | 45.8 | 0.0 | 0.0 | 100.0 | 100.0 |
| multiquery | 32K | 12 | 97.9 | 47.9 | 0.0 | 0.0 | 97.9 | 97.9 |
| Task | Length | n | Exact | RidgeFill | Identity | No fill | RidgeFill + rows | Identity + rows |
|---|---|---|---|---|---|---|---|---|
| multikey_1 | 8K | 10 | 100.0 | 70.0 | 80.0 | 0.0 | 100.0 | 100.0 |
| multikey_1 | 16K | 19 | 100.0 | 63.2 | 68.4 | 0.0 | 100.0 | 100.0 |
| multikey_1 | 32K | 21 | 100.0 | 81.0 | 66.7 | 0.0 | 100.0 | 100.0 |
| multikey_1 | 64K | 24 | 100.0 | 62.5 | 66.7 | n/a | 100.0 | 95.8 |
| multiquery | 16K | 6 | 100.0 | 33.3 | 58.3 | 0.0 | 100.0 | 100.0 |
| multiquery | 32K | 12 | 97.9 | 87.5 | 83.3 | 0.0 | 97.9 | 100.0 |
| Task | Length | n | Exact | RidgeFill | Identity | No fill | RidgeFill + rows | Identity + rows |
|---|---|---|---|---|---|---|---|---|
| multikey_1 | 8K | 10 | 100.0 | 90.0 | 50.0 | 0.0 | 100.0 | 80.0 |
| multikey_1 | 16K | 19 | 100.0 | 94.7 | 31.6 | 0.0 | 84.2 | 63.2 |
| multikey_1 | 32K | 21 | 95.2 | 90.5 | 23.8 | 0.0 | 85.7 | 38.1 |
| multiquery | 16K | 6 | 100.0 | 91.7 | 8.3 | 0.0 | 87.5 | 54.2 |
| multiquery | 32K | 12 | 95.8 | 81.2 | 16.7 | 0.0 | 70.8 | 31.2 |
| multivalue | 8K | 2 | 100.0 | 100.0 | 37.5 | 0.0 | 87.5 | 75.0 |
| Arm | Llama-3.1-8B | Qwen3.5-9B | Qwen3-8B |
|---|---|---|---|
| Samples | 100 | 100 | 75 |
| Exact | 100.0 | 100.0 | 100.0 |
| RidgeFill alone | 57.0 | 86.0 | 97.3 |
| RidgeFill + rows | 100.0 | 100.0 | 89.3 |
| Identity + rows | 100.0 | 99.0 | 54.7 |
| RidgeFill alone, second needle in the filled region | 46.9 | 82.7 | 96.6 |
| Model | Checkpoint | Revision | Note |
|---|---|---|---|
| Llama-3.1-8B | unsloth/Llama-3.1-8B-Instruct | 4699cc75b550 | |
| Qwen3.5-9B | Qwen/Qwen3.5-9B | c20223623576 | post-trained from Qwen3.5-9B-Base; text path only |
| Qwen3-8B | Qwen/Qwen3-8B | b968826d9c46 |
| Commit | Runs |
|---|---|
| 42028dfc | the original loss runs, decode agreement and second-question runs on all three models; Qwen3-8B’s RULER |
| 253f845a | Llama-3.1-8B’s RULER; the second loss runs on Llama-3.1-8B and Qwen3.5-9B |
| 21ccd244 | Qwen3.5-9B’s RULER; Qwen3-8B’s second loss run; Qwen3-8B’s 5/8-depth comparison with POP |
| 019ae056 | Llama-3.1-8B’s LongBench v1; the fresh documents; Llama-3.1-8B’s 5/8-depth comparison with POP |
| Deployment | File | Source and revision | sha256 |
|---|---|---|---|
| 27B | model, Qwen3.8-27B-UD-Q4_K_XL.gguf | unsloth/Qwen3.8-27B-GGUF, 4ca720788d1e01f1bff70c033e0d0028fd02e502 | 3f227079003add2511437e5b1e94812e363385225bf6a9b47b0054a72bc8b01e |
| 27B | maps, ridgefill-projector-qwen3.8-27b-i8-S16.gguf | TheCacheIsALie/RidgeFill-Projector-Qwen3.8-27B-i8, 69e3ea99 (GGUF unchanged since 07eb31cd) | 8646d41a5a28d60ef53cc5e1fee4b2bfb20d0775db88d3f89d0731ad9f6b629d |
| Flash-Next | model | tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ, 6fe3e746 | none; multi-file repository pinned by revision |
| Flash-Next | engine image | tcclaviger/vllm:29.08.10 | digest 809657ddb1533db10a0c08267c3ee6fde180b839e379713414efa121415914a3 |
| Flash-Next | determinism fix, replacing vllm/models/qwen4_exp/amd/ops/qsa.py | data repository, code/engine-fix/ (the diff and both files’ hashes) | eaf0dd1975799f4c8897cbea62791de44fd28322758fb9891cbe7ff0ef52db99 |
| Flash-Next | maps | TheCacheIsALie/RidgeFill-Projector-Qwen3.8-Flash-Next-i8, adb763a7 (map files unchanged since 0b6f2ce0) | manifest ridgefill.json 795363fa153f0e7e16454af6acabd644709e703d84de0dd9535565226bb51faa |
| Model | Rows | Median document (tokens) | Longest document (tokens) | Fit time |
|---|---|---|---|---|
| Llama-3.1-8B | 834,541 | 11,219 | 28,352, counting the beginning-of-sequence token | about 24 min, one RTX 4090 |
| Qwen3.5-9B | 839,652 | 11,563 | 30,465 | about 44 min, one A100, including the download |
| Qwen3-8B | 845,043 | 11,482 | 29,009 | 31 min, one R9700 |
| Qwen3.8-27B | 839,652 | 11,563 | 30,465 | 26 min, one RTX 5090 |
| File | What it proves | Used in |
|---|---|---|
| RidgeSums.lean | Summing over documents one at a time gives the same ridge solution as one solve over all rows, and the solution is unique for any ridge penalty > 0. | § 2.2 |
| StCorrection.lean | The scale, its uniqueness and the bound above, for scalar gates and a linear recurrence. | § 2.3 , Appendix H |
| GdnContraction.lean | A bound on how fast the full delta rule forgets earlier inputs. | Appendix H |
| AttnSensitivity.lean | The value and key bounds, for single-head softmax attention without a causal mask. | § 2.4 |
| ImpactBound.lean | Under a first-order impact model with errors of bounded size, keeping exact the rows of largest sensitivity gives the smallest bound. | § 2.4 |
| RandomControl.lean | Under an additive gain model, a random subset of the rows gains on average its share of the total, so a random control of the same size measures only the selector’s skill. | § 3.3 |
| Correction | Change in error, % [95 % CI] | Prompts improved |
|---|---|---|
| Jensen term, on each logit | +62.9 [+55.1, +70.5] | 0 of 23 |
| Jensen term, sign reversed | +2.5 [ 1.5, +6.1] | 11 of 23 |
| Jensen term, scaled per row | +80.2 [+60.2, +114.1] | 0 of 23 |
| Jensen term with the key–value cross-covariance | +70.1 [+61.7, +78.2] | 0 of 23 |
| Key–value cross-covariance alone | +1.8 [+0.9, +2.6] | 6 of 23 |
| Value map fitted to the attention output | +51.2 [+43.5, +57.7] | 1 of 23 |
| Model | Needles | Units | Exact digit rows | Exact rows by attention |
|---|---|---|---|---|
| Llama-3.1-8B | numeric, multi-key | 156 | +25 [+18, +32] | +22 [+14, +29] |
| Llama-3.1-8B | text | 660 | +2 [0, +3] | +15 [+12, +18] |
| Qwen3.5-9B | numeric, multi-key | 156 | +26 [+19, +33] | +21 [+13, +28] |
| Qwen3.5-9B | text | 264 | 0 [ 1, +1] | +16 [+11, +20] |
LLM Inference Acceleration
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
See all 1,228 papers