Text Speeding Itself Up: Recency and Restraint in Session-Local Drafting
Organizations: Utrecht University
Abstract
Speculative decoding lets a cheap draft model propose tokens that a large language model verifies in a single forward pass. When the verifier keeps a KV cache, verification is cheap and drafting must be nearly free. Session-local drafters, which predict from the prompt and the text generated so far, meet that bar without training. We compare a recency-weighted, backing-off, confidence-gated session memory against prompt lookup decoding in its reference form, which copies from the earliest earlier match, and in a variant that copies from the most recent one, with paired per-prompt bootstrap intervals throughout. On general, medical, and legal text with a Mistral-7B verifier, recency accounts for the speed: most-recent lookup gains up to 0.09 in speedup over earliest-match lookup on 500-token generations, and session memory's weighting and backoff add nothing measurable on top, at 30 to 40 times the drafting cost. Energy tells a different story. On medical and legal text, unselective drafters are 14% faster than plain decoding yet use as much energy per token, while drafters that abstain on weak evidence save 3 to 4 joules per 100 tokens; a three-token minimum match length on prompt lookup matches the confidence-gated session memory. Outputs are token-identical to greedy decoding in 94% of generations, with every divergence at a numerical near-tie. Under a repetition penalty, long-generation speedups shrink from 1.4-1.7x to 1.1-1.2x, but recency still helps and the length floor is the only drafter that saves energy in both domains.
Figures & tables
| Drafter | Draws on | Abstains if |
| Lookup, earliest | first match, | no full match |
| Lookup, most recent | last match, | no full match |
| Lookup, -token floor | last match, | no match |
| Lookup, agreement | all matches, top | successors differ |
| Session memory | decayed counts, | nothing matches |
| gated ( =0.5) | same |
| Speedup over plain decoding | Difference | |||||
|---|---|---|---|---|---|---|
| Domain | Lookup, earliest | Lookup, most recent | Session memory | recency | weighting | |
| General | 3 | 1.153 [1.122, 1.183] | 1.169 [1.139, 1.200] | 1.165 [1.134, 1.197] | +0.016 ∗ | -0.004 ∗ |
| General | 5 | 1.216 [1.170, 1.266] | 1.236 [1.187, 1.286] | 1.227 [1.179, 1.280] | +0.020 ∗ | -0.009 ∗ |
| General | 8 | 1.217 [1.159, 1.282] | 1.245 [1.181, 1.311] | 1.243 [1.177, 1.313] | +0.027 ∗ | -0.002 |
| Medical | 3 | 1.110 [1.089, 1.132] | 1.120 [1.100, 1.143] | 1.116 [1.095, 1.139] | +0.011 ∗ | -0.004 ∗ |
| Medical | 5 | 1.139 [1.106, 1.174] | 1.152 [1.119, 1.188] | 1.148 [1.115, 1.184] | +0.013 ∗ | -0.004 ∗ |
| Domain | Drafter | Acc. % | Abstain % | Tok./step | Speedup | Draft % wall |
|---|---|---|---|---|---|---|
| General | Lookup, earliest | 49.9 | 30 | 2.82 | 1.622 [1.500, 1.740] | 0.01 |
| General | Lookup, most recent | 56.3 | 32 | 2.98 | 1.711 [1.585, 1.829] | 0.01 |
| General | Session memory | 55.4 | 31 | 2.98 | 1.703 [1.578, 1.821] | 0.39 |
| General | Session memory, gated | 76.4 | 55 | 2.81 | 1.682 [1.550, 1.806] | 0.41 |
| Legal | Lookup, earliest | 30.7 | 28 | 2.11 | 1.366 [1.316, 1.419] | 0.01 |
| Legal | Lookup, most recent | 35.9 | 30 | 2.27 | 1.450 [1.384, 1.517] | 0.01 |
| Legal | General | |||||
|---|---|---|---|---|---|---|
| Speedup | J/100 | Speedup | J/100 | |||
| Drafter | greedy | penalised | penalised | greedy | penalised | penalised |
| Plain decoding | 1 | 1 | 106.4 | 1 | 1 | 101.7 |
| Lookup, earliest | 1.37 | 1.09 | 107.7 | 1.62 | 1.14 | 99.9 |
| Lookup, most recent | 1.45 | 1.12 | 105.9 | 1.71 | 1.16 | 98.7 |
| Lookup, 3-token floor | 1.40 | 1.09 | 102.0 ∗ | 1.67 | 1.16 | 95.3 ∗ |
| Generated tokens | ||||||
|---|---|---|---|---|---|---|
| Drafter | 1–100 | –200 | –300 | –400 | –500 | |
| Legal | Lookup, earliest | 0.41 | 0.79 | 1.34 | 1.62 | 2.22 |
| Legal | Lookup, most recent | 0.43 | 0.87 | 1.50 | 1.92 | 2.68 |
| Legal | Session memory, gated | 0.30 | 0.62 | 1.24 | 1.60 | 2.29 |
| General | Lookup, earliest | 0.44 | 1.55 | 2.06 | 2.07 | 2.34 |
| General | Lookup, most recent | 0.44 | 1.77 | 2.52 | 2.39 | 2.79 |
| 100 tokens, | 500 tokens, | |||||
|---|---|---|---|---|---|---|
| Drafter | Abstain % | General | Medical | Legal | General | Legal |
| Plain decoding | – | 92.7 | 93.7 | 98.2 | 105.3 | 108.2 |
| Lookup, most recent | 41 | 89.5 ∗ | 94.6 ∗ | 99.2 ∗ | 78.2 | 90.0 ∗ |
| Lookup, 2-token floor | 69 | 87.5 ∗ | 92.0 ∗ | 96.8 ∗ | 77.7 | 88.4 |
| Lookup, 3-token floor | 79 | 86.7 | 91.2 | 96.1 ∗ | 78.4 | 88.6 |
| Lookup, agreement | 64 | 88.6 ∗ | 93.4 ∗ | 96.7 ∗ | 78.7 ∗ | 89.7 ∗ |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Identical | % | ||
| Domain | General | 840/900 | 93.3 |
| Domain | Medical | 838/900 | 93.1 |
| Domain | Legal | 861/900 | 95.7 |
| Drafter | Lookup, earliest | 841/900 | 93.4 |
| Drafter | Lookup, most recent | 850/900 | 94.4 |
| Drafter | Session memory | 848/900 | 94.2 |
Language Model Decoding
Language model decoding studies how models choose and generate outputs, balancing quality, reliability, diversity, and inference cost.
See all 182 papers