Speculative decoding lets a cheap draft model propose tokens that a large language model verifies in a single forward pass. When the verifier keeps a KV cache, verification is cheap and drafting must be nearly free. Session-local drafters, which predict from the prompt and the text generated so far, meet that bar without training. We compare a recency-weighted, backing-off, confidence-gated session memory against prompt lookup decoding in its reference form, which copies from the earliest earlier match, and in a variant that copies from the most recent one, with paired per-prompt bootstrap intervals throughout. On general, medical, and legal text with a Mistral-7B verifier, recency accounts for the speed: most-recent lookup gains up to 0.09 in speedup over earliest-match lookup on 500-token generations, and session memory's weighting and backoff add nothing measurable on top, at 30 to 40 times the drafting cost. Energy tells a different story. On medical and legal text, unselective drafters are 14% faster than plain decoding yet use as much energy per token, while drafters that abstain on weak evidence save 3 to 4 joules per 100 tokens; a three-token minimum match length on prompt lookup matches the confidence-gated session memory. Outputs are token-identical to greedy decoding in 94% of generations, with every divergence at a numerical near-tie. Under a repetition penalty, long-generation speedups shrink from 1.4-1.7x to 1.1-1.2x, but recency still helps and the length floor is the only drafter that saves energy in both domains.
Figures & tables
Drafter
Draws on
Abstains if
Lookup, earliest
first match, n≤3
no full match
Lookup, most recent
last match, n≤3
no full match
Lookup, m -token floor
last match, m≤n≤3
no match n≥m
Lookup, agreement
all matches, top n
successors differ
Session memory
decayed counts, n≤4
nothing matches
gated ( β =0.5)
same
κ<β
Table 1: The session-local drafters compared. All draw only on the prompt and verifier-accepted tokens.
Speedup over plain decoding
Difference
Domain
k
Lookup, earliest
Lookup, most recent
Session memory
recency
weighting
General
3
1.153 [1.122, 1.183]
1.169 [1.139, 1.200]
1.165 [1.134, 1.197]
+0.016 ∗
-0.004 ∗
General
5
1.216 [1.170, 1.266]
1.236 [1.187, 1.286]
1.227 [1.179, 1.280]
+0.020 ∗
-0.009 ∗
General
8
1.217 [1.159, 1.282]
1.245 [1.181, 1.311]
1.243 [1.177, 1.313]
+0.027 ∗
-0.002
Medical
3
1.110 [1.089, 1.132]
1.120 [1.100, 1.143]
1.116 [1.095, 1.139]
+0.011 ∗
-0.004 ∗
Medical
5
1.139 [1.106, 1.174]
1.152 [1.119, 1.188]
1.148 [1.115, 1.184]
+0.013 ∗
-0.004 ∗
Table 2: Prompt lookup decoding versus session memory, KV-cached Mistral-7B, 100-token generations, 100 prompts per domain. Speedup is the per-prompt ratio to plain decoding in the same session, mean with 95% bootstrap interval over prompts. recency is most-recent minus earliest match (what recency alone buys); weighting is session memory minus most-recent lookup (what recency weighting and backoff add on top). ∗ : interval excludes zero.
Domain
Drafter
Acc. %
Abstain %
Tok./step
Speedup
Draft % wall
General
Lookup, earliest
49.9
30
2.82
1.622 [1.500, 1.740]
0.01
General
Lookup, most recent
56.3
32
2.98
1.711 [1.585, 1.829]
0.01
General
Session memory
55.4
31
2.98
1.703 [1.578, 1.821]
0.39
General
Session memory, gated
76.4
55
2.81
1.682 [1.550, 1.806]
0.41
Legal
Lookup, earliest
30.7
28
2.11
1.366 [1.316, 1.419]
0.01
Legal
Lookup, most recent
35.9
30
2.27
1.450 [1.384, 1.517]
0.01
Table 3: 500-token generations, k=5 , 30 prompts per domain. Acceptance is per proposed token; abstain is the share of draft calls that proposed nothing; tokens per step counts verifier calls. The gate raises acceptance by abstaining, which lowers tokens per step and speed.
Legal
General
Speedup
J/100
Speedup
J/100
Drafter
greedy
penalised
penalised
greedy
penalised
penalised
Plain decoding
1
1
106.4
1
1
101.7
Lookup, earliest
1.37
1.09
107.7
1.62
1.14
99.9
Lookup, most recent
1.45
1.12
105.9
1.71
1.16
98.7
Lookup, 3-token floor
1.40
1.09
102.0 ∗
1.67
1.16
95.3 ∗
Table 4: 500-token generations with a repetition penalty ( θ =1.2) on the verifier, k=5 , 30 prompts per domain, against greedy decoding. The penalty lowers the share of positions in the last 100 tokens whose preceding 4-gram occurred earlier from 81% to 28% (legal) and from 84% to 44% (general). Energy in joules per 100 tokens; ∗ : paired saving over plain decoding with a 95% bootstrap interval excluding zero.
Generated tokens
Drafter
1–100
–200
–300
–400
–500
Legal
Lookup, earliest
0.41
0.79
1.34
1.62
2.22
Legal
Lookup, most recent
0.43
0.87
1.50
1.92
2.68
Legal
Session memory, gated
0.30
0.62
1.24
1.60
2.29
General
Lookup, earliest
0.44
1.55
2.06
2.07
2.34
General
Lookup, most recent
0.44
1.77
2.52
2.39
2.79
Table 5: Accepted draft tokens per verifier step by position within 500-token greedy generations ( k=5 , 30 prompts per domain). Drafting value rises as the output accumulates, and the gap between most-recent and earliest-match lookup widens.
100 tokens, k=5
500 tokens, k=5
Drafter
Abstain %
General
Medical
Legal
General
Legal
Plain decoding
–
92.7
93.7
98.2
105.3
108.2
Lookup, most recent
41
89.5 ∗
94.6 ∗
99.2 ∗
78.2
90.0 ∗
Lookup, 2-token floor
69
87.5 ∗
92.0 ∗
96.8 ∗
77.7
88.4
Lookup, 3-token floor
79
86.7
91.2
96.1 ∗
78.4
88.6
Lookup, agreement
64
88.6 ∗
93.4 ∗
96.7 ∗
78.7 ∗
89.7 ∗
Table 6: Energy in joules per 100 generated tokens (CPU+GPU+ANE power integrated over each generation; mean over prompts). Abstain is the mean share of draft calls that proposed nothing. ∗ : paired difference from gated session memory has a 95% bootstrap interval excluding zero. A 3-token order floor on lookup comes within 1 J of gated session memory in every column.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Identical
%
Domain
General
840/900
93.3
Domain
Medical
838/900
93.1
Domain
Legal
861/900
95.7
Drafter
Lookup, earliest
841/900
93.4
Drafter
Lookup, most recent
850/900
94.4
Drafter
Session memory
848/900
94.2
Appendix
Table 7: Exact token agreement between speculative output and greedy decoding (2,700 100-token generations). The 47 distinct divergence points have a top-two logit margin of at most 0.023 (median 0.0) against a median of 3.03 over all generated positions; in 45 of 47 the speculative token is the reference’s runner-up.