cs.CLNov 14, 2026

Text Speeding Itself Up: Recency and Restraint in Session-Local Drafting

Authors: Antal van den Bosch

Organizations: Utrecht University

Abstract

Speculative decoding lets a cheap draft model propose tokens that a large language model verifies in a single forward pass. When the verifier keeps a KV cache, verification is cheap and drafting must be nearly free. Session-local drafters, which predict from the prompt and the text generated so far, meet that bar without training. We compare a recency-weighted, backing-off, confidence-gated session memory against prompt lookup decoding in its reference form, which copies from the earliest earlier match, and in a variant that copies from the most recent one, with paired per-prompt bootstrap intervals throughout. On general, medical, and legal text with a Mistral-7B verifier, recency accounts for the speed: most-recent lookup gains up to 0.09 in speedup over earliest-match lookup on 500-token generations, and session memory's weighting and backoff add nothing measurable on top, at 30 to 40 times the drafting cost. Energy tells a different story. On medical and legal text, unselective drafters are 14% faster than plain decoding yet use as much energy per token, while drafters that abstain on weak evidence save 3 to 4 joules per 100 tokens; a three-token minimum match length on prompt lookup matches the confidence-gated session memory. Outputs are token-identical to greedy decoding in 94% of generations, with every divergence at a numerical near-tie. Under a repetition penalty, long-generation speedups shrink from 1.4-1.7x to 1.1-1.2x, but recency still helps and the length floor is the only drafter that saves energy in both domains.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

    Nov 3, 2026Hoang Anh Duy Le, Huibing Dong, Alireza Khadem +6Speculative DecodingLLM Inference Acceleration

  2. Long-Context Drafting Does Not Require Long-Context Drafters

    Oct 14, 2026Aref Jafari, Vansh Bhatia, Matt Qin +5Speculative DecodingLLM Inference Acceleration