cs.CLNov 12, 2026

Towards Green Text Generation: Memory-Based Speculative Decoding

Authors: Antal van den Bosch

Organizations: Utrecht University

Abstract

Speculative decoding accelerates text generation with large language models by letting a cheap draft model propose tokens that a large verifier checks in a single forward pass. We introduce Olifant, a draft model built on a memory-based classifier. It predicts the next token from a four-token window in under 0.3 ms on CPU, requiring no GPU, no backpropagation, and no accelerator memory. We measure throughput and energy under two verifier regimes on consumer hardware. Without a KV cache, drafting is nearly free: a domain-matched Olifant model reaches 38.1% acceptance on EU legal text, delivering 2.43x throughput and 2.51x lower energy per token than the neural verifier model alone. With a KV-cached verifier, itself a ~10x energy saver, drafting costs matter: a TinyLlama draft model halves throughput and triples energy despite high acceptance, whereas Olifant still yields 1.25x throughput and the lowest energy in the study (93 J per 100 tokens, 14x below the uncached baseline). We further show that KV caching decouples time from energy: verification of longer drafts raises power draw, shifting the energy-optimal draft length below the speed-optimal one. Speculative decoding is not dead in the age of fast verifiers, provided drafting is nearly free.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Only Accepted Tokens Count: An Energy Accounting of Speculative Decoding Under Latency Targets

    Oct 29, 2026Bole Ma, Ayesha Afzal, Jan Eitzinger +2Speculative DecodingEnergy-Aware LLM Inference

  2. Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

    Nov 3, 2026Hoang Anh Duy Le, Huibing Dong, Alireza Khadem +6Speculative DecodingLLM Inference Acceleration