cs.LGNov 17, 2026

FACT: Fixed-Budget Allocation and Cross-Step Training for Speculative Decoding

Authors: Hadi Fares, Mayssa Hajj Hassan, Ali Salman, Ammar Mohanna

Organizations: American University of Beirut

Abstract

Speculative decoding accelerates language-model generation by verifying multiple draft tokens in a single target-model pass. However, better draft predictions and longer accepted continuations do not necessarily produce faster generation. To examine this gap, we present FACT (Fixed-Budget Allocation and Cross-Step Training), an experimental framework for studying how drafter training, candidate allocation, and execution conditions jointly affect acceptance and latency. Using a lightweight feature-prediction drafter with a frozen Llama-3-8B-Instruct target under greedy decoding, we first examine how training translates into accepted tokens. Backpropagating through predicted-feature rollouts with greater weight on early steps improves acceptance by 15-24% over uniformly weighted detached training, yet average rollout proxy cross-entropy does not reliably rank drafters by acceptance. Beyond prediction quality, the placement of draft candidates also matters: with a fixed 256-node budget and matched per-depth widths, replacing static allocation with confidence-guided allocation increases speedup over autoregressive decoding from 1.46×1.46\times to 2.02×2.02\times on HumanEval and from 1.31×1.31\times to 1.75×1.75\times on MT-Bench. This allocation benefit extends to an independently trained drafter, but stronger training does not consistently amplify it, showing that the two improvements do not necessarily reinforce each other. Even these speed gains depend on execution conditions: changing only host-thread settings can nearly eliminate speedup without changing acceptance. Together, these findings show why training and allocation must be evaluated jointly using both acceptance and end-to-end latency under controlled execution conditions.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Give the Drafter a Second Chance to Guess: Parallel Reconsideration for Speculative Decoding

    Nov 3, 2026Hoang Anh Duy Le, Huibing Dong, Alireza Khadem +6Speculative DecodingLLM Inference Acceleration

  2. Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds

    Oct 7, 2026Yunxiao Zhao, Changxiao CaiSpeculative DecodingParallel Speculative Decoding

  3. Acceptance-Aware Draft Model Training for Speculative Decoding

    Sep 21, 2026Tianhua Xia, Mugilan Ganesan, Yifei Feng +3Speculative DecodingLLM Inference Acceleration