cs.LGNov 14, 2026

LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers

Authors: Siqi Zhu, Suozhi Huang, Yi Zhao, Jiaxuan You

Organizations: University of Illinois Urbana-Champaign · Princeton University

Abstract

Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. During autoregressive decoding, however, generating each token requires several sequential passes through these layers, leading to high decoding latency. Pipelined self-speculative methods use shallow-loop predictions to process tokens at different loops in the same batch, but each token must reach the drafting depth before the next can enter the pipeline. We introduce LoopFill, a training-free speculative decoding algorithm for looped LLM that combines multi-token suffix drafts with shallow-loop predictions in a single draft tree. Both drafters contribute to one tree, so a suffix lookup can immediately extend a shallow prediction and several future tokens can enter the pipeline together. The shared block computes the highest-scoring draft candidates at different loop depths in one batch, and full-depth verification preserves the model's autoregressive output distribution. Across seven benchmarks, LoopFill's average speedup over greedy autoregressive decoding is 5.85×5.85\times on Ouro-2.6B and 5.13×5.13\times on Huginn-3.5B. The speedups exceed those of LoopAhead, the strongest baseline, by 40%40\% on Ouro-2.6B and 25%25\% on Huginn-3.5B.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models

    Oct 15, 2026Zixuan Liu, Nan Xue, Rui Bao +4Speculative DecodingLLM Inference Acceleration

  2. LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

    Sep 15, 2026SangLyul Cho, Langqing Cui, Sehoon Kim +2Efficient Transformer InferenceSpeculative Decoding

  3. DLoop: Looped Speculative Decoding

    Oct 6, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative DecodingLLM Inference Acceleration