cs.CLNov 18, 2026

LoopCache: Accelerating Long-Context Self-Speculative Decoding for Looped Transformers

Authors: Zizhuo Fu, Haochen Huang, Shengxuan Qiu, Linye Wei, Ziyu Tang, Runsheng Wang, Meng Li

Organizations: Peking University

Abstract

Looped Transformers achieve large effective depth with compact models by repeatedly applying a weight-shared block across R recurrent steps, each maintaining a loop-specific key-value (KV) cache. To accelerate sequential loop execution, recent wave-style self-speculative methods like WaveFront Decoding (WFD) and LoopSpec pipeline draft and target tokens across loops in a single step. However, they face two critical limitations. First, they draft only at fixed loops and ignore intermediate prediction changes, wasting computation on stale trajectories. Second, batching tokens from different loops loads up to Rx the KV cache per step, severely amplifying memory traffic in long-context scenarios. To address these challenges, we propose LoopCache, an efficient self-speculative decoding framework for long-context Looped Transformers. Based on the correlation between prediction switches and output confidence, LoopCache introduces confidence-guided dynamic draft updating to refine drafts in flight across all intermediate loops. Furthermore, LoopCache employs a cyclic loop KV cache strategy that rotates through loop-specific KV caches, restricting per-step KV traffic to a single loop while batching tokens at the nearest matching cycle for lossless verification. By also selectively retaining independent KV caches for a few loop-sensitive shallow layers, LoopCache improves acceptance rates with negligible overhead. Experiments show that LoopCache achieves average 10.25x and 14.56x speedup over WFD and LoopSpec.

Figures & tables

Explore similar work

CardsList
  1. LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

    Sep 15, 2026SangLyul Cho, Langqing Cui, Sehoon Kim +2Efficient Transformer InferenceSpeculative Decoding

  2. LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models

    Oct 15, 2026Zixuan Liu, Nan Xue, Rui Bao +4Speculative DecodingLLM Inference Acceleration

  3. LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers

    Nov 14, 2026Siqi Zhu, Suozhi Huang, Yi Zhao +1Speculative DecodingLLM Inference Acceleration