LoopCache: Accelerating Long-Context Self-Speculative Decoding for Looped Transformers
Organizations: Peking University
Abstract
Looped Transformers achieve large effective depth with compact models by repeatedly applying a weight-shared block across R recurrent steps, each maintaining a loop-specific key-value (KV) cache. To accelerate sequential loop execution, recent wave-style self-speculative methods like WaveFront Decoding (WFD) and LoopSpec pipeline draft and target tokens across loops in a single step. However, they face two critical limitations. First, they draft only at fixed loops and ignore intermediate prediction changes, wasting computation on stale trajectories. Second, batching tokens from different loops loads up to Rx the KV cache per step, severely amplifying memory traffic in long-context scenarios. To address these challenges, we propose LoopCache, an efficient self-speculative decoding framework for long-context Looped Transformers. Based on the correlation between prediction switches and output confidence, LoopCache introduces confidence-guided dynamic draft updating to refine drafts in flight across all intermediate loops. Furthermore, LoopCache employs a cyclic loop KV cache strategy that rotates through loop-specific KV caches, restricting per-step KV traffic to a single loop while batching tokens at the nearest matching cycle for lossless verification. By also selectively retaining independent KV caches for a few loop-sensitive shallow layers, LoopCache improves acceptance rates with negligible overhead. Experiments show that LoopCache achieves average 10.25x and 14.56x speedup over WFD and LoopSpec.
Figures & tables
| Model Metric | Ouro 1.4B | Ouro 2.6B | Huginn 3.5B | Raven Llama-3.2 | Raven OLMo | Raven TinyLlama |
|---|---|---|---|---|---|---|
| Params (B) | 1.40 | 2.60 | 3.50 | 1.38 | 1.35 | 0.84 |
| Loops | 4 | 4 | 32 | 32 | 32 | 32 |
| KV (GiB) | 6.00 | 12.00 | 2.58 | 0.375 | 1.50 | 0.25 |
LLM Inference Acceleration
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
See all 1,233 papers