cs.LGNov 17, 2026

InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers

Authors: Zhaofeng Sun, Lanxiang Hu, Yu-Yang Qian, Jiaqi Leng, Yuanbo Yang, Chi Zhang, Youyou Lu, Peng Zhao, 

Organizations: Tsinghua University · University of California, San Diego · Nanjing University

Abstract

Parallel Looped Transformers (PLTs) use otherwise idle decoding compute by overlapping recurrent iterations. However, vanilla PLTs scale poorly with depth: they exhibit training-loss spikes, and their quality gains shrink as more loops are added. The benefit of additional loops depends on the information available to each iteration, so we examine whether the existing computational dependencies limit effective depth scaling. We find that only the preceding in-trajectory state is passed across loop boundaries, even though completed cross-trajectory representations are already available. Based on this finding, we propose InfuseLoop, a cross-trajectory latent injection mechanism. It aggregates causally available hidden states from later prediction trajectories through lightweight attention residuals. This design preserves token-parallel training and concurrent loop execution, and requires only O(Rd)O(Rd) additional hidden-state storage. Experiments show that InfuseLoop eliminates training-loss spikes and improves validation-loss scaling up to eight loops. At eight loops, InfuseLoop reduces perplexity by 3.4--17.1% relative to vanilla PLTs and by 10.5--37.3% relative to the non-looped baseline. It also improves average downstream accuracy by 0.84 and 1.99 percentage points over these two baselines. InfuseLoop incurs only 0.3% decoding time over vanilla PLTs and 23.3% over the non-looped baseline, while keeping the same logical KV-cache footprint as vanilla PLTs.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LoopFold: Parallelizing Recurrent Depth in Looped Transformers

    Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Parallel Training of Recurrent NetworksRecurrent Transformers

  2. How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference

    Oct 29, 2026Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan +5Recurrent TransformersEfficient Transformer Inference

  3. FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

    Sep 24, 2026Wanqi Yang, Shiwei LiuEfficient Transformer InferenceRecurrent Transformers