cs.CLNov 4, 2026

LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models

Authors: Lele Yang, Muxi Diao, Zhonghao Yan, Jiankang Chen, Bin Wen, Tingting Gao, Wentao Zhang, Kongming Liang, 

Organizations: Beijing University of Posts and Telecommunications

Abstract

Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop count is hard to change after training. We propose LooM (Loop-invariant Memory), which addresses each in turn. In a causal encoder-decoder (CED), LooM loops only the decoder layers after the first, which read a global memory written once per token, and each of them publishes the key-value (KV) entry for a token only on its first visit, so the cache does not grow with the loop count. These layers loop independently, providing every budget from 3 to 21 layer executions per token, and one model trained on randomly sampled budgets covers all of them without a router. With enough loops, LooM outperforms a single-pass baseline trained on the same data on language modeling, zero-shot accuracy and long-context perplexity, while using the same cache size. Our analysis further shows that the training budget distribution decides what extra depth buys. These results suggest that a CED is a natural place to loop: memory is written once, and depth becomes a budget chosen at deployment.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

    Nov 9, 2026Jiaqi Leng, Minshen Zhang, Zhaofeng Sun +3LLM Inference AccelerationKV-Cache Management

  2. The Surprising Effectiveness of Shared Memory in Looped Transformers

    Oct 1, 2026Giovanni Monea, Keshav Ramji, Yousef El-Kurdi +4Memory-Efficient InferenceRecurrent Transformers

  3. LoopFold: Parallelizing Recurrent Depth in Looped Transformers

    Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Parallel Training of Recurrent NetworksRecurrent Transformers