cs.LGNov 14, 2026

Architectures and KV Replay for Depth-Recurrent Language Models

Authors: Lucia Licakova, Niklas Schmidinger, Lukas Hauzenberger, Sebastian Böck, Sepp Hochreiter

Organizations: NXAI · Johannes Kepler University Linz, Austria.

Abstract

Depth recurrence increases effective model depth by repeatedly applying a fixed computation block. However, sharing attention parameters does not share the key-value (KV) states generated at different recurrence iterations, which causes memory growth with recurrence depth during autoregressive decoding. We study how to reduce this memory cost through architectural and cache-sharing approaches within a common depth-recurrent framework. We first study a looped hybrid that repeats a stack of Transformer layers, with recurrent mLSTM or Gated DeltaNet blocks replacing most attention layers and periodic Transformer attention anchors retained. We then restrict recurrence to a dedicated recurrent workspace in a Parcae-style architecture and test its stabilising adapter as a wrapper for the looped hybrid. Finally, we train a looped Transformer with recurrent KV replay, where only selected recurrence iterations write KV states that later iterations reuse. This trains the model to operate with the same shared KV topology used during inference. We find that looped hybrids require sufficiently frequent attention anchors, with performance depending strongly on the recurrent mixer. For cross-iteration KV sharing, recurrent KV replay allows multiple recurrence iterations to reuse the same KV states while retaining performance close to the full looped Transformer with separate KV caches for each iteration.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

    Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Autoregressive Language ModelingSelf-Attention

  2. Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

    Nov 9, 2026Jiaqi Leng, Minshen Zhang, Zhaofeng Sun +3LLM Inference AccelerationKV-Cache Management

  3. Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

    Aug 15, 2026Amr Hegazy, Amr Alanwar, Mostafa ElhoushiRecurrent TransformersTransformer Expressivity