Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

Amr Hegazy; Amr Alanwar; Mostafa Elhoushi

cs.CLAug 15, 2026

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

Authors: Amr Hegazy, Amr Alanwar, Mostafa Elhoushi

Organizations: The German University in Cairo · Technical University of Munich · Cerebras Systems Inc.

Abstract

Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.

Figures & tables

Figure 1: Gated Recurrent Transformer overview. (a) A Gated Recurrent Unit (GRU) reuses the same weight matrices at every sequence position—recurrence over the sequence dimension. (b) A standard transformer stacks

L

independent layers, each with unique weights, applied once to the full sequence. (c) Gated Recurrent Transformer introduces recurrence over the depth dimension, applying each recurrence to the full sequence

Figure 2: Gated Recurrent Transformer architecture ( 2+5

\times

8+2 layout). Top: Two prelude blocks (gray) encode a fixed context representation

\mathbf{h}^{(\mathrm{pre})}

that is held constant across all recurrence steps. Five shared blocks (blue) are applied

R=8

times recurrently;

\mathbf{h}^{(\mathrm{pre})}

conditions the projection and gate at each step. Two coda blocks (green) project to logits. Bottom: Zoom-in of a single recurrence step

r

: the current state

\mathbf{h}^{(r-1)}

is projected together with

\mathbf{h}^{(\mathrm{pre})}

via

W_{\mathrm{proj}}

, processed by the shared blocks to produce proposal

\mathbf{o}^{(r)}

, and blended back via the elementwise gate

\mathbf{g}^{(r)}\in[0,1]^{T\times d}

Model	Config	Layers	FLOPs/fwd	Params	Val Loss $\downarrow$
iso FLOPs regime (matched FLOPs per forward pass)
GPT-2 Small
Baseline	12L	12	1.84 G	124M	3.15
MoR [ 25 ]	1+1 $\times$ 10+1	3	1.84 G	45M	3.30
Heavy-tail Poisson [ 9 ]	1+1 $\times$ 10+1	3	1.84 G	34M	3.23
Ouro [ 10 ]	1+1 $\times$ 10+1	3	1.84 G	35M	3.19

Table 1: Main results (validation loss). Upper : iso FLOPs regime— Gated Recurrent Transformer matches forward-pass FLOPs of the dense baseline with fewer unique parameters. Shaded rows are dense baselines. Lower : iso Params regime—matched unique parameters, higher FLOPs per step. Layers = unique transformer blocks.

\downarrow

lower is better. † RRT adds one LoRA adapter per recurrence, so its weights are not identical across steps.

Task	GRT iso FLOPs	GPT-2 Large	GRT iso Params
	1+5 $\times$ 6+5	(baseline)	3+30 $\times$ 6+3
	288M, 11L ( $-$ 63%)	774M, 36L	774M, 36L
ARC-Challenge ( $\uparrow$ )	25.43 $\uparrow$ 1.80	23.63	25.26 $\uparrow$ 1.63
ARC-Easy ( $\uparrow$ )	42.89 $\downarrow$ 0.42	43.31	45.54 $\uparrow$ 2.23
BoolQ ( $\uparrow$ )	60.18 $\uparrow$ 2.60	57.58	52.29 $\downarrow$ 5.29
HellaSwag ( $\uparrow$ )	35.62 $\downarrow$ 1.59	37.21	41.52 $\uparrow$ 4.31

Table 2: Downstream task evaluation (large scale). Zero-shot accuracy on standard benchmarks using lm-eval-harness .

\uparrow

higher is better; deltas relative to GPT-2 Large. Orange : best iso FLOPs result; blue : best iso Params result. The iso Params model outperforms the dense baseline on average by +2.10 points ; the iso FLOPs model matches GPT-2 Large at only 37% of its parameter count.

Panel of Figure 3 — Figure 3: Early-Exit Analysis Inference FLOPs are measured based on sequence length 1024, batch size 1.

Figure 4: Next-token probability at each recurrence step. Each row shows one prompt; the correct answer is highlighted in colour. Prelude-only output (leftmost column) captures partial signal; by step 2 the correct token dominates in three of four prompts. Step labels use 1-indexed notation consistent with the analysis section.

Model	Config	Params	Eager (ms/tok)	Compiled (ms/tok)	Peak mem.
GPT-2 Medium	24L	354M	2.95	2.54	747 MB
GRT Medium	2+5 $\times$ 4+2	127M	3.62	2.81	402 MB
GPT-2 Large	36L	774M	4.33	3.33	1570 MB
GRT Large	1+5 $\times$ 6+5	293M	5.33	3.67	639 MB

Table 3: Generation latency and peak memory ( iso FLOPs configurations). Batch size 4, prompt length 1024, 128 generated tokens, same hardware. Both arms are measured under torch.compile and in eager mode.

Configuration	HellaSwag	Mem. @ $B{=}1$	Mem. @ $B{=}32$
GPT-2 Medium (24L, 354M)	34.45	0.75 GiB (1.00 $\times$ )	3.66 GiB (1.00 $\times$ )
GRT iso FLOPs , full $R\times$ KV	33.65	0.41 GiB (0.55 $\times$ )	3.32 GiB (0.91 $\times$ )
GRT iso FLOPs , averaged KV	33.90	0.35 GiB (0.47 $\times$ )	1.44 GiB (0.39 $\times$ )

Table 4: End-to-end decoding memory (medium iso FLOPs checkpoint, bf16,

T=1024

). Weights

+

KV cache. Parenthesised values are relative to GPT-2 Medium.

Figure 5: Component ablation on Gated Recurrent Transformer -small (20,000 steps). Each bar adds one component to the one on its left. The dashed line is the dense 12-layer baseline at

3.157

, trained identically; only the full model falls below it. The 2,000-step counterpart on the medium configuration is in Appendix B.2 .

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Run	$p$	$b$	$R$	$c$	$d$	$h$	Batch tokens	Peak LR	Steps	Hardware
GPT-2 Small (baseline)	—	—	—	—	768	12	491,520	$6\times 10^{-4}$	20k	2 $\times$ H200
GRT Small (isoFLOP)	1	1	10	1	768	12	491,520	$6\times 10^{-4}$	20k	2 $\times$ H200
GPT-2 Medium (baseline)	—	—	—	—	1024	16	491,520	$6\times 10^{-4}$	20k	2 $\times$ H200
GRT Medium (isoFLOP)	2	5	4	2	1024	16	491,520	$6\times 10^{-4}$	20k	2 $\times$ H200
GRT Medium (isoParam)	2	20	4	2	1024	16	491,520	$6\times 10^{-4}$	20k	2 $\times$ H200
GPT-2 Large (baseline)	—	—	—	—	1280	20	491,520	$6\times 10^{-4}$	20k	2 $\times$ H200

Appendix

Table 5: Training configurations.

p

: prelude blocks;

b

: shared blocks;

R

: recurrence steps;

c

: coda blocks;

d

: embedding dimension;

h

: attention heads. Batch tokens = batch_size

\times

gradient_accumulation

\times

sequence_length

\times

num_GPUs. All runs use AdamW (

\beta_{1}=0.9

\beta_{2}=0.95

), weight decay 0.1, gradient clip 1.0, bfloat16 mixed precision, cosine LR schedule.

Model	Seed 1337	Seed 42	Seed 123	Mean $\pm$ std
Dense baseline (12L)	3.154	3.156	3.253	$3.188\pm 0.056$
GRT ( 1+1 $\times$ 10+1 )	3.141	3.146	3.148	$\mathbf{3.145\pm 0.004}$

Appendix

Table 6: Three-seed validation loss on the small configuration (20,000 steps). Both models use the iso FLOPs setting of Table 1 . Lower is better.

1	Dense baseline (non-recurrent)	3.621	—
	Configuration	Val. loss	$\boldsymbol{\Delta}$
2	Recurrence	4.163	$+$ 0.542
3	+ Prelude / coda ( $2{+}\_+2$ )	4.118	$-$ 0.045
4	+ State noise ( $\sigma_{x}{=}0.1$ )	4.099	$-$ 0.019
5	+ Prelude re-injection	3.901	$-$ 0.198
6	+ Elementwise gate (full Gated Recurrent Transformer )	3.786	$-$ 0.115

Appendix

Table 7: Component ablation on Gated Recurrent Transformer -medium at 2 000 steps. Short-horizon counterpart to Figure 5 . The dense baseline (row 1) is a standard 24-layer GPT trained identically.

Property	Dense	MoR	Poisson	RRT	Ouro	GRT
		[ 25 ]	[ 9 ]	[ 12 ]	[ 10 ]	(ours)
Full weight sharing	n/a	Yes	Yes	No (LoRA)	Yes	Yes
Variable inference depth	No	Yes	Yes	No	Yes	Yes
Input-dependent gating	No	Yes	No	No	Yes	Yes
Per-step noise injection	No	No	Init only	No	No	Yes
Per-iteration loss	No	No	No	No	Yes	No

Appendix

Table 8: Design properties of recurrent-depth methods , with small-scale iso FLOPs results for reference. Training times are wall-clock for the small configuration at 20,000 steps on identical hardware, and are not comparable against the dense column: the dense baseline was trained with torch.compile enabled, whereas every recurrent method including our own ran with compilation disabled, which did not work on our hardware. The dense column is therefore optimistic; the recurrent columns remain comparable to one another.

Table 15

Figure 8: Per-token gate activation $\bar{g}^{(r)}_{t}$ across recurrence steps. Each row of cells corresponds to one recurrence step (columns = token positions). Colour encodes $\bar{g}^{(r)}_{t}\in[0.70,1.00]$ : bright yellow-green = write-heavy (gate open, block output substantially absorbed); dark purple = copy-heavy (hidden state passed through unchanged). Token labels are shown below each step column. The per-step mean (annotated top-right of each panel) rises from step 2 onward, reflecting the write-to-copy transition identified in Figure 10 (b). Structurally informative tokens (content words, operators) consistently receive lower gate values than function words and punctuation across all steps.

Figure 9: Loss convergence across recurrence steps. (a) Validation loss at each recurrence step

r

for the trained 1+5

\times

6+5 model. The “Gate=1” baseline (loss

5.26

, green dashed) equals the prelude-only output, confirming that selective writing is essential: a fully open gate discards every block proposal, leaving the hidden state unchanged across all six steps. The “Gate=0” baseline (loss

12.03

) confirms that recurrence is essential, not a skip connection. Extended inference at

R\in\{8,10,12\}

does not improve beyond

R=6

(

2.69

2.72

, and

2.74

respectively), indicating representational convergence. (b) KL divergence of each step’s output distribution to the final-step distribution; 95% of the residual uncertainty is resolved within four recurrence steps.

Figure 10: Gate statistics across recurrence steps. (a) Gate mean and standard deviation. (b) Fraction of copy-saturated (

g>0.95

) and write-saturated (

g<0.05

, scaled

\times 1000

) dimensions per step. Copy-saturation grows steadily while write-saturation stays negligible. (c) Effective gate openness (applied/proposal

L_{2}

ratio), declining monotonically—early steps write substantially, late steps refine conservatively.

Figure 11: Update dynamics across recurrence steps. (a)

L_{2}

norms of the block’s raw proposal (near-constant,

\approx 94

) and the gate-applied update (declining

17.6\to 6.5

); the divergence is caused entirely by gate closure. (b) Cosine similarity between successive representations. (c) Local step gain estimate, peaking at step 2 and declining thereafter.

Figure 12: CKA representational geometry. (a) Within-model CKA; the sharp discontinuity between steps 1–2 and steps 3–6 (dashed lines) reveals a two-phase structure. (b) Cross-model CKA against GPT-2 Large; early steps map to shallow baseline layers, all later steps peak at layer $\approx 11$ .

Figure 13: Token difficulty routing. (a) Total improvement over six steps vs. initial difficulty decile ( $R^{2}=0.998$ ); recurrence implicitly allocates more work to uncertain tokens. (b) Per-step loss curves for selected deciles.

Anchor type	Val. loss (nats)	$\Delta$ vs. trained
Zeros	8.08	+5.40
Input embedding	3.73	+1.05
Prev. hidden state $\mathbf{h}^{(r-1)}$	3.38	+0.70
Trained model (fixed $\mathbf{h}^{(\mathrm{pre})}$ )	2.68	—

Appendix

Table 11: Anchor ablation: effect of replacing

\mathbf{h}^{(\mathrm{pre})}

with alternative anchors at each recurrence step. The “prev. hidden state” anchor drifts at each step (

\mathbf{h}^{(r-1)}

) while the learned adaptive gate remains active throughout. The

0.70

-nat gap between that condition and the trained model isolates the value of holding the prelude output fixed as a stable context reference.

Figure 14: Effective rank and fixed-point convergence. (a) Effective rank grows with depth; the late-delta subspace rank (dashed) is higher than the representation itself. (b) Fixed-point convergence (orange) declines monotonically;

\Delta h

cosine similarity (blue) increases—both consistent with convergent refinement.

Figure 15: Gate attribution across recurrence steps. (a) Fraction of gate logit variance from current state

x

(blue) vs. frozen anchor

h

(orange); the crossover from rough parity at step 1 to 74%/26% at step 6 is explained by growing divergence between

x^{(r)}

and

h

. (b) Token-level

x

-dominance and

x/h

sign-agreement fractions. (c) Gate weight singular value spectra; the 44% gap in leading singular values (

9.87

6.85

) is the structural origin of the state’s eventual dominance.

Figure 16: Recurrent projection weight analysis. (a) Singular value spectra of

W_{x}

and

W_{h}

; both are nearly full-rank. (b) Cumulative energy; both halves reach 90% at similar dimensionalities. (c) Frobenius norms of all four weight halves; the state half (

W_{x}

) consistently outweighs the anchor half (

W_{h}

) in both modules.

Figure 17: Attention entropy across recurrence steps. (a) Full

6\times 100

entropy heatmap; white lines separate blocks.

\star

: Block 0 Head 14 (previous-token specialist, entropy 0.30).

\triangle

: Block 0 Head 4 (broadcast head, entropy 4.87). Head roles are highly stable. (b) Mean entropy (blue) and diagonal mass (orange) per step; non-monotone entropy profile mirrors the two-phase CKA structure. (c) Head type classification across all 100 heads.

Figure 18: Token frequency class breakdown. (a) Per-step loss by BPE frequency band; rare tokens maintain persistently higher loss throughout. (b) Per-step relative improvement

(L^{(r-1)}-L^{(r)})/L^{(r-1)}

per frequency class; the first recurrence step alone resolves 47% of byte/punctuation loss vs. 23% for rare tokens—easy tokens converge fastest, hard tokens accumulate improvements gradually. (c) Initial loss vs. total improvement; the two axes are near-orthogonal, consistent with dual completion/narrowing roles.

Recurrent Transformers

Recurrent Transformers reuse layers or state across steps to extend context, reasoning depth, and sequence modeling with less parameter growth.

See all 179 papers

Apr 23, 2026

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

Transformers process tokens in parallel but are temporally shallow: at position

t

, each layer attends to key-value pairs computed based on the previous layer, yielding a depth capped by the number of layers. Recurrent models offer unbounded temporal depth but suffer from optimization instability and historically underutilize modern accelerators. We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations, yielding layerwise recurrent memory while preserving standard autoregressive decoding cost. We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-token recurrent updates under mild assumptions, while avoiding optimization instability. Naively, prefill/training appears bandwidth-bound with effective arithmetic intensity near

1

because keys and values are revealed sequentially; we give an exact tiling-based algorithm that preserves the mathematical computation while reducing HBM traffic from

Θ(N^2)

Θ(N\log N)

, increasing effective arithmetic intensity to

Θ(N/\log N)

for sequence length

N

. On 150M and 300M parameter C4 pretraining, Recurrent Transformers improve cross-entropy over a parameter-matched Transformer baseline and achieve the improvement with fewer layers (fixed parameters), suggesting that recurrence can trade depth for width, thus reducing KV cache memory footprint and inference latency.

Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3

Harvard University

Autoregressive Language Modeling Self-Attention Attention Mechanisms Efficient Transformer Inference Recurrent Transformers Efficient Attention+1

Ask PDF

Apr 2, 2026

Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulation

Recursive transformers reuse a shared weight block across multiple depth steps, trading parameters for compute. A core limitation: every step applies the same transformation, preventing the model from composing distinct operations across depth. We present Ouroboros, a system that attaches a compact Controller hypernetwork to a recursive transformer block. The Controller observes the current hidden state, produces a per-step diagonal modulation vector, and applies it to frozen SVD-initialized LoRA bases, making each recurrence step input-dependent. We combine this with gated recurrence (bias-initialized to 88% retention) and per-step LayerNorm for stable deep iteration. On Qwen2.5-3B split into a Prelude/Recurrent/Coda architecture (17 of 36 layers retained), Ouroboros reduces training loss by 43.4% over the unmodified 17-layer baseline, recovering 51.3% of the performance gap caused by layer removal. The full system adds only 9.2M trainable parameters (Controller, gate, and per-step norms) yet outperforms equivalently-sized static per-step LoRA by 1.44 loss points at depth 1 and remains ahead across all tested depths (1, 4, 8, 16) and ranks (8, 32, 64). We also find that gated recurrence is essential: without it, recursive layer application makes the model strictly worse. These gains are measured on the training distribution; on held-out text, the Controller does not yet improve over the baseline, a limitation we attribute to frozen downstream layers and discuss in detail. Code: https://github.com/RightNow-AI/ouroboros

Jaber Jaber, Osama Jaber

RightNow AI

Recurrent Transformers Decoder-Only Language Models Hypernetworks Low-Rank Adaptation

Ask PDF

Sep 18, 2026

Trading Depth for Time in Recurrent Transformers

Recurrent Transformers increase computational depth through temporal recurrence, feeding each token's high-level hidden state into the computation of the next. This raises a natural question: is additional computation better spent on more temporal steps or greater physical depth? We investigate this question using Latent Recurrent Transformers (LRT), which retain one backbone forward pass per vocabulary token during decoding and provide a controlled setting for comparing these two ways of adding computation. We insert a latent thought token between consecutive vocabulary tokens, adding a hidden-state refinement step through the shared backbone before predicting the next token. On 16- and 20-layer NanoChat MoE backbones, we compare this model with double-depth vanilla Transformers and LRT, two-loop Transformers, and PonderLM-2 with one thought token. All configurations execute 2L blocks per vocabulary token during decoding. Under matched training FLOPs, LRT with one thought token outperforms PonderLM-2 and overtakes the two-loop Transformer as compute increases. Its gap to vanilla double-depth narrows from 0.014 to 0.001 bits per byte, despite approximately 46% fewer parameters and approximately half as many unique training tokens. At the training budget used for our main comparison, one thought token recovers an average of 89% of the gains from doubling LRT depth across the two backbone sizes, with approximately 48% fewer parameters. These results highlight temporal thinking as a parameter-efficient way to extend computation in recurrent Transformers.

Zeyi Huang, Xuehai He, Yelong Shen +1

Microsoft · University of Wisconsin–Madison

Transformer Recurrent Transformers

Ask PDF

Abstract

Recurrent Transformers

Explore similar work

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulation

Trading Depth for Time in Recurrent Transformers