Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulation

Jaber Jaber; Osama Jaber

cs.LGApr 2, 2026

Ouroboros: Dynamic Weight Generation for Recursive Transformers via Input-Conditioned LoRA Modulation

Authors: Jaber Jaber, Osama Jaber

Organizations: RightNow AI

Abstract

Recursive transformers reuse a shared weight block across multiple depth steps, trading parameters for compute. A core limitation: every step applies the same transformation, preventing the model from composing distinct operations across depth. We present Ouroboros, a system that attaches a compact Controller hypernetwork to a recursive transformer block. The Controller observes the current hidden state, produces a per-step diagonal modulation vector, and applies it to frozen SVD-initialized LoRA bases, making each recurrence step input-dependent. We combine this with gated recurrence (bias-initialized to 88% retention) and per-step LayerNorm for stable deep iteration. On Qwen2.5-3B split into a Prelude/Recurrent/Coda architecture (17 of 36 layers retained), Ouroboros reduces training loss by 43.4% over the unmodified 17-layer baseline, recovering 51.3% of the performance gap caused by layer removal. The full system adds only 9.2M trainable parameters (Controller, gate, and per-step norms) yet outperforms equivalently-sized static per-step LoRA by 1.44 loss points at depth 1 and remains ahead across all tested depths (1, 4, 8, 16) and ranks (8, 32, 64). We also find that gated recurrence is essential: without it, recursive layer application makes the model strictly worse. These gains are measured on the training distribution; on held-out text, the Controller does not yet improve over the baseline, a limitation we attribute to frozen downstream layers and discuss in detail. Code: https://github.com/RightNow-AI/ouroboros

Figures & tables

System	Year	Per-Step	Input-	Gated	SVD	Trainable
		Adapters	Cond.	Recur.	Init	Params
Universal Transformer	2019	✗	✗	✗	✗	Full
Relaxed Recursive	2024	✓	✗	✗	✓	LoRA
RingFormer	2025	✓	✗	✗	✗	Signals
Huginn	2025	✗	✗	✗	✗	Full
Retrofitted Recurrence	2025	✗	✗	✗	✗	Full

Table 1: Comparison of recursive transformer approaches. Ouroboros is the only system where per-step weight modulation depends on the current hidden state (input-conditioned).

Panel of Figure 1 — Figure 1: Ouroboros architecture. A pretrained model is split into frozen Prelude and Coda sections. The middle Recurrent Block loops a single transformer layer $N$ times. At each step, the Controller observes the mean-pooled hidden state $\bar{h}$ and generates a diagonal modulation vector applied to frozen SVD-initialized LoRA bases. A gated recurrence (initialized to 88% retention) mixes the new and previous hidden states.

Component	Parameters	Status
Prelude (8 layers)	544M	Frozen
Recurrent Layer	86M	Frozen
Coda (8 layers)	544M	Frozen
Embeddings + LM Head	622M	Frozen
CompactController	0.7M	Trained
Recurrence Gate	8.4M	Trained

Table 2: Parameter breakdown for Ouroboros on Qwen2.5-3B. All base model components are frozen.

Configuration	Controller	Static LoRA	$\Delta$ (Controller wins by)
Depth = 1	5.082	6.519	1.437
Depth = 4	5.075	5.246	0.171
Depth = 8	5.080	5.119	0.039
Depth = 16	5.078	—	—
lr = 1e-3 (depth 8)	5.073	5.106	0.033
Rank = 8 (depth 8)	5.078	5.091	0.013

Table 3: Controller vs. static per-step LoRA on Qwen2.5-3B (Prelude/Recurrent/Coda). All runs: 300K steps, rank 32, FineWeb-edu. The 17-layer baseline without any recurrence has loss 8.975. Lower is better. The Controller wins in every configuration, with the largest margin at depth 1.

Variant	Final Loss	Improvement	vs. Baseline (8.975)
Depth = 1	5.082	$-$ 43.4%	$-$ 3.893
Depth = 2	5.081	$-$ 43.4%	$-$ 3.894
Depth = 4	5.075	$-$ 43.5%	$-$ 3.900
Depth = 8	5.080	$-$ 43.4%	$-$ 3.895
Depth = 16	5.078	$-$ 43.4%	$-$ 3.897
Rank = 8	5.078	$-$ 43.4%	$-$ 3.898

Table 4: Controller ablation across depths, ranks, and learning rates. All runs: 300K steps on a single H100. The system converges to nearly identical loss (

\approx 5.08

) across all configurations, demonstrating strong robustness to hyperparameter choices.

Model	Avg Loss	Beats 17-layer	Beats Full 3B
Full Qwen 3B (36 layers)	1.683	—	—
17-layer baseline	5.690	—	0/12
Ouroboros V2	5.961	3/12	0/12

Table 5: Held-out evaluation on 12 diverse passages. The Controller improves training-distribution loss but does not yet improve held-out loss. We attribute this to frozen coda layers that cannot adapt to modified hidden-state distributions.

Recurrent Transformers

Recurrent Transformers reuse layers or state across steps to extend context, reasoning depth, and sequence modeling with less parameter growth.

See all 179 papers

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation
Aug 15, 2026Amr Hegazy, Amr Alanwar, Mostafa ElhoushiRecurrent Transformers Transformer Expressivity
The German University in Cairo · Technical University of Munich · Cerebras Systems Inc.
How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
Apr 22, 2026Kristian Schwethelm, Daniel Rueckert, Georgios KaissisTruncated Backpropagation Through Time Language Model Scaling Laws
Chair for AI in Healthcare and Medicine, Technical University of Munich, Germany · Department of Computing, Imperial College London, UK · Munich Center for Machine Learning (MCML), Germany +1
LoopFold: Parallelizing Recurrent Depth in Looped Transformers
Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Recurrent Transformers Efficient Transformer Inference
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute

Aug 15, 2026J/K move · Enter open · S save

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

Amr Hegazy, Amr Alanwar, Mostafa Elhoushi

The German University in Cairo · Technical University of Munich · Cerebras Systems Inc.

Recurrent Transformers Transformer Expressivity Decoder-Only Language Models LLM Compression

Open paper AskPDF arXiv

Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.

Abstract

Recurrent Transformers

Explore similar work

Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models

LoopFold: Parallelizing Recurrent Depth in Looped Transformers