We describe an inference-time architectural enhancement for off-the-shelf foundation models that systematically reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase. Motivated by the fundamental limitation that state updates in feedforward transformers are bounded by model depth, our technique, recirculation, introduces a specific form of recurrence that allows the model to act as a dynamical system and track belief states. We distinguish this technique from chain-of-thought computation---which is better reserved for complex inferences rather than basic state tracking---as well as from popular depth-recurrence techniques (looping) and the costly training of recurrent transformers. We also propose and evaluate an adaptive variant of recirculation which requires only light tuning of hyperparameters while freezing the original model weights. Relative to the off-the-shelf baseline, adaptive recirculation achieves surprising gains on the Gemma3 family, including a consistent reduction in perplexity on a suite of datasets, a 21% increase in relative accuracy on GSM8k, and reliable improvements in accuracy on other downstream tasks. Our training-free approach succeeds by leveraging the model itself to inform architectural modifications, suggesting a route to architectural evolution guided by a trained network's properties rather than forced, arbitrary design choices.
Figures & tables
Figure 1: Example of a contextualization error that results from failure to track state. This example was produced by Gemini 2.5 Flash in 2025 and motivated by Lepori et al. (2025) . Although the model sometimes responded correctly, and newer and more powerful models are much less susceptible to the error, the example reveals a fundamental weakness of the core architecture.
Figure 2: A schematic depiction of transformer activations in processing the example of Figure 1 . The grid of rectangles denote transformer blocks, with the rows corresponding to network depth (bottom to top is shallow to deep) and columns corresponding to input steps (left to right is first to last). (a) The depth of a state representation in a transformer can limit its utility for inference (adapted from Lepori et al. , 2025 ). (b) Recirculation pushes representations from deep in the transformer to shallower layers, making the state available for information processing.
Figure 3: (a) A transformer in which recurrent connections are introduced from layer 6 back to layer 3. The diagram might correspond to very different recurrence dynamics depending on how the model is sequentially unrolled. (b) Unrolling the recurrent architecture in depth yields a looped transformer. (c) Unrolling the model in both depth and input steps yields recirculation. Recirculation can be viewed as a variant of a looped transformer in which the top stack of step i is merged with the bottom stack of step i+1 , but the read out occurs following the first iteration of a stack.
Figure 4: (a) Unrolled loop transformer and (b) unrolled recirculation transformer. The open colored rectangles depict state propagation. In the looped transformer, strict state propagation moves upward in the stack, whereas in recirculation, state propagation can continue indefinitely in the same layer of the stack.
Figure 5: Percentage change in perplexity on a portion of the arXiv dataset for Gemma3 4B with recirculation, sweeping over α (separate graphs), source layer (vertical axis) and destination layer (horizontal axis). Blue indicates a reduction in perplexity, red an increase.
Figure 6: Top row shows perplexity for a subset of arXiv, C4/WebTextLike and PG-19, using Gemma3 4B with recirculation over a range of source and destination layers (vertical and horizontal axes of heatmaps, respectively) for α=0.09 . The boxed heatmap averages the percentage change in perplexity across the three datasets.
Pretrained (PT)
Instruc. tuned (IT)
PT, missing BOS
sou=15, dest=8, α =.09
sou=33, dest=28, α =.15
sou=18, dest=9, α =.15
Dataset
# Tokens
baseline perplexity
% reduction
baseline LongPPL †
% reduction
baseline perplexity
% reduction
baseline perplexity
% reduction
arxiv
54.9M
7.86
0.25%
2.09
0.72%
14.37
9.22%
14.76
11.26%
pubmed
26.5M
7.78
0.15%
1.91
0.58%
14.24
12.27%
11.65
9.21%
pg19
10.9M
9.44
0.20%
2.48
1.18%
19.97
10.76%
19.49
15.72%
booksum/chapter
6.9M
13.42
0.19%
2.61
1.40%
26.63
11.22%
∗ 29.09
15.95%
Table 1: % Reduction in Perplexity with Recirculation for Gemma3 4B across four configurations
Figure 7: Recirculation shows promise for four diverse model families. Sweeps over source-destination pairs reveal a robust region in which reductions in perplexity are observed.
Figure 8: The three columns show layer sweeps for Gemma3 1B, 4B, and 12B models with arXiv documents. The first row is with recirculation, the second row is with looped transformers. The two techniques yield qualitatively different outcomes.
Figure 9: Recirculation analysis of the Gemma3 4B model with documents from the arXiv dataset. (a) Mean increase in log likelihood of a target token at position t+k as a result of recirculating the token at position t , as a function of the lag k . Mean is computed over sequences and t . Line is smoothed with a moving average filter. (b) Net increase in log likelihood of a target token at position t+k as a function of t , summed over different lag ranges— k∈[1,…,8] (green), k∈[1,…,64] (orange), k∈[1,…,256] (blue)—and smoothed with a moving-average filter. (c) Mean percentage change in perplexity for individually recirculated tokens grouped according to their part-of-speech (PoS) tags. (d) Recirculation benefit when all and only tokens of a given POS are recirculated, relative to the recirculation benefit when a matched-size set of random tokens is recirculated.
Figure 10: Performance of Gemma3 4B and 12B IT models on a simple in-context instruction-following task. We compare the baseline models with recirculation using preselected hyperparameters and recirculation using hyperparameters tuned to the task.
Figure 11: Question-answering accuracy on the Racing Thoughts datasets of Lepori et al. (2025) for Gemma3 IT models of sizes 1B, 4B, and 12B. Each graph shows accuracy on the three component datasets as a function of number of distractor sentences in the text. Solid lines are model performance without recirculation, dashed with.
Dataset
Baseline % Correct
Recirculation % Correct
Adaptive Recirculation % Correct
MMLU
MMLU
ARC
ARC
(test)
(train)
Easy
Challenge
MMLU
57.90
58.28
59.83 *
58.63
56.05
55.02
ARC Easy
81.78
82.07
82.79
82.20
82.87
81.73
ARC Challenge
54.44
54.86
54.95
52.39
53.50
53.58
PiQA
79.98
80.52
80.20
80.30
80.52
80.09
Table 2: Accuracy of Gemma3 4B PT baseline model (second column) and model with recirculation (third column) on eight standard single-token response datasets. Green shading indicates results for which recirculation matches or beats the baseline, red shading indicates otherwise. Recirculation improves performance on six of eight datasets, mean 0.23%. The last four columns report accuracy for adaptive recirculation (Section 4.6 ), a variant of recirculation involving fine tuning of recirculation coefficients α and β . The success of adaptive recirculation is highly dependent on the dataset used for fine tuning (shown at the top of the column); consistent improvements in accuracy are observed when the MMLU test set is used for fine tuning. * denotes overlap between the train and test distributions.
Figure 12: GSM8k and HumanEval accuracy for Gemma3 4B PT with pass@1 (left panel) and pass@128 (right panel). For each dataset, we compare the baseline (off-the-shelf) model against a version with recirculation and a version with adaptive recirculation (see Section 4.6 ).
Figure 13: Reduction in perplexity (%) with six variants of recirculation on Gemma3 1B for nine evaluation datasets. The mean percentage reduction in perplexity across datasets is shown via the gray dashed line above each variant. The variant labeled ‘fixed’ is recirculation with hyperparameters chosen previously. The next four variants involve learning coefficients α and β , either in a token conditional or unconditional manner and either as scalars or vectors. The final variant involves full fine tuning of the LLM, whose performance does not quite match that of the learned-conditional-vector variant, referred to in the text as adaptive recirculation .
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Recirculation architecture with two iterations of recirculation, which yields three passes through each stack.
Dataset
Documents
Total Tokens
arxiv
100
923,769
C4
2000
817,377
PG19
10
1,137,576
Appendix
Table B.1 : Sweep Data Statistics
Figure B.1 : Source and destination layer sweep of Gemma3 1B PT with recirculation and α=0.05 .
Figure B.2 : Source and destination layer sweep of Gemma3 12B PT with recirculation and α=0.08 .
Figure B.3 : Source and destination layer sweep of Gemma3 4B IT model, with recirculation and α=0.09 .
Figure B.4 : Source and destination layer sweep of Gemma3 4B PT with recirculation and α=0.10 , where the beginning-of-sequence (BOS) token is absent from half the sequences.
Model
Condition
Source
Destination
α
β
Ramp Steps
Gemma3 1B
PT
10
4
0.07
1.00
10
PT, BOS absent
11
4
0.07
0.93
Gemma3 4B
PT
15
8
0.09
1.00
0
PT, BOS absent
18
9
0.10
0
IT
33
28
0.15
PT, MMLU devel. set
16
5
0.09
Appendix
Table B.2 : Optimal hyperparameters found by a sweep. Other than the row labeled “MMLU tuning set,” all sweeps used subsets of arXiv, PG-19, and C4, as described in Appendix B.1 . PT = pretrained model, IT = instruction tuned model, BOS absent = only first window in a document included the BOS token.
Dataset
# tokens
Gemma3 1B PT
Gemma3 4B PT
Gemma3 12B PT
baseline ppl
recirc. ppl
% reduction
baseline ppl
recirc. ppl
% reduction
baseline ppl
recirc. ppl
% reduction
arxiv
51M
19.10
16.54
13.99%
14.76
13.10
11.26%
33.93
25.38
25.20%
big_patent
55M
10.89
9.90
9.13%
8.74
8.12
7.08%
17.55
13.20
24.76%
billsum
5.8M
4.71
4.65
1.25%
3.57
3.41
4.46%
4.50
4.09
10.96%
booksum/book
7.7M
32.48
27.30
15.95%
29.09
24.45
15.95%
77.02
51.67
32.91%
c4/webtextlike
4M
17.10
16.43
3.93%
14.26
13.64
4.37%
19.21
16.71
13.01%
Appendix
Table B.3 : Reduction in perplexity across ten language-modeling tasks with recirculation for Gemma3 models when BOS tokens are omitted.
Figure B.5 : Hyperparameter sweeps for various normalization schemes with Gemma3 1B PT.
Figure B.6 : Hyperparameter sweeps for various normalization schemes with Gemma3 4B PT.
Figure B.7 : Hyperparameter sweeps for various normalization schemes with Gemma3 12B PT.
Description
Formula
no normalization
d′=(1−α)d+αs=(1−α+ραcosθ)d+αs⊥
convex combination, magnitude ratio
d′=(1−α)d+αρs=(1−α+cosθ)d+αρs⊥
nonconvex combination, magnitude ratio
d′=d+αρs=(1+αcosθ)d+αρs⊥
convex combination, magnitude ratio reweighting, destination renormalized
d′d′=(1−α)d+αρs=(1−α+cosθ)d+αρs⊥←∥d′∥∥d∥d′
nonconvex combination, recirculate only component of source orthogonal to destination
d′=d+α(s−dTdsTdd)=d+αs⊥
convex combination (when aligned) or nonconvex (when orthogonal), magnitude ratio reweighting
d′=d+αρ(s−dTdsTdd)=d+αρs⊥
Appendix
Table B.4 : A variety of recirculation update rules, expressed both in terms of destination d and source s , as well as in terms of d and s⊥ , the orthogonal component of s : s⊥=s−dTdsTdd . Other notation: magnitude ratio ρ=∥s∥∥d∥ , and the angle θ=arccos(∥s∥∥d∥sTd) .
Figure C.1 : Recirculation hyperparameter sweeps on Gemma2 2B and Gemma4 12B on a subset of the arXiv dataset with α=0.07 .
Figure C.2 : Percentage change in perplexity on PG-19 eval dataset relative to baseline Gemma3 4B model for a range of softmax temperatures, both with (green) and without (grey) recirculation. (top panel) 4B PT, (middle panel) 4B IT, (bottom panel) 4B PT where beginning-of-sequence (BOS) tokens are omitted.
Figure C.3 : Effect of recirculating all tokens by a specific part-of-speech (blue bars) or count matched random tokens (grey bars).
Figure D.1 : Hyperparameter sweeps for Gemma3 1B, 4B, 12B IT models on the instruction-following task
Dataset /
Baseline
Recirculation
Relative
# Distractor Sent.
Error (%)
Error (%)
Reduction (%)
Factual
0
0.00
0.00
N/A
1
0.60
0.47
22.22%
2
0.63
0.49
21.43%
3
0.54
0.30
43.75%
Appendix
Table D.1: Comparison of Baseline vs. Recirculation Error Rate Reduction across Factual, Gender, and Polysemous datasets for the Gemma3 12B model.
Dataset
sou 16 dest 5 α .09
sou 15 dest 8 α .09
sou 18 dest 6 α .06
sou 16 dest 8 α .05
sou 15 dest 8 α .05
Baseline
Recirculation
Baseline
Recirculation
Baseline
Recirculation
Baseline
Recirculation
Baseline
Recirculation
MMLU (0-shot)
57.90%
58.28%
57.93%
58.12%
57.92%
58.05%
57.92%
58.06%
57.93%
58.25%
HellaSwag (0-shot)
75.89%
75.86%
75.94%
76.17%
76.51%
76.57%
76.51%
76.64%
76.51%
76.68%
WinoGrande (0-shot)
69.46%
69.38%
69.61%
69.85%
69.53%
69.93%
69.53%
69.77%
69.61%
70.48%
ARC Easy (0-shot)
81.78%
82.07%
81.52%
82.11%
82.37%
82.45%
82.37%
82.28%
82.32%
82.62%
ARC Challenge (0-shot)
54.44%
54.86%
54.35%
54.44%
54.61%
54.86%
54.61%
54.52%
54.61%
54.95%
Appendix
Table D.2 : Zero-shot performance comparison between Baseline and Recirculation across five model configurations. The model configurations were chosen based on the hyperparameter sweep to all be good candidates. We report separate baselines for each run because they were performed on different hardware, which affects detailed model performance.
Figure D.2 : (a) Perplexity of the response token for the 1531 problems in the MMLU development set for the Gemma3 4B PT model with recirculation. A sweep is conducted over source and destination layers, fixing α=.07 ; (b) Sweep of α , fixing source and destination at 16 and 5, the condition that yields the best performance in (a).
Repeating a small block of middle layers increases a language model's effective inference depth without adding parameters or generating extra tokens, and recent work shows that this latent recurrence improves reasoning. However, two design choices limit these gains. Each iteration sees only the previous output and cannot directly access earlier computations. Moreover, a fixed loop count wastes depth on easy inputs while leaving hard ones with too little computation. We introduce RecurTrace, which addresses both limitations using the loop's own trajectory. Specifically, Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone. A halting head then reads the loop state and predicts whether to continue, with supervision from an oracle that identifies when additional depth still reduces loss. In a controlled MathQA comparison on the same looped backbone, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute. By comparison, ACT and PonderNet collapse to one loop, and CALM reaches only 54.1% with 5.6 loops, while the stronger LoopUS-Conf and TaH-Mismatch baselines reach 55.3% at 3.2 loops and 55.7% at 2.1 loops. Finally, RecurTrace improves generation accuracy over same-budget fine-tuned baselines at 0.6B, 1.7B, 4B, and 8B, with the gain growing with model size from 0.6 to 3.4 points.
Yuxiang Wang, Kunyu Feng, Yingda Shen +3
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University +2
Transformers process tokens in parallel but are temporally shallow: at position t, each layer attends to key-value pairs computed based on the previous layer, yielding a depth capped by the number of layers. Recurrent models offer unbounded temporal depth but suffer from optimization instability and historically underutilize modern accelerators. We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations, yielding layerwise recurrent memory while preserving standard autoregressive decoding cost. We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-token recurrent updates under mild assumptions, while avoiding optimization instability. Naively, prefill/training appears bandwidth-bound with effective arithmetic intensity near 1 because keys and values are revealed sequentially; we give an exact tiling-based algorithm that preserves the mathematical computation while reducing HBM traffic from Θ(N2) to Θ(NlogN), increasing effective arithmetic intensity to Θ(N/logN) for sequence length N. On 150M and 300M parameter C4 pretraining, Recurrent Transformers improve cross-entropy over a parameter-matched Transformer baseline and achieve the improvement with fewer layers (fixed parameters), suggesting that recurrence can trade depth for width, thus reducing KV cache memory footprint and inference latency.
Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness. Through controlled experiments, we systematically examine (1) when recurrence helps, (2) where it should be applied, and (3) how its conditioning affects performance. Our evaluation covers inference budgets below, within, and beyond the training horizon under knowledge and reasoning tasks. (1) We find that recurrence can improve reasoning beyond the training horizon while degrading knowledge performance, but harder reasoning instances do not consistently benefit more. (2) Performance also depends on how distinct layers and recurrent iterations are allocated, showing that effective depth alone is insufficient to predict behavior. Non-recurrent output layers improve robustness to under-unrolling, while the preferred placement of input and output layers varies with inference budget. (3) Finally, we find that conventional initial-state injection offers limited robustness to varying recurrence depth. We therefore propose history-state injection as an alternative, and show that channel-wise history-state injection combined with timestep conditioning offers a low-cost and more effective design, better preserving knowledge under extended unrolling while improving robustness across inference budgets. Overall, our results clarify when recurrent computation helps, where it fails, and offer practical guidelines for designing LoopLMs across variable inference budgets.