Recirculation

Michael C. Mozer; Shoaib Ahmed Siddiqui; Danny Sawyer; Sunny Sanyal; Rosanne Liu

cs.LGAug 18, 2026

Recirculation

Authors: Michael C. Mozer, Shoaib Ahmed Siddiqui, Danny Sawyer, Sunny Sanyal, Rosanne Liu

Organizations: Google DeepMind · University of Texas, Austin

Abstract

We describe an inference-time architectural enhancement for off-the-shelf foundation models that systematically reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase. Motivated by the fundamental limitation that state updates in feedforward transformers are bounded by model depth, our technique, recirculation, introduces a specific form of recurrence that allows the model to act as a dynamical system and track belief states. We distinguish this technique from chain-of-thought computation---which is better reserved for complex inferences rather than basic state tracking---as well as from popular depth-recurrence techniques (looping) and the costly training of recurrent transformers. We also propose and evaluate an adaptive variant of recirculation which requires only light tuning of hyperparameters while freezing the original model weights. Relative to the off-the-shelf baseline, adaptive recirculation achieves surprising gains on the Gemma3 family, including a consistent reduction in perplexity on a suite of datasets, a 21% increase in relative accuracy on GSM8k, and reliable improvements in accuracy on other downstream tasks. Our training-free approach succeeds by leveraging the model itself to inform architectural modifications, suggesting a route to architectural evolution guided by a trained network's properties rather than forced, arbitrary design choices.

Figures & tables

Panel of Figure 1 — Figure 1: Example of a contextualization error that results from failure to track state. This example was produced by Gemini 2.5 Flash in 2025 and motivated by Lepori et al. (2025) . Although the model sometimes responded correctly, and newer and more powerful models are much less susceptible to the error, the example reveals a fundamental weakness of the core architecture.

Figure 2: A schematic depiction of transformer activations in processing the example of Figure 1 . The grid of rectangles denote transformer blocks, with the rows corresponding to network depth (bottom to top is shallow to deep) and columns corresponding to input steps (left to right is first to last). (a) The depth of a state representation in a transformer can limit its utility for inference (adapted from Lepori et al. , 2025 ). (b) Recirculation pushes representations from deep in the transformer to shallower layers, making the state available for information processing.

Figure 3: (a) A transformer in which recurrent connections are introduced from layer 6 back to layer 3. The diagram might correspond to very different recurrence dynamics depending on how the model is sequentially unrolled. (b) Unrolling the recurrent architecture in depth yields a looped transformer. (c) Unrolling the model in both depth and input steps yields recirculation. Recirculation can be viewed as a variant of a looped transformer in which the top stack of step

i

is merged with the bottom stack of step

i+1

, but the read out occurs following the first iteration of a stack.

Figure 4: (a) Unrolled loop transformer and (b) unrolled recirculation transformer. The open colored rectangles depict state propagation. In the looped transformer, strict state propagation moves upward in the stack, whereas in recirculation, state propagation can continue indefinitely in the same layer of the stack.

Figure 5: Percentage change in perplexity on a portion of the arXiv dataset for Gemma3 4B with recirculation, sweeping over $\alpha$ (separate graphs), source layer (vertical axis) and destination layer (horizontal axis). Blue indicates a reduction in perplexity, red an increase.

Figure 6: Top row shows perplexity for a subset of arXiv, C4/WebTextLike and PG-19, using Gemma3 4B with recirculation over a range of source and destination layers (vertical and horizontal axes of heatmaps, respectively) for $\alpha=0.09$ . The boxed heatmap averages the percentage change in perplexity across the three datasets.

		Pretrained (PT)				Instruc. tuned (IT)		PT, missing BOS
		sou=15, dest=8, $\alpha$ =.09				sou=33, dest=28, $\alpha$ =.15		sou=18, dest=9, $\alpha$ =.15
Dataset	# Tokens	baseline perplexity	% reduction	baseline LongPPL †	% reduction	baseline perplexity	% reduction	baseline perplexity	% reduction
arxiv	54.9M	7.86	0.25%	2.09	0.72%	14.37	9.22%	14.76	11.26%
pubmed	26.5M	7.78	0.15%	1.91	0.58%	14.24	12.27%	11.65	9.21%
pg19	10.9M	9.44	0.20%	2.48	1.18%	19.97	10.76%	19.49	15.72%
booksum/chapter	6.9M	13.42	0.19%	2.61	1.40%	26.63	11.22%	∗ 29.09	15.95%

Table 1: % Reduction in Perplexity with Recirculation for Gemma3 4B across four configurations

Figure 7: Recirculation shows promise for four diverse model families. Sweeps over source-destination pairs reveal a robust region in which reductions in perplexity are observed.

Figure 8: The three columns show layer sweeps for Gemma3 1B, 4B, and 12B models with arXiv documents. The first row is with recirculation, the second row is with looped transformers. The two techniques yield qualitatively different outcomes.

Figure 9: Recirculation analysis of the Gemma3 4B model with documents from the arXiv dataset. (a) Mean increase in log likelihood of a target token at position $t+k$ as a result of recirculating the token at position $t$ , as a function of the lag $k$ . Mean is computed over sequences and $t$ . Line is smoothed with a moving average filter. (b) Net increase in log likelihood of a target token at position $t+k$ as a function of $t$ , summed over different lag ranges— $k\in[1,\ldots,8]$ (green), $k\in[1,\ldots,64]$ (orange), $k\in[1,\ldots,256]$ (blue)—and smoothed with a moving-average filter. (c) Mean percentage change in perplexity for individually recirculated tokens grouped according to their part-of-speech (PoS) tags. (d) Recirculation benefit when all and only tokens of a given POS are recirculated, relative to the recirculation benefit when a matched-size set of random tokens is recirculated.

Figure 10: Performance of Gemma3 4B and 12B IT models on a simple in-context instruction-following task. We compare the baseline models with recirculation using preselected hyperparameters and recirculation using hyperparameters tuned to the task.

Figure 11: Question-answering accuracy on the Racing Thoughts datasets of Lepori et al. (2025) for Gemma3 IT models of sizes 1B, 4B, and 12B. Each graph shows accuracy on the three component datasets as a function of number of distractor sentences in the text. Solid lines are model performance without recirculation, dashed with.

Dataset	Baseline % Correct	Recirculation % Correct	Adaptive Recirculation % Correct
			MMLU	MMLU	ARC	ARC
			(test)	(train)	Easy	Challenge
MMLU	57.90	58.28	59.83 *	58.63	56.05	55.02
ARC Easy	81.78	82.07	82.79	82.20	82.87	81.73
ARC Challenge	54.44	54.86	54.95	52.39	53.50	53.58
PiQA	79.98	80.52	80.20	80.30	80.52	80.09

Table 2: Accuracy of Gemma3 4B PT baseline model (second column) and model with recirculation (third column) on eight standard single-token response datasets. Green shading indicates results for which recirculation matches or beats the baseline, red shading indicates otherwise. Recirculation improves performance on six of eight datasets, mean 0.23%. The last four columns report accuracy for adaptive recirculation (Section 4.6 ), a variant of recirculation involving fine tuning of recirculation coefficients

\alpha

and

\beta

. The success of adaptive recirculation is highly dependent on the dataset used for fine tuning (shown at the top of the column); consistent improvements in accuracy are observed when the MMLU test set is used for fine tuning. * denotes overlap between the train and test distributions.

Figure 12: GSM8k and HumanEval accuracy for Gemma3 4B PT with pass@1 (left panel) and pass@128 (right panel). For each dataset, we compare the baseline (off-the-shelf) model against a version with recirculation and a version with adaptive recirculation (see Section 4.6 ).

Figure 13: Reduction in perplexity (%) with six variants of recirculation on Gemma3 1B for nine evaluation datasets. The mean percentage reduction in perplexity across datasets is shown via the gray dashed line above each variant. The variant labeled ‘fixed’ is recirculation with hyperparameters chosen previously. The next four variants involve learning coefficients

\alpha

and

\beta

, either in a token conditional or unconditional manner and either as scalars or vectors. The final variant involves full fine tuning of the LLM, whose performance does not quite match that of the learned-conditional-vector variant, referred to in the text as adaptive recirculation .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Recirculation architecture with two iterations of recirculation, which yields three passes through each stack.

Dataset	Documents	Total Tokens
arxiv	100	923,769
C4	2000	817,377
PG19	10	1,137,576

Appendix

Table B.1 : Sweep Data Statistics

Figure B.1 : Source and destination layer sweep of Gemma3 1B PT with recirculation and $\alpha=0.05$ .

Figure B.2 : Source and destination layer sweep of Gemma3 12B PT with recirculation and $\alpha=0.08$ .

Figure B.3 : Source and destination layer sweep of Gemma3 4B IT model, with recirculation and $\alpha=0.09$ .

Figure B.4 : Source and destination layer sweep of Gemma3 4B PT with recirculation and $\alpha=0.10$ , where the beginning-of-sequence (BOS) token is absent from half the sequences.

Model	Condition	Source	Destination	$\alpha$	$\beta$	Ramp Steps
Gemma3 1B	PT	10	4	0.07	1.00	10
Gemma3 1B	PT, BOS absent	11	4	0.07	0.93	10
Gemma3 4B	PT	15	8	0.09	1.00	0
	PT, BOS absent	18	9	0.10		0
	IT	33	28	0.15
	PT, MMLU devel. set	16	5	0.09

Appendix

Table B.2 : Optimal hyperparameters found by a sweep. Other than the row labeled “MMLU tuning set,” all sweeps used subsets of arXiv, PG-19, and C4, as described in Appendix B.1 . PT = pretrained model, IT = instruction tuned model, BOS absent = only first window in a document included the BOS token.

Dataset	# tokens	Gemma3 1B PT			Gemma3 4B PT			Gemma3 12B PT
		baseline ppl	recirc. ppl	% reduction	baseline ppl	recirc. ppl	% reduction	baseline ppl	recirc. ppl	% reduction
arxiv	51M	19.10	16.54	13.99%	14.76	13.10	11.26%	33.93	25.38	25.20%
big_patent	55M	10.89	9.90	9.13%	8.74	8.12	7.08%	17.55	13.20	24.76%
billsum	5.8M	4.71	4.65	1.25%	3.57	3.41	4.46%	4.50	4.09	10.96%
booksum/book	7.7M	32.48	27.30	15.95%	29.09	24.45	15.95%	77.02	51.67	32.91%
c4/webtextlike	4M	17.10	16.43	3.93%	14.26	13.64	4.37%	19.21	16.71	13.01%

Appendix

Table B.3 : Reduction in perplexity across ten language-modeling tasks with recirculation for Gemma3 models when BOS tokens are omitted.

Figure B.5 : Hyperparameter sweeps for various normalization schemes with Gemma3 1B PT.

Figure B.6 : Hyperparameter sweeps for various normalization schemes with Gemma3 4B PT.

Figure B.7 : Hyperparameter sweeps for various normalization schemes with Gemma3 12B PT.

Description	Formula
no normalization	$\begin{aligned} d^{\prime}&=(1-\alpha)d+\alpha s\\ &=\left(1-\alpha+\frac{\alpha}{\rho}\cos\theta\right)d+\alpha s_{\perp}\end{aligned}$
convex combination, magnitude ratio	$\begin{aligned} d^{\prime}&=(1-\alpha)d+\alpha\rho s\\ &=\left(1-\alpha+\cos\theta\right)d+\alpha\rho s_{\perp}\end{aligned}$
nonconvex combination, magnitude ratio	$\begin{aligned} d^{\prime}&=d+\alpha\rho s\\ &=(1+\alpha\cos\theta)d+\alpha\rho s_{\perp}\end{aligned}$
convex combination, magnitude ratio reweighting, destination renormalized	$\begin{aligned} d^{\prime}&=(1-\alpha)d+\alpha\rho s\\ &=\left(1-\alpha+\cos\theta\right)d+\alpha\rho s_{\perp}\\ d^{\prime}&\leftarrow\frac{\\|d\\|}{\\|d^{\prime}\\|}d^{\prime}\\ \end{aligned}$
nonconvex combination, recirculate only component of source orthogonal to destination	$\begin{aligned} d^{\prime}&=d+\alpha\left(s-\frac{s^{T}d}{d^{T}d}d\right)\\ &=d+\alpha s_{\perp}\end{aligned}$
convex combination (when aligned) or nonconvex (when orthogonal), magnitude ratio reweighting	$\begin{aligned} d^{\prime}&=d+\alpha\rho\left(s-\frac{s^{T}d}{d^{T}d}d\right)\\ &=d+\alpha\rho s_{\perp}\end{aligned}$

Appendix

Table B.4 : A variety of recirculation update rules, expressed both in terms of destination

d

and source

s

, as well as in terms of

d

and

s_{\perp}

, the orthogonal component of

s

s_{\perp}=s-\frac{s^{T}d}{d^{T}d}d

. Other notation: magnitude ratio

\rho=\frac{\|d\|}{\|s\|}

, and the angle

\theta=\arccos\left(\frac{s^{T}d}{\|s\|\|d\|}\right)

Figure C.1 : Recirculation hyperparameter sweeps on Gemma2 2B and Gemma4 12B on a subset of the arXiv dataset with $\alpha=0.07$ .

Figure C.2 : Percentage change in perplexity on PG-19 eval dataset relative to baseline Gemma3 4B model for a range of softmax temperatures, both with (green) and without (grey) recirculation. (top panel) 4B PT, (middle panel) 4B IT, (bottom panel) 4B PT where beginning-of-sequence (BOS) tokens are omitted.

Figure C.3 : Effect of recirculating all tokens by a specific part-of-speech (blue bars) or count matched random tokens (grey bars).

Figure D.1 : Hyperparameter sweeps for Gemma3 1B, 4B, 12B IT models on the instruction-following task

Dataset /	Baseline	Recirculation	Relative
# Distractor Sent.	Error (%)	Error (%)	Reduction (%)
Factual
0	0.00	0.00	N/A
1	0.60	0.47	22.22%
2	0.63	0.49	21.43%
3	0.54	0.30	43.75%

Appendix

Table D.1: Comparison of Baseline vs. Recirculation Error Rate Reduction across Factual, Gender, and Polysemous datasets for the Gemma3 12B model.

Dataset	sou 16 dest 5 $\alpha$ .09		sou 15 dest 8 $\alpha$ .09		sou 18 dest 6 $\alpha$ .06		sou 16 dest 8 $\alpha$ .05		sou 15 dest 8 $\alpha$ .05
	Baseline	Recirculation	Baseline	Recirculation	Baseline	Recirculation	Baseline	Recirculation	Baseline	Recirculation
MMLU (0-shot)	57.90%	58.28%	57.93%	58.12%	57.92%	58.05%	57.92%	58.06%	57.93%	58.25%
HellaSwag (0-shot)	75.89%	75.86%	75.94%	76.17%	76.51%	76.57%	76.51%	76.64%	76.51%	76.68%
WinoGrande (0-shot)	69.46%	69.38%	69.61%	69.85%	69.53%	69.93%	69.53%	69.77%	69.61%	70.48%
ARC Easy (0-shot)	81.78%	82.07%	81.52%	82.11%	82.37%	82.45%	82.37%	82.28%	82.32%	82.62%
ARC Challenge (0-shot)	54.44%	54.86%	54.35%	54.44%	54.61%	54.86%	54.61%	54.52%	54.61%	54.95%

Appendix

Table D.2 : Zero-shot performance comparison between Baseline and Recirculation across five model configurations. The model configurations were chosen based on the hyperparameter sweep to all be good candidates. We report separate baselines for each run because they were performed on different hardware, which affects detailed model performance.

Figure D.2 : (a) Perplexity of the response token for the 1531 problems in the MMLU development set for the Gemma3 4B PT model with recirculation. A sweep is conducted over source and destination layers, fixing $\alpha=.07$ ; (b) Sweep of $\alpha$ , fixing source and destination at 16 and 5, the condition that yields the best performance in (a).

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 280 papers

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory
Sep 3, 2026Yuxiang Wang, Kunyu Feng, Yingda Shen +3Memory-Augmented Language Models Recurrent Transformers
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University +2
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Apr 23, 2026Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3Autoregressive Language Modeling Self-Attention
Harvard University
What Makes Recurrence Effective in Looped Language Models?
Sep 29, 2026Xinlin Zhuang, Siyuan Wang, Imran Razzak +1Recursive Language Models Efficient Language Model Inference
The Chinese University of Hong Kong · MBZUAI

Sep 3, 2026J/K move · Enter open · S save

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

Yuxiang Wang, Kunyu Feng, Yingda Shen, Haoning Xu+2

The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University+2

Memory-Augmented Language Models Recurrent Transformers LLM Inference Acceleration Adaptive Computation

Open paper AskPDF arXiv

Repeating a small block of middle layers increases a language model's effective inference depth without adding parameters or generating extra tokens, and recent work shows that this latent recurrence improves reasoning. However, two design choices limit these gains. Each iteration sees only the previous output and cannot directly access earlier computations. Moreover, a fixed loop count wastes depth on easy inputs while leaving hard ones with too little computation. We introduce RecurTrace, which addresses both limitations using the loop's own trajectory. Specifically, Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone. A halting head then reads the loop state and predicts whether to continue, with supervision from an oracle that identifies when additional depth still reduces loss. In a controlled MathQA comparison on the same looped backbone, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute. By comparison, ACT and PonderNet collapse to one loop, and CALM reaches only 54.1% with 5.6 loops, while the stronger LoopUS-Conf and TaH-Mismatch baselines reach 55.3% at 3.2 loops and 55.7% at 2.1 loops. Finally, RecurTrace improves generation accuracy over same-budget fine-tuned baselines at 0.6B, 1.7B, 4B, and 8B, with the gain growing with model size from 0.6 to 3.4 points.

Abstract

LLM Inference

Explore similar work

RecurTrace: Adaptive Latent Reasoning with Loop-Time Memory

The Recurrent Transformer: Greater Effective Depth and Efficient Decoding

What Makes Recurrence Effective in Looped Language Models?