From Chain-of-Thought to Loops: Non-Autoregressive Latent Reasoning via Looped Transformers

Gerard Grau García; Arnau Padrés Masdemont; Niccolò Grillo; Jordi Ros-Giralt; Arash Behboodi; Victor Conchello Vendrell

cs.AIOct 8, 2026

From Chain-of-Thought to Loops: Non-Autoregressive Latent Reasoning via Looped Transformers

Authors: Gerard Grau García, Arnau Padrés Masdemont, Niccolò Grillo, Jordi Ros-Giralt, Arash Behboodi, Victor Conchello Vendrell

Organizations: Qualcomm AI Research · MIT · University of Cambridge · Harvard University

Abstract

Chain-of-thought (CoT) reasoning often improves language-model performance by giving models additional computation before answering. However, explicit CoT expresses this computation as a sequence of autoregressively generated tokens. Latent reasoning replaces these tokens with compact continuous states, but most autoregressive latent-reasoning methods retain a left-to-right dependency among latent vectors. We introduce LLoCoT: a looped latent-reasoning framework that replaces left-to-right latent generation with iterative refinement of a compact latent workspace. A shared transformer is reapplied for a small number of refinement iterations, jointly updating the latent slots based on the prompt and the evolving workspace state. Using the refined state, a probabilistic head predicts a distribution from which latent tokens are sampled in parallel and used to condition an autoregressive decoder for answer generation. Training uses continuous representations derived from explicit CoT together with a final-answer prediction loss and likelihood-based supervision of the latent states. Across HumanEval and MBPP, LLoCoT achieves the highest mean among the evaluated methods, performing on par in accuracy with Reasoning SFT, our explicit-CoT baseline, while outperforming the base model, answer-only SFT and NF-CoT. Relative to Reasoning SFT, LLoCoT reduces time to the first answer token by approximately $36\times$ and reasoning-phase latency by approximately $42\times$ , while increasing end-to-end throughput by $9.2\%$ . This design replaces serial thought generation with parallel latent-slot refinement while retaining probabilistic latent modeling and autoregressive answer decoding.

Figures & tables

Figure 1: Overview of the LLoCoT architecture. A shared transformer repeatedly refines K loop hidden states, with each non-final Gaussian mean providing feedback to the next loop. The final Gaussian head generates all final latent thoughts jointly for autoregressive answer decoding; VAE-derived targets and auxiliary intermediate supervision are used only during training.

Mode	HumanEval	HumanEval+	MBPP	MBPP+	Mean
Qwen3-1.7B-Base	$59.9\pm 1.2$	$52.6\pm 1.3$	$63.3\pm 0.8$	$53.4\pm 0.8$	$57.3$ (+0.0)
Base SFT	$68.6\pm 1.1$	$\underline{63.9\pm 1.1}$	$63.4\pm 0.7$	$\underline{55.0\pm 0.6}$	$62.7$ (+5.4)
Reasoning SFT	$\mathbf{70.3\pm 1.0}$	$\mathbf{64.4\pm 1.0}$	$63.5\pm 0.7$	$54.7\pm 0.7$	$\underline{63.2}$ (+5.9)
NF-CoT	$69.2\pm 1.1$	$63.5\pm 1.1$	$62.2\pm 0.7$	$52.9\pm 0.7$	$62.0$ (+4.7)
No-feedback LLoCoT, $R=2$	$\underline{69.3\pm 1.1}$	$63.8\pm 1.1$	$\underline{64.3\pm 0.7}$	$\underline{55.0\pm 0.6}$	$63.1$ (+5.8)
LLoCoT	$\underline{69.3\pm 1.0}$	$\underline{63.9\pm 1.0}$	$\mathbf{65.1\pm 0.6}$	$\mathbf{56.2\pm 0.6}$	$\mathbf{63.6}$ (+6.3)

Table 1: Main code-generation results. Values are percentages. Benchmark entries are point estimate

\pm

the 95% fixed-task sampling-interval half-width. The mean is the unweighted arithmetic mean of the four benchmark point estimates. Parenthetical values in the Mean column are changes relative to Qwen3-1.7B-Base. Best and second-best available entries are shown in bold and underlined , respectively.

Mode	HumanEval	HumanEval+	MBPP	MBPP+	Mean
No-feedback LLoCoT, $R=1$	$69.1\pm 1.0$	$63.3\pm 1.1$	$\mathbf{64.4\pm 0.7}$	$\underline{54.9\pm 0.6}$	$62.9$ (-0.2)
No-feedback LLoCoT, $R=2$	$\underline{69.3\pm 1.1}$	$\underline{63.8\pm 1.1}$	$\underline{64.3\pm 0.7}$	$\mathbf{55.0\pm 0.6}$	$\underline{63.1}$ (+0.0)
No-feedback LLoCoT, $R=3$	$\mathbf{69.4\pm 1.0}$	$\mathbf{64.0\pm 1.1}$	$\underline{64.3\pm 0.6}$	$\mathbf{55.0\pm 0.6}$	$\mathbf{63.2}$ (+0.1)
No-feedback LLoCoT, $R=4$	$68.2\pm 1.0$	$63.1\pm 1.0$	$63.8\pm 0.7$	$54.8\pm 0.6$	$62.5$ (-0.6)
No-feedback LLoCoT, $R=6$	$68.6\pm 1.1$	$63.6\pm 1.1$	$60.4\pm 0.7$	$52.0\pm 0.6$	$61.2$ (-1.9)
No-feedback LLoCoT, $R=8$	$68.0\pm 1.0$	$63.0\pm 1.0$	$62.2\pm 0.6$	$53.6\pm 0.6$	$61.7$ (-1.4)

Table 2: Loop-depth ablation for No-feedback LLoCoT. Values are percentages and report point estimate

\pm

the 95% fixed-task sampling-interval half-width. The mean is the arithmetic average of the four benchmark point estimates. Parenthetical values in the mean column are changes relative to No-feedback LLoCoT,

R=2

. Green, red, and gray indicate positive, negative, and zero changes, respectively.

Mode	HumanEval	HumanEval+	MBPP	MBPP+	Mean
No-feedback LLoCoT, $R=2$	$\mathbf{69.3\pm 1.1}$	$\underline{63.8\pm 1.1}$	$64.3\pm 0.7$	$55.0\pm 0.6$	$63.1$ (+0.0)
+ Answer-mean	$61.7\pm 1.2$	$55.3\pm 1.2$	$60.2\pm 0.7$	$51.6\pm 0.7$	$57.2$ (-5.9)
+ Auxiliary-NLL loss	$68.1\pm 1.1$	$62.6\pm 1.0$	$65.0\pm 0.7$	$56.0\pm 0.6$	$62.9$ (-0.2)
+ Feedback-mean + auxiliary NLL (LLoCoT)	$\mathbf{69.3\pm 1.0}$	$\mathbf{63.9\pm 1.0}$	$\underline{65.1\pm 0.6}$	$\underline{56.2\pm 0.6}$	$\underline{63.6}$ (+0.5)
+ Feedback-sampled + auxiliary NLL	$67.9\pm 1.1$	$62.3\pm 1.0$	$64.2\pm 0.6$	$55.3\pm 0.6$	$62.4$ (-0.7)

Table 3: Feedback, conditioning, and auxiliary-supervision ablations. Values are percentages and report point estimate

\pm

the 95% fixed-task sampling interval half-width. The mean is the arithmetic average of the four benchmark point estimates. Parenthetical values in the mean column are changes relative to No-feedback LLoCoT,

R=2

. Green, red, and gray indicate positive, negative, and zero changes, respectively.

Metric	Base model	Reason. SFT	NF-CoT	LLoCoT
Time to first answer token (s) $\downarrow$	$\mathbf{0.03}$	$3.17$	$2.01$ ( $\times 0.6$ )	$\underline{0.09}$ ( $\times 0.03$ )
End-to-end latency (s) $\downarrow$	$22.59$	$15.16$	$\underline{13.73}$ ( $\times 0.9$ )	$\mathbf{11.47}$ ( $\times 0.8$ )
Serial reasoning/latent phase (s) $\downarrow$	$\mathbf{0}$	$2.47$	$1.98$ ( $\times 0.8$ )	$\underline{0.06}$ ( $\times 0.02$ )
Conditioning/selection phase (s) $\downarrow$	$\mathbf{0}$	$0.70$	$\underline{0.03}$ ( $\times 0.04$ )	$\underline{0.03}$ ( $\times 0.04$ )
Peak allocated (GiB) $\downarrow$	$\underline{3.4}$	$\mathbf{3.3}$	$6.3$ ( $\times 1.9$ )	$6.3$ ( $\times 1.9$ )
Reasoning/latent forward calls $\downarrow$	$\mathbf{1.0}$	$81.5$	$64.0$ ( $\times 0.8$ )	$\underline{2.0}$ ( $\times 0.02$ )

Table 4: Sampled inference-efficiency measurements. Values are point estimates. Parenthetical multiplicative ratios for NF-CoT and LLoCoT are relative to Reasoning SFT. Ratios are rounded to the displayed precision; values below

0.1

retain one significant digit. Best and second-best values in each row are shown in bold and underlined, respectively. Colors indicate favorable, unfavorable, and neutral changes relative to Reasoning SFT.

Explore similar work

Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
Jun 30, 2026Ying Fan, Anej Svete, Kangwook LeeCoT Reasoning Recurrent Transformers
Microsoft Research · 2ETH Zürich · 3KRAFTON +1
Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate
Jun 15, 2026Hanyu Lin, Min Cai, Jiawei Wen +1Language Model Decoding Efficient Language Model Reasoning
1Shenzhen University · University of Alberta
Latent Reasoning with Normalizing Flows
Jun 4, 2026Guancheng Tu, Xiangjun Fu, Suhao Yu +5Normalizing Flows Continuous Latent Reasoning
University of Pennsylvania · UC San Diego · Meta

Jun 30, 2026, cs.LGJ/K move · Enter open · S save

Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

Ying Fan, Anej Svete, Kangwook Lee

Microsoft Research · 2ETH Zürich · 3KRAFTON+1

CoT Reasoning Recurrent Transformers Continuous Latent Reasoning Efficient Language Model Inference Latent CoT Reasoning

Open paper AskPDF arXiv

Language models typically reason via explicit chain-of-thought (CoT), generating intermediate steps token-by-token. Latent CoT offers an alternative: it performs multi-step reasoning in the model's hidden states, replacing decoded tokens with continuous representations for greater efficiency. However, existing latent CoT methods underperform explicit CoT beyond 1B parameters, and the gap widens with scale. Looped, or recurrent-depth, Transformers, which reuse their weights to increase computation depth without adding parameters, are a natural fit for latent reasoning. We therefore ask whether looped Transformers can bridge this gap. We answer affirmatively with a simple recipe: a looped padded Transformer that processes K latent blocks in parallel for R iterations, with a cross-entropy loss on each latent position's gold CoT-step token, similar to explicit CoT supervision. We instantiate it as LOTUS (Looped Transformers with parallel supervision on latents). LOTUS is, to our knowledge, the first latent-CoT method to bridge the gap to explicit CoT at the 3B scale, while cutting thought-phase latency by 2.5x-6.9x from compact math expressions to natural language. Projecting LOTUS's post-loop latents through the base LM head recovers the gold reasoning steps and even surfaces alternative valid intermediate steps, evidence that its latent space is interpretable and CoT-aligned. Ablations confirm that both the looped backbone and the parallel supervision on gold CoT tokens are essential. Code is available at https://github.com/yingfan-bot/lotus.

Abstract

Explore similar work

Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

Tyler: Typed Latent Reasoning for Language Models -- When to Think, What to Compute, and How Much to Allocate

Latent Reasoning with Normalizing Flows