Looped Actor: Depth-Recurrent Reasoning Models for Reinforcement Learning
Authors: T. Konstantin Rusch, Tim Seyde, Jared Boyer, Zach J. Patterson, Daniela Rus
Organizations: ELLIS Institute Tübingen & Max Planck Institute for Intelligent Systems & Tübingen AI Center & Liquid AI · Liquid AI · CSAIL, MIT · Case Western Reserve University
Looped reasoning models repeatedly apply a shared set of parameters, enabling more computation without increasing the model size. These models also support input-dependent computation by dynamically deciding when to stop looping. Motivated by the recent success of looped transformers in language modeling and reasoning, we investigate whether dynamic looping can similarly benefit sequential decision-making. We provide a complexity-theoretic motivation for this approach by showing that there exist Markov decision processes in which a state-adaptive policy achieves the optimal return with asymptotically less expected computation than any optimal fixed-runtime policy. To learn compute-adaptive policies in practice, we introduce Looped Actor, a transformer-based policy that repeatedly refines a latent representation toward a fixed point using a shared computational block. This allows the model to allocate computation adaptively by varying the number of loops based on the current state. We evaluate Looped Actor on 22 tasks across six environments, ranging from combinatorial puzzles to robotic manipulation and spanning online and offline reinforcement learning (RL) with discrete and continuous actions. Looped Actor matches or exceeds the performance of an untied baseline with 16× more parameters, with the largest gains in environments where action selection requires substantial multistep planning. For the Boxoban environment, we find that the computation allocation is structured: the number of loops increases with the number of remaining pushes and future optimal pushes become increasingly predictable from the latent state over successive loops. Together, these results highlight actor looping as a simple and efficient way to equip RL agents with adaptive computation and improve their planning capabilities. Code is available at https://github.com/camail-official/LoopedActor
Figures & tables
Figure 1
Environment / task
Iso-params
Iso-FLOPs ( 16×# params)
Looped Actor
Cube double
15.2 ± 7.6
66.9 ± 4.2
67.7 ± 4.6
single pnp
50.8 ± 22.0
95.6 ± 3.0
97.2 ± 1.8
double pnp 1
7.2 ± 7.6
88.8 ± 4.6
89.6 ± 13.1
double pnp 2
3.2 ± 4.6
84.8 ± 8.3
93.2 ± 5.2
swap
0.0 ± 0.0
12.8 ± 3.0
6.4 ± 2.6
stack
14.8 ± 14.1
52.4 ± 8.6
52.0 ± 7.7
Table 1: Success rate of the Looped Actor and the Iso-parameters and Iso-FLOPs baselines on OGBench tasks (%, mean ± std over 5 seeds, final checkpoint at 1M gradient steps).
Figure 3: Accuracy in decoding the optimal plan from Looped Actor latents on Boxoban.
Figure 4
Figure 6: Adaptive vs Fixed-runtime success rates for Looped Actor on Boxoban.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Examples of the two game environments.
Figure 8: OGBench environments for Cube double, Cube triple, Puzzle 3x3, and scene.
Hyperparameter
OGBench
Boxoban
Rush Hour
Layers per iteration
2
2
2
Model width
128
128
128
Attention heads
4
4
4
MLP expansion ratio
4
4
4
Grid convolution
No
Yes
Yes
Train iterations (min/max)
2/16
2/16
2/16
Appendix
Table 2: Model hyperparameters.
Hyperparameter
OGBench
Boxoban / Rush Hour
Algorithm
GCIQL
PPO
Optimizer
Adam
Adam
Learning rate
3×10−4
10−4
Gradient clipping (global norm)
1.0
1.0
Batch / minibatch size
1024
1024
Training budget
106 gradient steps
108 environment steps
Appendix
Table 3: Optimization hyperparameters. Boxoban and Rush Hour share all settings listed here.
Figure 9: Success rate of Looped Actor, Iso-FLOPs, and Iso-Parameters models on all four OGBench environments considered in this work during training (mean and standard deviation over 5 seeds).
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Looped architectures scale computation by reusing the same parameters across recurrent steps, and recent work shows that they substantially improve deep reinforcement learning policies on long-horizon tasks. Since recurrent depth directly controls computation, one may expect looped policies to naturally support elastic inference across recurrent depths. Surprisingly, we find that pretrained looped policies exhibit severe recurrent-depth specialization: reliable decisions are concentrated near the full trained depth, tying deployment computation to this depth even when less computation may suffice. Achieving depth elasticity, i.e., reliable decisions across recurrent depths with adaptive computation at deployment, therefore remains a key challenge. To address this, we propose FlexLoop, a novel post-training framework that converts pretrained fixed-depth looped policies into depth-elastic policies. FlexLoop keeps training on the original RL objective to preserve full-depth capability while performing adjacent-depth policy distillation to progressively transfer decision quality from deeper to shallower recurrent steps. The resulting policy supports reliable inference across recurrent depths and enables state-wise adaptive inference through recurrent-depth consistency. Experiments on 30 online and offline long-horizon goal-conditioned environments show that FlexLoop preserves full-depth performance while making shallower depths effective. Keeping competitive performance, FlexLoop reduces average recurrent depth by up to 43% and achieves up to 1.34× wall-clock speedup in a stress test.
Xun Wang, Ruishuo Chen, Yu Chen +2
Institute for Interdisciplinary Information Sciences, Tsinghua University
How does the amount of compute available to a reinforcement learning (RL) policy affect its learning? Can policies using a fixed amount of parameters, still benefit from additional compute? The standard RL framework does not provide a language to answer these questions formally. Empirically, deep RL policies are often parameterized as neural networks with static architectures, conflating the amount of compute and the number of parameters. In this paper, we formalize compute bounded policies and prove that policies which use more compute can solve problems and generalize to longer-horizon tasks that are outside the scope of policies with less compute. Building on prior work in algorithmic learning and model-free planning, we propose a minimal architecture that can use a variable amount of compute. Our experiments complement our theory. On a set 31 different tasks spanning online and offline RL, we show that (1) this architecture achieves stronger performance simply by using more compute, and (2) stronger generalization on longer-horizon test tasks compared to standard feedforward networks or deep residual network using up to 5 times more parameters.
Raj Ghugare, Michał Bortkiewicz, Alicja Ziarko +1
Department of Computer Science, Princeton University · Warsaw University of Technology · University of Warsaw +2
Looped language models (LoopLMs) perform iterative latent computation to refine internal representations, offering an alternative to explicit chain-of-thought (CoT) reasoning. However, existing reinforcement learning (RL) paradigms primarily target output tokens, creating a structural mismatch with looped architectures whose reasoning unfolds implicitly. In this work, we propose LoopRPT, a reinforcement pre-training framework tailored for LoopLMs. By reframing next-token prediction as a next-token reasoning task, LoopRPT assigns reinforcement signals directly to latent steps using an EMA teacher reference and noisy latent rollouts. This formulation enables RL to directly shape intermediate representations, compressing effective reasoning into fewer iterations. We instantiate LoopRPT on Ouro across multiple model scales. Results show that LoopRPT improves per-step representation quality and achieves Pareto dominance in accuracy-computation trade-offs. Large gains on hard tokens indicate that LoopRPT enhances early-stage reasoning rather than merely encouraging premature exits. Our findings highlight reinforcement pre-training as a principled paradigm for learning efficient latent reasoning in LoopLMs.
Guo Tang, Shixin Jiang, Heng Chang +7
Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China · The Hong Kong University of Science and Technology, Guangzhou, China