Adaptive Mixing of Policies from Searching and Policies from Learning

Gavin B. Rens

cs.AIAug 16, 2026

Adaptive Mixing of Policies from Searching and Policies from Learning

Authors: Gavin B. Rens

Organizations: Computer Science Division, Stellenbosch University, Stellenbosch, South Africa

Abstract

Background: Distillation of training targets generated thru search/planning has proven useful in reinforcement learning, but search can take exceedingly long. Objectives: Rather than perform search to the same depth every time (typically at a fixed period of steps), reduce the search depth proportionally to the quality of the policy network priors. Methods: We describe Flexer, an architecture that, for each step, mixes the policy from a neural network and the policy from Monte Carlo tree search. The mixing factor favors the MCTS policy as the policy imitation error of the network and the environment models' variance increases. Results: Flexer outperforms a version of AlphaZero (and DQN and ADP) for some experiments on three toy symbolic problems.

Figures & tables

Panel of Figure 1 — Figure 1: Change in mixing factor and its influence (left to right): mixing factor ( $\mu$ ), number of nodes in MCTS tree. running times, return. Results are averages over 30 problem instances.

Figure 2: Conceptual representation of the AZLike architecture

Figure 3: Conceptual representation of the Flexer architecture

Panel of Figure 4 — Figure 4: Environments (left to right): SimpleGrid[10,0.15], BlocksWorld[4,4], MovingNumbers[5,2]

Panel of Figure 5 — Figure 5: Results of experiments running Flexer and AZLike on the SimpleGrid[10,0.15] environment.

Panel of Figure 6 — Figure 6: Results of experiments running Flexer and AZLike on the BlocksWorld[3,3] environment.

Panel of Figure 7 — Figure 7: Results of experiments running Flexer and AZLike on the MovingNumbers[5,2] environment.

Panel of Figure 8 — Figure 8: Results of experiments running Flexer and AZLike on the SimpleGrid[15,0.3] environment.

Panel of Figure 9 — Figure 9: Results of experiments running Flexer and AZLike on the BlocksWorld[4,4] environment.

Panel of Figure 10 — Figure 10: Results of experiments running Flexer and AZLike on the MovingNumbers[10,3] environment.

Panel of Figure 11 — Figure 11: Results of experiments running best Flexers, with and without value boosting on the SimpleGrid[15,0.3] environment.

Panel of Figure 12 — Figure 12: Results of experiments running best Flexers, with and without value boosting on the BlocksWorld[4,4] environment.

Panel of Figure 13 — Figure 13: Results of experiments running best Flexers, with and without value boosting on the MovingNumbers[10,3] environment.

Panel of Figure 14 — Figure 14: Results of experiments running best Flexers, DQN and ADP on the SimpleGrid[15,0.3] environment.

Panel of Figure 15 — Figure 15: Results of experiments running best Flexers, DQN and ADP on the BlocksWorld[4,4] environment.

Panel of Figure 16 — Figure 16: Results of experiments running best Flexers, DQN and ADP on the MovingNumbers[10,3] environment.

Explore similar work

FlexLoop: Depth-Elastic Looped Policies for Adaptive Test-Time Computation in Deep RL
Sep 28, 2026Xun Wang, Ruishuo Chen, Yu Chen +2Efficient Neural Network Inference Reinforcement Learning
Institute for Interdisciplinary Information Sciences, Tsinghua University
Particle Monte Carlo Tree Search
May 9, 2026Yaniv Oren, Viliam Vadocz, Joery A. de Vries +3Monte Carlo Tree Search Model-Based Planning
Department of Intelligent Systems, TU Delft · Centrum Wiskunde & Informatica, Amsterdam · Department of Computer Science, ETH Zürich +2
Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
Aug 13, 2026Haoze Wu, Chuqiao Kuang, Tianyi Zhuang +1Group Relative Policy Optimization Credit Assignment in RL
The Hong Kong University of Science and Technology · Huawei Technologies Ltd.

Sep 28, 2026J/K move · Enter open · S save

FlexLoop: Depth-Elastic Looped Policies for Adaptive Test-Time Computation in Deep RL

Xun Wang, Ruishuo Chen, Yu Chen, Zhuoran Li+1

Institute for Interdisciplinary Information Sciences, Tsinghua University

Efficient Neural Network Inference Reinforcement Learning Recurrent Neural Networks RL Post-Training

Open paper AskPDF arXiv

Looped architectures scale computation by reusing the same parameters across recurrent steps, and recent work shows that they substantially improve deep reinforcement learning policies on long-horizon tasks. Since recurrent depth directly controls computation, one may expect looped policies to naturally support elastic inference across recurrent depths. Surprisingly, we find that pretrained looped policies exhibit severe recurrent-depth specialization: reliable decisions are concentrated near the full trained depth, tying deployment computation to this depth even when less computation may suffice. Achieving depth elasticity, i.e., reliable decisions across recurrent depths with adaptive computation at deployment, therefore remains a key challenge. To address this, we propose FlexLoop, a novel post-training framework that converts pretrained fixed-depth looped policies into depth-elastic policies. FlexLoop keeps training on the original RL objective to preserve full-depth capability while performing adjacent-depth policy distillation to progressively transfer decision quality from deeper to shallower recurrent steps. The resulting policy supports reliable inference across recurrent depths and enables state-wise adaptive inference through recurrent-depth consistency. Experiments on

30

online and offline long-horizon goal-conditioned environments show that FlexLoop preserves full-depth performance while making shallower depths effective. Keeping competitive performance, FlexLoop reduces average recurrent depth by up to

\bf{43\%}

and achieves up to

\bf{1.34\times}

wall-clock speedup in a stress test.

Abstract

Explore similar work

FlexLoop: Depth-Elastic Looped Policies for Adaptive Test-Time Computation in Deep RL

Particle Monte Carlo Tree Search

Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents