cs.LGNov 9, 2026

Off-policy Learning with Excursion Policies

Authors: Jiamin He, Mark Rowland, Zhaohan Daniel Guo, Hado van Hasselt, Diana Borsa

Organizations: University of Alberta & Amii · Google DeepMind

Abstract

Off-policy reinforcement learning (RL) tries to learn the value of a target policy while data comes from a different behavior policy. This often requires trading off bias for variance or stability. One way to trade these off is to linearly interpolate between the behavior and target policies, but this represents only one point in a broader space of interpolation mechanisms. We introduce excursion policies--a structured alternative where the agent follows the target policy for a multi-step "excursion" before reverting to the behavior policy. We derive dynamic programming results for these policies, establishing the theoretical basis for their use in RL. We further develop provably sound off-policy temporal-difference (TD) algorithms for excursion value estimation. We find that excursion policies provide an efficiency-stability trade-off comparable to mixture policies in continuous control benchmarks, and our analysis illustrates conceptual differences in their behavior.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think

    May 27, 2026Otmane Sakhi, Aleksei Arzhantsev, Imad Aouali +1Reinforcement LearningRL for Language Model Reasoning

  2. MInTRL: Off-policy Intervention can boost On-policy RL

    Sep 11, 2026Mingyu Chen, Yefan Tao, Gerald Friedland +2Reinforcement LearningReinforcement Learning with Verifiable Rewards

  3. Decoupling Policy Extraction for Offline Reinforcement Learning

    Aug 21, 2026Xuyao Lin, Yixiang Shan, Jinru Duan +7Offline RLPolicy Learning