Off-policy reinforcement learning (RL) tries to learn the value of a target policy while data comes from a different behavior policy. This often requires trading off bias for variance or stability. One way to trade these off is to linearly interpolate between the behavior and target policies, but this represents only one point in a broader space of interpolation mechanisms. We introduce excursion policies--a structured alternative where the agent follows the target policy for a multi-step "excursion" before reverting to the behavior policy. We derive dynamic programming results for these policies, establishing the theoretical basis for their use in RL. We further develop provably sound off-policy temporal-difference (TD) algorithms for excursion value estimation. We find that excursion policies provide an efficiency-stability trade-off comparable to mixture policies in continuous control benchmarks, and our analysis illustrates conceptual differences in their behavior.
Figures & tables
Figure 1: Analysis of the Highway MDP example . Left : Schematic of the MDP featuring a “slow lane” (reward 0.01 , destination TA ) and a “fast lane” (reward 1.0 , terminal penalty −15.0 at TB ). The discount factor γ is 0.9 . Middle and Right : Value of the initial state under various policies without switching costs c=0 (middle) and with a cost of c=−1.0 (right), both based on an extended 10 -state chain. The behavior policy μ prefers the safe slow lane, while the target policy π prefers the fast lane. This example illustrates that a one-time excursion policy ( πβ ) can be better than a mixture policy ( πα ) when lane-switching incurs a penalty, as the latter induces frequent, costly oscillations.
Figure 2: Discounted deviation equivalence : An example of discounted timelines for a random mixture and a random excursion. In expectation, the total area of the orange blocks is the same.
Algorithm
Correction Weighting Ct:k
Convergence Guaranteed
Fixed Point
Uncorrected TD (λ)
Ct:k=1
Yes
vμ
Off-policy TD (λ)
Ct:k=ρtCt+1:k
No
vπ
α -TD (λ)
Ct:k=[(1−α)+αρt]Ct+1:k
No
vπα
Ex-TD (λ,β)
Ct:k=(1−β)+βρtCt+1:k
No
vπβ,λ
V-trace( λ )
Ct:k=ρ^tCt+1:k
Yes
vπρˉ
α -trace( λ )
Ct:k=[(1−α)+αρ^t]Ct+1:k
Yes
vπρˉα ∗
Table 1: Comparison of off-policy multi-step TD algorithms. Here, convergence means the algorithm converges in the tabular setting for any μ with full support, any π , and any values of relevant parameters, including λ,α , and β . ∗ : See Appendix C for the characterization of vπρˉα and vπρˉβ,λ .
Figure 3: Contraction-bias-variance trade-offs across off-policy learning targets. Quantities are computed for a fixed evaluation task within a randomly generated MDP. Each panel visualizes the relationship between two trade-off variables on the axes, while the point size encodes the magnitude of the third. We compare Ex-Retrace (orange), α -Retrace (purple), n -step (blue), and corrected n -step (green) across their respective parameter ranges. See Section E.1 for detailed definitions of the measured variables and task configurations, and Section F.1 for additional results.
Figure 4: Aggregate learning curves of Ex-Retrace on dm_control across β s ( SPI=32 ), averaged over seven tasks across three seeds, with shaded regions indicating the min-max range.
Figure 5: Aggregate performance on dm_control across Samples-Per-Insert (SPI) ratios. High SPI ratios correspond to increased levels of “off-policyness”. Results are averaged over the last 10% training steps and seven tasks across three seeds, with shaded regions indicating the min-max range. The left panel depicts Ex-Retrace ’s performance as a function of β , shown under the log scale of 1−β1=1+1−ββ , i.e., the mean excursion length including the terminating step. The right panel shows the performance for α -Retrace .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The MDP in Proposition C.2 . Labels give the rewards.
Domain
Task(s)
Observation dimension
Action dimension
dog
stand, walk, run, trot
223
38
humanoid
stand, walk, run
67
21
Appendix
Table 2: Observation and action dimensions of dm_control tasks.
Hyperparameter
Value
Optimizer
Adam
Learning rate (actor / critic)
3×10−4
Discount factor ( γ )
0.99
Trajectory Length for Network Update
20
Minibatch size
256
Number of hidden layers
3
Appendix
Table 3: Hyperparameters of MPO for dm_control .
Figure 7: Trade-offs across off-policy learning targets in a randomly generated evaluation task with seed 0. Top: Original target policy. Middle: Tempered target policy with T=0.5 . Bottom: Tempered target policy with T=0.25 . See Section E.1 for details.
Figure 8: Trade-offs across off-policy learning targets in a randomly generated evaluation task with seed 1. Top: Original target policy. Middle: Tempered target policy with T=0.5 . Bottom: Tempered target policy with T=0.25 . See Section E.1 for details. Since the variance of corrected n -step is too large in this example, its variance estimation is unstable, which is why the estimated variance of n=5 could be lower than that of n=3 .
Figure 9: Trade-offs across off-policy learning targets in a randomly generated evaluation task with seed 2. Top: Original target policy. Middle: Tempered target policy with T=0.5 . Bottom: Tempered target policy with T=0.25 . See Section E.1 for details.
Figure 10: Trade-offs across off-policy learning targets in a randomly generated evaluation task with seed 3. Top: Original target policy. Middle: Tempered target policy with T=0.5 . Bottom: Tempered target policy with T=0.25 . See Section E.1 for details.
Figure 11: Trade-offs across off-policy learning targets in a randomly generated evaluation task with seed 4. Top: Original target policy. Middle: Tempered target policy with T=0.5 . Bottom: Tempered target policy with T=0.25 . See Section E.1 for details.
Figure 12: Learning curves on the dog-walk task. We compare Retrace ( β=1.0 ) against Ex-Retrace with smaller values of β∈{0.0,0.9} . Panels show the return (left), estimated action-value (middle, log-scale), and clipped importance-sampling (IS) weights (right). Results for three independent seeds are plotted to illustrate correlations between metrics.
Figure 13: Learning curves of Ex-Retrace on dm_control . Results are averaged over three seeds, with shaded regions indicating the min-max range.
Figure 14: Learning curves of α -Retrace on dm_control . Results are averaged over three seeds, with shaded regions indicating the min-max range.
Figure 15: Aggregate Learning curves of Ex-Retrace on dm_control across varying SPI ratios. Results are averaged over seven high-dimensional tasks across three seeds, with shaded regions indicating the min-max range.
Figure 16: Aggregate Learning curves of α -Retrace on dm_control across varying SPI ratios. Results are averaged over seven high-dimensional tasks across three seeds, with shaded regions indicating the min-max range.
Figure 17: Learning curves of n -step on dm_control . Results are averaged over three seeds, with shaded regions indicating the min-max range.
Figure 18: Aggregate Learning curves of n -step on dm_control . Results are averaged over seven high-dimensional tasks across three seeds, with shaded regions indicating the min-max range.
Off-policy reinforcement learning studies how agents learn from data generated by other or earlier policies, improving sample efficiency while managing distribution shift.
Large scale reinforcement learning has become a central tool for improving reasoning in large language models. At this scale, generation is often lagged or asynchronous, so updates are performed on data collected by older policies. This makes learning inherently off-policy. Most existing approaches nevertheless remain rooted in PPO-style trust-region objectives, treating training as approximately on-policy and using importance weights to correct distribution mismatch. These corrections can introduce high variance, destabilize optimization, and accelerate entropy collapse. Recent work suggests an alternative: rather than correcting the mismatch, one can embrace off-policy data and remove importance weights, often yielding stronger algorithms. In this paper, we provide an intuitive construction of off-policy objectives that include successful off-policy objectives and show that their effectiveness can be understood through implicit pessimism: they optimize toward target policies that are more conservative than their nominal objectives suggest. This perspective explains why some particular implementation choices improve stability: they implicitly control the effective target distribution. We then propose a principled modification that stabilize this induced distribution and improve off-policy learning.
Reinforcement learning with verifiable rewards is typically performed on-policy, keeping training data close to the current policy but limiting learning to trajectories that the policy can discover itself. Off-policy methods such as supervised fine-tuning, on the other hand, can leverage external knowledge beyond the base model's capabilities, but may suffer from large distribution shift. The key challenge is thus to expand exploration without sacrificing learnability. In this work, we introduce Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts. During generation, a judge-intervention policy periodically reviews the current policy's output, replaces erroneous suffixes with short corrections, and immediately returns control to the policy. During training, MInTRL adopts a sequence-level advantage-regression objective that eliminates the need for importance sampling. We show that sparse, local interventions can substantially improve coverage beyond finite-budget on-policy sampling while preserving the overall on-policy nature of the resulting trajectories. Across math and code benchmarks, MInTRL consistently outperforms standard on-policy and off-policy baselines. Ablations show that MInTRL remains effective with self-intervention and across different judge policies, while performance peaks at moderate intervention intensity, highlighting the importance of intervening minimally. These results establish minimal intervention as an effective paradigm for enhancing on-policy RL.
Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled learning process is well motivated in online RL, where an improved actor collects new data that can further update the actor and the critic. However, training data remains fixed in offline RL, making actor-side policy improvement unable to generate new data to validate or correct the critic. Moreover, retaining this coupled paradigm leads to two related challenges. Firstly, actor updates can drift toward high-valued but potentially out-of-distribution (OOD) actions and amplify critic overestimation. Secondly, conservative value estimation or behavior-cloning regularization creates a difficult trade-off between suppressing OOD actions and selecting high-value actions within the data-supported region. Motivated by this observation, we revisit the conventional offline RL paradigm and propose decoupling policy improvement from actor training. Specifically, we train the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic. We refer to this paradigm as the decoupled policy extraction paradigm. Under such paradigm, the actor provides behavior-supported action candidates, while the critic performs value-based selection within this candidate set. Extensive experiments show that the decoupled policy extraction paradigm outperforms both behavior cloning and jointly learned offline RL methods, while remaining effective even with a naive Q-learning critic.
Xuyao Lin, Yixiang Shan, Jinru Duan +7
Simplexity Robotics, Rensselaer Polytechnic Institute · Simplexity Robotics · Simplexity Robotics, Northeastern University