cs.LGNov 18, 2026

MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

Authors: Seungyub Han, Taehyun Cho, Dohyeong Kim, Kyungjae Lee, Jungwoo Lee

Organizations: Seoul National University · Korea University

Abstract

Many multi-step off-policy temporal-difference methods handle policy mismatch through products of per-step trace coefficients, leading to a trade-off between variance control and long-horizon signal. The maximum trace length consequently becomes a sensitive hyperparameter. We propose MINT (Meeting-time INdicators for Truncation), which replaces these products with a binary stopping indicator: one until coupled behavior and target trajectories first meet, zero thereafter. At the true value function, the indicator's idempotence yields a variance bound governed by meeting-time survival. For practical implementation, a single sampled action provides a model-free surrogate for the stopping rule, requiring no transition model or mixing-time estimate. We derive separate sufficient conditions for contraction of the state-coupled operator and its practical surrogate under fixed-policy evaluation. On MuJoCo and DeepMind Control tasks, MINT outperforms the reported multi-step baselines; the main MuJoCo comparison uses a common cap without per-task cap-length tuning. We also demonstrate the effectiveness of MINT on Atari 100k, RWRL, and OGBench experiments; MINT adapts the backup horizon to reduce target variability relative to long fixed-horizon backups across tasks and dynamics.

Figures & tables

Appendix figures & tables61 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. MInTRL: Off-policy Intervention can boost On-policy RL

    Sep 11, 2026Mingyu Chen, Yefan Tao, Gerald Friedland +2Reinforcement LearningReinforcement Learning with Verifiable Rewards

  2. Offline Policy Evaluation for Manipulation Policies via Discounted Liveness Formulation

    May 12, 2026Hao Wang, Joshua Bowden, Colton Crosby +1RL for RoboticsReinforcement Learning

  3. Behavior-Induced Mirror-Prox Temporal-Difference Learning for Faster Off-Policy Prediction

    May 16, 2026Xingguo Chen, Yuchen Shen, Shangdong Yang +3Off-Policy EvaluationTemporal-Difference Learning