Offline-To-Online RL

RL: Reinforcement Learning

Overview

Offline-to-online reinforcement learning studies how agents can use datasets, demonstrations, or pretrained policies as a starting point and improve through new environment interactions. Common approaches include behavior-cloning initialization, offline actor-critic methods such as IQL, reuse and adaptive mixing of offline data, and uncertainty-guided exploration; newer work also adapts diffusion and flow policies, often for robotics. Methods are tested on continuous-control benchmarks such as D4RL and OGBench as well as simulated and real-robot tasks. A central tension is preserving useful offline behavior while retaining enough policy and critic plasticity to adapt when the data or environment differs from deployment.

Latest papers

CardsList

76 papers12 papers in the last four weeks, up 33% on the four weeks before. 0.1% of all new papers.

  1. Towards Optimism-Pessimism Trade-off in Model-based Offline-to-Online Reinforcement Learning

    Nov 17, 2026Guochen Zhou, Yijun Yang, Qiqi Duan +6Reinforcement LearningOffline-To-Online RL

  2. SAFER: Navigating the Exploration-Exploitation Dilemma in World-Model-Based Planning for Autonomous Driving

    Nov 15, 2026Yansong Qu, Zilin Huang, Zihao Sheng +2World Model-Based PlanningRL for Autonomous Driving

  3. Imitating the Imperfect: Offline-to-Online Robust Imitation Learning from Heterogeneous Demonstrators

    Nov 11, 2026Cheng Pan, Nan Lu, Kai Arulkumaran +1Robust Imitation LearningImitation Learning

  4. Uncertainty-Aware Hybrid Offline Reinforcement Learning with Online Action Selection Enabled by Sparse Gaussian-Process Critics

    Nov 10, 2026Marco Chan, Christos Evripidou, Liqiu Dong +2Offline RLOffline-To-Online RL

  5. Making Strides in Symbolic Imitation Learning: Offline Demonstrations, Online Tuning

    Nov 9, 2026Bryce MacInnis, Malcolm I. HeywoodImitation LearningReinforcement Learning

  6. ANCHOR: KL-Anchored Adaptation for Stable Offline-to-Online RL

    Nov 1, 2026Geonwoo Cho, Jaegyun Im, Doyoon Kim +1Offline-To-Online RLRL Stability

  7. Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

    Oct 26, 2026Sami Nur Islam, Johan Obando-Ceron, Doina Precup +2RL Fine-TuningOffline-To-Online RL

  8. RoboLongR: Causally Consistent Rewards Teach Robots to Progressively Master Long-Horizon Manipulation

    Oct 24, 2026Zixuan Chen, Lin Cao, Fenggang Zhang +5Long-Horizon Robotic ManipulationRobot Policy Adaptation

  9. RBGA: Residual-Budget Guide Arbitration for Offline-to-Online Safe Reinforcement Learning in Dexterous Manipulation

    Oct 24, 2026Zongyou Yu, Ao Zhou, Xiongbo Shi +7Reinforcement LearningSafe RL

  10. Reinforcement Learning with Multi-Step Transition Predictions: A Bellman Theory

    Oct 23, 2026Chenbei Lu, Zaiwei Chen, Tongxin Li +2Reinforcement LearningModel-Based RL

  11. DT-JEPA: Learning to Predict Future Representations for Decision-Making

    Oct 23, 2026Changxu Ni, Yiyang Ni, Jun Li +4Transformer-Based RLSelf-Supervised RL

  12. PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer

    Oct 22, 2026Ruiqi Wang, Dezhong Zhao, Xuesu Xiao +3Preference-Based RLZero-Shot RL

  13. Simulation Priors for Rapid Adaptation in Model-Based RL for Autonomous Racing

    Oct 19, 2026Adrian Remonda, Jiajun Xi, Nicklas Hansen +5RL for Autonomous DrivingModel-Based RL

  14. Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration

    Oct 17, 2026Ishir Garg, Vivek Myers, Qiyang Li +3EmpowermentRL Exploration

  15. Value as Memory: Online Value-Guided Post-training for General Robot Policies

    Oct 15, 2026Wenye Yu, Yijin Chen, Kaiyi Xiong +7Robotic RLRL Post-Training

  16. Completion-Aware Cross-Fidelity Offline-to-Online Reinforcement Learning for Multi-Line Bus Holding

    Sep 30, 2026Yifan Zhang, Qifan Zhang, Liang ZhengRL ControlReinforcement Learning

  17. Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis

    Sep 27, 2026Yuling Jiao, Lican Kang, Jerry Zhijian Yang +1Diffusion PolicyRL for Diffusion Models

  18. Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation

    Sep 27, 2026Yuheng Huang, Yunpeng Qing, Yixiao Chi +2Offline-To-Online RL

  19. MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimization

    Sep 26, 2026Guowei Zou, Haonan Chen, Haitao Wang +3Cooperative MARLMulti-Agent Coordination

  20. Visuomotor Robotic Pruning in Planar Orchards Using Hybrid Reinforcement Learning

    Sep 21, 2026Abhinav Jain, Cindy Grimm, Stefan LeeVisuomotor Policy LearningRobotics

  21. Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

    Sep 14, 2026Emek Barış Küçüktabak, Karankumar Patel, Zhaodong Yang +3Dexterous ManipulationContact-Rich Robotic Manipulation

  22. SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy

    Sep 11, 2026Xiefeng Wu, Shu Zhang, Zhaojie Chu +1Offline-To-Online RLConservative RL

  23. From Prior-Guided Heuristics to Deployable Agents: Accelerating Demonstration-Driven Reinforcement Learning for Deadline-Constrained Network Control

    Sep 3, 2026Vincenzo Norman Vitale, Mohammad Solki, Antonia Maria Tulino +2Learning from DemonstrationMulti-Agent Reinforcement Learning

  24. Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management

    Aug 28, 2026Yiming Zhang, Kun Yang, Cong Shen +1Wireless Resource AllocationDecentralized MARL

  25. CoDrift: Compositional Drifting for Offline Reinforcement Learning

    Aug 25, 2026Xiewei Ni, Ruofeng Mei, Xiangyu XuOffline RLPolicy Learning

  26. Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning

    Aug 20, 2026Tanachai Anakewat, Takayuki Osa, Tatsuya HaradaReinforcement LearningOffline-To-Online RL

  27. HAF: Adapting Generalist VLAs to Humanoid Whole-Body Loco-manipulation via Hierarchical Action Flow and Spectral Latent RL

    Aug 17, 2026Langzhe Gu, Chengkai Hou, Meng Li +14Humanoid Loco-ManipulationWhole-Body Robotic Manipulation