Offline-To-Online RL

RL: Reinforcement Learning

Ask about this topic

Overview

Offline-to-online reinforcement learning studies how agents can use datasets, demonstrations, or pretrained policies as a starting point and improve through new environment interactions. Common approaches include behavior-cloning initialization, offline actor-critic methods such as IQL, reuse and adaptive mixing of offline data, and uncertainty-guided exploration; newer work also adapts diffusion and flow policies, often for robotics. Methods are tested on continuous-control benchmarks such as D4RL and OGBench as well as simulated and real-robot tasks. A central tension is preserving useful offline behavior while retaining enough policy and critic plasticity to adapt when the data or environment differs from deployment.

Latest papers

CardsList

69 papers11 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

  1. Freeze, Don't Clone: Fine-Tuning with Expressive Policies Without Retaining Offline Data

    Oct 26, 2026Sami Nur Islam, Johan Obando-Ceron, Doina Precup +2RL Fine-TuningOffline-To-Online RL

    Mila – Québec AI Institute · McGill University · Université de Montréal +1

  2. RoboLongR: Causally Consistent Rewards Teach Robots to Progressively Master Long-Horizon Manipulation

    Oct 24, 2026Zixuan Chen, Lin Cao, Fenggang Zhang +5Long-Horizon Robotic ManipulationRobot Policy Adaptation

    Nanjing University · Southeast University · Fudan University

  3. RBGA: Residual-Budget Guide Arbitration for Offline-to-Online Safe Reinforcement Learning in Dexterous Manipulation

    Oct 24, 2026Zongyou Yu, Ao Zhou, Xiongbo Shi +7Reinforcement LearningSafe RL

    Tongji University · Shanghai Innovation Institute · Xiaomi EV +2

  4. Reinforcement Learning with Multi-Step Transition Predictions: A Bellman Theory

    Oct 23, 2026Chenbei Lu, Zaiwei Chen, Tongxin Li +2Reinforcement LearningModel-Based RL

    Duffield College of Engineering, Cornell University · Edwardson School of Industrial Engineering, Purdue University · School of Data Science, The Chinese University of Hong Kong (Shenzhen) +2

  5. DT-JEPA: Learning to Predict Future Representations for Decision-Making

    Oct 23, 2026Changxu Ni, Yiyang Ni, Jun Li +4Transformer-Based RLSelf-Supervised RL

    Southeast University · Jiangsu Second Normal University · The University of Hong Kong +1

  6. PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer

    Oct 22, 2026Ruiqi Wang, Dezhong Zhao, Xuesu Xiao +3Preference-Based RLZero-Shot RL

    University of Southern California, Los Angeles, CA, USA. · Beijing University of Chemical Technology, Beijing, China. · George Mason University, Fairfax, VA, USA. +1

  7. Simulation Priors for Rapid Adaptation in Model-Based RL for Autonomous Racing

    Oct 19, 2026Adrian Remonda, Jiajun Xi, Nicklas Hansen +5RL for Autonomous DrivingModel-Based RL

    Toyota Research Institute, Los Altos, CA, USA. · University of California San Diego (UCSD), La Jolla, CA, USA.

  8. Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration

    Oct 17, 2026Ishir Garg, Vivek Myers, Qiyang Li +3EmpowermentRL Exploration

    UC Berkeley · Princeton University

  9. Value as Memory: Online Value-Guided Post-training for General Robot Policies

    Oct 15, 2026Wenye Yu, Yijin Chen, Kaiyi Xiong +7Robotic RLRL Post-Training

    Shanghai Jiao Tong University · Shanghai Innovation Institute · Noematrix Ltd.

  10. Beyond Safe Execution: Safe Online Reinforcement Learning for Autonomous Driving with Value Calibration and Adaptive Guidance

    Oct 13, 2026Zeqiao Li, Yijing Wang, Haoyu Wang +2RL for Autonomous DrivingSafe RL

  11. Completion-Aware Cross-Fidelity Offline-to-Online Reinforcement Learning for Multi-Line Bus Holding

    Sep 30, 2026Yifan Zhang, Qifan Zhang, Liang ZhengRL ControlReinforcement Learning

    Central South University, Changsha, Hunan, China

  12. Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows

    Sep 29, 2026Junhyun Ha, Juho Lee, Byoungwoo ParkPolicy OptimizationOffline RL

    KAIST

  13. Schrödinger--Föllmer Actor--Critic: Diffusion Policy Improvement with Finite-Sample Analysis

    Sep 27, 2026Yuling Jiao, Lican Kang, Jerry Zhijian Yang +1Diffusion PolicyRL for Diffusion Models

    School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. · School of Mathematics and Statistics, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. +1

  14. Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation

    Sep 27, 2026Yuheng Huang, Yunpeng Qing, Yixiao Chi +2Offline-To-Online RL

    Zhejiang University · Carnegie Mellon University · Nanyang Technological University +1

  15. MA-FPPO: Multi-Agent Flow-Pretrained Policy Optimization

    Sep 26, 2026Guowei Zou, Haonan Chen, Haitao Wang +3Cooperative MARLMulti-Agent Coordination

    Sun Yat-sen University · National University of Singapore

  16. Visuomotor Robotic Pruning in Planar Orchards Using Hybrid Reinforcement Learning

    Sep 21, 2026Abhinav Jain, Cindy Grimm, Stefan LeeVisuomotor Policy LearningRobotics

    Collaborative Robotics and Intelligent Systems (CoRIS) Institute, Oregon State University, Corvallis OR 97331, USA

  17. Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

    Sep 14, 2026Emek Barış Küçüktabak, Karankumar Patel, Zhaodong Yang +3Dexterous ManipulationContact-Rich Robotic Manipulation

    Honda Research Institute USA, San Jose, CA, USA · Skild AI, San Mateo, CA, USA · Georgia Institute of Technology, Atlanta, GA, USA

  18. SCQ: Stabilizing Conservative Q-Learning with Sigmoid-Bounded Entropy

    Sep 11, 2026Xiefeng Wu, Shu Zhang, Zhaojie Chu +1Offline-To-Online RLConservative RL

    School of Computer Science, Wuhan University, Wuhan, China · School of Electronic Information, Wuhan University, Wuhan, China · School of Internet, Anhui University, Hefei, China

  19. From Prior-Guided Heuristics to Deployable Agents: Accelerating Demonstration-Driven Reinforcement Learning for Deadline-Constrained Network Control

    Sep 3, 2026Vincenzo Norman Vitale, Mohammad Solki, Antonia Maria Tulino +2Learning from DemonstrationMulti-Agent Reinforcement Learning

    DIETI, University of Naples Federico II, Via Claudio 21, Naples, 80125, Italy · New York University, 6 MetroTech Center, Brooklyn, New York, 11201, USA · University of Southern California, 3740 McClintock Ave., Los Angeles, 90089, California, USA +2

  20. Harnessing human expertise for high-precision robotic assembly in industrialized construction: A sample-efficient installer-in-the-loop interactive reinforcement learning framework

    Sep 2, 2026Zekai Jin, Huiguang Wang, Xiaoning Sun +1Robotic AssemblyReinforcement Learning

    Department of Civil Engineering, McGill University, Montreal, Quebec, Canada

  21. Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management

    Aug 28, 2026Yiming Zhang, Kun Yang, Cong Shen +1Wireless Resource AllocationDecentralized MARL

    Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL 60208 USA · Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, VA 22904 USA

  22. CoDrift: Compositional Drifting for Offline Reinforcement Learning

    Aug 25, 2026Xiewei Ni, Ruofeng Mei, Xiangyu XuOffline RLPolicy Learning

    Xi’an Jiaotong University

  23. Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning

    Aug 20, 2026Tanachai Anakewat, Takayuki Osa, Tatsuya HaradaReinforcement LearningOffline-To-Online RL

    The University of Tokyo · RIKEN Center for Advanced Intelligence Project

  24. HAF: Adapting Generalist VLAs to Humanoid Whole-Body Loco-manipulation via Hierarchical Action Flow and Spectral Latent RL

    Aug 17, 2026Langzhe Gu, Chengkai Hou, Meng Li +14Humanoid Loco-ManipulationWhole-Body Robotic Manipulation

    State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Innovation Center of Humanoid Robotics · Nankai University +1

  25. Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL

    Aug 12, 2026Martin Schuck, Maks Sorokin, Simone Manni +5Robotic Loco-ManipulationRobot Skill Learning

    RAI Institute · Technical University of Munich · ETH Zurich

  26. Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

    Aug 11, 2026Daoyi Li, Yixian Zhang, Wenbo Ding +2Reinforcement LearningCritic-Free RL

    Tsinghua University

  27. Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

    Jul 29, 2026Perry Dong, Ron Polonsky, Dorsa Sadigh +1Continuous ControlRL Fine-Tuning

    Stanford University

  28. A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics

    Jul 28, 2026Zheshun Wu, Renjie Zheng, Jinhang Zuo +2Non-Stationary RLRegret Minimization in RL

    Southern University of Science and Technology · Harbin Institute of Technology, Shenzhen Shenzhen, Guangdong, China · Southern University of Science and Technology Shenzhen, Guangdong, China +4

  29. Active Offline-to-Online Reinforcement Learning

    Jul 13, 2026Alper Kamil Bozkurt, Shangtong Zhang, Yuichi MotaiOffline RLRL Fine-Tuning

    Virginia Commonwealth University, USA · University of Virginia, USA

  30. Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning

    Jul 10, 2026Wenke Xia, Pei Ren, Wenbo Yu +10Value Function EstimationRobotic Manipulation

    1Gaoling School of Artificial Intelligence, Renmin University of China · 2Beijing Innovation Center of Humanoid Robotics · 3Beijing Forestry University +4