Offline-To-Online RL

RL: Reinforcement Learning

Offline-to-online reinforcement learning studies how agents can use datasets, demonstrations, or pretrained policies as a starting point and improve through new environment interactions. Common approaches include behavior-cloning initialization, offline actor-critic methods such as IQL, reuse and adaptive mixing of offline data, and uncertainty-guided exploration; newer work also adapts diffusion and flow policies, often for robotics. Methods are tested on continuous-control benchmarks such as D4RL and OGBench as well as simulated and real-robot tasks. A central tension is preserving useful offline behavior while retaining enough policy and critic plasticity to adapt when the data or environment differs from deployment.

Latest papers

69 papers11 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
Mar 28, 2026Dwait Bhatt, Shih-Chieh Chou, Nikolay AtanasovOffline-To-Online RL Sample-Efficient RL
Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, USA · Department of Mechanical and Electro-Mechanical Engineering, National Sun Yat-sen University, Taiwan
Escaping Offline Pessimism: Vector-Field Reward Shaping for Safe Frontier Exploration
Mar 18, 2026Amirhossein Roknilamouki, Arnob Ghosh, Eylem Ekici +1RL Exploration Reward Design for RL
Department of Electrical and Computer Engineering, The Ohio State University · Department of Electrical and Computer Engineering, New Jersey Institute of Technology · Department of Computer Science and Engineering, The Ohio State University.
From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
Mar 10, 2026Zhanyi Sun, Shuran SongRL for Robotics Robot Policy Learning
1Stanford University.
SPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Space
Mar 10, 2026Swaminathan S K, Aritra HazraReinforcement Learning Offline-To-Online RL
Department of Computer Science and Engineering, IIT Kharagpur
Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control
Mar 9, 2026Riccardo De Monte, Matteo Cederle, Gian Antonio SustoReinforcement Learning Continuous Control
Department of Information Engineering University of Padova
Hybrid Offline-Online Reinforcement Learning for Sensorless, High-Precision Force Regulation in Surgical Robotic Grasping
Feb 27, 2026Edoardo Fazzari, Omar Mohamed, Khalfan Hableel +2Force-Controlled Robotic Manipulation Surgical Robotics
Dept. of Robotics, Mohamed bin Zayed University of AI, Abu Dhabi, UAE.
Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
Feb 20, 2026Yongjae Shin, Jongseong Chae, Jongeui Park +1Offline-To-Online RL Sample-Efficient RL
KAIST
SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer
Feb 19, 2026Nathan Samuel de Lara, Florian ShkurtiOffline RL Offline-To-Online RL
Department of Computer Science, University of Toronto, Toronto, Canada · Vector Institute
The Three Regimes of Offline-to-Online Reinforcement Learning
Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline RL RL Fine-Tuning
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair

Mar 28, 2026J/K move · Enter open · S save

Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning

Dwait Bhatt, Shih-Chieh Chou, Nikolay Atanasov

Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, USA · Department of Mechanical and Electro-Mechanical Engineering, National Sun Yat-sen University, Taiwan

Offline-To-Online RL Sample-Efficient RL Learning from Demonstration Behavior Cloning

Open paper AskPDF arXiv

Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.