Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning

Dwait Bhatt; Shih-Chieh Chou; Nikolay Atanasov

cs.ROMar 28, 2026

Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning

Authors: Dwait Bhatt, Shih-Chieh Chou, Nikolay Atanasov

Organizations: Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, USA · Department of Mechanical and Electro-Mechanical Engineering, National Sun Yat-sen University, Taiwan

Abstract

Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.

Figures & tables

Panel of Figure 2 — Fig. 3 : Comparison of SAC vs TD3 success rates for Strategy A and C approaches.

Panel of Figure 3 — Fig. 4 : Comparing Sample Efficiency Improvement (SEI) scores for single strategy approaches.

Panel of Figure 4 — Fig. 5 : Comparison of Sample Efficiency Improvement (SEI) scores for the top ten hybrid variants across different strategy combinations.

Fig. 6 : Comparison of Sample Efficiency Improvement scores for the best performing variant of each strategy combination.

Fig. 7 : Distribution of normalized impact scores for individual algorithmic components across all task and robot configurations. Scores are independently normalized to

[-1,1]

per configuration. Positive values indicate that adding the component improved sample efficiency compared to matched baselines, while negative values indicate a degradation. The spread of the whiskers illustrates each component’s sensitivity to different task difficulties.

Fig. 8 : Normalized impact of individual components as a function of task difficulty. Tasks are ranked from easiest to hardest, defined by the number of steps required for SAC to reach peak success (values shown in parentheses below each task). Results are grouped by strategy to highlight how the utility of each algorithmic choice shifts in more complex settings.

Fig. 9 : Comparison of Sample Efficiency Improvement (SEI) scores for all methods, aggregated across the four evaluated task-robot configurations. Higher SEI scores indicate faster learning speed and final performance relative to the SAC baseline (SEI = 0). Component abbreviations are defined as follows: demos/rollouts indicates the data source; prefill/auxbc denotes the Strategy A direct data usage method; bc/mcq/calql/cqlrho/cqlh specifies the Strategy B pretraining method; and resrl/ibrl/cheq represents the Strategy C action mixing mechanism.

Online RL Hyperparameters
Hyperparameter	Value
Horizon ( $h$ )	3
Discount factor ( $\gamma$ )	0.8
Target smoothing ( $\tau$ )	0.01
Batch size	1024
Learning rate	$3\times 10^{-4}$

TABLE I : Hyperparameters

Offline-To-Online RL

Offline-to-online reinforcement learning adapts policies learned from fixed datasets through further interaction with the environment.

See all 68 papers

RAD: Retrieval High-quality Demonstrations to Enhance Decision-making
Jul 21, 2025Lu Guo, Yixiang Shan, Zhengbang Zhu +5Offline RL Generalization in RL
School of Artificial Intelligence, Jilin University, Changchun, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning
Oct 20, 2026Shixuan Liu, Cheng Tang, Yuzheng Hu +3Training Data Attribution Influence Functions
University of Illinois Urbana-Champaign
The Three Regimes of Offline-to-Online Reinforcement Learning
Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline RL RL Fine-Tuning
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair

Jul 21, 2025J/K move · Enter open · S save

RAD: Retrieval High-quality Demonstrations to Enhance Decision-making

Lu Guo, Yixiang Shan, Zhengbang Zhu, Qifan Liang+4

School of Artificial Intelligence, Jilin University, Changchun, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China

Offline RL Generalization in RL

Open paper AskPDF arXiv

Offline reinforcement learning (RL) learns policies from fixed datasets, thereby avoiding costly or unsafe environment interactions. However, its reliance on finite static datasets inherently restricts the ability to generalize beyond the training distribution. Prior solutions based on synthetic data augmentation often fail to generalize to unseen scenarios in the (augmented) dataset. To address these challenges, we propose Retrieval High-quAlity Demonstrations (RAD) for decision-making, which innovatively introduces a retrieval mechanism into offline RL. Specifically, RAD retrieves high-return and reachable states from the offline dataset as target states, and leverages a generative model to generate sub-trajectories conditioned on these targets for planning. Since the targets are high-return states, once the agent reaches such a target, it can continue to obtain high returns by following the associated high-return actions, thereby improving policy generalization. Extensive experiments confirm that RAD achieves competitive or superior performance compared to baselines across diverse benchmarks, validating its effectiveness. Our code is available at https://anonymous.4open.science/r/RAD_0925_1-690E.

Abstract

Offline-To-Online RL

Explore similar work

RAD: Retrieval High-quality Demonstrations to Enhance Decision-making

RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning

The Three Regimes of Offline-to-Online Reinforcement Learning