Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
Authors: Dwait Bhatt, Shih-Chieh Chou, Nikolay Atanasov
Organizations: Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA, USA · Department of Mechanical and Electro-Mechanical Engineering, National Sun Yat-sen University, Taiwan
Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.
Figures & tables
Figure 1
Fig. 3 : Comparison of SAC vs TD3 success rates for Strategy A and C approaches.
Fig. 4 : Comparing Sample Efficiency Improvement (SEI) scores for single strategy approaches.
Fig. 5 : Comparison of Sample Efficiency Improvement (SEI) scores for the top ten hybrid variants across different strategy combinations.
Fig. 6 : Comparison of Sample Efficiency Improvement scores for the best performing variant of each strategy combination.
Fig. 7 : Distribution of normalized impact scores for individual algorithmic components across all task and robot configurations. Scores are independently normalized to [−1,1] per configuration. Positive values indicate that adding the component improved sample efficiency compared to matched baselines, while negative values indicate a degradation. The spread of the whiskers illustrates each component’s sensitivity to different task difficulties.
Fig. 8 : Normalized impact of individual components as a function of task difficulty. Tasks are ranked from easiest to hardest, defined by the number of steps required for SAC to reach peak success (values shown in parentheses below each task). Results are grouped by strategy to highlight how the utility of each algorithmic choice shifts in more complex settings.
Fig. 9 : Comparison of Sample Efficiency Improvement (SEI) scores for all methods, aggregated across the four evaluated task-robot configurations. Higher SEI scores indicate faster learning speed and final performance relative to the SAC baseline (SEI = 0). Component abbreviations are defined as follows: demos/rollouts indicates the data source; prefill/auxbc denotes the Strategy A direct data usage method; bc/mcq/calql/cqlrho/cqlh specifies the Strategy B pretraining method; and resrl/ibrl/cheq represents the Strategy C action mixing mechanism.
Offline reinforcement learning (RL) learns policies from fixed datasets, thereby avoiding costly or unsafe environment interactions. However, its reliance on finite static datasets inherently restricts the ability to generalize beyond the training distribution. Prior solutions based on synthetic data augmentation often fail to generalize to unseen scenarios in the (augmented) dataset. To address these challenges, we propose Retrieval High-quAlity Demonstrations (RAD) for decision-making, which innovatively introduces a retrieval mechanism into offline RL. Specifically, RAD retrieves high-return and reachable states from the offline dataset as target states, and leverages a generative model to generate sub-trajectories conditioned on these targets for planning. Since the targets are high-return states, once the agent reaches such a target, it can continue to obtain high returns by following the associated high-return actions, thereby improving policy generalization. Extensive experiments confirm that RAD achieves competitive or superior performance compared to baselines across diverse benchmarks, validating its effectiveness. Our code is available at https://anonymous.4open.science/r/RAD_0925_1-690E.
Lu Guo, Yixiang Shan, Zhengbang Zhu +5
School of Artificial Intelligence, Jilin University, Changchun, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
Online reinforcement learning (RL) has achieved significant success in decision-making tasks, but modern online RL remains highly sensitive to the quality of training experience. Understanding the contribution of individual training instances is therefore central to data-centric RL, with applications ranging from experience selection to policy interpretation. Existing data attribution methods for online RL primarily focus on local influence within a single training round, overlooking the non-local data influence across multiple training rounds. In this work, we formalize data attribution in online RL through a trajectory-level leave-one-out quantity, RL-LOO, which measures how one training sample influences a downstream target after subsequent online updates have taken place. We then derive RL-IF, a first-order estimator that propagates data influence through both optimization effects and policy-induced sampling effects, and provide theoretical guarantees on its approximation accuracy under smoothness assumptions. Our analysis further disentangles these two effects and shows both theoretically and empirically that the full RL-IF estimator can often be well approximated using the optimization effect alone. Building on this observation, we develop RINSE, a practical data filtering method for online RL. Experiments on standard control benchmarks and an RLHF-style toxicity-mitigation setting show that RINSE improves training efficiency and final performance over standard PPO and local attribution baselines. These results suggest that RL-IF provides a practical and principled framework for understanding and improving online RL.
Offline-to-online reinforcement learning (RL) has emerged as a practical paradigm that leverages offline datasets for pretraining and online interactions for fine-tuning. However, its empirical behavior is highly inconsistent: design choices of online fine-tuning that work well in one setting can fail completely in another. Guided by the stability--plasticity principle, we propose a framework that can explain this inconsistency: We argue that efficient fine-tuning must preserve the utility of the stronger offline prior, whether that is the pretrained policy or the offline dataset, while maintaining sufficient plasticity. This perspective identifies three regimes of online fine-tuning, each requiring distinct stability properties. We validate this framework through a large-scale empirical study, finding that the results strongly align with its predictions in 45 out of 63 cases, with only 3 opposite mismatches. This work provides a framework for guiding design choices in offline-to-online RL based on the relative performance of the offline dataset and the pretrained policy.
Lu Li, Tianwei Ni, Yihao Sun +1
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair