Sample-Efficient Ensemble Reinforcement Learning at Scale
Organizations: Department of Computer Science, University of California, Irvine
Abstract
Ensemble methods such as MeanQ have been shown to improve the sample efficiency of relatively small model-free deep reinforcement learning (RL) architectures, including DQN and similarly sized variants such as Rainbow. However, to our knowledge, the effects of ensembling on large-scale architectures such as Bigger Better Faster (BBF) have not been studied. In this work, we present MeanQ-BBF, a hybrid model-free algorithm that combines the principles of MeanQ and BBF. Through statistically rigorous evaluation on the Atari 100K benchmark, we show that in contrast to naive scaling of the network size of BBF, which degrades performance, MeanQ-BBF is able to leverage the increase in model capacity to improve performance over BBF. Moreover, MeanQ-BBF does this in a computationally efficient way by organizing the added parameters into largely independent learners. Finally, we discuss the performance-cost tradeoff in the context of practical sample-efficient RL.
Figures & tables
| Technique | MeanQ | BBF | MeanQ-BBF |
|---|---|---|---|
| Ensemble-mean targets | |||
| Shared exploration policy | N/A | ||
| Replay buffer | |||
| Replay sampling priorities | |||
| Lagging target network | |||
| Large networks |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Environment | Random | Human | MeanQ | BBF | Scaled BBF | MeanQ-BBF | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| Alien | 184.8 | 7128.0 | 864.8 | (309.7) | 1092.4 | (93.0) | 1088.1 | (97.1) | 1204.5 | (221.0) |
| Amidar | 11.8 | 1720.0 | 112.5 | (28.3) | 214.1 | (35.6) | 256.7 | (46.6) | 291.9 | (58.3) |
| Assault | 233.7 | 742.0 | 894.4 | (120.7) | 1891.5 | (270.9) | 2141.5 | (297.9) | 2393.7 | (207.1) |
| Asterix | 248.8 | 8503.0 | 971.5 | (246.7) | 3023.0 | (766.3) | 1950.4 | (105.2) | 2872.4 | (615.5) |
| BankHeist | 15.0 | 753.0 | 215.6 | (282.4) | 590.7 | (342.1) | 864.0 | (322.9) | 743.9 | (413.8) |
| BattleZone | 2895.0 | 37188.0 | 13980.0 | (3225.0) | 19242.0 | (4506.0) | 22630.0 | (3769.0) | 25806.0 | (3733.1) |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,852 papers