Informed Posterior Sampling: More Efficient Online Learning with Few Offline Demonstrations in Average-Reward MDPs
Organizations: University of Southern California · DeepMind
Abstract
We study efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with. We assume that the offline dataset is generated by an expert but with an unknown level of competence, i.e., it is not perfect and not necessarily using the optimal policy. We show that if the learning agent models the behavioral policy (parameterized by a competence parameter) used by the expert, it can do substantially better in terms of minimizing cumulative regret than if it does not do that. We establish an upper bound on regret of the exact informed PSRL (iPSRL) algorithm that scales as . This requires a novel prior-dependent regret analysis of Bayesian online learning algorithms for the infinite horizon setting. We then propose the informed RLSVI (iRLSVI) algorithm to efficiently approximate the iPSRL algorithm. Empirical results show that small offline expert datasets are sufficient for iRLSVI to outperform online-only algorithms on hard exploration domains.
Figures & tables
| Environment | uRLSVI | piRLSVI | iRLSVI | IMED-RL | piIMED-RL | UCRL3 | piUCRL3 | |||
| RiverSwim-6 | 6 | 2 | 3 | 11.7k | 2.2k | 180 | 80 | 59 | 565 | 549 |
| RiverSwim-25 | 25 | 2 | 3 | 38.6k | 33.5k | 108 | 24.8k | 314 | 4.6k | 4.6k |
| RiverSwim-50 | 50 | 2 | 3 | 38.6k | 38.7k | 137 | 37.9k | 1.1k | 12.4k | 11.0k |
| RiverSwim-100 | 100 | 2 | 3 | 38.5k | 38.6k | 202 | 37.9k | 4.6k | 32.6k | 33.2k |
| ClusterRing-5 | 50 | 10 | 50 | 10.4k | 3.7k | 168 | 5.4k | 1.4k | 11.7k | 12.0k |
| ClusterRing-10 | 100 | 10 | 50 | 5.0k | 3.0k | 213 | 3.5k | 2.0k | 5.0k | 5.1k |
| RiverSwim-50 | ClusterRing-10 | Grid4Room | ||||
|---|---|---|---|---|---|---|
| Expert policy | iRLSVI | piRLSVI | iRLSVI | piRLSVI | iRLSVI | piRLSVI |
| Boltzmann, , | 5,978 | 38,544 | 925 | 3,278 | 655 | 3,490 |
| Boltzmann, , scalar | 2,338 | 38,745 | 588 | 2,871 | 1,713 | 3,476 |
| Boltzmann, , | 8,993 | 38,605 | 1,219 | 3,122 | 2,946 | 3,782 |
| -greedy, | 1,793 | 38,658 | 787 | 3,881 | 1,480 | 3,880 |
| Balanced mixture | 5,051 | 39,335 | 638 | 3,517 | 1,252 | 3,733 |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| (122) | ||||
| (123) | ||||
| (124) |
| Parameter | Value |
|---|---|
| Episode length ( ) | |
| Target update interval | |
| Optimizer for iRLSVI update | L-BFGS-B (200 iters) |
| TD-error prior variance ( ) | |
| Parameter prior covariance ( ) | |
| Expert deliberateness prior rate ( ) | 0.01 |
| Expert | Quality | iRLSVI | piRLSVI | piIMED-RL |
|---|---|---|---|---|
| Boltzmann, , scalar (Table 1 ) | 93.32% | 137 79 | 38,726 240 | 1,134 107 |
| Boltzmann, , | 83.25% | 5,978 4,351 | 38,544 193 | 8,342 7,086 |
| Boltzmann, , | 71.65% | 2,338 1,287 | 38,745 245 | 1,114 108 |
| Boltzmann, , | 83.34% | 8,993 4,787 | 38,605 210 | 13,859 8,453 |
| -greedy, | 62.71% | 1,793 1,775 | 38,658 237 | 2,878 3,864 |
| Balanced mixture | 72.57% | 5,051 2,975 | 39,335 623 | 37,837 8 |
| Expert | Quality | iRLSVI | piRLSVI | piIMED-RL |
|---|---|---|---|---|
| Boltzmann, , scalar (Table 1 ) | 82.70% | 213 78 | 2,986 226 | 2,015 120 |
| Boltzmann, , | 71.19% | 925 142 | 3,278 212 | 2,060 140 |
| Boltzmann, , | 80.81% | 588 83 | 2,871 243 | 2,073 115 |
| Boltzmann, , | 70.97% | 1,219 115 | 3,122 192 | 2,060 108 |
| -greedy, | 69.80% | 787 91 | 3,881 167 | 2,078 89 |
| Balanced mixture | 73.86% | 638 94 | 3,517 212 | 2,025 93 |
| Expert | Quality | iRLSVI | piRLSVI | piIMED-RL |
|---|---|---|---|---|
| Boltzmann, , scalar (Table 1 ) | 96.84% | 161 118 | 3,228 166 | 70 10 |
| Boltzmann, , | 78.33% | 655 142 | 3,490 158 | 66 9 |
| Boltzmann, , | 81.27% | 1,713 450 | 3,476 190 | 73 10 |
| Boltzmann, , | 46.19% | 2,946 328 | 3,782 139 | 199 44 |
| -greedy, | 65.97% | 1,480 226 | 3,880 142 | 137 39 |
| Balanced mixture | 64.48% | 1,252 299 | 3,733 118 | 89 29 |
| Environment | iRLSVI | piRLSVI | RLPD-tab | CQL-FT |
|---|---|---|---|---|
| RiverSwim-25 | 108 87 | 33,455 5,067 | 3,233 731 | 7,157 3,676 |
| RiverSwim-50 | 137 79 | 38,726 240 | 3,553 791 | 10,921 4,326 |
| RiverSwim-100 | 202 82 | 38,631 202 | 10,416 3,679 | 17,728 2,673 |
| ClusterRing-10 | 213 78 | 2,986 226 | 1,771 202 | 1,318 105 |
| Grid4Room | 161 118 | 3,228 166 | 4,713 189 | 266 145 |
| RiverSwim-50 | ClusterRing-10 | Grid4Room | |
|---|---|---|---|
| 1 | 42,813 1 | 919 152 | 1,029 157 |
| 2 | 4,484 6,136 | 619 89 | 621 101 |
| 3 | 206 95 | 478 88 | 442 100 |
| 5 | 248 97 | 325 73 | 202 69 |
| 10 | 257 152 | 171 81 | 51 27 |
| RiverSwim-50 | ClusterRing-10 | Grid4Room | |
|---|---|---|---|
| 1 | 14,563 9,148 | 234 83 | 24 11 |
| 10 | 209 94 | 257 71 | 100 42 |
| 100 | 220 84 | 288 79 | 185 60 |
| 1000 | 1,455 755 | 283 81 | 301 204 |
| 136 80 | 312 77 | 265 91 |
Offline-To-Online RL
Offline-to-online reinforcement learning adapts policies learned from fixed datasets through further interaction with the environment.
See all 75 papers