The Blindfolded Controller: Decision Model-Guided Training for Reinforcement Learning
Organizations: Coventry University, Coventry, UK
Abstract
Reinforcement learning (RL) provides a framework for continuous improvement through interaction with an environment. However, RL agents struggle against tasks that offer exploration challenges and sparse rewards, because learning efficient policies from scratch is slow and sample-inefficient. In contrast, decision models (DMs), given a description of the environment and task, can perform well in sparse-reward tasks, but, as they do not learn from reward, they are not able to improve towards the optimal solution. In this paper, we present Decision Model-Guided Training (DMGT), a technique that uses a DM with a description of the environment and task to initialise an RL policy, combining the DM's ability to act without training with RL's ability to improve from reward. Instead of exploring from scratch, the agent reaches the goal and receives reward from its first episodes, and RL then improves its policy beyond what the DM achieves. We demonstrate through experiments on seven tasks that our approach outperforms RL in all seven and DM in five. In addition, we provide comprehensive guidelines, derived from ablation studies, for describing an environment and task to a DM.
Figures & tables
| PPO from | DM | DMGT | DMGT | |
|---|---|---|---|---|
| Environment | scratch | alone | (BC) | (DAgger) |
| LunarLander-v3 | 185.3 86.1 | 209.0 125.8 | 263.5 26.6 | 269.3 18.1 |
| MountainCarContinuous-v0 | 0.0 0.0 | 88.0 1.0 | 93.3 0.7 | 93.2 0.6 |
| highway-fast-v0 | 21.7 0.4 | 22.7 2.2 | 21.5 4.2 | 22.9 0.5 |
| parking-v0 | 16.9 0.9 | 19.4 9.3 | 15.6 5.7 | 26.2 40.7 |
| MiniGrid-MultiRoom-N4-S5-v1 | 0.000 0.000 | 0.763 0.084 | 0.679 0.253 | 0.727 0.186 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Environment | Observation | Actions | Reward | Episode |
|---|---|---|---|---|
| Lunar Lander | 8 values: position, velocity, tilt, turning rate and the contact of each leg | 4: no engine, or the left, main or right engine; the DM chooses among the same four | shaped towards coming to rest on the pad: for coming to rest, for a crash, for each leg on the ground, small costs for firing the engines | 1000 steps, 50 per second |
| Mountain Car | 2 values: position and velocity | a force in ; the DM’s three options apply , 0 or | at the flag, per step for the force | 999 steps |
| Highway-fast | : presence, position and velocity of the car and of the 4 nearest cars | 5: change lane left, keep going, change lane right, faster, slower; the DM chooses among the same five | up to 1 per second for speed between 20 and 30 m/s and for the right lane; a collision ends the drive | 30 decisions, one per second |
| Parking | 18 values: position, velocity and heading of the car and of its space | acceleration and steering in ; the DM’s seven options (forward or backward with the wheels left, straight or right, or stop) set these controls | minus the square root of the weighted distance to the space at every step, and for hitting a wall; parked when the reward is above | 100 s, five decisions per second |
| MultiRoom N4-S5 and N6 | the image of the agent’s view | MiniGrid’s 7 actions; the DM chooses among the same seven | on reaching the goal after steps, 0 otherwise | or 120 steps |
| Skiing | object features read off the screen | 3: left, none, right; the DM chooses among the same three | minus the time on the game clock, with 5 s added for each of the 20 gates missed | until the finish, or 5 minutes on the clock |
| Hyperparameter | Value |
|---|---|
| Policy and value networks | separate, two hidden layers of 64 units, tanh |
| Environments steps per update | |
| Minibatch size | 64 |
| Epochs per update | 10 |
| Learning rate | |
| Discount factor | 0.99 |
| Environment | Demonstrations | DAgger |
|---|---|---|
| Lunar Lander | 1 (357 decisions) | (2909 labels) |
| Mountain Car | 8 (903) | (454–457) |
| Highway-fast | 100 (2828) | (1390) |
| Parking | 100 (4559) | (3140) |
| MultiRoom N4-S5 | 100 (2075) | (5360) |
| MultiRoom N6 | 100 (5292) | (19802) |
| Guideline | Evidence |
|---|---|
| 1. The goal | Highway-fast: three rewordings of the goal change the return by at most 0.1. |
| 2. Accurate action descriptions | Mountain Car: names without descriptions, no run reaches the flag (0 of 2), pushing right on all 300 steps. Neutral names without descriptions: no episode parks in Parking (0 of 6), and MultiRoom N4-S5 needs 23.8 more steps per episode. |
| 3. Actions explained once | Mountain Car: the description of the environment also said that the car can be pushed left, pushed right or not pushed. Without that sentence, so that only the options explain the actions, the car still reaches the flag (in 112 steps instead of 110), and the shortest query leaves the sentence out. |
| 4. Facts the DM cannot guess | Highway-fast (return out of 30): the facts matter as a group. Removed one at a time, each of the six lowers the return by at most 1.3, with no collision; removed all together, they lower it by 2.8, and the car collides in 1 of 3 episodes. A false fact, that the car brakes on its own, lowers the return by 1.1. |
| 5. The agent’s point of view | MultiRoom N4-S5 : with a shorter query, without the view and what is in front, 9 instead of 19 of 20 episodes reach the goal. |
| 6. No repetition | Highway-fast: every value only as a number or only in words changes the return by at most 0.4. Without both copies of the other cars, a shortened query collides in 2 of 3 episodes. |
| PPO time | DM calls per | DMGT calls, | Time per | |
|---|---|---|---|---|
| Environment | per run | test episode | BC; DAgger | DM call |
| Lunar Lander | 18 min | 348 | 480; 5143 | 0.38 s |
| Mountain Car | 2–4 min | 120 | 903; 1357–1360 | – |
| Highway-fast | 4.1–4.2 h | 29 | 2866; 4256 | 0.61 s |
| Parking | 2.3–2.4 h | 54 | 5076; 8475 | 0.54 s |
| MultiRoom N4-S5 | 43–44 min | 21 | 2075; 4172 | 1.50 s |
Reinforcement Learning
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
See all 1,852 papers