Teaching Agents the "Principle", Not the "Reward": Prescriptive Policy Learning
Organizations: National University of Singapore
Abstract
Reinforcement learning has achieved strong performance in many tasks by letting agents learn from nothing more than a reward signal. However, in more complex tasks, agents still struggle to learn good, generalizable policies, even with carefully designed rewards and abundant samples. We argue that this difficulty stems from the reward decoding burden: designers encode their knowledge of the desired behavior into rewards and penalties, and agents must infer this knowledge from long sequences of noisy scalars. To address this burden, we propose prescriptive policy learning (PPL), which directly teaches agents higher-level knowledge about the tasks, termed principles of behavior, such as how to maintain balance, how to accumulate energy, and how to land smoothly. Through appropriately selected key quantities and references, PPL can directly indicate at every step whether the agent follows the principles and to what extent. Our experiments show that agents can learn good, generalizable policies without any reward. Taught only simple heuristic principles, PPL agents outperform reward-based methods with far fewer samples. Their policies also transfer directly to harder settings and keep performing well, demonstrating their strong generalization ability. In summary, PPL provides a new approach in which rough knowledge of the desired behavior is enough for agents to learn good, generalizable policies, even when the system dynamics cannot be accurately modeled.
Figures & tables
| Task | Principle | Key quantity | Reference |
|---|---|---|---|
| Obstacle avoidance | Stay as far from the barriers as possible. | , the vertical velocity | , the vertical velocity that a conservative deceleration brings to zero at the height of the gap center |
| Lunar Lander | Stay upright, approach the pad, and avoid descending too fast. | , the vertical, angular, and horizontal velocities | , , and , computed from the height above the pad, the tilt, and the horizontal position and velocity |
| Reacher | Turn each joint toward the configuration that places the fingertip on the target. | , the joint angular velocities | , computed from the angle that joint has left to turn |
| Success (%) | Steps to ( ) | Clearance | ||||
|---|---|---|---|---|---|---|
| Method | Moderate | Strong | Moderate | Strong | Moderate | Strong |
| PPL | 4.24 | 4.18 | 0.166 | 0.155 | ||
| PPO | 10.0 | 10.0 | 0.112 | 0.107 | ||
| DQN | 75.0 | 85.0 | 0.097 | 0.087 | ||
| PPO-R | 20.0 | 20.0 | 0.053 | 0.048 | ||
| Success (%) | Steps to ( ) | |||
| Method | Without wind | With wind | Without wind | With wind |
| PPL | 19.7 | 29.3 | ||
| PPO | 520 | 790 | ||
| DQN | 135 | 390 | ||
| Discrete SAC | – | – | ||
| PPO-R | – | – | ||
| Method | Standard | Coupled joints | Gravity | Gravity and disturbances |
|---|---|---|---|---|
| PPL | ||||
| PPO | ||||
| SAC | ||||
| PPO-R | ||||
| Principle controller, best gain |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Setting | Obstacle avoidance | Lunar Lander | Reacher |
|---|---|---|---|
| Policy | Categorical, 3 actions | Categorical, 4 actions | Gaussian, 2 torques |
| Policy input | Raw observation | Raw observation | Normalized observation |
| Baseline input | State | State | State, other torque, error |
| Trajectories per batch | |||
| Epochs per batch | |||
| Minibatch size |
Generalization in RL
Reinforcement learning generalization studies how agents transfer learned behavior to unseen tasks, environments, dynamics, and observations.
See all 97 papers