Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences
Organizations: Walker Department of Mechanical Engineering, The University of Texas at Austin, Austin, TX, USA · Independent researcher.
Abstract
Robots increasingly plan by searching against a reward model learned from human preferences. We measure what this search does to the true objective. On HalfCheetah and Hopper with crowd-sourced labels, and on Meta-World drawer-open and plate-slide with synthetic and blind human labels, harder MPPI search cuts true return by up to 76% while held-out label accuracy stays at 0.75-0.97; with human labels the drawer opens in 0 of 20 runs, against 10 of 10 for the cloned policy the planner starts from. The effect persists with a learned dynamics model and an image scorer. We prove that -regularized MPPI at a temperature set by the scorer's error spread on the proposals is worst-case optimal, and that no gain can be certified once that spread exceeds the scorer's own. Calibrated from 100 ranked proposal pairs, this rule is the safest fix we test, keeping 86-94% of the cloned policy's return on drawer-open and plate-slide.
Figures & tables
| Cheetah | Hopper | Drawer 500 | Drawer human | Plate 500 | |
| Cloned policy, run directly | 412 114 | 502 32 | 898 1 (10/10) | 898 1 (10/10) | 1111 5 (10/10) |
| Uniform averaging (no search) | 233 78 | 418 38 | 852 1 (10/10) | 852 1 (10/10) | 1051 6 (10/10) |
| Perfect scorer, | 980 21 † | 396 26 † | 1090 7 (10/10) | 1090 7 (10/10) | 1139 7 (10/10) |
| Learned scorer, gentle ( ) | 480 38 † | 468 31 † | 707 41 † (18/20) | 851 4 † (20/20) | 950 21 † (20/20) |
| Learned scorer, hard search ( ) | |||||
| Standard (exp. weights) | 262 25 † | 356 39 † | 504 74 † (11/20) | 216 10 † (0/20) | 436 65 † (15/20) |
Reward Hacking
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
See all 202 papers