cs.RONov 6, 2026

Goodhart in the Planner: Sampling-Based Control Exploits Rewards Learned from Human Preferences

Authors: Soovadeep Bakshi, Apurba Bose

Organizations: Walker Department of Mechanical Engineering, The University of Texas at Austin, Austin, TX, USA · Independent researcher.

Abstract

Robots increasingly plan by searching against a reward model learned from human preferences. We measure what this search does to the true objective. On HalfCheetah and Hopper with crowd-sourced labels, and on Meta-World drawer-open and plate-slide with synthetic and blind human labels, harder MPPI search cuts true return by up to 76% while held-out label accuracy stays at 0.75-0.97; with human labels the drawer opens in 0 of 20 runs, against 10 of 10 for the cloned policy the planner starts from. The effect persists with a learned dynamics model and an image scorer. We prove that χ2χ^2-regularized MPPI at a temperature set by the scorer's error spread on the proposals is worst-case optimal, and that no gain can be certified once that spread exceeds the scorer's own. Calibrated from 100 ranked proposal pairs, this rule is the safest fix we test, keeping 86-94% of the cloned policy's return on drawer-open and plate-slide.

Figures & tables

Explore similar work

CardsList
  1. Freeform Preference Learning for Robotic Manipulation

    Jun 30, 2026Marcel Torne, Anubha Mahajan, Abhijnya Bhat +1Long-Horizon Robotic ManipulationRobot Skill Learning

  2. UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning

    Jun 17, 2026Mohamed Nabail, Leo Kaixuan Cheng, Jingmin Wang +1Preference-Based RLModel-Based RL