Robots increasingly plan by searching against a reward model learned from human preferences. We measure what this search does to the true objective. On HalfCheetah and Hopper with crowd-sourced labels, and on Meta-World drawer-open and plate-slide with synthetic and blind human labels, harder MPPI search cuts true return by up to 76% while held-out label accuracy stays at 0.75-0.97; with human labels the drawer opens in 0 of 20 runs, against 10 of 10 for the cloned policy the planner starts from. The effect persists with a learned dynamics model and an image scorer. We prove that χ2-regularized MPPI at a temperature set by the scorer's error spread on the proposals is worst-case optimal, and that no gain can be certified once that spread exceeds the scorer's own. Calibrated from 100 ranked proposal pairs, this rule is the safest fix we test, keeping 86-94% of the cloned policy's return on drawer-open and plate-slide.
Figures & tables
Fig. 1: True return against MPPI temperature (harder search to the right), mean ±1 standard error over 10–20 runs. A learned scorer loses return under harder search on every task except window-open; the perfect scorer keeps or improves its return except on hopper; 300 human labels are exploited fastest.
Fig. 2: One drawer start under three planners (true returns). Uniform averaging opens the drawer. Standard MPPI ( λ=0.01 ) on the 500-label scorer lifts the hand into a part of the workspace the data never visited; on the human-label scorer ( λ=0.1 ) it pulls in front of the handle without touching it.
Fig. 3: (a) True return gained over uniform averaging, as a fraction of the perfect scorer’s best return, against the distance d=KL (arm tasks: 500 labels). Circles: exponential weights; squares: χ2 weights; one point per temperature; curves: fits of ad−bd2 ; open diamonds: perfect scorer ( ±1 standard error). (b) The drawer for five label budgets and the human labels. (c) Best-of- N on the drawer (solid) and plate-slide (dashed), relative to the clean cloned action (dash-dot); N=1 (open markers) is one noisy action.
Fig. 4: (a) Drawer: true return against label budget at gentle and hard search; green: 500 labels, half on the planner’s own trajectories (Section V-D ). (b) Change in the scorer’s estimate of the executed episode from λ=3 to λ=0.01 , against the ratio of true returns; diamonds: human labels. (c) Each fix at λ=0.01 , relative to standard MPPI at λ=3 .
Cheetah
Hopper
Drawer 500
Drawer human
Plate 500
Cloned policy, run directly
412 ± 114
502 ± 32
898 ± 1 (10/10)
898 ± 1 (10/10)
1111 ± 5 (10/10)
Uniform averaging (no search)
233 ± 78
418 ± 38
852 ± 1 (10/10)
852 ± 1 (10/10)
1051 ± 6 (10/10)
Perfect scorer, λ=0.01
980 ± 21 †
396 ± 26 †
1090 ± 7 (10/10)
1090 ± 7 (10/10)
1139 ± 7 (10/10)
Learned scorer, gentle ( λ=3 )
480 ± 38 †
468 ± 31 †
707 ± 41 † (18/20)
851 ± 4 † (20/20)
950 ± 21 † (20/20)
Learned scorer, hard search ( λ=0.01 )
Standard (exp. weights)
262 ± 25 †
356 ± 39 †
504 ± 74 † (11/20)
216 ± 10 † (0/20)
436 ± 65 † (15/20)
TABLE I: True return at hard search, mean ± standard error over 10 runs per cell ( † 20); successes over runs in parentheses on the arm. ‡ Mean ± standard deviation across five independent 100-pair calibration sets ranked by an oracle, drawn on start states the evaluation never uses, each calibrating its own controller. Bold: best learned-scorer method per column.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
Planning with learned world models combines online trajectory optimization with learned value and policy functions for high-dimensional control. Because the planner determines the experience used for learning, while the learned critic and actor in turn score and propose future plans, planning and learning form a closed feedback loop. TD-MPC is a prominent instance of this design. Recent policy-constrained variants strengthen one part of the loop by aligning the learned policy with planner behavior. We introduce PL-MPC (Planning-Learning MPC), which additionally modifies critic supervision and planner terminal-value estimation. Hybrid multi-step TD targets expose critic updates to more realized rewards before bootstrapping; disagreement-aware terminal estimates reduce the influence of uncertain critic values during MPPI planning; and return-weighted actor distillation emphasizes planner-executed actions from high-return episodes. The world-model architecture and MPPI optimizer are otherwise unchanged. On HumanoidBench, the largest gains occur on balance-hard, where Total Average Return (TAR) increases from 98±18 to 387±255, and hurdle, from 199±13 to 466±200; performance across the broader benchmark remains task dependent, and PL-MPC remains competitive on DMControl. Controlled ablations show different component interactions across the two tasks. We further demonstrate zero-shot sim-to-real transfer on wrench-nut alignment with a 7-DoF KUKA IIWA14, obtaining higher observed success than TD-M(PC)^2 on the training object size and two unseen sizes. Code and data will be available at: https://pl-mpc-humanoid.github.io.
Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/
Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning. We introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics, and value functions. Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories according to a unified score that combines expected reward, terminal value, and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics. Under standard regularity assumptions, we establish sublinear regret guarantees for both finite-horizon and infinite-horizon settings. Empirically, experiments on the Meta-World benchmark show UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.
Mohamed Nabail, Leo Kaixuan Cheng, Jingmin Wang +1
Learning, Embodied Autonomy, and Forecasting (LEAF) Lab, University of Toronto