Reward learning typically assumes the human's input covers everything that matters for the task. In practice, humans can only attend to so much at once, so their teaching naturally captures some features---or aspects of the task---better than others. Prior work has shown how the robot can identify which features were poorly supervised directly from its teaching data. But the current solution is to ask the human to try again, putting the burden on someone who already struggled to convey those features. We propose shifting this burden to the robot. Our key insight is that the same information that tells the robot what it doesn't know can help it ask better questions. Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering. We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features. In simulation, the informed prior beats the standard active preference learning baseline by up to 30.8% in the low-query regime, where reducing human effort matters most. In a user study, the informed prior and explanations each significantly improve reward recovery and amplify each other.
Figures & tables
Figure 1: A user teaches a robot to carry a mug of coffee but pays little attention to how close it gets to the human. Using prior work [ 3 ] , the robot diagnoses from the demonstrations D which features were well-supervised (green), underspecified (red), or irrelevant (gray), and uses this partition o in two ways. (1) The robot shapes its belief P(w∣D) (true weights marked by ▲ ) into an informed prior P(w∣o) : the robot stays confident about the laptop, doesn’t commit to anything about the human, and assumes the apple doesn’t matter. (2) The robot tells the user what to focus on. Since the robot is only unsure about distance to human, it asks about two trajectories that differ in exactly that, and each answer sharpens P(w∣P,o) toward what it still needs to learn.
Figure 2: (a) Stove : the robot carries a utensil near a stove and a human. (b) Coffee : the robot carries a coffee cup near a laptop and a human. (c) A Franka Emika FR3 arm transports a knife in a kitchen.
Figure 3: Average normalized reward versus number of preference queries in Stove (a, b) and Coffee (c, d), with one (a, c) or two (b, d) features initially underspecified. Results are averaged over all four choices of the underspecified feature (a, c) and all six pairs of underspecified features (b, d). Curves show the mean and shaded bands ±1 SE across seeds. Left panels compare methods without explanations; right panels compare explanation variants alongside their base methods.
Figure 4: User study (a) normalized reward and (b) Likert score per condition. (c) Robot study mean normalized reward at K=10 by feature initially underspecified. Error bars are ±1 SE.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Normalized reward with (a, c) one underspecified feature and (b, d) two underspecified features for the Stove (a, b) and Coffee (c, d) environments. Across conditions, the informed prior recovers the true reward faster than the baselines, with the largest gains in the low-query regime.
Scale
Questions
Cronbach’s α
F
p
Ease
I found it easy to choose between the two trajectories in each query.
0.94
15.31
0.002
The queries were difficult to answer.
Under.
I understood what each query was trying to learn.
0.83
6.66
0.026
The queries seemed random or unmotivated.
Expr.
The queries covered important parts of the task.
0.75
2.96
0.113
Queries were wasted on things that didn’t matter.
Appendix
Table 1: ANOVA results for the main effect of explanation on Likert scales.
Figure 6: An example preference query from the user study in the Coffee environment: participants are shown two trajectories side by side and select the one they prefer.
Figure 7: The simulated model of the kitchen used to generate the trajectory bank for the real robot demos. A box is used as a visual placeholder for the cat in the actual environment.
Figure 8: The three highest-ranked test trajectories executed on the Franka Emika FR3 arm after K=10 preference queries, for Informed Prior and Standard when cat (top two rows) or stove (bottom two rows) was initially underspecified. Each panel traces the path of the knife in orange.
Learning reward functions from demonstrations assumes that demonstrations provide adequate supervision over all features -- or task-relevant aspects of behavior. In practice, demonstrations are often imperfect: humans may under-emphasize certain features due to cognitive load or physical difficulty, or the training regime may fail to sufficiently cover all relevant situations. In either case, important features may be underspecified, leading to ambiguity in the learned reward function and misaligned behavior at deployment. We propose a framework that detects such underspecified features and actively solicits targeted corrective demonstrations. Our key insight is that demonstrations implicitly reveal which features are well specified: features that are consistently optimized show little variation across demonstrations, while features that are underspecified vary widely. We leverage this statistical signal to infer which features may have been insufficiently demonstrated. The robot then explains which features it is uncertain about in natural language and queries for demonstrations that explicitly address the identified gaps. We evaluate our approach in a simulated tabletop manipulation domain and in a user study with a real Franka robot. Targeted, explanation-guided queries significantly improve reward recovery compared to random querying and passive data collection, reducing ambiguity that would otherwise persist in learning from imperfect demonstrations.
Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/
Comparative feedback, asking people which of two behaviors they prefer, has become a standard way to align robot and agent behavior with human intent when the reward itself cannot be specified directly. Preference-based reward learning typically casts the human teacher as a passive oracle answering learner-generated queries. We argue this forfeits the teacher's defining advantage: knowledge of the objective. A teacher who knows the target can construct training examples more efficiently than any learner-driven acquisition strategy, an advantage that widens as the reward's feature dimension grows. However, exploiting this advantage requires an accurate model of what the learner currently knows. We therefore recast preference learning as a human-autonomy team problem coupling two behavioral models: the teacher maintains a model of the learner to design an informative curriculum, and the learner maintains a second-order model of the teacher's model, emitting structured preference constraints (understanding statements) that keep the teacher's model of the learner synchronized. In simulation, an informed teacher outperforms learner-led selection; teacher-model drift under alternating teachers erodes this advantage; and understanding statements repair it, with second-order (ToM-2) statements outperforming mean-belief statements when the teacher's error about the learner is concentrated in a particular direction rather than spread evenly.