In Human-Robot Interaction, the standard approach to learn a reward model that represents human preferences for robot behavior consists of three steps. First, the robot collects limited direct evidence from human feedback (e.g., positive or negative binary feedback). Then, the robot utilizes the direct evidence to derive accepted or rejected labels to feasible but unchosen actions using fixed implication rules. Finally, the robot updates the reward model with both the direct and derived evidence. Unfortunately, the fixed rule can hinder preference learning: in a user study with two collaborative simulation environments, human-provided implication labels often differed from the standard fixed rule, and using the human labels substantially improved reward learning with the Preference Learning from Implicit and Explicit Feedback (PIE) algorithm. Consequently, we propose IMPLIED, an implication modeling method that treats fixed-rule implications as an initial guide while learning to infer and revise accepted and rejected action labels over time. Across evaluations on recorded human-robot interaction trajectories and a physical robot pizza-making study, IMPLIED predicts human implications more accurately than the fixed rule approach and LLM baselines, approaching the performance of a human-label oracle. In turn, IMPLIED reduces preference-estimation error and leads to robot actions that are more often rational with respect to a combined reward (which includes the true preference reward and a task-specific reward) compared to baselines. By learning to reason about the implications of human feedback, this work enables more faithful and efficient robot behavior adaptation during human-robot collaboration.
Figures & tables
Figure 1: (a) In a collaboration, person provides direct evidence, rejecting the robot’s executed action. (b) Typical implication applies fixed-rule to label other feasible actions given the direct evidence. (c) Our approach, IMPLIED, uses direct evidence overtime to revise implication labels for unchosen actions.
Figure 2: Preference-learning performance, with mean and std. error across participants. Human implications improved learning performance. The dashed line indicates the earliest task-completion step per task.
Figure 3: F1 score of classifying unchosen actions over time.
Figure 4: Collaboration setup.
Implication
L2 Error ↓
Conflict Prop. ↓
IMPLIED
0.86±0.07
0.16±0.03
Fixed-Rule
1.14±0.03
0.25±0.03
Table 2: Online learning performance for the physical human-robot collaboration study.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: MuJoCo tasks. For both tasks, a human controls an agent (shown on the left side of both images). The other agent (the robot) is autonomous. It estimates preferences and acts according to them during our user study. These MuJoCo tasks were implemented by adapting and extending RoCo simulation components [ 38 ] .
Figure 6: Graphical user interface used in the MuJoCo simulation study. At each timestep, participants (a) selected their own action, (b) provided explicit feedback on the robot’s action, (c) labeled implications for the robot’s unchosen actions, and (d) labeled implications for their own unchosen actions.
Parameter
Value
Description
learning_rate
1×10−4
Step size for preference-weight updates
weight_samples
10
Number of samples representing the belief over weight
max_iteration
1000
Maximum iterations for each preference update
convergence_threshold
1×10−3
Threshold for preference update convergence
βR
1.0
Robot action rationality coefficient
βexp
1.0
Explicit-feedback rationality coefficient
Appendix
Table 3: Preference-learning parameters for user study in MuJoCo simulation.
Task
τ
F1 ↑
L2 Error ↓
Conflict Prop. ↓
Sandwich-making
0.25
0.77±0.03
0.84±0.06
0.18±0.04
0.50
0.84±0.01
0.71±0.06
0.13±0.01
0.75
0.76±0.02
0.80±0.07
0.17±0.03
Block-sorting
0.25
0.72±0.04
0.56±0.08
0.16±0.05
0.50
0.87±0.01
0.42±0.07
0.12±0.03
0.75
0.74±0.01
0.49±0.07
0.16±0.04
Appendix
Table 4: Sensitivity analysis for the threshold τ .
Figure 7: F1 score for classifying unchosen robot actions over time with explicit feedback only. See first column of Tab. 5 for legend.
Figure 8: F1 score for classifying unchosen human actions over time using implicit feedback only . See first column of Tab. 6 for legend.
Figure 9: Layout of the tabletop pizza-making station in the physical human-robot collaboration study.
Reinforcement learning (RL) policies optimized for task rewards may produce behaviors that are efficient but misaligned with human expectations, safety preferences, or intent. This paper presents a reason-guided preference-based reinforcement learning (PbRL) framework that learns a human reward model from trajectory-level feedback. In addition to preference labels, the framework incorporates reason tags explaining why a behavior is preferred, enabling the learned reward to capture both human choices and their underlying rationale. Compared with preference-only reward learning, the reason-guided formulation provides additional supervision for learning human-aligned rewards and improves the interpretability of the resulting policy. The learned human reward is combined with the environment reward during policy optimization, allowing the agent to improve task performance while producing more interpretable, human- aligned, and preference-sensitive behavior
Anshul Nayak, Shahil Shaik, Jonathan Smereka +1
Clemson University, Clemson, SC 29631 USA · U.S. Army DEVCOM Ground Vehicle Systems Center, Warren, MI 48092 USA
Comparative feedback, asking people which of two behaviors they prefer, has become a standard way to align robot and agent behavior with human intent when the reward itself cannot be specified directly. Preference-based reward learning typically casts the human teacher as a passive oracle answering learner-generated queries. We argue this forfeits the teacher's defining advantage: knowledge of the objective. A teacher who knows the target can construct training examples more efficiently than any learner-driven acquisition strategy, an advantage that widens as the reward's feature dimension grows. However, exploiting this advantage requires an accurate model of what the learner currently knows. We therefore recast preference learning as a human-autonomy team problem coupling two behavioral models: the teacher maintains a model of the learner to design an informative curriculum, and the learner maintains a second-order model of the teacher's model, emitting structured preference constraints (understanding statements) that keep the teacher's model of the learner synchronized. In simulation, an informed teacher outperforms learner-led selection; teacher-model drift under alternating teachers erodes this advantage; and understanding statements repair it, with second-order (ToM-2) statements outperforming mean-belief statements when the teacher's error about the learner is concentrated in a particular direction rather than spread evenly.
Reward learning typically assumes the human's input covers everything that matters for the task. In practice, humans can only attend to so much at once, so their teaching naturally captures some features---or aspects of the task---better than others. Prior work has shown how the robot can identify which features were poorly supervised directly from its teaching data. But the current solution is to ask the human to try again, putting the burden on someone who already struggled to convey those features. We propose shifting this burden to the robot. Our key insight is that the same information that tells the robot what it doesn't know can help it ask better questions. Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering. We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features. In simulation, the informed prior beats the standard active preference learning baseline by up to 30.8% in the low-query regime, where reducing human effort matters most. In a user study, the informed prior and explanations each significantly improve reward recovery and amplify each other.