Environment Edits as a Corrective Signal for Reward Learning

Isaac Ngui; Courtney McBeth; Nancy M. Amato

cs.ROOct 26, 2026

Environment Edits as a Corrective Signal for Reward Learning

Authors: Isaac Ngui, Courtney McBeth, Nancy M. Amato

Organizations: Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, United States

Abstract

Robot corrections allow people to adjust a robot's behavior, but a single correction changes many features. Also, preferences about object attributes and relationships between objects cannot be expressed by moving the robot. We propose environment edits as a complementary form of feedback. The robot's motion is held constant as the human edits the scene, so the only explanation for the edit is the moved object. We introduce Feature Learning from Environmental and Robot-Relational Edit Traces (FERRET), which learns preferences from both robot and environment corrections. We show that environment edits are able to disambiguate features and learn preferences that robot corrections alone cannot.

Figures & tables

Figure 1: Our XR system for collecting robot and environment feature traces.

Panel of Figure 2 — Figure 2: Ground truth and FERL learned features for ’avoid laptop’. Higher feature values indicate lower reward/higher cost. The laptop outline is shown in black. Feature traces, either robot or environment, are shown in white. For environment traces, the robot path is shown in teal.

Panel of Figure 3 — Figure 3: MSE, PR-AUC, and Spearman correlation for the ’avoid laptop’ scenario with robot (top row) and environment (bottom row) feature traces.

Panel of Figure 4 — Figure 4: Ground truth and Relational model learned features for the ambiguous ’avoid laptop’ scenario. Higher feature values indicate lower reward/higher cost. The laptop and cup’s initial positions are shown in red and teal respectively. The laptop’s test-time outline is shown in black. Feature traces are shown in white. For environment traces, the robot path is shown in teal.

Panel of Figure 5 — Figure 5: MSE, PR-AUC, and Spearman correlation for the ambiguous ’avoid laptop’ scenario with robot (top row) and environment (bottom row) feature traces.

Panel of Figure 6 — Figure 6: Ground truth and learned rewards for the feature ’avoid laptop when carrying an open cup’. Rewards are shown when evaluated for the robot carrying an open and a closed cup. Robot feature traces are overlaid on the reward. ’No preference’ feature traces for the closed cup are shown in green. The ground truth for the closed cup is uniformly zero.

Panel of Figure 7 — Figure 7: MSE for both open and closed cup conditions, and PR-AUC and Spearman correlation for the open cup condition.

Robot Learning

Robot learning develops methods for robots to acquire skills and adapt from demonstrations, interaction, data, and physical models.

See all 321 papers

Enhancing Goal Inference via Correction Timing
Feb 20, 2026Anjiabei Wang, Shuangge Wang, Tesca FitzgeraldRobot Learning Human-Robot Collaboration
Yale University, New Haven, CT, United States
Flexible Language-Guided Representation Editing for Adaptive Robot Behavior
Oct 20, 2026Audrey Lee, Nathaniel Dennler, Ilia Sucholutsky +1Robot Policy Adaptation Language-Conditioned Robot Control
MIT Computer Science & Artificial Intelligence Laboratory · New York University
Robots That Know What to Ask: Recovering Misaligned Rewards through Targeted Explanations
May 21, 2026Helena Merker, Nick Walker, Andreea BobuReinforcement Learning Learning from Demonstration
Massachusetts Institute of Technology

Feb 20, 2026J/K move · Enter open · S save

Enhancing Goal Inference via Correction Timing

Anjiabei Wang, Shuangge Wang, Tesca Fitzgerald

Yale University, New Haven, CT, United States

Robot Learning Human-Robot Collaboration RL from Human Feedback

Open paper AskPDF arXiv

Corrections offer a natural modality for people to provide feedback to a robot, by (i) intervening in the robot's behavior when they believe the robot is failing (or will fail) the task objectives and (ii) modifying the robot's behavior to successfully fulfill the task. Each correction offers information on what the robot should and should not do, where the corrected behavior is more aligned with task objectives than the original behavior. Most prior work on learning from corrections involves interpreting a correction as a new demonstration (consisting of the modified robot behavior), or a preference (for the modified trajectory compared to the robot's original behavior). However, this overlooks one essential element of the correction feedback, which is the human's decision to intervene in the robot's behavior in the first place. This decision can be influenced by multiple factors including the robot's task progress, alignment with human expectations, dynamics, motion legibility, and optimality. In this work, we investigate whether the timing of this decision can offer a useful signal for inferring these task-relevant influences. In particular, we investigate three potential applications for this learning signal: (1) identifying features of a robot's motion that may prompt people to correct it, (2) quickly inferring the final goal of a human's correction based on the timing and initial direction of their correction motion, and (3) learning more precise constraints for task objectives. Our results indicate that correction timing results in improved learning for the first two of these applications. Overall, our work provides new insights on the value of correction timing as a signal for robot learning.

Abstract

Robot Learning

Explore similar work

Enhancing Goal Inference via Correction Timing

Flexible Language-Guided Representation Editing for Adaptive Robot Behavior

Robots That Know What to Ask: Recovering Misaligned Rewards through Targeted Explanations