Organizations: Division of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology (NAIST), Ikoma-shi, Nara 630-0192, Japan
Imitation learning frameworks that learn robot control policies from demonstrators' motions via hand-mounted demonstration interfaces have attracted increasing attention. However, due to differences in physical characteristics between demonstrators and robots, this approach faces two limitations: i) the demonstration data do not include robot actions, and ii) the demonstrated motions may be infeasible for robots. These limitations make policy learning difficult. To address them, we propose Feasibility-Aware Behavior Cloning from Observation (FABCO). FABCO integrates behavior cloning from observation, which complements robot actions using robot dynamics models, with feasibility estimation. In feasibility estimation, the demonstrated motions are evaluated using a robot-dynamics model, learned from the robot's execution data, to assess reproducibility under the robot's dynamics. The estimated feasibility is used for multimodal feedback and feasibility-aware policy learning to improve the demonstrator's motions and learn robust policies. Multimodal feedback provides feasibility through the demonstrator's visual and haptic senses to promote feasible demonstrated motions. Feasibility-aware policy learning reduces the influence of demonstrated motions that are infeasible for robots, enabling the learning of policies that robots can execute stably. We conducted experiments with 15 participants on two tasks and confirmed that FABCO improves imitation learning performance by more than 3.2 times compared to the case without feasibility feedback.
Figures & tables
Fig. 1: Comparison between conventional and proposed feasibility-aware imitation learning frameworks. (a) In demonstrations performed with a hand-mounted interface, feasibility constraints are not reflected in demonstrations, resulting in learned policies that may produce unpredictable or undesirable motions. (b) Proposed framework incorporates feasibility in both demonstration and learning, promoting feasible demonstrations and improving performance of learned policy toward desired motions.
Fig. 2: Overview of FABCO framework. This framework incorporates feasibility into both demonstration and policy learning in imitation learning using a hand-mounted demonstration interface. (a) The demonstrator receives feasibility feedback on demonstrated motions, which is computed using FDM and IDM, and refines the demonstration based on this feedback. (b) Policy learning uses the feasibility of the demonstration data, permitting the learned policy to generate feasible robot motions.
Fig. 3: Hand-mounted demonstration interface. (a) Overview of interface. An encoder is attached to measure finger position. Motion-capture markers are attached to track the interface pose. (b) Internal structure of handle. It is equipped with two vibration motors.
Fig. 4: Experimental setup for human-subject experiment. (a) Experimental environment: In this environment, the workspace on the desk is shared with a demonstrator and the robot. The motion is captured by a motion-capture camera. The demonstrator receives the feasibility through the monitor and the interface. (b) Visual feasibility feedback system: This system provides two types of visualizations: pose-level feasibility visualization and overall feasibility visualization.
Fig. 5: Task environments. (a) The peg-insertion task requires moving a grasped peg from start to goal while avoiding obstacles. (b) The circle-tracing task requires tracing a circular path on a table. In each task, the green region denotes the workspace in which the robot must complete motions.
Fig. 6: Transitions of estimated feasibility during demonstration: (a) transition in the peg-insertion task and (b) transition in the circle-tracing task. Horizontal axis represents number of demonstrated motions, and vertical axis represents mean feasibility of demonstrated motions. Each plot shows mean and standard deviation of feasibility transitions. For No FB, statistics are computed across a single experiment by 15 subjects. For the other methods, statistics are computed across a single experiment by five subjects each.
Fig. 7: Transitions of demonstrated motions collected under each demonstration method in peg-insertion task. Demonstrated motions were collected using (a) visuo-haptic feedback demonstration and (b) No FB demonstration.
Fig. 8: Transitions of demonstrated motions collected under each demonstration method in circle-tracing task. Horizontal axis represents number of demonstrated motions, and vertical axis represents mean speed of demonstrated motions. Each plot shows mean and standard deviation of the demonstrated motion’s speed. For No FB, statistics are computed across a single experiment by 15 subjects. For the other methods, statistics are computed across a single experiment by five subjects each.
Fig. 9: NASA-TLX performance subscale in peg-insertion task for all demonstration methods. “*” indicates a statistically significant difference with p<0.05 in t-test.
Fig. 10: Results of NASA-TLX for all demonstration methods in circle-tracing task: (a) effort and (b) frustration subscales. “*” indicates a statistically significant difference with p<0.05 in t-test.
Fig. 11: Results of questionnaire on subjects’ preferences for demonstration methods. (a) Questionnaire results evaluating whether overall visualization or pose-level visualization was more beneficial for subjects using visuo-haptic and visual feedback demonstrations. Lower values indicate a stronger preference for pose-level visualization, while higher values indicate a stronger preference for overall visualization. (b) Questionnaire results evaluating whether visual feedback or haptic feedback was more beneficial for subjects using a visuo-haptic feedback demonstration. Lower values indicate a stronger preference for visual feedback, while higher values indicate a stronger preference for haptic feedback. In both questionnaires, responses were rated on a 20-point scale.
Fig. 12: Robot motions using the policy learned by FABCO with visuo-haptic feedback demonstration in (a) peg-insertion task and (b) circle-tracing task.
TABLE I: Pairwise significance comparison of success rates among demonstration methods in peg-insertion task. Pairwise t-test results are shown for FABCO, with visuo-haptic, haptic, and visual feedback, and for BCO. “+” indicates that the method in the row achieves a higher success rate than the method in the column, while “–” indicates the opposite. “*” denotes statistical significance ( p<0.05 ). Bottom row shows the success rate (%) for each method.
Fig. 13: Dynamic Time Warping (DTW) distances between mean demonstrated motion and each demonstrated motion for each subject in peg-insertion task. Mean demonstrated motion is computed by averaging positions at each time step across subjects’ temporally normalized demonstrated motions. Statistical significance of DTW differences among methods is evaluated using pairwise t-tests across all combinations of demonstration methods, where “*” indicates a statistically significant difference ( p<0.05 ).
TABLE II: Pairwise significance comparison of success rates among demonstration methods in circle-tracing task. Pairwise t-test results are shown for FABCO, with visuo-haptic, haptic, and visual feedback, and for BCO. “+” indicates that the method in the row achieves a higher success rate than the method in the column, while “–” indicates the opposite. “*” denotes statistical significance ( p<0.05 ). The bottom row shows the success rate (%) for each method.
Fig. 14: Robot motion trajectories of policies learned in peg-insertion task. Policies learned by (a) FABCO with visuo-haptic demonstration, (b) FABCO with haptic demonstration, (c) FABCO with visual demonstration, and (d) BCO with No FB demonstration. Red lines and green areas indicate robot motion trajectories and the workspace, respectively.
Fig. 15: Robot motions of learned policies in circle-tracing task. Policies learned by (a) FABCO with visuo-haptic demonstration, (b) FABCO with haptic demonstration, (c) FABCO with visual demonstration, and (d) BCO with No FB demonstration. Red lines and green points indicate robot motion and demonstrated motion, respectively.
Peg-insertion task
Circle-tracing task
FABCO
FABCO w/o FPL
FABCO
FABCO w/o FPL
First-half
76.0 †
32.0 †
64.0 ‡
36.0 ‡
Full
90.0
88.0
88.0
88.0
TABLE III: Comparison of success rates for policies learned by FABCO from visuo-haptic demonstrations, with and without feasibility-aware policy learning (FPL), in peg-insertion and circle-tracing tasks. Variant learned without FPL denoted as FABCO w/o FPL. “ † ” and “ ‡ ” indicate a statistically significant difference with p<0.05 in pairwise t-test.
Fig. 16: Robot motion of learned policies in peg-insertion task. Policies learned by (a) FABCO using first half of demonstration data, (b) FABCO w/o feasibility-aware policy learning (FPL) using first half of demonstration data, (c) FABCO using full demonstration data, and (d) FABCO w/o FPL using full demonstration data. Red lines and green areas indicate robot motion trajectories and the workspace, respectively.
Fig. 17: Robot motion trajectories of learned policies in circle-tracing task. Policies learned by (a) FABCO using first half of demonstration data, (b) FABCO w/o feasibility-aware policy learning (FPL) using first half of demonstration data, (c) FABCO using full demonstration data, and (d) FABCO w/o FPL using full demonstration data. Red lines and green points indicate robot motion trajectories and demonstration data, respectively.
Fig. 18: Comparison of Hausdorff distance between demonstrated and robot motions generated by learned policies with and without feasibility-aware policy learning (FPL). (a) Results for peg-insertion task. (b) Results for circle-tracing task. “*” indicates a statistically significant difference with p<0.05 in pairwise t-test.
With the increasing use of robot-free demonstration interfaces that provide state trajectories without action labels, imitation from observation has become a promising approach for learning robot behaviors from human demonstrations. However, due to differences in embodiment and dynamics between humans and robots, demonstrated human motions may not be feasible for the robot, potentially degrading policy performance. In this study, we propose Experience-Based Feasibility-Aware Generative Adversarial Imitation from Observation (EF-GAIfO), which estimates the feasibility of state-only demonstrations from the robot's own experience rather than relying on explicit dynamics models or large prior exploration datasets. A key feature of EF-GAIfO is that the notion of feasibility evolves with policy learning: as the policy improves and the robot experiences a broader range of state transitions, the feasible region is progressively expanded, allowing additional demonstrations to be incorporated into learning. This enables feasibility-aware imitation that adapts to the current stage of policy learning, rather than relying on a pre-designed feasibility criterion. We validate the effectiveness of EF-GAIfO on a locomotion task in simulation and on a real quadruped robot performing a object-reaching-and-grasping task.
Yoshiki Takebayashi, Giovanni Perantoni, Hikaru Sasaki +2
Division of Information Science, Graduate School of Information Science, Nara Institute of Science and Technology (NAIST), Nara, Japan · Department of Industrial Engineering of the University of Trento, Trento, Italy
Visuomotor imitation learning has demonstrated success for manipulation tasks. However, the trained policies remain brittle to visual nuisances', with even minor task-preserving variations such as lighting, distractions or changes in colour result in heavy degradation of the trained policy's performance. While increasing data diversity can improve robustness, it is unclear which additional demonstrations are informative for a particular trained policy. We propose Counterfactual Nuisance Behaviour Cloning (CFNBC), an offline data-selection framework for targeted robustness repair. Starting from a nominal policy trained on clean' demonstrations, CFNBC generates paired clean and nuisance observations that preserve the expert action, then measures action drift: the change in the policy's predicted action under a nuisance that should not alter the desired behaviour. This provides a policy-specific sensitivity signal for selecting a compact, response-diverse repair set from a larger candidate pool, without requiring rollout success labels or online policy execution. We show in MuJoCo bimanual cube transfer and SimplerEnv cube stacking that action drift correlates with nuisance-induced failure, and that response-guided repair with only 20--30 selected candidates substantially outperforms matched-budget random selection while approaching the performance of much larger random repair budgets. These results support a data-centric view of robustness repair: the most useful data are not necessarily the most numerous, visually diverse, or obviously difficult, but the examples that cover fragile response modes of the current policy.
Giovanni D'urso, Kaushik Roy, Nicholas Lawrance +1
Behavior cloning for robot manipulation relies on expert demonstrations. However, for tasks that require dynamic stability, precise contact timing, or dexterous coordination, human operators may find it hard or even impossible to collect data. We study this infeasible-demonstration regime and propose GLIDE: Guardrails for Learning from Infeasible Demonstrations Efficiently, a framework that infers task-specific failure modes and converts them into executable guardrails for data collection and policy deployment. Given a task description and the conditioning teleoperation code, GLIDE writes guardrails that use system states to filter teleoperation and policy commands, constrain failure-prone actions, and iteratively improve from trajectory feedback. Across three tasks, GLIDE discovers emergent guardrails that go beyond domain-expert hardcoded ones, improving data collection over naive VR teleoperation and domain-expert hardcoded guardrails. After refinement, GLIDE raises data-collection success from 0-10 percent to 70-90 percent across the three tasks. During policy execution, mixed-data guarded policies reach 70 percent, 60 percent, and 60 percent success on Tomato plate transfer, Marker handover and stand, and Wine serving tasks. These results show that GLIDE can support policy learning when direct demonstrations are infeasible. Project website: http://guardrail-policy.github.io/