cs.ROFeb 17, 2026
SaveFeasibility-aware Imitation Learning from Observation with Multimodal Feedback
Organizations: Division of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology (NAIST), Ikoma-shi, Nara 630-0192, Japan
Abstract
Imitation learning frameworks that learn robot control policies from demonstrators' motions via hand-mounted demonstration interfaces have attracted increasing attention. However, due to differences in physical characteristics between demonstrators and robots, this approach faces two limitations: i) the demonstration data do not include robot actions, and ii) the demonstrated motions may be infeasible for robots. These limitations make policy learning difficult. To address them, we propose Feasibility-Aware Behavior Cloning from Observation (FABCO). FABCO integrates behavior cloning from observation, which complements robot actions using robot dynamics models, with feasibility estimation. In feasibility estimation, the demonstrated motions are evaluated using a robot-dynamics model, learned from the robot's execution data, to assess reproducibility under the robot's dynamics. The estimated feasibility is used for multimodal feedback and feasibility-aware policy learning to improve the demonstrator's motions and learn robust policies. Multimodal feedback provides feasibility through the demonstrator's visual and haptic senses to promote feasible demonstrated motions. Feasibility-aware policy learning reduces the influence of demonstrated motions that are infeasible for robots, enabling the learning of policies that robots can execute stably. We conducted experiments with 15 participants on two tasks and confirmed that FABCO improves imitation learning performance by more than 3.2 times compared to the case without feasibility feedback.
Figures & tables
Fig. 1: Comparison between conventional and proposed feasibility-aware imitation learning frameworks. (a) In demonstrations performed with a hand-mounted interface, feasibility constraints are not reflected in demonstrations, resulting in learned policies that may produce unpredictable or undesirable motions. (b) Proposed framework incorporates feasibility in both demonstration and learning, promoting feasible demonstrations and improving performance of learned policy toward desired motions.
Fig. 2: Overview of FABCO framework. This framework incorporates feasibility into both demonstration and policy learning in imitation learning using a hand-mounted demonstration interface. (a) The demonstrator receives feasibility feedback on demonstrated motions, which is computed using FDM and IDM, and refines the demonstration based on this feedback. (b) Policy learning uses the feasibility of the demonstration data, permitting the learned policy to generate feasible robot motions.
Fig. 3: Hand-mounted demonstration interface. (a) Overview of interface. An encoder is attached to measure finger position. Motion-capture markers are attached to track the interface pose. (b) Internal structure of handle. It is equipped with two vibration motors.
Fig. 4: Experimental setup for human-subject experiment. (a) Experimental environment: In this environment, the workspace on the desk is shared with a demonstrator and the robot. The motion is captured by a motion-capture camera. The demonstrator receives the feasibility through the monitor and the interface. (b) Visual feasibility feedback system: This system provides two types of visualizations: pose-level feasibility visualization and overall feasibility visualization.
Fig. 5: Task environments. (a) The peg-insertion task requires moving a grasped peg from start to goal while avoiding obstacles. (b) The circle-tracing task requires tracing a circular path on a table. In each task, the green region denotes the workspace in which the robot must complete motions.
Fig. 6: Transitions of estimated feasibility during demonstration: (a) transition in the peg-insertion task and (b) transition in the circle-tracing task. Horizontal axis represents number of demonstrated motions, and vertical axis represents mean feasibility of demonstrated motions. Each plot shows mean and standard deviation of feasibility transitions. For No FB, statistics are computed across a single experiment by 15 subjects. For the other methods, statistics are computed across a single experiment by five subjects each.
Fig. 7: Transitions of demonstrated motions collected under each demonstration method in peg-insertion task. Demonstrated motions were collected using (a) visuo-haptic feedback demonstration and (b) No FB demonstration.
Fig. 8: Transitions of demonstrated motions collected under each demonstration method in circle-tracing task. Horizontal axis represents number of demonstrated motions, and vertical axis represents mean speed of demonstrated motions. Each plot shows mean and standard deviation of the demonstrated motion’s speed. For No FB, statistics are computed across a single experiment by 15 subjects. For the other methods, statistics are computed across a single experiment by five subjects each.
Fig. 9: NASA-TLX performance subscale in peg-insertion task for all demonstration methods. “*” indicates a statistically significant difference with in t-test.
Fig. 10: Results of NASA-TLX for all demonstration methods in circle-tracing task: (a) effort and (b) frustration subscales. “*” indicates a statistically significant difference with in t-test.
Fig. 11: Results of questionnaire on subjects’ preferences for demonstration methods. (a) Questionnaire results evaluating whether overall visualization or pose-level visualization was more beneficial for subjects using visuo-haptic and visual feedback demonstrations. Lower values indicate a stronger preference for pose-level visualization, while higher values indicate a stronger preference for overall visualization. (b) Questionnaire results evaluating whether visual feedback or haptic feedback was more beneficial for subjects using a visuo-haptic feedback demonstration. Lower values indicate a stronger preference for visual feedback, while higher values indicate a stronger preference for haptic feedback. In both questionnaires, responses were rated on a 20-point scale.
Fig. 12: Robot motions using the policy learned by FABCO with visuo-haptic feedback demonstration in (a) peg-insertion task and (b) circle-tracing task.
TABLE I: Pairwise significance comparison of success rates among demonstration methods in peg-insertion task. Pairwise t-test results are shown for FABCO, with visuo-haptic, haptic, and visual feedback, and for BCO. “+” indicates that the method in the row achieves a higher success rate than the method in the column, while “–” indicates the opposite. “*” denotes statistical significance ( ). Bottom row shows the success rate (%) for each method.
Fig. 13: Dynamic Time Warping (DTW) distances between mean demonstrated motion and each demonstrated motion for each subject in peg-insertion task. Mean demonstrated motion is computed by averaging positions at each time step across subjects’ temporally normalized demonstrated motions. Statistical significance of DTW differences among methods is evaluated using pairwise t-tests across all combinations of demonstration methods, where “*” indicates a statistically significant difference ( ).
TABLE II: Pairwise significance comparison of success rates among demonstration methods in circle-tracing task. Pairwise t-test results are shown for FABCO, with visuo-haptic, haptic, and visual feedback, and for BCO. “+” indicates that the method in the row achieves a higher success rate than the method in the column, while “–” indicates the opposite. “*” denotes statistical significance ( ). The bottom row shows the success rate (%) for each method.
Fig. 14: Robot motion trajectories of policies learned in peg-insertion task. Policies learned by (a) FABCO with visuo-haptic demonstration, (b) FABCO with haptic demonstration, (c) FABCO with visual demonstration, and (d) BCO with No FB demonstration. Red lines and green areas indicate robot motion trajectories and the workspace, respectively.
Fig. 15: Robot motions of learned policies in circle-tracing task. Policies learned by (a) FABCO with visuo-haptic demonstration, (b) FABCO with haptic demonstration, (c) FABCO with visual demonstration, and (d) BCO with No FB demonstration. Red lines and green points indicate robot motion and demonstrated motion, respectively.
| Peg-insertion task | Circle-tracing task | |||
| FABCO | FABCO w/o FPL | FABCO | FABCO w/o FPL | |
| First-half | 76.0 † | 32.0 † | 64.0 ‡ | 36.0 ‡ |
| Full | 90.0 | 88.0 | 88.0 | 88.0 |
TABLE III: Comparison of success rates for policies learned by FABCO from visuo-haptic demonstrations, with and without feasibility-aware policy learning (FPL), in peg-insertion and circle-tracing tasks. Variant learned without FPL denoted as FABCO w/o FPL. “ ” and “ ” indicate a statistically significant difference with in pairwise t-test.
Fig. 16: Robot motion of learned policies in peg-insertion task. Policies learned by (a) FABCO using first half of demonstration data, (b) FABCO w/o feasibility-aware policy learning (FPL) using first half of demonstration data, (c) FABCO using full demonstration data, and (d) FABCO w/o FPL using full demonstration data. Red lines and green areas indicate robot motion trajectories and the workspace, respectively.
Fig. 17: Robot motion trajectories of learned policies in circle-tracing task. Policies learned by (a) FABCO using first half of demonstration data, (b) FABCO w/o feasibility-aware policy learning (FPL) using first half of demonstration data, (c) FABCO using full demonstration data, and (d) FABCO w/o FPL using full demonstration data. Red lines and green points indicate robot motion trajectories and demonstration data, respectively.
Fig. 18: Comparison of Hausdorff distance between demonstrated and robot motions generated by learned policies with and without feasibility-aware policy learning (FPL). (a) Results for peg-insertion task. (b) Results for circle-tracing task. “*” indicates a statistically significant difference with in pairwise t-test.