A Unified Framework for Probabilistic Dynamic-, Trajectory- and Vision-based Virtual Fixtures
Organizations: School of Computation, Information and Technology, Sensor Based Robotic Systems and Intelligent Assistance Systems, Technical University of Munich, Friedrich-Ludwig-Bauer-Str. 3, Garching, Germany. · German Aerospace Center (DLR), Robotics and Mechatronics Center (RMC), Münchener Str. 20, 82234 Weßling, Germany. · Idiap Research Institute, Martigny, Switzerland. · École Polytechnique Fédérale de Lausanne (EPFL), Switzerland.
Abstract
Probabilistic Virtual Fixtures (VFs) enable the adaptive selection of the most suitable haptic feedback for each phase of a task, based on learned or perceived uncertainty. While keeping the human in the loop remains essential, for instance, to ensure high precision, partial automation of certain task phases is critical for productivity. We present a unified framework for probabilistic VFs that seamlessly switches between manual fixtures, semi-automated fixtures (with the human handling precise tasks), and full autonomy. We introduce a novel probabilistic Dynamical System-based VF for coarse guidance, enabling the robot to autonomously complete certain task phases while keeping the human operator in the loop. For tasks requiring precise guidance, we extend probabilistic position-based trajectory fixtures with automation, allowing for seamless human interaction, geometry-awareness and optimal impedance gains. For manual tasks requiring very precise guidance, we also extend visual servoing fixtures with the same geometry-awareness and impedance behavior. We validate our approach on different robots, including an evaluation with expert users, showcasing operation modes, the ease of programming fixtures and lower interaction forces and favorable usability compared to a baseline.
Figures & tables
| Fixture | Input | Output | Accuracy | Workspace |
|---|---|---|---|---|
| Dynamical System ( V ) | Pose | Velocity | Coarse | Everywhere |
| Position-based Trajectory ( VI ) | Pose | Pose | Precise | Near trajectory |
| Visual servoing ( VII ) | Image | Pose | Very precise | Near target |
| Section | Automation | Interaction | Evaluation Metrics |
|---|---|---|---|
| VIII-A | manual | hand-guided | time, precision, usability |
| VIII-B | manual | hand-guided | qualitative |
| VIII-C | full | without | repeatability |
| VIII-D | full | without | tracking error |
| VIII-E | full | without | qualitative |
| VIII-F | semi | teleop & hand-g. | time, forces, usability |
| Metric | Var. Stiffness | No Stiffness | Wilcoxon signed rank |
|---|---|---|---|
| Time | , | ||
| Rot. Error | , | ||
| NASA TLX | , | ||
| SUS | , |
| Metric | With Arbitration | Without Arbitration | Wilcoxon signed rank |
|---|---|---|---|
| Time | , | ||
| Measured Forces | , | ||
| Perceived Forces | , n.s. | ||
| NASA TLX | , | ||
| SUS | , |
| Metric | With Arbitration | Without Arbitration | Wilcoxon signed rank |
|---|---|---|---|
| Time | , n.s. | ||
| Measured Forces | , | ||
| Perceived Forces | , n.s. | ||
| NASA TLX | , n.s. | ||
| SUS | , n.s. |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Number of data points per demonstration | ||
| , , | Number of fixtures of each type | |
| Number of Gaussian components in a fixture | ||
| Attractor of the -th fixture | ||
| Wrench of the -th fixture | ||
| Covariance of the -th fixture | ||
| Stiffness matrix of the -th fixture |
| Maximilian Mühlbauer received his B.Sc. and M.Sc. degrees in Mechanical Engineering respectively Robotics from the Technical University of Munich (TUM). He is a research scientist in the department of Cognitive Robotics at the Institute of Robotics and Mechatronics at the German Aerospace Center (DLR) and a Ph.D. student at the Technical University of Munich. His research focuses on probabilistic machine learning and control methods for human-robot collaboration with applications in space, medical and industrial robotics. |
| Bernhard Weber received his Dipl.-Psych. and PhD degree at the University of Würzburg, Germany, in 2004 and 2008, respectively. From 2008–2010, he was with the German Aerospace Center (DLR), Institute of Flight Guidance, Brunswick, Germany and since 2010 at the DLR Institute of Robotics and Mechatronics, Wessling, Germany, as a human factors expert. His main research interests are human factors in telerobotic systems, evaluation of haptic interaction technology, and sensorimotor performance under conditions of microgravity. |
| Sylvain Calinon received the Ph.D. degree in robotics from the École Polytechnique Fedérale de Lausanne (EPFL), Lausanne, Switzerland, in 2007. He is currently a Senior Research Scientist with the Idiap Research Institute, Martigny, Switzerland, and a Lecturer with the EPFL. From 2009 to 2014, he was a Team Leader with the Italian Institute of Technology, Genoa, Italy. From 2007 to 2009, he was a Postdoc with EPFL. His research interests include human–robot collaboration, robot learning, robot manipulation skills, geometry and model-based optimization. |
| Freek Stulp received the doctorate degree in computer science from the Technical University of Munich, Munich, Germany, in 2007. He is currently the Head of the Department of Cognitive Robotics, Institute of Robotics and Mechatronics, German Aerospace Center. Previously, he was an Assistant Professor with the École Nationale Supérieure de Techniques Avancées. |
| Alin Albu-Schäffer received his M.S. in electrical engineering from the Technical University of Timisoara, Romania in 1993 and his Ph.D. in automatic control from the Technical University of Munich in 2002. Since 2012 he is the head of the Institute of Robotics and Mechatronics at the German Aerospace Center (DLR). Moreover, he is a professor at the Technical University of Munich, holding the Chair for ”Sensor Based Robotic Systems and Intelligent Assistance Systems” at the School of Computation, Information and Technology. His personal research interests include robot design, modeling and control, nonlinear control, flexible joint and variable compliance robots, impedance and force control, physical human-robot interaction, bio-inspired robot design and control. He received several awards, including the IEEE King-Sun Fu Best Paper Award of the Transactions on Robotics in 2012 and 2014; several ICRA and IROS Best Paper Awards as well as the DLR Science Award. He was strongly involved in the development of the DLR light-weight robot and its commercialization through technology transfer to KUKA. He is the coordinator of euROBIN, the European network of excellence on intelligent robotics, IEEE Fellow and RAS-member. |
| João Silvério is a senior researcher and group leader at the German Aerospace Center (DLR) in Munich, where he leads the Interactive Skill Learning group since 2022. He received a PhD in Robotics from the Italian Institute of Technology (IIT), Genoa, in 2017. From 2017 to 2019, he was a postdoctoral researcher at IIT, and from 2019 to 2022, he held a postdoctoral position at the Idiap Research Institute in Martigny, Switzerland. His research focuses on machine learning for robotics, with particular interest in learning for control, imitation learning, and reinforcement learning. |