A Unified Framework for Probabilistic Dynamic-, Trajectory- and Vision-based Virtual Fixtures
Authors: Maximilian Mühlbauer, Bernhard Weber, Sylvain Calinon, Freek Stulp, Alin Albu-Schäffer, João Silvério
Organizations: School of Computation, Information and Technology, Sensor Based Robotic Systems and Intelligent Assistance Systems, Technical University of Munich, Friedrich-Ludwig-Bauer-Str. 3, Garching, Germany. · German Aerospace Center (DLR), Robotics and Mechatronics Center (RMC), Münchener Str. 20, 82234 Weßling, Germany. · Idiap Research Institute, Martigny, Switzerland. · École Polytechnique Fédérale de Lausanne (EPFL), Switzerland.
Probabilistic Virtual Fixtures (VFs) enable the adaptive selection of the most suitable haptic feedback for each phase of a task, based on learned or perceived uncertainty. While keeping the human in the loop remains essential, for instance, to ensure high precision, partial automation of certain task phases is critical for productivity. We present a unified framework for probabilistic VFs that seamlessly switches between manual fixtures, semi-automated fixtures (with the human handling precise tasks), and full autonomy. We introduce a novel probabilistic Dynamical System-based VF for coarse guidance, enabling the robot to autonomously complete certain task phases while keeping the human operator in the loop. For tasks requiring precise guidance, we extend probabilistic position-based trajectory fixtures with automation, allowing for seamless human interaction, geometry-awareness and optimal impedance gains. For manual tasks requiring very precise guidance, we also extend visual servoing fixtures with the same geometry-awareness and impedance behavior. We validate our approach on different robots, including an evaluation with expert users, showcasing operation modes, the ease of programming fixtures and lower interaction forces and favorable usability compared to a baseline.
Figures & tables
Fig. 1 : Overview of our unified framework. We propose a new type of learned, Dynamical Systems based Virtual Fixture to assist an operator with coarse guidance in progressing along the task while staying near the training data. For precise guidance , position-based trajectory fixtures support the human operator while geometric visual servoing fixtures provide very precise guidance . Input to the fixtures is the end effector pose xee and an image I while the output of each fixture is a probabilistic wrench wi∼N(μi,Σi) . Core of our framework is a novel variable impedance control scheme as well as an optimal arbitration of all probabilistic fixture wrenches.
Fixture
Input
Output
Accuracy
Workspace
Dynamical System ( V )
Pose
Velocity
Coarse
Everywhere
Position-based Trajectory ( VI )
Pose
Pose
Precise
Near trajectory
Visual servoing ( VII )
Image
Pose
Very precise
Near target
TABLE I : Types of fixtures considered in our framework.
Fig. 2 : Manifolds used in this work inspired by and using the notation of [ 22 ] . Depending on the task, properties can be expressed more efficiently in cylindrical ( M2 ) or spherical ( M3 ) compared to Cartesian ( M1 ) coordinates. The coordinate systems in each image depict the orientation basis, i.e., the unit quaternion (0,0,0,1)⊤ for different positions on the manifold.
Section
Automation
Interaction
Evaluation Metrics
VIII-A
manual
hand-guided
time, precision, usability
VIII-B
manual
hand-guided
qualitative
VIII-C
full
without
repeatability
VIII-D
full
without
tracking error
VIII-E
full
without
qualitative
VIII-F
semi
teleop & hand-g.
time, forces, usability
TABLE II : Automation, interaction and evaluation metrics.
Metric
Var. Stiffness
No Stiffness
Wilcoxon signed rank
Time
7.66±±1.20s
11.47±±4.04s
W=0 , p<0.05
Rot. Error
1.43±±1.77\SIUnitSymbolDegree
15.79±±18.25\SIUnitSymbolDegree
W=0 , p<0.05
NASA TLX
3.1±0.6
5.0±2.1
W=0 , p<0.05
SUS
94.2±2.9
79.6±17.4
W=0 , p<0.05
TABLE III : Variable stiffness: quant. evaluation (M ± SD).
Fig. 14 : Left image: The RoboGrav robot approaching the spring S using an automated position-based VF during the 0g phase of a parabolic flight. Also shown are the launch adapter ring segment LAR and the launch lock LL . Right image: PB1 denotes a position-based fixture for approaching and pressing the spring assembly S while PB2 shows the second position-based fixture for the robot moving along the launch adapter ring segment LAR .
Fig. 16 : DS -based VF DS combined with a position-based fixture PB on the RoboGrav space robot. The DS -based VF guides the robot towards the start of the automated position-based fixture performing a docking of the interface mounted at the end effector to the interface mounted on the rack DOCK .
Metric
With Arbitration
Without Arbitration
Wilcoxon signed rank
Time
23.98±±4.15s
28.10±±11.22s
W=1 , p<0.1
Measured Forces
15.28±±1.90N
19.70±±3.06N
W=0 , p<0.05
Perceived Forces
5.8±1.5
6.6±2.0
W=6 , n.s.
NASA TLX
3.86±1.3
6.8±1.6
W=0 , p<0.05
SUS
85.0±14.7
66.3±14.0
W=1 , p<0.1
TABLE IV : Tube experiment: quantitative evaluation (M ± SD).
Fig. 20 : Picking up a bottle B for moving it into a crate C using position-based trajectory, velocity and visual servoing fixtures. The rotationally symmetric bottle can be picked up from multiple angles thanks to a probabilistic trajectory fixture on M2 with variable stiffness (left). After B has been picked up, the velocity fixture takes over, fusing a stabilizing with the learned policy (right). This fixture aids the operator to move to C where a probabilistic visual servoing VF enables selection of placement position.
Metric
With Arbitration
Without Arbitration
Wilcoxon signed rank
Time
10.76±±2.92s
12.16±±4.41s
W=6 , n.s.
Measured Forces
6.18±±1.19N
10.81±±2.32N
W=0 , p<0.05
Perceived Forces
6.7±1.1
6.0±0.8
W=5.5 , n.s.
NASA TLX
4.7±1.4
4.7±1.5
W=9.5 , n.s.
SUS
89.0±6.6
87.9±10.0
W=7 , n.s.
TABLE V : Bottle experiment: quantitative evaluation (M ± SD).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
N∈N
≜
Number of data points per demonstration
NDS , NPB , NVS
≜
Number of fixtures of each type
M
≜
Number of Gaussian components in a fixture
xVF,i
≜
Attractor of the i -th fixture
wVF,i
≜
Wrench of the i -th fixture
ΣVF,i=PVF,i−1
≜
Covariance of the i -th fixture
KVF,i
≜
Stiffness matrix of the i -th fixture
Appendix
TABLE VI : Key notations used in our framework.
Fig. 22 : 2D projection of xee and the closest poses μj and μj+1 on a trajectory (red points) in M1 ( xcart ) as well as M23 ( xcyl ). The x axis through xee as well as the circle with the same radius on S1×R2×S3 is plotted in dashed gray, note that according to the manifold metric the closest pose is much further to the right than for Cartesian coordinates.
Maximilian Mühlbauer received his B.Sc. and M.Sc. degrees in Mechanical Engineering respectively Robotics from the Technical University of Munich (TUM). He is a research scientist in the department of Cognitive Robotics at the Institute of Robotics and Mechatronics at the German Aerospace Center (DLR) and a Ph.D. student at the Technical University of Munich. His research focuses on probabilistic machine learning and control methods for human-robot collaboration with applications in space, medical and industrial robotics.
Appendix
Table 13
Bernhard Weber received his Dipl.-Psych. and PhD degree at the University of Würzburg, Germany, in 2004 and 2008, respectively. From 2008–2010, he was with the German Aerospace Center (DLR), Institute of Flight Guidance, Brunswick, Germany and since 2010 at the DLR Institute of Robotics and Mechatronics, Wessling, Germany, as a human factors expert. His main research interests are human factors in telerobotic systems, evaluation of haptic interaction technology, and sensorimotor performance under conditions of microgravity.
Appendix
Table 14
Sylvain Calinon received the Ph.D. degree in robotics from the École Polytechnique Fedérale de Lausanne (EPFL), Lausanne, Switzerland, in 2007. He is currently a Senior Research Scientist with the Idiap Research Institute, Martigny, Switzerland, and a Lecturer with the EPFL. From 2009 to 2014, he was a Team Leader with the Italian Institute of Technology, Genoa, Italy. From 2007 to 2009, he was a Postdoc with EPFL. His research interests include human–robot collaboration, robot learning, robot manipulation skills, geometry and model-based optimization.
Appendix
Table 15
Freek Stulp received the doctorate degree in computer science from the Technical University of Munich, Munich, Germany, in 2007. He is currently the Head of the Department of Cognitive Robotics, Institute of Robotics and Mechatronics, German Aerospace Center. Previously, he was an Assistant Professor with the École Nationale Supérieure de Techniques Avancées.
Appendix
Table 16
Alin Albu-Schäffer received his M.S. in electrical engineering from the Technical University of Timisoara, Romania in 1993 and his Ph.D. in automatic control from the Technical University of Munich in 2002. Since 2012 he is the head of the Institute of Robotics and Mechatronics at the German Aerospace Center (DLR). Moreover, he is a professor at the Technical University of Munich, holding the Chair for ”Sensor Based Robotic Systems and Intelligent Assistance Systems” at the School of Computation, Information and Technology. His personal research interests include robot design, modeling and control, nonlinear control, flexible joint and variable compliance robots, impedance and force control, physical human-robot interaction, bio-inspired robot design and control. He received several awards, including the IEEE King-Sun Fu Best Paper Award of the Transactions on Robotics in 2012 and 2014; several ICRA and IROS Best Paper Awards as well as the DLR Science Award. He was strongly involved in the development of the DLR light-weight robot and its commercialization through technology transfer to KUKA. He is the coordinator of euROBIN, the European network of excellence on intelligent robotics, IEEE Fellow and RAS-member.
Appendix
Table 17
João Silvério is a senior researcher and group leader at the German Aerospace Center (DLR) in Munich, where he leads the Interactive Skill Learning group since 2022. He received a PhD in Robotics from the Italian Institute of Technology (IIT), Genoa, in 2017. From 2017 to 2019, he was a postdoctoral researcher at IIT, and from 2019 to 2022, he held a postdoctoral position at the Idiap Research Institute in Martigny, Switzerland. His research focuses on machine learning for robotics, with particular interest in learning for control, imitation learning, and reinforcement learning.
In many teleoperation applications, collecting a large number of demonstrations as required for traditional probabilistic learning from demonstration (LfD) approaches may not be feasible. To still give operators the ability to intuitively create trajectories as Virtual Fixtures (VFs), we propose to leverage a motion prior in the learning process. Particularly, by using Linear Quadratic Tracking (LQT), users are able to define guiding trajectories from the demonstration of just a few via points. To account for orientation guidance, we further reformulate classical LQT on Riemannian manifolds, introducing an additional geometric prior. Through a probabilistic interpretation of the LQT solution, we derive a covariance estimate at each trajectory point which we use to modulate the stiffness of the resulting fixture, resulting in strong guidance around the via points and softer guidance when far away. The covariance information is also used to define a validity region of the fixture, allowing the operator to leave its influence area and conduct unmodeled tasks. We evaluate the proposed Riemannian LQT formulation in a set of toy examples and the full framework on a cutting task requiring high precision in a minimally invasive surgery setting on the da Vinci Research Kit (dVRK).
Maximilian Mühlbauer, Marcella Piacentino, Raffaella Mancino +10
German Aerospace Center (DLR), Robotics and Mechatronics Center (RMC), Münchener Str. 20, 82234 Weßling, Germany. · Department of Computer Engineering, Technical University of Munich, Friedrich-Ludwig-Bauer-Str. 3, Garching, Germany. · C.R.E.A.T.E. Consorzio di Ricerca per l’Energia, l’Automazione e le Tecnologie dell’Elettromagnetismo, Universit`a degli Studi di Napoli Federico II, Napoli, Italy +5
Vision-Language-Action (VLA) models excel at mapping visual inputs and natural language instructions directly to robotic control policies. However, because they are trained primarily to fit behavioural demonstration data, they do not explicitly enforce fundamental physical principles such as rigid-body dynamics or contact constraints. This exposes a critical physics gap: standard temporal smoothing applied on top of single-step or chunked VLAs trades trajectory quality for added failures that short-term memory cannot resolve. To bridge this gap, we introduce PhysVLA (Physics-VLA), a plug-and-play, inference-time framework designed to wrap any frozen VLA backbone without retraining, fine-tuning, or weight access, with less than 1 ms of overhead per control step. PhysVLA intercepts the predicted control action, captures only the simulator or system state, and applies a dual-layered correction: (i) a phase-aware finite-state machine that structures discrete task segments (approach, grasp, transport, and place), and (ii) a selective Euler-Lagrange gate that activates only when a dynamics oracle detects kinodynamic inconsistency. Evaluated across OpenVLA, OpenVLA-OFT, Force-VLA, and Generalist-VLA on LIBERO-Spatial with a 7-DoF Franka Panda, the framework delivers absolute success rate increases of up to 17% and stability increases of up to 19% with no per-task regressions, improves trajectory efficiency by up to 15% across all four backbones, and shows up to a 10x improvement in trajectory jerk robustness on a Robosuite Lift cross-simulator sweep. We further validate the framework on a real Agilex Piper arm with a pick-and-place task, confirming that PhysVLA transfers to physical hardware without retraining, with success-rate improvements of up to 50%, establishing physical awareness as a composable, backbone-agnostic runtime module.
Namai Chandra, Shriram Damodaran, Lin Wang
Electronic Systems IIT Madras, India · EmPACT Lab Nanyang Technological University Singapore
Natural-language manipulation instructions specify the task goal but leave the underlying robot trajectory unspecified. We present AntiGrounding, a visual action-selection framework built around a dual geometric-visual trajectory interface. After feasibility filtering, each retained short trajectory is both an explicit motion plan for execution and a rendered prompt for instruction-conditioned vision-language model (VLM) evaluation. Structured multi-view visual question answering (VQA) scores safety, task alignment, efficiency, and physical plausibility; weighted view fusion aggregates the trajectory scores. These scores guide subsequent translational trajectory proposals; separate orientation and gripper controls coordinate interaction. An initialized digital twin provides the planning state and validates selected segments before the real robot executes the same waypoint sequences. Across eight real-world manipulation tasks, AntiGrounding with a single GPT-6 Astra evaluator achieves 71.25% overall success, compared with 50.00% for pi0.5 and 47.50% for a PIVOT-style visual proposal-selection baseline using the same evaluator under the reported deployment protocol. Component ablations and evaluator-sensitivity analyses examine trajectory evaluation, proposal search, orientation control, and evaluator choice. The interface connects general-purpose multimodal reasoning to executable trajectories, with performance bounded by digital-twin fidelity and physical interaction.
Wenbo Li, Yiteng Chen, Wenhao Li +1
School of Software Engineering, South China University of Technology