Safety-Critical Bilateral Teleoperation for Omnidirectional Aerial Manipulation Using Force-Sensorless Haptic Feedback
Authors: Yubin Kim, Jinwoo Lee, Yongjun You, H. Jin Kim, Jeonghyun Byun
Organizations: Department of Aerospace Engineering, Seoul National University, Seoul, South Korea · Department of Aerospace Engineering, Automation and System Research Institute (ASRI), and Institute of Advanced Aerospace Technology (IAAT), Seoul National University, Seoul, South Korea · Automation and System Research Institute (ASRI), Seoul National University, Seoul, South Korea
This paper presents a safety-critical bilateral teleoperation framework for omnidirectional aerial manipulators that integrates visual and force-sensorless haptic wrench feedback. Unlike existing approaches that either rely on onboard force/torque sensors or use model-dependent wrench estimates, which may become unreliable under model uncertainties or induce unintended feedback during free-flight, our method implements a hierarchical safety filter based on control barrier functions to avoid such limitations. The safety filter, being the key contribution, explicitly accounts for tracking errors arising from physical interaction between the aerial manipulator and its surroundings while enforcing thrust limits, a factor overlooked despite its critical importance for flight safety. This safety filter adjusts the command from the operator to ensure safe and stable aerial manipulation and avoid motor saturation. The adjustment made by the filter is mapped to haptic feedback, which is intuitive to the operator and conveys information on physical interaction and impending motor saturation. By actual experiments with a hexarotor-based omnidirectional aerial manipulator, we demonstrate that the proposed method avoids haptic feedback during free-flight, provides directionally consistent feedback under physical interaction, and can be operated for diverse manipulative tasks. Moreover, an ablation study further shows that the saturation filter improves interaction stability by explicitly preventing motor saturation and informing the operator of corrective actions.
Figures & tables
Fig. 1 : Illustration of our haptic-based bilateral teleoperation for an omnidirectional aerial manipulator (remote environment) teleoperated by a human operator through a haptic master device (local environment).
Fig. 2 : Three-dimensional models of (a) the AM and (b) the master device, including their corresponding body frames, with motor indices labeled on the AM.
Fig. 3 : Outline of the teleoperation framework.
Fig. 4 : The behaviors of the AM and master device during free-flight using haptic feedback generated by (a) the momentum-based wrench estimation [ 2 , 27 ] and (b) the proposed method. Note that (a) exhibits unintended drift while hovering, whereas (b) remains stable.
Fig. 5 : Comparison between the momentum-based wrench estimation and the proposed haptic feedback method. In this figure, we let Δ denote the deviation from the initial value, and ϕ , θ , and ψ represent the roll, pitch, and yaw angles of the AM, respectively.
Fig. 6 : The behaviors of the AM and master device when the AM is manually displaced from its desired pose by pulling. It can be observed that the end-effector of the master device moves in accordance with the AM’s deviation from the desired pose.
Fig. 7 : Measured wrench feedback under externally induced AM pose deviations. In this figure, ϕ , θ , and ψ represent the roll, pitch, and yaw angles of the AM, respectively.
Fig. 8 : Snapshots and wrench feedback results from (a) cart pulling, (b) plug pulling, and (c) push-and-slide.
Fig. 9 : The AM lifting up the box during the pick-and-place task, along with the corresponding pose and haptic feedback wrenches.
Fig. 10 : Ablation study on the saturation filter conducted with a static pushing scenario.
Fully actuated unmanned aerial vehicles (UAVs) are usually certified through rank conditions on a control-allocation matrix or through free-flight tracking performance. For aerial physical interaction, this certification may be incomplete. During sustained contact, part of the available wrench is consumed by the interaction task, and only the residual wrench remains available for stabilization, disturbance rejection, and maneuvering. This paper introduces a control-theoretic framework for \emph{contact-persistent full actuation}. A rigid-body model on R3×\SO(3) is combined with a morphology-dependent wrench map that captures fixed-tilt, variable-tilt, coaxial, and overactuated multirotor architectures. We define feasible wrench sets under actuator limits, residual wrench sets under task loading, and residual authority margins that strengthen the usual rank-based notion of full actuation. The main result shows that contact-persistent full actuation is equivalent to interiority of the task wrench in the constrained feasible wrench polytope, and that the residual authority radius is exactly the distance to the polytope boundary. We further introduce a signed residual-margin certificate for infeasible and boundary cases, a slack-maximizing allocation certificate, and a robust implementability condition that can be used as a margin-aware safety filter. Numerical evaluation on an abstract tilted hexarotor shows that full row rank alone does not imply feasible contact operation. Intermediate tilt angles preserve residual authority during pushing, whereas small or excessive tilts fail because of lateral-force deficiency or hover-margin loss.
Guidance, Autonomy, Learning, and Control for Intelligent Systems (GALACxIS) Lab, Department of Aerospace Engineering and Engineering Mechanics, University of Cincinnati, OH 45221, USA · Intelligent Systems and Control (ISaC) Lab, Department of Aerospace Engineering, Indian Institute of Technology Bombay, Powai, Mumbai 400076, India
Aerial manipulation extends robotic operations to previously inaccessible aerial environments. Unlike arm-equipped aerial systems, tiltable-multirotors can directly generate six-degree-of-freedom wrenches through their flight bases, enabling both efficient movement and omnidirectional operation by tilting the thrust direction. This work presents a design analysis and a wrench-based control framework for tiltable-multirotors in aerial manipulation. We show that a four-rotor tiltable configuration provides a balance between interference-free propeller sizing and hovering efficiency across different attitudes, and its null-space redundancy is crucial for traversing singular configurations under physical constraints. We further show that an upward end-effector placement yields a favorable trade-off between geometric clearance and available wrench. To address disturbances, we propose a dual strategy consisting of a modified integral term for model error and an acceleration-based estimator for external wrenches. Building on these insights, we develop an effector-centric nonlinear model predictive control (NMPC) framework that integrates design choices, singularity handling, and disturbance compensation into a unified formulation. The proposed framework runs fully onboard at 100 Hz on a custom-built tiltable-quadrotor. Real-world experiments, including a 90-deg step cartwheel rotation, whiteboard pushing, and continuous 360-deg valve turning, demonstrate the feasibility of wrench-based omnidirectional manipulation with singularity traversal on a one-DoF-per-arm tiltable-quadrotor.
Jinjie Li, Yicheng Chen, Johannes Kübel +3
DRAGON Lab, Department of Mechanical Engineering, The University of Tokyo, Tokyo 113-8656, Japan
In robot teleoperation, haptic feedback can be used to help human operators accomplish dexterous manipulation tasks. However, existing haptic feedback methods try to replicate high-fidelity sensory haptics that are felt in real world interactions, which are constrained by the sensing and feedback hardware capability and may lead to higher workload. To addresses these limitations, this work introduces semantic haptics for teleoperation, which uses abstract haptic patterns to convey critical information about robot states. We categorize robot states into "Confirmations" and "Exceptions", implement a modular haptic rendering pipeline in robot simulation, and deliver semantic haptic feedback to operators through pneumatic and vibrotactile wristbands. This simplifies hardware requirements and enables one-to-many mappings between haptic patterns and robot states. Through three evaluation studies, we identify the most effective semantic haptic design for a common pick and place teleoperation task and compare semantic haptics to other teleoperation feedback approaches including sensory haptics and visual feedback. Results suggest that while semantic haptics performs similarly as other feedback in unimanual tasks, it achieves superior performance in bimanual tasks, with reduced task workload, increased situational awareness, and overall preference.