cs.RONov 3, 2026

Haptic Shared Control for Multi-Robot Systems

Authors: Maxime Bernard

Organizations: Unité de recherche : IRISA UMR6074

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multi-agent systems offer significant advantages in terms of flexibility and redundancy, enabling the efficient distribution of complex tasks among multiple robots. However, their deployment requires advanced control strategies to ensure optimal cooperation between autonomous agents and human operators. We propose a modification of the global connectivity maintenance algorithm to incorporate measurements taken in agent's local frame, using sensors with limited fields of view. This algorithm, responsible for the autonomous behavior of the multi-robot system, is combined with different adaptive supervision strategies, allowing the user to exert control at varying scales over the entire group. (i) Single Agent Control (S): The operator directly controls a single robot, (ii) Pinning Control (P): The operator imposes constraints on multiple robots to influence the group, (iii) Centroid Control (C): The operator acts on the system's center of mass to steer the entire fleet. In conjunction with the connectivity maintenance algorithm, these methods ensure compliance with inter-agent constraints while providing haptic feedback to the user regarding the system's connectivity state. The effectiveness of these approaches was evaluated through a user study, comparing performance with and without haptic feedback across multiple scenarios tailored to the different control modes.

Figures & tables

Figure 1.1: MAS application examples. Fig. 1.1a : Simulation of a micro-robot team in a 3D endovascular environment recreated from CT scans. The robots are represented by the red and blue spheres. Fig. 1.1b : KUKA robots working together in the automotive industry.
Figure 1.2: MRS applications examples. Fig. 1.2a : Hercules robots being tested at Amazon’s BOS27 Robotics Innovation Hub ( AFP, 2022a ) . Fig. 1.2b : Stationary robot arm inside Amazon’s facility, designed to grasp and handle packages on conveyors and shelves ( Alan S. Brown, 2021a ) .
Figure 1.3: Examples of decentralization. Fig. 1.3a : a group of ants collaboratively forming a bridge to reach a distant location, studied in Reid et al., 2015a . Fig. 1.3b : Osaka’s Festival of Fire and Water, contingent collectively carrying a Mikoshi (portable shrine).
Figure 1.4: Fig. 1.4a : complete graph, Fig. 1.4b : strongly connected graph, Fig. 1.4c : connected graph, Fig. 1.4d : disconnected graph.
Figure 1.5: Graphs with different connectivity eigenvalues λ2\lambda_{2}
Figure 1.6: Drone race opposing human pilots and an autonomous drone, presented in Kaufmann et al., 2023a . The autonomous drone was trained using a trajectory dataset captured via global sensing techniques (laser tracker) and dynamically adapts its flight-path mid-race by detecting racing gates with local sensing techniques (onboard camera).
Figure 1.7: UVDAR and UVDAR-COM systems. Fig. 1.7a : Configuration of the UVDAR system, featuring UV LEDs circled in green and cameras circled in red. Fig. 1.7b : UVDAR-COM system workflow, including time tracking of UV markers using 4D Hough Transformation, frame reconstruction, error detection, and data decoding.
Figure 2.1: Fig. 2.1a : Mechanical teleoperation system developed by Goertz in 1954. The system allowed operators to manipulate radioactive materials from a safe location. Fig. 2.1b : PHANToM device, 6 Degreed of Freedom haptic device developed by Sensable Technologies.
Figure 2.2: Fig. 2.2a : dense point cloud reconstructions generated by ORB-SLAM2, utilizing estimated keyframe poses and sensor depth maps, presented in Mur-Artal & Tardós, 2017a . Fig 2.2b : map and pose graph from a Swarm-SLAM experiment involving three robots, detailed in Lajoie & Beltrame, 2023a .
Figure 2.3: Fig. 2.3a : The Revolute-Revolute-Spherical fingertip device presented in Chinello et al., 2017a . Fig 2.3b : An electrode, placed in contact with the finger pad, is controlled by an electrical stimulator attached to the forearm ( Vizcay et al., 2021a )
Figure 2.4: Virtuose 6D haptic device, Haption.
Figure 2.5: Formation control of a group of robots maintaining a desired shape while tracking a moving target and avoiding obstacles, see Dang et al., 2019a .
Figure 2.6: Teleoperation setup. The human operator uses a Virtuose 6D grounded haptic interface (Haption, France) to control a 7 degree of freedom Panda robotic manipulator (Franka Emika, Germany), used as a 6-DoF manipulator. The user receives haptic feedback through the haptic interface and visual feedback from an LCD screen ( Rahal et al., 2023a ) .
Figure 2.7: The shape of Vλ(λ2)V^{\lambda}(\lambda_{2}) with λ2min=0.01\lambda^{min}_{2}=0.01 and λ2max=3.01\lambda^{max}_{2}=3.01 .
Figure 3.1: Onboard sensors: The green areas in the figures represent the sensing directions of the respective sensors.
Figure 4.1: The shape of the two considered rising/falling activation functions.
Figure 4.2: Relative orientation estimation. Fig. 4.2a Robot ii senses robot jj and measures its velocity. Fig. 4.2b The angle between iRjvj{}^{i}\textbf{R}_{j}{\boldsymbol{v}}_{j} and vj\boldsymbol{v}_{j} on the XYXY -plane is the relative orientation ψij\psi_{ij} .
dmin=1\penalty md_{min}=1\penalty\ mamin=0.4a_{min}=0.4dminobs=1.0\penalty md^{obs}_{min}=1.0\penalty\ mk1=10k_{1}=10
D1=3\penalty mD_{1}=3\penalty\ mamax=0.95a_{max}=0.95dmaxobs=3.5\penalty md^{obs}_{max}=3.5\penalty\ mkˉ2=10\bar{k}_{2}=10
D2=7\penalty mD_{2}=7\penalty\ moC=[1,1,0]2\mathbf{o}_{C}=\dfrac{[1,1,0]}{\sqrt{2}}λ2min=0.01\lambda^{min}_{2}=0.01k3=10k_{3}=10
dmax=9\penalty md_{max}=9\penalty\ m λ2max=3.01\lambda^{max}_{2}=3.01 
Table 4.1: Algorithm parameters used in our validation.
Figure 4.3: Experimental setup for simulation with Connectivity Maintenance.
Figure 4.4: Drones trajectories during the experiment. Drone 1 is the drone controlled by the human user. Each dot in the trajectory corresponds to the position of the drone at a 5 seconds interval.
Figure 4.5: Validation in simulation. (Top) True λ2\lambda_{2} vs. the six estimations λ~2i\tilde{\lambda}^{i}_{2} calculated by the robots over time; the green phases indicate when the human user is providing an external input to the robot leader (robot 11 ). (Bottom) Number of edges in the graph, which changes dynamically as the task evolves.
Figure 4.6: Estimations errors for ψ~12\tilde{\psi}_{12} and ψ~13\tilde{\psi}_{13} . Blue phases indicate when robot 11 is measuring robot jj ( f1j∗≠0f^{*}_{1j}\neq 0 , see Sect. 4.1.1 ) but robot jj does not measure robot 11 ( fj1∗=0f^{*}_{j1}=0 ); yellow phases indicate when robot jj is measuring robot 11 but robot 11 does not measure robot jj ; green phases indicate when both robot 11 and jj are measuring each other. Note how the estimation errors always remain very small apart from the white phase (bottom plot) since, in this case, the two robots are completely disconnected. The error however quickly converges back to zero as soon as one robot starts measuring the other one (vertical red dashed line).
Figure 5.1: The Virtuose device is a high-end 6-DoF haptic interface able to provide 3-DoF translational and 3-DoF rotational feedback.
Figure 5.2: The shape of FV\boldsymbol{F}_{V} on the xx axis with pV0=0\boldsymbol{p}^{0}_{V}=0 , pVmin=0.03\boldsymbol{p}^{min}_{V}=0.03 , pVmax=1\boldsymbol{p}^{max}_{V}=1 , kd=100k_{d}=100 , kc=20k_{c}=20 , and kl=500k_{l}=500 .
Figure 5.3: Pinning control. pp\boldsymbol{p}_{p} and pj\boldsymbol{p}_{j} are the position of the virtual point and the agent jj , respectively. djpmind^{min}_{jp} is the minimum distance between agent jj and the virtual point, η~V\widetilde{\boldsymbol{\eta}}_{V} is the user rotational input and vjE\boldsymbol{v}^{E}_{j} is the velocity control input controlling agent jj .
Figure 5.4: The projection of the linear elements of the connectivity maintenance force onto the vector pjp\boldsymbol{p}_{jp} and the tangent plan containing pjp×η~V\boldsymbol{p}_{jp}\times\widetilde{\boldsymbol{\eta}}_{V} .
Figure 5.5: Software implementation. For each simulated drone, one instance of the control and simulation nodes is executed.
Figure 5.6: “Plains” environment. Unity 3D visualization of the robots when using the Single Agent Control method. The leader robot is highlighted.
Figure 5.7: “Desert” environment. Unity 3D visualization of the robots when using the Pinning Control method. The pinned robots are highlighted.
Figure 5.8: Unity 3D visualization of the robots when using the Centroid Control method. The centroid of the group is depicted by the red arrow.
  Control conditions
  Single AgentPinningCentroid
 without Haptic FeedbackSNHPNHCNH
Feedback conditionswith Haptic FeedbackSHPHPNH
Table 5.1: Name of the different control and feedback conditions.
Figure 5.9: Experimental setup for the user study. The environment is displayed on a large screen. The user controls the agents using the Virtuose haptic interface.
Figure 5.10: Experimental results. (a) User perceived effectiveness and intuitiveness, and (b) Measured completion time for the Single Agent Control scenario.
Figure 5.11: User Perceived Effectiveness and Intuitiveness, and Measured Completion Time for Pinning Control Experiment.
Figure 5.12: User Perceived Effectiveness and Intuitiveness, and Proportion of Discovered Environment for Centroid Control Experiment.
Figure 5.13: Agents trajectories throughout the experiment. Each dot in the trajectory corresponds to the position of an agent at a 5 seconds interval.
Figure 5.15: The shape of λ2\lambda_{2} throughout the experiment and the number of connected agents.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: ROS 2 nodes running onboard each agent under the same namespace (drone ii ) and their input/output topics.
Figure A.2: Software implementation for the haptic interface.
Figure A.3: State machine of the centralized_controller_node .
More in this topic

Robotic Control

Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.

See all 606 papers

Explore similar work

CardsList
Feb 25, 2026

ADM-DP: Adaptive Dynamic Modality Diffusion Policy through Vision-Tactile-Graph Fusion for Multi-Agent Manipulation

Multi-agent robotic manipulation remains challenging due to the combined demands of coordination, grasp stability, and collision avoidance in shared workspaces. To address these challenges, we propose the Adaptive Dynamic Modality Diffusion Policy (ADM-DP), a framework that integrates vision, tactile, and graph-based (multi-agent pose) modalities for coordinated control. ADM-DP introduces four key innovations. First, an enhanced visual encoder merges RGB and point-cloud features via Feature-wise Linear Modulation (FiLM) modulation to enrich perception. Second, a tactile-guided grasping strategy uses Force-Sensitive Resistor (FSR) feedback to detect insufficient contact and trigger corrective grasp refinement, improving grasp stability. Third, a graph-based collision encoder leverages shared tool center point (TCP) positions of multiple agents as structured kinematic context to maintain spatial awareness and reduce inter-agent interference. Fourth, an Adaptive Modality Attention Mechanism (AMAM) dynamically re-weights modalities according to task context, enabling flexible fusion. For scalability and modularity, a decoupled training paradigm is employed in which agents learn independent policies while sharing spatial information. This maintains low interdependence between agents while retaining collective awareness. Across seven multi-agent tasks, ADM-DP achieves 12-25% performance gains over state-of-the-art baselines. Ablation studies show the greatest improvements in tasks requiring multiple sensory modalities, validating our adaptive fusion strategy and demonstrating its robustness for diverse manipulation scenarios.

Enyi Wang, Wen Fan, Dandan Zhang

Department of Bioengineering, Imperial-X Initiative, Imperial College London, London, United Kingdom.

Multi-Robot CoordinationDiffusion PolicyMultimodal Sensor FusionRobotic Grasping
AskPDF
Jun 5, 2026

A Multi-Operator Mixed-Reality Interface for Multi-Robot Control and Coordination: Co-Located and Private Workspace Collaboration

Multi-operator control of robot teams requires not only access to the same mission information, but also mechanisms for maintaining shared awareness and preventing conflicting interventions. Building on our previous HORUS interface (Holistic Operational Reality for Unified Systems) we present a mixed-reality interface that extends single-operator multi-robot supervision to collaborative multi-operator use. The system supports two complementary modes: a co-located shared workspace, in which operators observe and manipulate the same mini-map in the same physical location, and a private-workspace mode, in which operators work on the same mission through independently placed local workspaces. The architecture combines registration-driven scene construction, lightweight shared-session synchronization, and per-robot control leases to support collaborative monitoring, tasking, and teleoperation while preventing conflicting commands. We evaluated the approach in a human-subject study with 36 participants (18 pairs) controlling three Nova Carter mobile robots in two search environments. The performance of the objective task was comparable across the two modes, indicating that both modes supported effective mission execution. However, the co-located shared workspace significantly improved perceived collaboration, shared understanding, and handoff clarity, and was the preferred collaborative mode. These results indicate that physically co-locating the MR workspace improves how operators coordinate even when the underlying robot-control tools remain unchanged.

Omotoye Shamsudeen Adekoya, Antonio Sgorbissa, Carmine Tommaso Recchiuto

DIBRIS Department, RICE Laboratory, University of Genoa, Italy

Robot TeleoperationMulti-Robot CoordinationHuman-Robot CollaborationHuman-Robot Interaction
AskPDF
Oct 11, 2026

A Reinforcement Learning Framework for Safety-Aware Haptic Shared Control in Surgical Robotics

Haptic shared control in surgical robotics combines human-side and robot-side forces at a shared end effector while preserving operator involvement. However, the effects of geometric safety choices on task execution and force coordination remain unclear. We model the interaction as a two-agent cooperative Markov game under centralized training and decentralized execution, comparing learned human control with a task-tuned fixed-impedance baseline. This model-free multi-agent reinforcement learning framework supports joint policy adaptation without requiring a complete analytical interaction model. It compares MADDPG and rMAPPO with an epigraph-constrained approach under common geometric task definitions. In simulation, rMAPPO generally achieved the highest return and tracking performance, whereas the epigraph approach produced more conservative motions and lower geometric exposure, including no observed collision or dangerous-region exposure after geometry transfer. With real operator input, fixed-impedance-trained policies were faster and successful in all recorded trials. Policies trained with a learned partner exhibited greater command opposition, while the fixed epigraph condition produced the most aligned commands. This work provides a benchmark for studying human-control representation and constraint-handling trade-offs in haptic shared control.

Zihao Zhang, Karl Handwerker, Balint Varga

Institute of Control Systems, Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany

Robotic ControlHuman-Robot CollaborationSurgical RoboticsSafe RLCooperative MARL
AskPDF