Human-robot collaboration aims to extend human ability through cooperation with robots. This technology is currently helping people with physical disabilities, has transformed the manufacturing process of companies, improved surgical performance, and will likely revolutionize the daily lives of everyone in the future. Being able to enhance the performance of both sides, such that human-robot collaboration outperforms a single robot/human, remains an open issue. For safer and more effective collaboration, a new control scheme has been proposed for redundant robots in this paper, consisting of an adaptive vision-based control term in task space and an interactive control term in null space. Such a formulation allows the robot to autonomously carry out tasks in an unknown environment without prior calibration while also interacting with humans to deal with unforeseen changes (e.g., potential collision, temporary needs) under the redundant configuration. The decoupling between task space and null space helps to explore the collaboration safely and effectively without affecting the main task of the robot end-effector. The stability of the closed-loop system has been rigorously proved with Lyapunov methods, and both the convergence of the position error in task space and that of the damping model in null space are guaranteed. The experimental results of a robot manipulator guided with the technology of augmented reality (AR) are presented to illustrate the performance of the control scheme.
Figures & tables
Fig. 1: Block diagram of the proposed control scheme. In the presence of an uncalibrated camera, the exact information about the image Jacobian matrix is unknown and denoted as J^s(r) . In vision space, the robot end-effector is controlled to move to the desired position, that is, x→xd ; In null space, the redundant joints are controlled to suit human intention under the desired damping model, that is, cdq˙=d , where cd is a positive constant and d denotes the control efforts from a human.
Fig. 2: The human can input via various interfaces, e.g., head-mounted display, haptic device, or tactile sensor, to shape the redundant configuration without affecting the robot end-effector.
Fig. 3: (a) Experimental setup of an AR-guided robot manipulator (b) A snapshot of the human-robot interface deployed on HoloLens 2. The sliders were placed on the left side, and the virtual manipulator model was on the right side, overlapped with the real manipulator. The human operator was pointing at joint 3 and giving a sign of moving up. The virtual manipulator model could also be placed somewhere else according to the situations.
Fig. 4: Snapshots of task 1. The manipulator was controlled to move to the desired point in vision space, while the operator exerted additional control efforts to make another human subject more comfortable via the AR interface. (a) t=0.0s : The end-effector started at the lower-right corner of the camera frame, and would move to the desired point at the center in 1.6s . (b) t=6.0s : The operator began to drag sliders corresponding to joint 2 and joint 3 to change the manipulator’s configuration because another human subject entered the workspace and stretched his body in an uncomfortable pose to fetch tools. (c) t=20.1s : The manipulator adjusted its pose according to the command, while the position of the end-effector in vision space remained the same. Meanwhile, it enabled the human subject to have a safer and more comfortable workspace. (d) t=30.3s : After the subject left, the robot’s joint moved back to its original position according to the operator’s command.
Fig. 5: Results of task 1: (a) top: The position errors of the end-effector in vision space; bottom: The human’s control efforts applied to the joints. (b) The trajectory of the end-effector in vision space.
Fig. 6: Results of ablation study: (a) The position errors of the end-effector in vision space during the first 6.5s, without adaptation law (top) and with adaptation law (bottom). (b) The estimated and actual depth in the whole process of task 1.
Fig. 7: Snapshots of task 2. The manipulator end-effector was performing a path-following task in vision space while the operator was guiding the manipulator body through the AR interface to avoid the collision. (a) t=0.0s : Initial state. (b) t=5.4s : The end-effector began to follow the circular path. (c) t=15.8s : A human subject placed a toolbox on the workbench, which raised the safety issue. (d) t=31.1s : The manipulator adjusted its pose according to the operator’s inputs, while the end-effector’s position in vision space remained unaffected. (e) t=57.3s : Guided by the operator, the manipulator resumed its original pose after the toolbox was removed.
Fig. 9: Results of task 2: (a) top: the position errors of the end-effector in vision space; bottom: the human intention applied to the joints (b) The trajectory of the end-effector in vision space.