Active Perception of Upper-Limb Functional Mobility through Human-Robot Handovers
Authors: Francesco Semeraro, Marta Lagomarsino, Alberto Ranavolo, Arash Ajoudani
Organizations: Human-Robot Interfaces and Interaction (HRI2) Laboratory, Italian Institute of Technology, 16163, Genoa, Italy. · Department of Occupational and Environmental Medicine, Epidemiology and Hygiene, INAIL, 00078, Rome, Italy.
The reintegration of people in the workplace after an upper-limb injury is often a long process. Collaborative robots and humanoids operating in shared workspaces must adapt their behaviour to the physical capabilities of their human partners. Achieving this requires robots to autonomously estimate task-relevant human mobility during interaction, without relying on external clinical assessments. In this work, we formulate the estimation of upper-limb functional mobility as an active perception problem. We model task-available upper-limb functional ranges of motion through impairment coefficients that parameterise reductions of healthy joint limits, and integrate this representation into a human model used by the robot during interaction. Based on this formulation, we propose an online active perception pipeline in which the robot actively probes the user through repeated handover requests while progressively updating a Bayesian belief over the impairment coefficients. This enables the robot to continuously refine its internal model of the user's joint-level capabilities through interaction rather than passive observation. The proposed framework was evaluated through simulation studies and a proof-of-concept physical validation with a robotic platform and enacted human impairments, demonstrating that the pipeline can accurately recover the impairment coefficients during real human-robot handovers.
Figures & tables
Fig. 1: A person with an impairment enacted by means of a harness performs a handover requested by the robot. An underlying optimisation process computes the inverse kinematics of the human arm to the best extent allowed by the current belief on the impairment. This last gets updated based on the mismatch between the estimate and the real upper-limb joint values measured during the interaction.
Fig. 2: Diagram of the active perception pipeline for fROM estimation.
Fig. 3: Kinematic chain and rendering of the human model.
Fig. 4: Pictures from the physical validations with enacted impairments. From left to right: αSy∗=1.0 , αSy∗=0.5 , αEy∗=1.0 and αEy∗=0.5 . In some instances of the interactions, the participant was not able to reach the object. This is expected, as the pipeline does not have prior knowledge of the user’s real impairment. Nonetheless, the pipeline is able to perform the updates, because the underlying optimisation used enforces the constraint xobj(q)=ht at the best extent allowed by the impairment, but it is able to return an IK solution even when the constraint cannot be met (see Subsection II-B ).
Fig. 5: Rendering of the experimental setting in Rviz 2. The human model was being acquired through the Real-Move setup. The green cube represents the action space the robot ould requests handovers within. The orange cube centered in the frame with the blue z axis oriented towards the user’s head represents ht and the mug handle used in the handovers.
Configuration
Error
N. of Handovers
Mean
SD
Mean
SD
αSy∗≤0.5 , αEy∗≤0.5
0.0867
0.0867
23.86
3.51
αSy∗≤0.5 , αEy∗>0.5
0.0504
0.0582
21.56
5.84
αSy∗>0.5 , αEy∗≤0.5
0.0833
0.0615
23.62
4.11
αSy∗>0.5 , αEy∗>0.5
0.0453
0.0590
19.91
6.80
TABLE I: Performance metrics across shoulder (S) and elbow (E) flexion impairment configurations. Each configuration was evaluated over 400 trials.
Fig. 6: Trends on the joint estimate of αSy∗ and αEy∗ in the physical validations. They all start from 0.5 because of the way At was defined (see Subsection II-C ). The shaded areas were plotted based on the related σα,t . Trends terminating before iteration 25 indicate that the run was able to meet the early termination criterion (see Subsection II-C ).
Fig. 7: Visual comparisons between the result of the IK solution given by the α^ at the current iteration and the recorded joint values (shaded mannequin) of the position performed by the participant during the handover.
Robot-to-human object handover is an essential skill for robot assistants, from serving drinks at home to passing surgical tools in the operating room. We expect robots to perform handover robustly -- to release the object only after a firm human grasp while ignoring incidental touches. Existing passive-sensing methods struggle to generalize across diverse objects and human behaviors, as they lack informative perturbations to disambiguate different contact conditions, such as firm grasp versus incidental touch. We propose an active sensing approach for robust handovers: the robot applies information-gathering motions and senses the resulting human-applied forces to infer the contact state. A firm grasp produces forces in multiple directions, while an accidental touch does not. To capture this distinction, we model the contact state with a Bayesian linear model: a distribution over piecewise-linear mappings from robot motions to human-applied forces. This model enables firm grasp detection and active information gathering. In experiments with 12 participants and 30 diverse rigid objects, our method achieved a 97.5% success rate -- over 30% higher than two common baselines.
Multi-agent manipulation naturally produces multiple task-driven viewpoints: every arm carries a wrist camera and moves through the scene while acting. Yet these observations are typically underutilized, and active perception in manipulation is still often treated as requiring a dedicated sensing agent. We introduce MAAP (Multi-Agent Active Perception), in which every arm is dual-purpose: it executes manipulation actions and, through the wrist camera it carries, simultaneously serves as a moving viewpoint for the team. We pair this with RAIL (Role-Aware Imitation Learning), a controller that predicts each arm's current role alongside its action chunk and conditions action generation on it, representing role-dependent actions within one network. Across four simulated tasks, widening the perception regime lifts average success from 56.5% with a fixed camera to 62.5% with one active wrist view and 70.0% with all of them, while MAAP+RAIL reaches 79.2%. RAIL's additional gain is concentrated on the three-arm Microwave task, where success rises from 47% to 82% on identical multi-wrist inputs. On a dual-arm platform, MAAP+RAIL succeeds in 14 of 20 placement trials compared with 0 of 20 for fixed-view ACT. Collaborative manipulation can thus serve as an active perception mechanism in its own right.
Bruno N. Y. Chen, Li Kang, Heng Zhou +4
Carnegie Mellon University · Shanghai Jiao Tong University · University of Science and Technology of China +3
We focus on human-robot collaborative transport, a challenging task of broad relevance spanning logistics, manufacturing, and the home, in which a user and a robot work together to relocate a large or heavy object. To act as an effective partner, the robot should reduce the user's effort by contributing to efficient relocation of the object while remaining physically responsive to them. Prior work often addresses these capabilities separately, producing robots that may move the object efficiently but resist user input, or accommodate the user but depend on continuous guidance. Our key insight is that obstacle-constrained collaborative transport requires integrating predictions of human collaborative behavior with compliant robot control. To this end, we introduce PROACT, a framework for human-robot collaborative transport that incorporates anticipation into compliant whole-body control through a learned model of human collaborative behavior. Trained on a large-scale, real-world dataset of dyadic human transport demonstrations, our transformer architecture distills collaborative behavior into predictions of future object motion. Across 108 real-world trials with a 9-DoF mobile manipulator, PROACT reduces mean interaction work by 59.2% and 20.4%, and mean completion time by 12.9% and 6.9%, relative to compliance-only and MPC baselines, respectively. Footage from our experiments can be found at https://youtu.be/qAGvQfVPjbk.
Elvin Yang, Christoforos Mavrogiannis
Department of Robotics, University of Michigan, Ann Arbor, United States