Ensuring safety and reliability in human-robot interaction (HRI) requires the timely detection of unexpected events that could lead to system failures or unsafe behaviours. Anomaly detection thus plays a critical role in enabling robots to recognize and respond to deviations from normal operation during collaborative tasks. While reconstruction models have been actively explored in HRI, approaches that operate directly on feature vectors remain largely unexplored. In this work, we propose MADRI, a framework that first transforms video streams into semantically meaningful feature vectors before performing reconstruction-based anomaly detection. Additionally, we augment these visual feature vectors with the robot's internal sensors' readings and a Scene Graph, enabling the model to capture both external anomalies in the visual environment and internal failures within the robot itself. To evaluate our approach, we collected a custom dataset consisting of a simple pick-and-place robotic task under normal and anomalous conditions. Experimental results demonstrate that reconstruction on vision-based feature vectors alone is effective for detecting anomalies, while incorporating other modalities further improves detection performance, highlighting the benefits of multimodal feature reconstruction for robust anomaly detection in human-robot collaboration.
Figures & tables
Fig. 1: Human Robot Cup Handover Failure. The human did not position the cup exactly as expected, resulting in a grasp failure and the robot dropping the cup.
Fig. 2: Overview of the proposed failure detection framework. Video input is processed by a Swin3D backbone (without classification head) to extract visual features, which can optionally be fused with sensor data. The resulting feature vector is passed through a feature reconstruction model composed of an encoder-decoder architecture. Reconstruction error is then compared against a predefined threshold to classify failure and generate the final output
Fig. 3: Confusion matrix for the action recognition model across seven action classes. The diagonal values indicate correct classifications, while off-diagonal values represent misclassification. The model demonstrates high accuracy for classes such as ’Move No Cup’ and ’Move with Cup’, with some confusion between visually similar actions like ’Place’ and ’Robot Picking Object’.
Fig. 4: Representative examples from the recorded task, illustrating four distinct actions: robot picking up an object (top-left), human-robot handover (top-right), robot moving with a cup (bottom-left), and robot moving without a cup (bottom-right).
Fig. 5: Reconstruction errors per clip during a joint torque failure. The left plot shows errors from the vision-only model, while the right plot shows errors from the multimodal model. The failure occurs at clip 11. The multimodal plot contains fewer clips due to robot shutdown following the internal failure.
Fig. 6: Reconstruction errors per clip during a failure event in which the robot drops the cup at clip 19. The left plot shows results from the vision-only model, while the right plot presents the multimodal model’s reconstruction.
School of Artificial Intelligence, Xidian University, China · School of Artificial Intelligence and Robotics, Hunan University, China · National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China