RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation

Shijie Zhou; Bin Zhu; Jiarui Yang; Xiangyu Zhao; Jingjing Chen; Yu-Gang Jiang

cs.CVMar 11, 2026

RC-NF: Robot-Conditioned Normalizing Flow for Real-Time Anomaly Detection in Robotic Manipulation

Authors: Shijie Zhou, Bin Zhu, Jiarui Yang, Xiangyu Zhao, Jingjing Chen, Yu-Gang Jiang

Organizations: Institute of Trustworthy Embodied AI, Fudan University · Shanghai Key Laboratory of Multimodal Embodied AI · Singapore Management University

Abstract

Recent advances in Vision-Language-Action (VLA) models have enabled robots to execute increasingly complex tasks. However, VLA models trained through imitation learning struggle to operate reliably in dynamic environments and often fail under Out-of-Distribution (OOD) conditions. To address this issue, we propose Robot-Conditioned Normalizing Flow (RC-NF), a real-time monitoring model for robotic anomaly detection and intervention that ensures the robot's state and the object's motion trajectory align with the task. RC-NF decouples the processing of task-aware robot and object states within the normalizing flow. It requires only positive samples for unsupervised training and calculates accurate robotic anomaly scores during inference through the probability density function. We further present LIBERO-Anomaly-10, a benchmark comprising three categories of robotic anomalies for simulation evaluation. RC-NF achieves state-of-the-art performance across all anomaly types compared to previous methods in monitoring robotic tasks. Real-world experiments demonstrate that RC-NF operates as a plug-and-play module for VLA models (e.g., pi0), providing a real-time OOD signal that enables state-level rollback or task-level replanning when necessary, with a response latency under 100 ms. These results demonstrate that RC-NF noticeably enhances the robustness and adaptability of VLA-based robotic systems in dynamic environments.

Figures & tables

Figure 2: Overview of our framework. Our Robot-Conditioned Normalizing Flow (RC-NF) operates as a real-time runtime monitor for robotic manipulation tasks. (Left) SAM2 extracts object masks from video streaming, which are then grid-sampled into point sets. Task prompts are encoded using spherical uniform encoding, and robot proprioception provides joint, gripper, and pose states. (Center) Our RC-NF leverages these signals as conditions within the affine coupling layers in the proposed RCPQNet (Sec . 3.3 ) to apply $K$ invertible transformations and compute anomaly scores for the current task. (Right) When the anomaly score exceeds the threshold, the Anomaly Detection and Handling module triggers corrective behaviors by task replanning for task-level OOD and task rollback for state-level OOD.

Figure 3: Robot-Conditioned Point Query Network (RCPQNet) is introduced as the affine coupling layer in RC-NF that generates shift and scale parameters.

Method	Gripper Open		Gripper Slippage		Spatial Misalignment		Average
	AUC	AP	AUC	AP	AUC	AP	AUC	AP
GPT-5 [ 29 ]	0.9137	0.9642	0.8941	0.8720	0.4904	0.4015	0.8500	0.8507
Gemini 2.5 Pro [ 8 ]	0.8644	0.9333	0.8633	0.8505	0.5167	0.4271	0.8186	0.8313
Claude 4.5 [ 4 ]	0.8754	0.9401	0.8551	0.8285	0.5292	0.4290	0.8214	0.8249
FailDetect [ 47 ]	0.7883	0.9032	0.6665	0.6932	0.6557	0.5820	0.7181	0.7700
RC-NF(Ours)	0.9312	0.9781	0.9195	0.9180	0.9676	0.9585	0.9309	0.9494

Table 1: Quantitative comparison of anomaly detection performance on the LIBERO-Anomaly-10 benchmark.

Figure 4: Visualization of the anomaly scores for the task Pick up the book and place it in the back compartment of the caddy . The x-axis denotes the time steps, and the y-axis indicates the anomaly score. The red dashed line marks $t_{\text{anomaly}}$ .

Row	Method	Gripper Open		Gripper Slippage		Spatial Misalignment		Average
		AUC	AP	AUC	AP	AUC	AP	AUC	AP
1	RC-NF(Ours)	0.9312	0.9781	0.9195	0.9180	0.9676	0.9585	0.9309	0.9494
2	w/o Task Embedding	0.8769	0.9603	0.8668	0.8680	0.8139	0.8118	0.8643	0.9008
3	w/o Robot State	0.6327	0.8621	0.7443	0.8116	0.8929	0.8617	0.7152	0.8401
4	w/o Pos. Residual branch	0.9045	0.9712	0.8971	0.9085	0.8543	0.8072	0.8947	0.9225
5	w/o Dyn. Shape branch	0.7666	0.9234	0.7763	0.8108	0.1022	0.2755	0.6841	0.7899

Table 2: Ablation study of the proposed Robot-Conditioned Point Query Network (RCPQNet). RC-NF (Ours) denotes the full model.

Figure 5: A real-world comparison of $\pi_{0}$ and $\pi_{0}$ + RC-NF (ours) during the task placing the ball into the open drawer , when the drawer closes midway.

Figure 6: Comparison of $\pi_{0}$ + RC-NF (Ours) and $\pi_{0}$ model in handling two consecutive task-level OOD events during real-world deployment. $t_{1a}$ represents the moment when the ball is repositioned on the table. $t_{2a}$ marks when the ball rolls behind the gripper. $t_{3r}$ indicates when control is returned to the VLA after homing. The timeline of $\pi_{0}$ + RC-NF (Ours) is shown in Fig. 7 .

Figure 7: Anomaly score plot for a state-level OOD sequence where the ball is repositioned at $t_{1a}$ and rolls behind the gripper at $t_{2a}$ . The yellow curve indicates the detection of an anomaly, with the trajectory corrected by the homing procedure. The green curve represents the operation of the VLA model $\pi_{0}$ .

SAM2 Inference	Grid Sampling	RC-NF Inference	Other Latency	Overall (RTX 3090)
50 ms	1.7 ms	30 ms	5 ms	86.7 ms

Table 9

Figure 8: Diagram illustrating Spherical Uniform Encoding. The left side displays the ten tasks from the training set along with their descriptions, while the right side provides an example demonstration.

Figure 9: Comparison of performance metrics before and after incorporating three tasks with spatial misalignment (left, front and right) into the training set.

Figure 10: Prompts for generating bounding boxes with Gemini 2.5 Pro

Figure 11: VLMs prompts for robotic anomaly detection scoring

Explore similar work

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction
May 18, 2025Zewei Ye, Weifeng Lu, Minghao Ye +4Robot Failure Recovery Robustness of VLA Models
School of AI, Shanghai Jiao Tong University · Harbin Institute of Technology, Shenzhen
OFlow: Injecting Object-Aware Temporal Flow Matching for Robust Robotic Manipulation
Apr 20, 2026Kuanning Wang, Ke Fan, Chenhao Qiu +5VLM Robustness Flow Matching
Fudan University Shanghai, China · University of Southern California Los Angeles, CA, USA · KAUST Saudi Arabia
ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control
Apr 17, 2026Lingling Chen, Zongyao Lyu, William J. BeksiUncertainty Estimation for VLA Models Robot Failure Detection
Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA

May 18, 2025J/K move · Enter open · S save

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction

Zewei Ye, Weifeng Lu, Minghao Ye, Tao Lin+3

School of AI, Shanghai Jiao Tong University · Harbin Institute of Technology, Shenzhen

Robot Failure Recovery Robustness of VLA Models Vision-Language-Action Models Fault Diagnosis

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have recently advanced robotic manipulation by translating natural-language instructions and visual observations into control actions. However, existing VLAs are primarily trained on successful expert demonstrations and lack structured supervision for failure diagnosis and recovery, limiting robustness in open-world scenarios. To address this limitation, we propose the Robotic Failure Analysis and Correction (RoboFAC) framework. We construct a large-scale failure-centric dataset comprising 9,440 erroneous manipulation trajectories and 78,623 QA pairs across 53 scenes in both simulation and real-world environments, with systematically categorized failure types. Leveraging this dataset, we develop a lightweight multimodal model specialized for task understanding, failure analysis, and failure correction, enabling efficient local deployment while remaining competitive with large proprietary models. Experimental results demonstrate that RoboFAC achieves a 34.1% higher failure analysis accuracy compared to GPT-4o. Furthermore, we integrated RoboFAC as an external supervisor in a real-world VLA control pipeline, yielding a 29.1% relative improvement across four tasks while significantly reducing latency relative to GPT-4o. These results demonstrate that RoboFAC enables systematic failure diagnosis and recovery, significantly enhancing VLA recovery capabilities. Our model and dataset are publicly available at https://github.com/MINT-SJTU/RoboFAC.

Abstract

Explore similar work

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction

OFlow: Injecting Object-Aware Temporal Flow Matching for Robust Robotic Manipulation

ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control