Vision-Language-Action Models

Also known as VLA

These models combine vision-language backbones with action generators that produce continuous motion or discrete action tokens, often in chunks and with proprioceptive input. They are commonly trained on robot demonstrations, with flow-matching and diffusion policies, reinforcement fine-tuning, and predictive world models used to improve control and planning. Research spans tabletop and bimanual manipulation, mobile and aerial navigation, autonomous driving, and humanoid control, evaluated on benchmarks such as LIBERO, CALVIN, RoboTwin, and driving simulators. A central challenge is making broad visual and language competence translate into reliable closed-loop behavior under new objects, viewpoints, embodiments, delays, and safety constraints; benchmark success can still conceal weak instruction grounding and sim-to-real failures.

Latest papers

1,202 papers131 papers in the last four weeks, down 41% on the four weeks before. 0.9% of all new papers.

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction
May 18, 2025Zewei Ye, Weifeng Lu, Minghao Ye +4Robot Failure Recovery Robustness of VLA Models
School of AI, Shanghai Jiao Tong University · Harbin Institute of Technology, Shenzhen
3D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generation
Jan 28, 2025Yueen Ma, Zenglin Xu, Irwin King3D Spatial Reasoning Mixture-of-Experts Inference
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR

May 18, 2025J/K move · Enter open · S save

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction

Zewei Ye, Weifeng Lu, Minghao Ye, Tao Lin+3

School of AI, Shanghai Jiao Tong University · Harbin Institute of Technology, Shenzhen

Robot Failure Recovery Robustness of VLA Models Vision-Language-Action Models Fault Diagnosis

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have recently advanced robotic manipulation by translating natural-language instructions and visual observations into control actions. However, existing VLAs are primarily trained on successful expert demonstrations and lack structured supervision for failure diagnosis and recovery, limiting robustness in open-world scenarios. To address this limitation, we propose the Robotic Failure Analysis and Correction (RoboFAC) framework. We construct a large-scale failure-centric dataset comprising 9,440 erroneous manipulation trajectories and 78,623 QA pairs across 53 scenes in both simulation and real-world environments, with systematically categorized failure types. Leveraging this dataset, we develop a lightweight multimodal model specialized for task understanding, failure analysis, and failure correction, enabling efficient local deployment while remaining competitive with large proprietary models. Experimental results demonstrate that RoboFAC achieves a 34.1% higher failure analysis accuracy compared to GPT-4o. Furthermore, we integrated RoboFAC as an external supervisor in a real-world VLA control pipeline, yielding a 29.1% relative improvement across four tasks while significantly reducing latency relative to GPT-4o. These results demonstrate that RoboFAC enables systematic failure diagnosis and recovery, significantly enhancing VLA recovery capabilities. Our model and dataset are publicly available at https://github.com/MINT-SJTU/RoboFAC.

Latest papers

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction

3D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generation