cs.RONov 19, 2026

AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control

Authors: Yiyang Du, Yuxuan Kuang, Qitao Zhao, Yixuan Wang, Zhanqiu Guo, Yi Xu, Xin Li, Wenbin He, 

Organizations: Carnegie Mellon University · University of Illinois at Urbana-Champaign · Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI)

Abstract

Frontier multimodal language agents such as GPT-6 Astra have demonstrated strong capabilities in controlling robots directly via low-level numerical actions without high-level primitives. However, existing benchmarks report only episode-level success rates, which show neither how these agents succeed nor where they fail. In this paper, we present a systematic action-level analysis of frontier agents on direct robot control across 12 manipulation benchmarks. We introduce AgentRoboCritic, an intent-grounded critique framework in which the policy agent states a local intent alongside each action, and a critic evaluates the action's outcome and failure mode against this intent. Our action-level analysis demonstrates the promising capabilities of frontier agents on direct robot control, including closed-loop recovery from local execution failures, reliable semantic grounding across diverse scenes, and strong zero-shot manipulation without task-specific adaptation. It also identifies their current limitations, including task-relevant physical execution bottlenecks such as grasping and alignment, and uneven recovery across failure modes. Finally, we validate our critic on a 400-action human-annotated benchmark and release a dataset of 12,844 annotated actions alongside a fine-tuned open-source critic, Muse-Glimmer-30B, whose failure-mode accuracy rises from 32.58% to 74.24%, enabling low-cost action-level analysis.

Figures & tables

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy

    Oct 26, 2026Yifei Yao, Yude Zou, Guanbo Jiao +12Language Model-Based ControlRobotic Policy Evaluation

  2. RoboARCade: Benchmarking Agentic Robot Control

    Nov 5, 2026Satvik Sharma, Hengyuan Hu, Mihika Kamat +2Embodied Agent BenchmarksRobot Manipulation Benchmarks

  3. LIBERO-Agent: Evaluating General-Purpose Agents for Direct Embodied Manipulation

    Sep 30, 2026Zijie Diao, Yitong Chen, Sicheng Xie +7Long-Horizon Robotic ManipulationContact-Rich Robotic Manipulation