AgentRoboCritic: Understanding the Capability of Frontier Agents on Direct Robot Control
Authors: Yiyang Du, Yuxuan Kuang, Qitao Zhao, Yixuan Wang, Zhanqiu Guo, Yi Xu, Xin Li, Wenbin He, Liu Ren, Chenyan Xiong
Organizations: Carnegie Mellon University · University of Illinois at Urbana-Champaign · Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI)
Frontier multimodal language agents such as GPT-6 Astra have demonstrated strong capabilities in controlling robots directly via low-level numerical actions without high-level primitives. However, existing benchmarks report only episode-level success rates, which show neither how these agents succeed nor where they fail. In this paper, we present a systematic action-level analysis of frontier agents on direct robot control across 12 manipulation benchmarks. We introduce AgentRoboCritic, an intent-grounded critique framework in which the policy agent states a local intent alongside each action, and a critic evaluates the action's outcome and failure mode against this intent. Our action-level analysis demonstrates the promising capabilities of frontier agents on direct robot control, including closed-loop recovery from local execution failures, reliable semantic grounding across diverse scenes, and strong zero-shot manipulation without task-specific adaptation. It also identifies their current limitations, including task-relevant physical execution bottlenecks such as grasping and alignment, and uneven recovery across failure modes. Finally, we validate our critic on a 400-action human-annotated benchmark and release a dataset of 12,844 annotated actions alongside a fine-tuned open-source critic, Muse-Glimmer-30B, whose failure-mode accuracy rises from 32.58% to 74.24%, enabling low-cost action-level analysis.
Figures & tables
Figure 1: Overview of AgentRoboCritic. (a) Local intent makes action-level critique fine-grained and well-posed. (b) Capabilities of frontier agents on direct robot control. (c) Evaluation of action-level critics on the human-labeled benchmark.
Figure 2: Intent-grounded action-level critique. (1) The policy agent states a local intent gt alongside every action at , in the same tool call. (2) The critic receives the global goal G , the local intent gt and the observations Ot , and returns the outcome, the failure mode and a note. (3) Human annotators label the same actions to validate the reliability of critics.
Figure 3: Action-level failure modes ( M ) in AgentRoboCritic. Each of the six modes shown is illustrated by one failed action from our annotated episodes. Apart from the six modes, Unresolved marks failures with occluded or ambiguous evidence.
Figure 4: Action-level failure modes per benchmark. Each bar shows the distribution of failure modes among the failed actions of one benchmark. ∗ denotes the vision-only settings of visuo-tactile benchmarks. RoboDojo is shown by its 5 evaluation dimensions separately.
Figure 5: Closed-loop recovery from local execution failures. Overall combines all 12 benchmarks. (a) Share of successful episodes that contain at least one failed action. (b) Episode success rate conditioned on the number of local failures within an episode. (c) Share of failed actions with a verified recovery within a given number of later actions, up to the end of the episode. (d) Verified recovery rate against the official success rate of episodes with a local failure.
Figure 6: Zero-shot direct robot control comparison against VLAs and WAMs. (a) Head-to-head zero-shot episode success rates comparing a frontier agent (GPT-6 Astra) against representative VLA and WAM ( π0.5 and InternW0- Δ ) without task-specific fine-tuning. (b) Qualitative rollout trajectories illustrating successful zero-shot task completion and closed-loop adjustment by the frontier agent alongside failure modes of the VLA baseline.
Figure 7: Recovery by failure mode and Grasp sub-categories. (a) Share of failed actions with a verified recovery by the end of the episode. (b) Share of each sub-category among all Grasp failures across the 12 benchmarks.
Critic Model
Outcome
Failure Mode
Overall Accuracy (%)
Cost ($)
Acc. (%)
κ
Acc. (%)
κ
Proprietary Models
GPT-6 Astra
79.25
0.497
80.68
0.600
71.25
26.06
w/o local intent gt
25.25
0.102
25.00
0.078
24.50
25.18
GPT-6.1 Sol
74.50
0.487
62.12
0.314
69.75
5.00
GPT-6.1 Terra
74.50
0.470
61.36
0.303
66.75
2.36
Table 1: Action-level critic evaluation on the human-annotated test set. Outcome is scored on all actions and Failure Mode on the failed actions. Overall Accuracy requires both to be correct. Inference costs on the test set are reported in US dollars. Bold and underlined values mark the best and second-best results.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Sub-category
Stage
At closure
Closure above the object
Closure
Closure beside the object
Closure
Pinch on a rim, tip, or edge
Closure
Closure pushes or tips the object away
Closure
The support or another object is caught
Closure
Appendix
Table 2: Grasp sub-categories and their stages. Figure 7 (b) combines the last four closure errors into other closure errors, the two carry losses into loss during carry, and failed transfers and releases into one category.
Figure 8: System prompt of the critic. The codebook that follows the prompt is given in Appendix J , and Table 14 lists the names used in this paper for its identifiers.
Figure 9: User prompt of each critic request. The header is the original JSON header with two-space indentation and the keys annotation_task , target_action_alias , task_identifier , task_instruction_verbatim , task_instruction_origin , reference_context , original_note_or_reason_verbatim , note_scope , intent_basis , explicit_local_objective , native_command_inputs , target_video_interval , alignment_and_missing_information , later_context , later_context_rule , in this order. Each source image is sent in order, preceded by the metadata line shown above [PNG image] . Playback time uses six digits after the decimal point.
Benchmark
Episodes
Success (%)
Actions
Success (%)
RoboDojo
2,100
22
105,429
77
StationeryBench
100
7
1,544
58
Astranomy
56
52
9,003
66
UniVTAC ∗
80
76
1,297
86
OpenViTac ∗
130
80
2,303
91
ContactBench 1.0
71
48
1,161
44
Appendix
Table 3: Main dataset composition. Success is the percentage of scored episodes that succeed and the percentage of annotated actions judged achieved . A darker cell marks a lower success rate. ∗ denotes the vision-only settings of visuo-tactile benchmarks.
Figure 10: Recovery over actions by failure mode. Share of failed actions with a verified recovery within a given number of later actions.
Benchmark
Failures
Recovered (%)
Episodes
Success (%)
RoboDojo
9,923
57
1,659
17
StationeryBench
411
32
100
7
Astranomy
869
63
49
49
UniVTAC ∗
109
17
12
42
OpenViTac ∗
164
66
42
57
ContactBench 1.0
38
42
33
45
Appendix
Table 4: Recovery analysis per benchmark. Recovered is the percentage of analyzed failed actions with a verified recovery by the end of the episode. Success is the official success rate of the graded episodes that contain at least one failed action. ∗ denotes the vision-only settings of visuo-tactile benchmarks.
Successes
Episode success by failed actions
Failures recovered
Benchmark
With a failure
No failure
One or two
Three or more
Within 10 actions
By episode end
RoboDojo
62
63
34
9
41
57
StationeryBench
100
–
27
4
31
32
Astranomy
83
71
75
44
25
63
UniVTAC ∗
8
82
50
40
15
17
OpenViTac ∗
23
91
69
50
40
66
Appendix
Table 5: Combined statistics per benchmark , in percent. The sample-wise average combines all episodes or failed actions, and the benchmark-wise average gives each of the 12 benchmarks equal weight. DexJoCo and StationeryBench have no episode without a failed action. ∗ denotes the vision-only settings of visuo-tactile benchmarks.
Share of n (%)
Benchmark
Achieved
Failed
Partial
Unverifiable
Not applicable
n
RoboDojo
Precision
61
8
29
2
18,315
Open
79
11
9
1
13,984
Long-horizon
78
11
10
< 1
23,299
Generalization
80
11
8
1
34,430
Appendix
Table 6: Action outcomes , as a percentage of the n annotated actions in each row.
Share of n (%)
Benchmark
Grasp
Alignment
Interference
Effect
Incomplete motion
Unresolved
Choice
n
RoboDojo
Precision
63
17
15
2
4
–
1,500
Open
68
9
16
4
3
1,489
Long-horizon
49
20
26
1
4
< 1
2,586
Generalization
47
15
32
4
2
< 1
3,646
Appendix
Table 7: Failure modes of failed actions , as a percentage of the n failed actions with a failure-mode label in each row.
At closure
After capture
Other
Benchmark
Closure
Handover
Lift
Carry
Release
Empty gripper
Unresolved contact
n
RoboDojo
Precision
35
2
5
8
29
20
944
Open
70
< 1
6
14
< 1
5
5
1,012
Long-horizon
82
< 1
4
10
< 1
2
1
1,272
Generalization
65
3
6
18
< 1
7
1
1,703
Appendix
Table 8: Grasp failures by stage , grouped as in Figure 7 (b), as a percentage of the n Grasp failures in each row.
Figure 11: Annotation interface. The annotator judges one action at a time from the task instruction, the local intent and synchronized frames of all camera views.
Split
Achieved
Failed
Total
Training
3,211
9,633
12,844
Validation
387
1,161
1,548
Test
484
1,452
1,936
Appendix
Table 9: Splits of the critic dataset in annotated actions.
Configuration
Supervised Fine-tuning
Base model
Muse-Glimmer-30B
Trainable parameters
LoRA adapter only
LoRA rank
32
Peak learning rate
5×10−5
Learning rate schedule
Cosine with warmup
Warmup steps
25
Appendix
Table 10: Key hyperparameters for fine-tuning Muse-Glimmer-30B as the critic.
Figure 12: Closed-loop recovery and semantic grounding in two successful episodes. Top, seven consecutive actions in which a missed grasp is corrected by a regrasp. Bottom, selected actions in which the agent presses the buttons in the count and order of its first local intent. Each frame shows the end of the action and the critic’s outcome.
Figure 13: Failed actions over time. Frames run from before the action to its end or shortly after. Each row gives the local intent and the critic’s failure reason.
Figure 14: Feedback context of the repeated attempt. The control attempt receives the same text without the last four lines, which hold the critic’s assessment of the three actions.
Figure 15: Control and feedback attempts on the DexJoCo water-plant task. Each frame shows the scene at the end of the labeled action. The feedback row shows the first carry and the last actions of the episode, and the inset magnifies the spray in its last frame.
Figure 16: Zero-shot success rates on matched tasks , the percentage of tasks in each suite that each policy completes. All combines the five suites.
Tasks
GPT-6 Astra (s)
π0.5 (s)
RoboTwin 2.0
38.4 [19.2, 54.7]
0.109 [0.108, 0.111]
YAMLab
34.1 [15.5, 45.4]
0.142 [0.141, 0.143]
All matched tasks
31.9 [4.8, 46.8]
0.110 [0.108, 0.128]
Appendix
Table 12: Decision latency , the median over tasks of each run’s median interval from an observation to the next executed action, in seconds, with the interquartile range in brackets.
Critic
Labeled achieved
GPT-6 Astra
19 (7.2%)
Claude Fable 5.2
75 (28.4%)
Claude Opus 5.5
77 (29.2%)
GPT-6.1 Sol
81 (30.7%)
GPT-6.1 Terra
71 (26.9%)
Muse-Glimmer-30B (Zero-Shot)
67 (25.4%)
Appendix
Table 13: Failed actions labeled achieved by each critic, out of the 264 actions that the annotators judge as failed in the human-annotated test set.