Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks
Organizations: Mohamed bin Zayed University of Artificial Intelligence · Peking University
Abstract
Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree, recreates those scenarios in simulation, and runs the original checker. Multi-view renderings are then used to verify whether the observed behavior satisfies the instruction. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predicted checker verdict. Visual verification and deduplication yielded 61 distinct defects across 49 tasks, including 40 phantom successes and 21 silent failures. Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0. These findings suggest that validating success checkers requires testing both incomplete executions and valid alternatives to expert demonstrations, and motivate more rigorous evaluation protocols for future benchmarks.
Figures & tables
| RoboTwin 2.0 | RoboCasa365 | RoboDojo | Total | |
| Defects | 31 | 18 | 12 | 61 |
| Tasks with a defect | 25 | 13 | 11 | 49 |
| By direction of the error | ||||
| Phantom success | 20 | 17 | 3 | 40 |
| Silent failure | 11 | 1 | 9 | 21 |
| By primary cause | ||||
| Success (%) | Phan. | Silent | ||||
| Policy | Orig. | Verif. | succ. | fail. | Flip | |
| RoboTwin 2.0 | ||||||
| LingBot-VA | 93.1 | 90.0 | -3.1 | 3.3 | 0.0 | 3.1 |
| LingBot-VLA 2.0 | 93.9 | 90.8 | -3.1 | 3.4 | 1.5 | 3.3 |
| Evo-1 | 68.3 | 65.5 | -2.7 | 5.8 | 3.9 | 5.2 |
| X-VLA | 64.9 | 61.6 | -3.4 | 6.2 | 2.0 | 4.7 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Cause | The checker … |
| Specification: tests conditions that differ from the instruction | |
| Missing condition | omits a required condition, such as bottle motion in “shake bottle”. |
| Unstated requirement | adds a condition, such as a centre target in “pick dual bottles”. |
| Measurement: tests a condition on an unsuitable quantity | |
| Proxy signal | reads an indirect signal, such as the commanded gripper value for a grasp. |
| Loose geometry | uses a region or threshold that is too permissive, such as a straw across the rim of a cup counted as inside it. |
| LingBot | |||||
| Verdict change | VA | VLA 2 | Evo-1 | X-VLA | |
| Press without contact | 6 | 23 | 20 | 24 | 19 |
| Missing condition | 0 | 2 | 4 | 3 | 9 |
| Command-based grasp | 0 | 0 | 5 | 3 | 7 |
| Transient success | 2 | 9 | 13 | 11 | 15 |
| Silent failures | 0 | 1 | 13 | 7 | 9 |
Automated Evaluation
Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.
See all 422 papers