Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree, recreates those scenarios in simulation, and runs the original checker. Multi-view renderings are then used to verify whether the observed behavior satisfies the instruction. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predicted checker verdict. Visual verification and deduplication yielded 61 distinct defects across 49 tasks, including 40 phantom successes and 21 silent failures. Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0. These findings suggest that validating success checkers requires testing both incomplete executions and valid alternatives to expert demonstrations, and motivate more rigorous evaluation protocols for future benchmarks.
Figures & tables
Figure 1: A checker validated on the expert can be wrong elsewhere. On the expert demonstration (left), the checker agrees with the instruction. On other executions (right), the same checker accepts a bottle that is lifted but not shaken, and rejects two bottles that are picked up away from the centre.
Figure 2: SimAudit workflow. (A) An LLM agent reads the task instruction and checker code. (B) It proposes cases likely to expose checker errors. (C) A probe in the unmodified simulator triggers a success verdict while the mouse is still held above the stand. (D) A separate VLM reviewer uses multi-view renderings to confirm that the task is incomplete, identifying a phantom success.
Figure 3: Verified checker defects across three benchmarks. Each tile pairs the task instruction with the observed state at the checker’s verdict. Phantom successes appear above and silent failures below. cmd denotes the commanded gripper value. The tiles show 51 defects, and the inset counts all 61, including ten additional RoboTwin 2.0 successes reported while the object was still held.
RoboTwin 2.0
RoboCasa365
RoboDojo
Total
Defects
31
18
12
61
Tasks with a defect
25
13
11
49
By direction of the error
Phantom success
20
17
3
40
Silent failure
11
1
9
21
By primary cause
Table 1: Verified checker defects in the three benchmarks. The defects are split first by the direction of the error and then by their primary cause.
Figure 4: Checker verdicts on four RoboTwin 2.0 rollouts. Each row shows a rollout scored by the original and Verified checkers; green denotes a true verdict. The top three rows show phantom successes: the gripper stops above the bell or beside the stapler, or holds the roller with one gripper. The bottom row shows a silent failure: the stapler is placed on the mat, yet the original checker rejects it for a yaw requirement absent from the instruction.
Success (%)
Phan.
Silent
Policy
Orig.
Verif.
Δ
succ.
fail.
Flip
RoboTwin 2.0
LingBot-VA
93.1
90.0
-3.1
3.3
0.0
3.1
LingBot-VLA 2.0
93.9
90.8
-3.1
3.4
1.5
3.3
Evo-1
68.3
65.5
-2.7
5.8
3.9
5.2
X-VLA
64.9
61.6
-3.4
6.2
2.0
4.7
Table 2: Re-scoring released policies on the same rollouts. Δ is the Verified minus original success rate in percentage points. Phan. succ. and Silent fail. are the percentages of original successes and failures reversed, respectively. Flip is the percentage of all episodes whose verdict changes.
Figure 5: Per-task success rates. Bars show the original checker and lines the Verified checker on the same rollouts.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Cause
The checker …
Specification: tests conditions that differ from the instruction
Missing condition
omits a required condition, such as bottle motion in “shake bottle”.
Unstated requirement
adds a condition, such as a centre target in “pick dual bottles”.
Measurement: tests a condition on an unsuitable quantity
Proxy signal
reads an indirect signal, such as the commanded gripper value for a grasp.
Loose geometry
uses a region or threshold that is too permissive, such as a straw across the rim of a cup counted as inside it.
Appendix
Table 3: Checker defect taxonomy. Five causes in three categories. Each category corresponds to one step of a checker’s verdict.
LingBot
Verdict change
VA
VLA 2
Evo-1
X-VLA
π0.5
Press without contact
6
23
20
24
19
Missing condition
0
2
4
3
9
Command-based grasp
0
0
5
3
7
Transient success
2
9
13
11
15
Silent failures
0
1
13
7
9
Appendix
Table 4: Verdict changes by repair type and policy. The first four rows count corrected phantom successes, and the fifth counts corrected silent failures.
Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fall below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate together with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.
Alex Zhou, Junhao Wu, Xueyang Zhou +5
College of AI, Tsinghua University · University of Chinese Academy of Sciences
Imitation-learning policies for robot manipulation inherit the quality of the success labels attached to their training episodes, and those labels are usually produced by the robot's own success check. A particularly damaging error is the false success: an episode the robot logs as a success when the task outcome was actually wrong. We ask a narrow but practical question about these episodes. Once an episode has already been flagged as a success, how much of the information needed to overturn that label is present in proprioception, and how much requires vision? We build a simulated testbed on two bimanual ALOHA tasks, induce failures through environment perturbations rather than label edits, label every episode by privileged simulator state that the detector never sees, and keep only episodes the robot flagged as successful. We then compare detectors restricted to proprioception against a vision-based detector. We find that recoverability spans a wide range: in cube transfer the false successes are almost fully recoverable from joint data alone, while in peg insertion proprioception recovers only part of them and a vision detector closes most of the gap. We also show that the proprioceptive separability we measure rests on velocity differences far below any realistic sensor noise floor, so it is best read as an optimistic upper bound that a noiseless simulator inflates. We release the generation and evaluation pipeline.
Aarav Bedi
University of California, Berkeley · Department of Mechanical Engineering University of California, Berkeley Berkeley, California, USA
Simulated evaluation is widely used to benchmark AI agents, yet how much evidence a simulated pass provides about physical deployment has not been systematically quantified. We present SimVerity, a verdict-transfer assurance framework: it replays matched scenarios on target smart home deployments and cross-validates agent execution against independently qualified physical witnesses. Our evaluation highlights that deployment success is a real-world process, not a static property in simulation: completion, reported state, observable effect, and settled outcome diverged within the same execution. Although an advanced simulator cleared all 240 light trials, a camera caught 42 sub-second failures invisible to settled-state checks. False clearance was predictable: a risk profile learned from measured trials and locked before evaluation predicted failures on a path it never physically measured, beating a property-blind baseline in all eleven held-out sessions across two cohorts. Agent auditability was also measurable: switching one agent loop's model-client/serving configuration raised its scenario-matching share from 52-88% to 100%. Finally, a second qualified simulator added no independent cross-check: it never disagreed on any overlapping case, and only physical measurement exposed their shared blind spots. SimVerity turns verdict transfer into an explicit decision: clear, abstain, or escalate before deployment.