When the Simulator Fails, Not the Policy: Auditing Contact Artifacts in Simulated Robot Benchmarks
Organizations: College of AI, Tsinghua University · University of Chinese Academy of Sciences
Abstract
Simulation benchmarks score robot policies by success rate, and every failed episode counts against the policy. Many of these failures are the simulator's fault, not the policy's; we find the cause and fix it. Testing a vision language action model and a world action model in simulation, we find that the policy often places the object and the simulator then ejects it, and that the benchmark's own scripted demonstrations fail the same way. We present SimuGuard, an auditing plugin for LIBERO, RoboTwin, RoboCasa and RoboDojo. It leaves the evaluation unchanged. At every physics step, it records the state and flags an object that starts moving at a contact faster than the robot's contact or gravity account for. It replays a recorded episode bit for bit, and it reruns the episode with one simulation parameter changed. With it we trace these episodes on RoboTwin to the collision detection of the physics engine: for geometry one to two centimeters apart, the engine reports a penetration of about two centimeters, and the solver throws the object out at several meters per second. The cause is one constant: the distance algorithm discards triangles below a fixed area threshold, which valid triangles fall below at the centimeter scale. With that constant changed and nothing else, episodes with an artifact fall from 12 in 100 to none, and the success rate on five container tasks rises from 89% to 98%. Success rates in simulation thus count failures that the simulator causes. Benchmarks on PhysX 5.3.1 or earlier should rerun in a repaired engine. Where the engine cannot change, they should report the success rate together with the range it spans when every flagged episode fails or passes: 75.8% to 98.5% around a reported 83.1% for LingBot-VA.
Figures & tables
| LingBot-VA (world action model) | HyVLA (vision language action) | |||||||
| Task | Rep. | Replay | Excl. | CL | Rep. | Replay | Excl. | CL |
| Place Can Basket | 74.4 | 86.7 | 96.2 | 90.0 | 80.0 | 75.0 | 81.3 | 80.0 |
| Stack Bowls Three | 76.0 | 90.0 | 100.0 | 100.0 | 90.0 | 95.0 | 100.0 | 100.0 |
| Place Object Basket | 87.5 | 90.0 | 94.1 | 50.0 | 95.0 | 95.0 | 95.0 | 45.5 |
| Place Bread Basket | 97.5 | 100.0 | 100.0 | 85.0 | 100.0 | 100.0 | 100.0 | 70.0 |
| Stack Bowls Two | 92.5 | 100.0 | 100.0 | 100.0 | 85.0 | 100.0 | 100.0 | 100.0 |
| Contacts | Expert, artifact | Expert, failed | ||||
| Engine | False (20) | Real (4) | Can | Bowls | Can | Bowls |
| PhysX 5.3.1 (RoboTwin) | 20 | 4 | 18 | 6 | 6 | 5 |
| PhysX 5.3.1, threshold of 5.4.0 | 0 | 4 | 1 | 0 | 1 | 0 |
| PhysX 5.9.0 | 0 | 4 | – | – | – | – |
| LingBot-VA | HyVLA | ||||
|---|---|---|---|---|---|
| Task | Bench. | Offset | Repaired | Bench. | Repaired |
| Place Can Basket | 15 (5) | 19 (3) | 20 (0) | 16 (4) | 18 (0) |
| Stack Bowls Three | 15 (6) | 17 (3) | 19 (0) | 18 (3) | 20 (0) |
| Place Object Basket | 20 (0) | 20 (0) | 20 (0) | 19 (0) | 18 (0) |
| Place Bread Basket | 19 (1) | 19 (0) | 20 (0) | 20 (0) | 18 (1) |
| Stack Bowls Two | 20 (0) | 20 (0) | 19 (0) | 17 (3) | 20 (0) |
| Benchmark | Engine | Policy | Episodes | With artifact | Peak (m/s) | Limited (m/s) |
|---|---|---|---|---|---|---|
| RoboTwin 2.0 | PhysX, SAPIEN | LingBot-VA | 2060 | 80 | 21.8 | 1.39 |
| HyVLA | 220 | 10 | – | – | ||
| RoboDojo | PhysX, Isaac Sim | XR-1 | 40 | 7 | 6.30 | 1.88 |
| LIBERO | MuJoCo | 400 | 2 | 1.57 | – | |
| RoboCasa | MuJoCo | XR-1 | 300 | 5 | 1.19 | – |
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Successes | Reported (%) | Artifact | Skipped | |
|---|---|---|---|---|---|
| Place Can Basket | 20 | 16 | 80.0 | 1/4 | 5 |
| Stack Bowls Three | 20 | 18 | 90.0 | 2/2 | 18 |
| Place Object Basket | 20 | 19 | 95.0 | 0/1 | 8 |
| Place Bread Basket | 20 | 20 | 100.0 | 0/0 | 5 |
| Stack Bowls Two | 20 | 17 | 85.0 | 3/3 | 7 |
| Blocks Ranking Size | 20 | 17 | 85.0 | 0/3 | 2 |
| Stage | Parameter | Value |
| Contact ejection | recent contact window | 40 ms |
| minimum speed | 0.5 m/s | |
| minimum velocity change in one 4 ms step | 0.35 m/s | |
| confirmation window | 240 ms | |
| minimum displacement | 4 cm | |
| free flight | 16 ms |
| Check | Platform | Result |
| Replay | ||
| Fresh process, 96 failures (2.42 M steps); events and outcomes reproduced | RoboTwin | 96/96 exact; 43/43, 96/96 |
| Stack Bowls Three, 12 failures: outcomes with / without gripper restore | RoboTwin | 12/12 / 7/12 |
| Another machine, 25 episodes | RoboTwin | 25/25 exact |
| Prefix before a rerun switch, 273 replays | RoboTwin | 273/273 exact |
| Full episodes, 400 (1.88 M steps); without the recorded warm start | LIBERO | 400/400 exact; 0/33 |
| Platform | Episodes | Flagged | After gravity | After carrier |
|---|---|---|---|---|
| RoboTwin, Put Bottles Dustbin (LingBot-VA) | 10 | 20 | 0 | 0 |
| RoboTwin, 40 tasks, second pass (LingBot-VA) | 800 | 12 (12) | 10 (10) | 10 (10) |
| RoboTwin, eight tasks, second pass (LingBot-VA) | 200 | 169 (58) | 22 (10) | 17 (10) |
| RoboCasa, ten placement tasks (XR-1) | 100 | 7 (7) | 3 (3) | 3 (3) |
| RoboCasa, the other 40 tasks (XR-1) | 200 | 15 (15) | 2 (2) | 2 (2) |
| RoboDojo, four tasks (XR-1) | 40 | 25 (15) | 12 (7) | 12 (7) |
| Version | Can, false (4) | Bowls, false (16) | Bowls, real (4) |
|---|---|---|---|
| 5.3.1 | 4 | 16 | 4 |
| 5.3.1 with the simplex reduction of 5.4.0 | 0 | 0 | 4 |
| 5.3.1 with the hull extent of 5.4.0 | 4 | – | – |
| 5.4.0, 5.4.2, 5.5.1, 5.6.1 | 0 | – | – |
| 5.9.0 | 0 | 0 | 4 |
| Contact offset | Episodes with an artifact | Events | Failures that pass | Successes kept |
| 0.5 mm | 3 | 10 | 18/21 | 10/16 |
| 1 mm | 4 | 14 | 19/21 | 12/16 |
| 2 mm | 9 | 29 | 18/21 | 14/16 |
| 3 mm | 11 | 25 | 18/21 | 12/16 |
| 5 mm | 13 | 46 | 16/21 | 10/16 |
| 10 mm | 37 | 152 | – | – |
| Task | Benchmark | Offset 1 mm | Repaired | Manifold off |
|---|---|---|---|---|
| Place Can Basket | 15 (5), 4 | 19 (3), 4 | 20 (0), 2 | 20 (0), 3 |
| Stack Bowls Three | 15 (6), 18 | 17 (3), 16 | 19 (0), 11 | 18 (2), 12 |
| Place Object Basket | 20 (0), 14 | 20 (0), 12 | 20 (0), 12 | 20 (1), 3 |
| Place Bread Basket | 19 (1), 6 | 19 (0), 5 | 20 (0), 4 | 20 (0), 3 |
| Stack Bowls Two | 20 (0), 7 | 20 (0), 7 | 19 (0), 7 | 19 (0), 7 |
| Blocks Ranking Size | 18 (0), 1 | 17 (0), 1 | 19 (0), 2 | 17 (0), 3 |
| Engine and setting | Thrown (%) | Exit speed (m/s) | Distance (cm) | Removal (ms) |
|---|---|---|---|---|
| PhysX, RoboTwin settings | 91 | 8.77 | 155 | 2.6 |
| PhysX, can mass 100 g (10 g) | 90 | 6.86 | 73 | 3.3 |
| PhysX, solver iterations 32/8 (10/1) | 90 | 16.78 | 200 | 1.4 |
| PhysX, depenetration 1 m/s | 65 | 1.36 | 10 | 16.5 |
| PhysX, depenetration 0.1 m/s | 0 | 0.14 | 3.7 | 169 |
| MuJoCo, default (time constant 20 ms) | 5 | 0.56 | 2.8 | 38.6 |
| Place Can Basket failures | Stack Bowls Three | ||
|---|---|---|---|
| Change | with artifact (21) | without artifact (2) | failures (12) |
| Depenetration 1 m/s | 19 | 2 | 8 |
| Depenetration 0.1 m/s | 9 | 2 | 5 |
| Object mass 100 g | 17 | 2 | 6 |
| Solver iterations 32/8 | 14 | 1 | 5 |
| Speed clamp 2 m/s | 14 | 0 | 0 |
| Place Can Basket | Stack Bowls Three | |||
|---|---|---|---|---|
| Change | Rescued | Peak (m/s) | Rescued | Peak (m/s) |
| Contact offset 1 mm (10 mm) | 19/21 | 0.91 | 4/12 | 2.50 |
| Depenetration 1 m/s | 18/21 | 1.39 | 7/12 | 0.65 |
| Object mass 100 g | 14/21 | 3.72 | 5/12 | 2.12 |
| Solver iterations 32/8 | 12/21 | 11.20 | 4/12 | 7.00 |
| Place Can Basket (21) | Stack Bowls Three (12) | Three tasks (3) | Five tasks | |||
| Limit | Rescued | Peak (m/s) | Rescued | Peak (m/s) | Rescued | Rescued |
| 0.1 m/s | 18 | 0.91 | 7 | 0.40 | 3 | 28/36 |
| 0.5 m/s | 18 | 1.05 | 7 | 0.63 | 2 | 27/36 |
| 1 m/s | 18 | 1.25 | 7 | 0.65 | 3 | 28/36 |
| 2 m/s | 16 | 1.95 | 4 | 1.09 | 3 | 23/36 |
| Task | Reported | Limit on |
|---|---|---|
| LingBot-VA | ||
| Place Can Basket | 67/90 | 81/90 |
| Stack Bowls Three | 16/20 ∗ | 20/20 |
| Stack Bowls Two | 19/20 | 20/20 |
| Place Bread Basket | 19/20 | 17/20 |
| Place Object Basket | 18/20 | 10/20 |
| Configuration | Rep. 1 | Rep. 2 | Rep. 3 | Total | Mean std (%) |
|---|---|---|---|---|---|
| RoboTwin (can 10 g) | 23/30 | 24/30 | 20/30 | 67/90 | |
| Can mass 100 g | 29/30 | 30/30 | 27/30 | 86/90 | |
| Solver position iterations 32 | 26/30 | 26/30 | – | 52/60 |
| Actor | Episodes | Failures | With an ejection | Pass with the limit |
|---|---|---|---|---|
| LingBot-VA (world action model) | 90 | 23 | 21 | 18 |
| HyVLA (vision language action) | 20 | 4 | 1 | 1 |
| Scripted expert (ground truth poses) | 22 | 22 | 16 | 13 |
| Depenetration limit | Exit speed (m/s) |
|---|---|
| As run, 3 m/s by default (replay identical to the recording) | 4.31 |
| 1 m/s | 1.88 |
| 0.3 m/s | 0.62 |
| Suite | Initial states | Contact ejection | After gravity | Contact artifacts | Tasks affected |
|---|---|---|---|---|---|
| LIBERO-Spatial | 500 | 3 | 0 | 0 | 0/10 |
| LIBERO-Object | 500 | 500 | 500 | 500 | 10/10 |
| LIBERO-Goal | 500 | 0 | 0 | 0 | 0/10 |
| LIBERO-10 | 500 | 152 | 150 | 150 | 3/10 |
| LingBot-VA | HyVLA | |||||
| Task | Art. | Resc. | Lost | Art. | Resc. | Lost |
| Place Can Basket | 21/23 | 18 | 7/16 | 1/4 | 1 | 2/3 |
| Stack Bowls Three | 12/12 | 7 | – | 2/2 | 1 | 0/1 |
| Place Object Basket | 3/5 | 3 | 2/3 | 0/1 | – | – |
| Place Bread Basket | 1/1 | 1 | – | 0/0 | – | – |
| Stack Bowls Two | 3/3 | 3 | – | 3/3 | 3 | – |
| Task | Episodes | Seed, recorded replayed |
| LingBot-VA | ||
| Place Can Basket (rep. 1) | 13 | 100000 F S, 100002 S S, 100004 S F, 100006 F S, 100007 F S, 100008 F S, 100014 S S, 100017 S F, 100021 F S, 100022 S S, 100024 S F, 100029 F S, 100031 F F |
| Place Can Basket (rep. 2) | 11 | 100004 S S, 100011 F S, 100012 F S, 100017 S F, 100019 F S, 100021 S S, 100022 F S, 100025 S F, 100026 S S, 100027 S F, 100029 F F |
| Place Can Basket (rep. 3) | 13 | 100000 S S, 100003 F S, 100009 F S, 100016 F F, 100019 F S, 100021 F S, 100022 F S, 100023 F S, 100027 F S, 100028 F S, 100029 S F, 100030 S S, 100032 S S |
| Stack Bowls Three | 12 | 100002 F S, 100009 F F, 100023 F F, 100028 F F, 100047 F S, 100067 F S, 100071 F S, 100073 F F, 100075 F S, 100077 F S, 100078 F S, 100089 F F |
| Place Object Basket | 6 | 100013 F S, 100015 S S; pass 2: 100001 F S, 100009 S F, 100014 S F, 100016 F S |
Robotics Simulation
Robotics simulation builds virtual robots and environments to model, train, and evaluate autonomous systems before real-world deployment.
See all 349 papers