Closed-Loop Physics-Based Manipulation Evaluation at a Commodity Price: What 4.4 L4 GPU hours at $3.50 Buys You
Abstract
Running evaluations of contact-rich robotic manipulation at statistical scale has largely been possible only at well-resourced labs, highlighting the need for affordable and efficient options for broader audiences. We present a reproducible evaluation harness wrapper based on a previous study at 200-rollout scale on serverless commodity GPUs, at a cost of approximately $3.50 USD and 38 minutes of wall-clock per replication (4.4 L4-GPU-hours across eight parallel shards). Using the harness, we report a partially controlled negative result on action chunking with sparse data: three independently trained chunked checkpoints produce zero genuine successes across 780 rollouts, while three single-step checkpoints range from 2.6% to 11.8% success rate. A per-step-replanning control partially isolates chunk prediction, rather than open-loop chunk execution as the mechanism, with the failure localized to the training formulation and multi-step prediction. This work further democratizes physics-based policy development and evaluation through our reusable harness.
Figures & tables
| Wall-clock (fan-out) | 37.7 min (8 parallel shards) |
|---|---|
| Eval compute | 4.36 L4-GPU-hr |
| Cost (USD) | 0.80 / L4-hr |
| Mean rollout duration | 78 s |
| Train + one replication | $3.85 |
| All work in Table II | $34 to reproduce |
| Checkpoint | Decoder | Success Rate | 95% Wilson CI | |
| v1 (orig.) | BC | (1,1) | 5.1% (7.5%) | [2.8, 9.2] |
| v1 seed1 | BC | (1,1) | 11.8% (14%) | [8.0, 17.1] |
| v1 seed2 | BC | (1,1) | 2.6% (5%) | [1.1, 5.9] |
| v2a | DDPM | (1,1) | 0% (2.5%) | [0, 1.9] |
| v2b † | DDPM | (16,8) | 0% (2.5%) | [0, 1.9] |
| v2b-bc | BC | (8,4) | 0% (2.5%) | [0, 1.9] |
Robotic Policy Evaluation
Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.
See all 149 papers