Auditing Grasp Feasibility Predictors with Planner-Verified Counterfactuals
Organizations: IIIT Hyderabad, India
Abstract
Grasp-feasibility predictors report near-perfect accuracy on Task and Motion Planning benchmarks, but high accuracy on these benchmarks does not verify that a predictor uses the collision geometry that the motion planner checks. We propose a planner-verified counterfactual (CF) test: holding the grasp pose fixed, we insert a solid obstacle into the approach corridor, add its points to the model input, and let a planner re-derive the label, which flips from feasible to infeasible. Across eight predictors spanning point clouds, voxels, depth, and object tokens, CF-pair AUROC falls from high natural AUROC (0.920-0.996) to chance (0.470-0.504), and the flip holds across four planners, two arms, and a real-object obstacle. Predictors fail at different resolutions, from a bounding box to a 12.5 cm voxel. In the point-cloud models we dissect, the obstacle either moves the representation in a direction unrelated to whether it blocks the grasp or is not registered at all. A clearance-aware read of the corridor points, trained on the unmodified benchmark with no counterfactual scenes, reaches 0.983 CF-pair AUROC, but it remains bound to its trained obstacle placements; a placement-invariant clearance representation is still open. We release the counterfactual labels, metrics, datasheet, and relabeling pipeline at https://github.com/codewithsajid/graspfc-cf.
Figures & tables
| Predictor | Input | Par. | Regime | Nat. | CF-pair |
|---|---|---|---|---|---|
| GraspFC-NNet [ 3 ] | pose+crop | 1.65M | released | 0.995 | 0.491 |
| GraspFC-Conv3D [ 3 ] | pose+voxels | 9.7M | released | 0.976 | 0.497 |
| GraspFC-PTX [ 3 ] | pose+cloud | 9.0M | released | 0.996 | 0.498 |
| SceneCollisionNet [ 4 ] | voxels | 26M | re-impl.+rel. | 0.964 | 0.470 |
| PIGINet [ 7 ] | tokens | 31M | re-impl.+rel. | 0.960 | 0.500 |
| NFC † [ 6 ] | depth | 22.8k | re-impl. | 0.965 | 0.504 |
| Model (run as-is) | CF-iid | CF-adj. | CF-dist. | max |
|---|---|---|---|---|
| GraspFC-PTX (9.0M) | 0.501 | 0.499 | 0.497 | 0.001 |
| PIGINet | 0.500 | 0.508 | 0.493 | 0.014 |
| NFC | 0.504 | 0.511 | 0.532 | 0.014 |
| AGFP-Net | 0.498 | 0.497 | 0.499 | 0.007 |
| SceneCollisionNet (26M) | 0.470 | 0.473 | 0.554 | 0.052 |
| Oracle / obstacle | Flip | Null | CF-pair | |
|---|---|---|---|---|
| Corridor box, cm standoff | 1.00 | 0.02 | 0.498 | 0.001 |
| Corridor box, cm standoff | 1.00 | 0.02 | 0.499 | 0.000 |
| Corridor box, cm standoff | 1.00 | 0.02 | 0.500 | 0.000 |
| Resting pillar (on the table) | 0.74 | 0.03 | 0.499 | 0.000 |
| Transit-path box | 0.56 | 0.01 | 0.491 | 0.001 |
| Dataset object (on the table) | 1.00 | 0.00 | 0.500 | 0.001 |
| Predictor | Responds to | Stops below | Native | Interv. |
| GRN † | presence, bbox | sub-bbox shape | 0.995 | 0.500 |
| SceneCollisionNet † | contact, coarsely | cm voxel | 0.969 | 0.821 |
| PIGINet † | object tokens | corridor geometry | 0.86 | 0.470 |
| GraspFC-NNet | presence | whether it blocks | — | 0.491 |
| GraspFC-Conv3D | voxel grid | any obstacle | — | 0.497 |
| GraspFC-PTX | (no response) | any obstacle | — | 0.498 |
| Encoder | Supervision | feat. | Probe |
|---|---|---|---|
| PointNet++ | recognition | 0.03 | |
| PointNeXt-S | recognition | 0.05 | |
| PTv3 | feasibility | 0.01 | |
| Clearance tokens (ours) | clearance | 0.71 |
| Intervention | Nat. | CF-pair | Verdict |
|---|---|---|---|
| More natural data | 0.990 | 0.498 | term not identified |
| Fine-tune deployed | 0.990 | 0.495 to 0.508 | does not transfer |
| Train from scratch on CF | 0.990 | 0.991 | fits intervention |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Robot | Planner | Agree | Flip | |
|---|---|---|---|---|
| Panda | BiRRT-Connect, re-run (control) | 0.995 | 0.990 | 1.000 |
| Panda | BiRRT-Connect, other hyper-params | 0.990 | 0.977 | 1.000 |
| Panda | single-tree goal-biased RRT | 0.973 | 0.940 | 1.000 |
| Panda | Lazy-PRM | 0.992 | 0.983 | 1.000 |
| xArm6 | BiRRT-Connect, re-run (control) | 0.988 | 0.973 | 1.000 |
| xArm6 | BiRRT-Connect, other hyper-params | 0.989 | 0.975 | 1.000 |
| Condition | Planner | Failure mode | model |
|---|---|---|---|
| Clear approach | feasible | none | 0.975 |
| Floating corridor box | infeasible | blocked approach | 0.976 |
| Resting pillar | infeasible | blocked approach | 0.975 |
| Transit-path box | infeasible | blocked transit | 0.975 |
| Model | Representation (what enters the net) | Training pool | Par. | Objective | Authors’ reported (own data) | Nat. |
|---|---|---|---|---|---|---|
| GraspFC-NNet | pose m grasp-local crop, pooled MLP | benchmark train split | 1.65M | BCE | [ 3 ] | 0.995 |
| GraspFC-Conv3D | pose voxel grid, 3D CNN | benchmark train split | 9.7M | BCE | [ 3 ] | 0.976 |
| GraspFC-PTX | pose point cloud, PTv3 backbone | benchmark train split | 9.0M | BCE | [ 3 ] | 0.996 |
| SceneCollisionNet | per-point MLP, scatter-max to cm voxel, 3D conv; PointNet++ gripper | k labels | 26M | hard-neg BCE | acc , AP [ 4 ] | 0.964 |
| PIGINet | per-object CLIP ViT-B/32 image pose token, attention blocks | k labels | 31M | pos-weighted BCE | val acc , plan-time cut [ 7 ] | 0.960 |
| NFC | object-centered -ch depth (ch. 2 neighbors), conv FC | k labels | 22.8k | weighted BCE | acc , time saved [ 6 ] | 0.965 |
| Model | Resolves at | Nat. | CF-pair | Failure mechanism |
|---|---|---|---|---|
| GraspFC-NNet | crop pool | 0.995 | 0.491 | registers the obstacle; feature displacement is direction-random |
| GraspFC-Conv3D | voxel grid | 0.976 | 0.497 | obstacle in the voxel grid, discarded |
| GraspFC-PTX | point pool | 0.996 | 0.498 | obstacle in the cloud, discarded |
| SceneCollisionNet | cm voxel | 0.964 | 0.470 | contact boundary below the cm voxel |
| PIGINet | object token | 0.960 | 0.500 | one object token, no clearance encoded |
| NFC | -ch depth | 0.965 | 0.504 | predicts from pose, does not use neighbor channel |
| Model | clear | obstacle | |
|---|---|---|---|
| GraspFC-PTX | 0.947 | 0.947 | |
| SceneCollisionNet | 0.70 | 0.72 | |
| PIGINet | 0.766 | 0.766 | |
| NFC | 0.798 | 0.797 | |
| AGFP-Net | 0.702 | 0.701 | |
| GRN (released) | 0.990 | 0.007 |
| Hole | GRN AUROC | Mislabels |
|---|---|---|
| Shape | 0.500 | (false block) |
| Tilt | 0.500 | (false clear) |
| Concavity | 0.500 | (false block) |
Counterfactual Evaluation
Counterfactual evaluation tests how models, explanations, and policies behave under hypothetical changes to inputs, evidence, actions, or environments.
See all 103 papers