Grasp-feasibility predictors report near-perfect accuracy on Task and Motion Planning benchmarks, but high accuracy on these benchmarks does not verify that a predictor uses the collision geometry that the motion planner checks. We propose a planner-verified counterfactual (CF) test: holding the grasp pose fixed, we insert a solid obstacle into the approach corridor, add its points to the model input, and let a planner re-derive the label, which flips from feasible to infeasible. Across eight predictors spanning point clouds, voxels, depth, and object tokens, CF-pair AUROC falls from high natural AUROC (0.920-0.996) to chance (0.470-0.504), and the flip holds across four planners, two arms, and a real-object obstacle. Predictors fail at different resolutions, from a bounding box to a 12.5 cm voxel. In the point-cloud models we dissect, the obstacle either moves the representation in a direction unrelated to whether it blocks the grasp or is not registered at all. A clearance-aware read of the corridor points, trained on the unmodified benchmark with no counterfactual scenes, reaches 0.983 CF-pair AUROC, but it remains bound to its trained obstacle placements; a placement-invariant clearance representation is still open. We release the counterfactual labels, metrics, datasheet, and relabeling pipeline at https://github.com/codewithsajid/graspfc-cf.
Figures & tables
Figure 1: The counterfactual test. ( left ) A feasible grasp. ( right ) The same grasp becomes infeasible once a cube (red) is inserted in its approach path; points sampled from the cube’s surface are added to the point-cloud input. The classifier (GraspFC-PTX) still predicts p(feasible)=0.86 in both scenes. Over 1,065 such pairs, its CF-pair AUROC is 0.498 (chance =0.5 ).
Figure 2: The counterfactual protocol on a GraspNet-1B scene (deployed GraspFC-PTX). One grasp is held fixed; each of three obstacles (a floating corridor box, a resting pillar, a transit-path box) makes the planner relabel it infeasible (red), while the clear grasp stays feasible (green). The model returns the same p(feasible) , 0.975 to 0.976 , for all four, so its score does not separate the clear scene from any of the three blocked ones.
Predictor
Input
Par.
Regime
Nat.
CF-pair
GraspFC-NNet [ 3 ]
pose+crop
1.65M
released
0.995
0.491
GraspFC-Conv3D [ 3 ]
pose+voxels
9.7M
released
0.976
0.497
GraspFC-PTX [ 3 ]
pose+cloud
9.0M
released
0.996
0.498
SceneCollisionNet [ 4 ]
voxels
26M
re-impl.+rel.
0.964
0.470
PIGINet [ 7 ]
tokens
31M
re-impl.+rel.
0.960
0.500
NFC † [ 6 ]
depth
22.8k
re-impl.
0.965
0.504
Table 1: The eight audited predictors: input modality, parameters, evidence regime, and natural vs. counterfactual AUROC on the shared benchmark (GRN on its own, below the rule). released : the authors’ released weights; re-impl. : re-implemented and trained on the planner labels (“+rel.”: also run from released weights on its authors’ own data, Tab. 4 ). † No public code or weights; re-implemented from the published description. Nat. : ranking feasible vs. infeasible grasps on the unmodified benchmark. CF-pair : ranking the clear vs. obstacle-inserted version of a grasp at the identical pose, on the planner-verified pair set ( n=1,065 ); 0.5 is the no-signal baseline. ∗ GRN detects presence but does not resolve sub-bounding-box shape, so its entry is the same-bounding-box shape swap ( Tab. 4 ). The methods’ original (non-AUROC) headlines and full interventional scores are in supp. and .
Model (run as-is)
CF-iid
CF-adj.
CF-dist.
max ∣Δp∣
GraspFC-PTX (9.0M)
0.501
0.499
0.497
< 0.001
PIGINet
0.500
0.508
0.493
0.014
NFC
0.504
0.511
0.532
0.014
AGFP-Net
0.498
0.497
0.499
0.007
SceneCollisionNet (26M)
0.470
0.473
0.554
0.052
Table 2: Placement-invariance (inference-only, no retraining). CF-pair AUROC with the obstacle at the in-distribution corridor offset (iid), an unseen adjacent offset ( n=561 ), and a distal offset outside the crop ( n=250 ), all in held-out scenes. The one cell above 0.5 at distance (SceneCollisionNet, 0.554 ) is ≈1.4 SE from 0.5 and falls below it at the other two offsets. max ∣Δp∣ is the largest score change the obstacle produces.
Figure 3: CF-pair AUROC vs. obstacle distance along the approach (inference-only). The three counterfactual-cured variants (the deployed global PTX, and two crop-input models using raw corridor points vs. clearance-aware tokens) react only near the gripper; the original model and a no-cloud control stay at 0.5 .
Oracle / obstacle
Flip
Null
CF-pair
∣Δp∣
Corridor box, 10 cm standoff
1.00
0.02
0.498
0.001
Corridor box, 5 cm standoff
1.00
0.02
0.499
0.000
Corridor box, 15 cm standoff
1.00
0.02
0.500
0.000
Resting pillar (on the table)
0.74
0.03
0.499
0.000
Transit-path box
0.56
0.01
0.491
0.001
Dataset object (on the table)
1.00
0.00
0.500
0.001
Table 3: Robustness of the flip and of the model’s insensitivity to oracle standoff, obstacle type, planner, and arm. Flip : fraction of grasps the inserted obstacle relabels infeasible (planner ground truth). Null : false-flip rate of a matched off-corridor box that should stay feasible (ideal 0 ). CF-pair (chance 0.5 ) and ∣Δp∣ (score change with vs. without the obstacle) are for the deployed model (GraspFC-PTX). The planner relabels every variant while the model stays at chance. Labels themselves differ across planners and arms ( 607 of 957 Panda-feasible grasps are out of xArm6 reach); the flip does not. Per-condition values: supp. .
Predictor
Responds to
Stops below
Native
Interv.
GRN †
presence, bbox
sub-bbox shape
0.995
0.500
SceneCollisionNet †
contact, coarsely
12.5 cm voxel
0.969
0.821
PIGINet †
object tokens
corridor geometry
0.86
0.470
GraspFC-NNet
presence
whether it blocks
—
0.491
GraspFC-Conv3D
voxel grid
any obstacle
—
0.497
GraspFC-PTX
(no response)
any obstacle
—
0.498
Table 4: The physical-sensitivity spectrum on one protocol, for all eight audited predictors. Responds to is the coarsest geometry that moves the score; stops below is where the response no longer tracks the planner. † Released weights on the method’s own data and robot: Native is our AUROC, Interv. the counterfactual score on that same set (GRN and SceneCollisionNet: shape swap; PIGINet: insert-a-box). Each reproduces its authors’ published headline first, none of which is an AUROC: GRN’s action, grasp, and IK F1 and obstruction MAE [ 2 ] , SceneCollisionNet’s trajectory accuracy and AP [ 4 ] , PIGINet’s validation plan-feasibility and planning-time cut [ 7 ] ; values in supp. . The remaining rows are on the shared benchmark, where Native is Tab. 1 ’s Nat. column, so Interv. compares within a regime and not across. GraspFC-NNet and GraspFC-PTX differ by 28× in how strongly the obstacle moves their score ( ∣Δp∣ = 0.027 / 0.001 ), yet neither tracks collision. Per-model mechanisms are in supp. .
Figure 4: PIGINet on the authors’ own PR2 kitchen ( 35 -body saved scene, top-left; released weights). At one fixed arm pose: a clear approach, a harmless box beside the target, and a box blocking the approach. The planner relabels only the blocking case infeasible (red), yet PIGINet returns the identical p=1.00 in all three, so its score does not register the corridor box.
Figure 5: SceneCollisionNet on a cluttered ACRONYM-style tabletop, released checkpoint. A gripper-held cube is swept against a solid obstacle from overlapping it, through contact, to a 6 cm clear gap. The true label flips (red to teal), yet the score holds at pcoll=0.76 ( 0.75 to 0.77 over the sweep, never below 0.5 ), where an ideal checker steps from 1 to 0 . The whole sweep is shorter than the 12.5 cm voxel it reasons over.
Figure 6: GRN on its own panda_3d_4 scenes ( 30 -body environment, left; released weights). GRN detects inserted geometry, dropping from feasible 1.00 (clear) to 0.00 when a multi-object cluster blocks the approach. What it does not resolve is sub-bounding-box shape (rightmost, close-up from a rollout): a thin pole leaning 40∘ into the approach corridor is read as an upright box and called feasible ( p=0.85 ), yet the gripper drives straight through it. GRN reads the bounding box, not the shape ( Tab. 4 ).
Encoder
Supervision
Δ feat.
Probe R2
PointNet++
recognition
−0.014
0.03
PointNeXt-S
recognition
+0.001
0.05
PTv3
feasibility
+0.002
0.01
Clearance tokens (ours)
clearance
+1.48
0.71
Table 5: Frozen-encoder response to a feasibility-flipping obstacle. Supervision is what each encoder was trained to encode (recognition on ModelNet40 [ 32 ] , feasibility, or clearance); PTv3 is the deployed model’s own backbone, so it is the in-domain control. Δ feat. is the pooled-feature change in native units (within-row only, not comparable across rows); Probe R2 is the cross-encoder readout of how well a linear map recovers corridor clearance. Only the clearance tokenizer both responds and is readable. These are frozen encoders probed in isolation, and are not comparable to the deployed forward-pass displacements quoted in the text.
Intervention
Nat.
CF-pair
Verdict
More natural data
0.990
0.498
term not identified
Fine-tune deployed
0.990
0.495 to 0.508
does not transfer
Train from scratch on CF
0.990
0.991
fits intervention
Table 6: More data does not teach a transferable collision term; the from-scratch cure fits the intervention and is placement-bound ( Fig. 3 ).
Figure 7: Train-time learning on the unmodified benchmark (plain BCE, no counterfactual scenes). Three representations of the same corridor points, all high on the benchmark; only the clearance-aware tokenization separates under the protocol (CF-pair 0.983 ). Dashed: the param-free geometric readout.
Figure 8: Out-of-distribution transfer of the three cured models of Sec. 4.5 (generic, slab, clearance). Each panel is a held-out obstacle family, planner-relabeled; bars are AUROC in- versus out-of-distribution and the dashed line is chance ( 0.5 ). Cures transfer at or near their in-distribution score; box-to-sphere error bars are bootstrap 95% CIs ( n=532 ID, 250 OOD).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Left : every predictor in falls from near-perfect natural AUROC to near 0.5 under the same-pose counterfactual (one line each; GraspFC-NNet red). Right : for GraspFC-NNet, the blocked grasp gets essentially the clear grasp’s score, so the same-pose pairs whose label flips lie on the y=x diagonal rather than in the region a collision-aware model would occupy (shaded).
Figure 10: The obstacle is a GraspNet-1B object (from the scene segmentation, same Novel split) resting on the table next to the target. The planner relabels the grasp infeasible; the deployed SOTA returns the identical score. Over 150 such pairs the planner flips every one while the model cannot tell the two apart (CF-pair AUROC 0.5 ): the result extends to dataset objects, not only inserted boxes.
Robot
Planner
Agree
κ
Flip
Panda
BiRRT-Connect, re-run (control)
0.995
0.990
1.000
Panda
BiRRT-Connect, other hyper-params
0.990
0.977
1.000
Panda
single-tree goal-biased RRT
0.973
0.940
1.000
Panda
Lazy-PRM
0.992
0.983
1.000
xArm6
BiRRT-Connect, re-run (control)
0.988
0.973
1.000
xArm6
BiRRT-Connect, other hyper-params
0.989
0.975
1.000
Appendix
Table 7: What the causal edit does to the label, under four planning algorithms and two kinematic chains. The labels themselves are not identical across conditions; the flip is. Only the planner object or the robot changes; the scene, the grasp poses, the contact tolerance, and the 21 -step approach discretization are the same code. Agree is label agreement with the released oracle and κ is Cohen’s κ , both over the clear, causal, and null conditions. Flip is the fraction of causal edits the planner relabels infeasible. The causal edit flips the label in 8 of 8 conditions without exception, over 8,412 planner calls. Residual disagreement sits on the clear and null branches and is bounded by the RNG-only control in the first row of each block. The xArm6 is the harder embodiment: 607 of 957 Panda-feasible candidate grasps are out of its reach, and every corridor insertion still flips the label.
Condition
Planner
Failure mode
model p
Clear approach
feasible
none
0.975
Floating corridor box
infeasible
blocked approach
0.976
Resting pillar
infeasible
blocked approach
0.975
Transit-path box
infeasible
blocked transit
0.975
Appendix
Table 8: Per-condition scores for the worked example of : one grasp under four conditions (deployed SOTA, released weights).
Figure 11: as a chart: planner relabel rate (teal) versus deployed-SOTA CF-pair AUROC (amber) across three standoffs and two further obstacle types (a gravity-resting pillar and a transit-path box). The dashed line at 0.5 applies only to the amber AUROC, not to the teal flip rate.
Figure 12: Dense multi-object clutter on a GraspNet scene: a cluster of four solid obstacles inserted into the approach makes the planner relabel the grasp infeasible (right), while the deployed SOTA returns the same p(feasible) as the clear scene.
Figure 13: Closed-loop safety demo on one counterfactual scene (Novel split, scene 160 ), path-traced in Isaac Sim by replaying the pybullet link poses. Left : the pose-shortcut model’s top pick drives the gripper into the inserted box (pybullet-verified collision). Right : the clearance-aware screen of the same 20 candidates returns a collision-free top-1 (BiRRT-verified).
Figure 14: PIGINet on its own PR2 kitchen (released weights), aggregated over 450 counterfactual pairs. Its binarized feasibility call is flat ( 87 to 90% ) across a clear approach, a harmless box beside the target, and a box blocking the approach, while the planner ground truth drops to zero for the blocking case.
Figure 15: The protocol at the plan level. A multi-step rearrangement plan is plan-feasible when the scene is clear (top); inserting an obstacle at one step makes the planner relabel that step and every step after it infeasible (bottom), yet a plan-feasibility predictor returns the same score.
Figure 16: GRN on its panda_3d_4 benchmark, released weights. Left: GRN detects an inserted obstacle (feasibility drops only when the box is on the path). Right: a box and its inscribed same-bounding-box ellipsoid get an identical score (shape-swap AUROC 0.500 ).
Model
Representation (what enters the net)
Training pool
Par.
Objective
Authors’ reported (own data)
Nat.
GraspFC-NNet
pose +0.3 m grasp-local crop, pooled MLP
benchmark train split
1.65M
BCE
[ 3 ]
0.995
GraspFC-Conv3D
pose + voxel grid, 3D CNN
benchmark train split
9.7M
BCE
[ 3 ]
0.976
GraspFC-PTX
pose + point cloud, PTv3 backbone
benchmark train split
9.0M
BCE
[ 3 ]
0.996
SceneCollisionNet
per-point MLP, scatter-max to 12.5 cm voxel, 3D conv; PointNet++ gripper
Table 9: Per-model representation, training pool, and in-distribution accuracy. Nat. is the shared-benchmark natural AUROC of (GRN on its own panda_3d_4 task); Authors’ reported is each method’s published headline on its own data, none of which is AUROC, which is why we re-measure all of them under one Nat. The re-implementations are trained on planner labels under one protocol; PIGINet loads its released head and the GRN run matches the paper’s F1/MAE.
Model
Resolves at
Nat.
CF-pair
Failure mechanism
GraspFC-NNet
crop pool
0.995
0.491
registers the obstacle; feature displacement is direction-random
GraspFC-Conv3D
voxel grid
0.976
0.497
obstacle in the voxel grid, discarded
GraspFC-PTX
point pool
0.996
0.498
obstacle in the cloud, discarded
SceneCollisionNet
12.5 cm voxel
0.964
0.470
contact boundary below the 12.5 cm voxel
PIGINet
object token
0.960
0.500
one object token, no clearance encoded
NFC
2 -ch depth
0.965
0.504
predicts from pose, does not use neighbor channel
Appendix
Table 10: Representation-to-failure taxonomy. “Resolves at” is the coarsest collision geometry each representation can express; Nat. and CF-pair are from . ∗ GRN moves on presence (it has a supervised obstruction head fed the box), so its entry is the same-bounding-box shape swap, dissected in Appendix E .
Model
clear p
obstacle p
Δp
GraspFC-PTX
0.947
0.947
0.000
SceneCollisionNet
0.70
0.72
+0.02
PIGINet
0.766
0.766
0.000
NFC
0.798
0.797
−0.001
AGFP-Net
0.702
0.701
+0.001
GRN (released)
0.990
0.007
−0.983
Appendix
Table 11: Behavioral gallery: predicted p(feasible) on a matched set before and after inserting the flip-causing obstacle, which the planner relabels feasible → infeasible in every row. Values are means over the matched set; the worked example of ( 0.975 ) is one such pair.
Hole
GRN AUROC
Mislabels
Shape
0.500
14% (false block)
Tilt
0.500
72.5% (false clear)
Concavity
0.500
100% (false block)
Appendix
Table 12: GRN passes presence and bounding-box clearance but is at chance (AUROC 0.5 ) for geometry finer than its bounding box: three sub-bounding-box holes (shape, tilt, concavity), each a matched pair GRN scores identically while the corridor oracle flips. “Mislabels” is the fraction of probed targets the hole gets wrong; the failures span both error directions (false block and false clear).
Figure 17: The clearance-aware tokenization of , the implemented 23 k-parameter network. The grasp-local cloud and pose enter in the gripper frame; the 10 cm approach corridor is sliced into K=13 axial slabs, each emitting a three-number token (minimum lateral clearance ρk , occupancy, slab center). A shared per-slab MLP, a max-pool over slabs, and the global minkρk form a clearance feature hC fused with a pose feature hR (the R⊗C factorization).
Counterfactual evaluation tests how models, explanations, and policies behave under hypothetical changes to inputs, evidence, actions, or environments.