Equivariant Volumetric Grasping

Pinhao Song; Yutong Hu; Pengteng Li; Renaud Detry

Figure 1: An illustration of rotational equivariance in robotic grasp prediction. When the scene ( left ) is rotated by

90^{\circ}

( bottom ), the predicted grasps ( right ) also rotate by the same amount, ensuring consistent grasp orientations for the transformed scene.

Figure 2: (a) Trivial representation and its transformation. (b) Irreducible representation and its transformation. (c) Regular representation and its transformation under

\textnormal{C}_{4}

. For (a-c), see text for details. (d) The diagram illustrates that

h

, an equivariant function, commutes with the group action. Applying a group transformation

g

to the input feature

f_{\text{in}}

results in transformed output

f_{\text{out}}

, where the transformation is carried through

h

via consistent actions defined by

\rho_{\text{in}}

and

\rho_{\text{out}}

. Thus, applying

g

before

h

(top-right path) is equivalent to applying

g

after (bottom-left path).

Figure 3: Illustration of equivariant manifestations of

90^{\circ}

Z

-axis rotations of a 3D scene composed of a cube and a cone. Rows labeled

XY

,

XZ

,

YZ

show projections of the scene on the three planes. Each column corresponds to a

90^{\circ}

Z

-axis rotation of the one that precedes it. The

XY

plane transforms equivariantly under the

\textnormal{C}_{4}

group.

XZ

and

YZ

do not exhibit trivial equivariance, but they do exhibit a predictable response to

90^{\circ}

Z

-axis rotations in the form of

XZ

-

YZ

permutations occasionally accompanied by an in-plane reflection. The black star is the queried position.

Figure 4: The architecture of the Equivariant Tri-plane UNet. Given a TSDF input, the model applies a single 3D steerable convolution to lift the volumetric data into a 3D feature field, which is then projected onto three canonical planes. The

XY

-plane features are processed by a

\textnormal{C}_{4}

-equivariant UNet, while the

XZ

- and

YZ

-plane features are handled by a reflection-equivariant UNet. The outputs from both branches are fused into a unified tri-plane feature field. A vertical-to-tableplane feature augmentation (V2TP) module enables communication between the two equivariant branches, enhancing feature representation through structured interactions.

Figure 5: Illustration of the vertical-to-tableplane feature augmentation. Features from the vertical planes (XZ and YZ) are mean-pooled along the Z-axis to reconstruct auxiliary XY-plane features. These reconstructed features are then fused with the original XY-plane features to form the updated XY-plane representation for the features at the next UNet layer.

Figure 6: Illustration of the EquiGIGA and EquiIGD workflows. This figure is based on graphical elements from GIGA ( Jiang et al., 2021 ) and IGD ( Song et al., 2024 ) . Given a feature queried at position

\mathbf{p}

from the tri-plane feature grid, (a) EquiGIGA directly regresses the rotation

\mathbf{r}

and predicts the graspness

a

at

\mathbf{p}

; (b) EquiIGD uses Equivariant Rotation Flow to sample

\mathbf{r}

and additionally evaluates grasps with the grasp classifier

v

.

Figure 7: (a) The architecture of Equivariant Deformable Attention. (b) The architecture of Equivariant Grasp-conditioned Deformable Attention Module. This figure is based on graphical elements from DAM and GraspDAM in IGD ( Song et al., 2024 ) . See text for details.

Figure 8: (a) Experimental setup for the real-world declutter experiment. (b) An illustration of the packed scene and the objects. (c) An illustration of the pile scene and the objects. (d) An illustration of the adversarial scene and the objects.

Method	Packed		Pile		Latency (ms)
Method	GSR (%)	DR (%)	GSR (%)	DR (%)	Latency (ms)
VGN* ( Breyer et al., 2021 )	72.5 $\pm$ 2.6	76.7 $\pm$ 1.7	59.3 $\pm$ 2.9	43.5 $\pm$ 2.9	9
GIGA* ( Jiang et al., 2021 )	84.8 $\pm$ 2.2	85.1 $\pm$ 2.5	69.5 $\pm$ 1.3	49.0 $\pm$ 3.4	24
GraspNet-1B Baselines* ( Fang et al., 2020 )	49.9 $\pm$ 2.3	40.1 $\pm$ 2.2	50.2 $\pm$ 4.2	30.0 $\pm$ 2.3	77
GSNet* ( Wang et al., 2021 )	67.8 $\pm$ 2.5	60.1 $\pm$ 3.2	58.3 $\pm$ 3.8	51.3 $\pm$ 4.6	156
GPD* ( Ten Pas et al., 2017 )	41.8 $\pm$ 2.9	34.1 $\pm$ 3.4	22.7 $\pm$ 1.1	9.0 $\pm$ 0.7	2138

Table 1: Simulated clutter-removal performance with a single fixed camera and gamma noise. We report mean

\pm

standard deviation for GSR and DR.

N

is the number of sampling rounds used by IGD and EquiIGD. Results marked “*” are from ( Song et al., 2024 ) , which uses the same experimental protocol and random seeds.

\dagger

means that we use their open-source checkpoint to evaluate on this benchmark. Latency is measured from a depth map to predicted grasps on an RTX 4070 GPU, except for OrbitGrasp, which is measured on an RTX A6000 because it requires more than 40 GB of GPU memory. The best and second-best reported values in each column are shown in blue boldface and pink underlined text, respectively.

Method	Packed		Pile		Latency (ms)
Method	GSR (%)	DR (%)	GSR (%)	DR (%)	Latency (ms)
PointNetGPD ( Liang et al., 2019 )	79.3 $\pm$ 1.8	82.5 $\pm$ 2.9	75.6 $\pm$ 2.3	77.0 $\pm$ 2.8	149
VGN ( Breyer et al., 2021 )	80.2 $\pm$ 1.6	86.2 $\pm$ 2.0	64.9 $\pm$ 2.2	69.1 $\pm$ 3.2	9
GIGA ( Jiang et al., 2021 )	89.9 $\pm$ 1.7	87.6 $\pm$ 2.0	76.3 $\pm$ 2.4	80.9 $\pm$ 4.1	24
GIGA (HR) ( Jiang et al., 2021 )	91.4 $\pm$ 1.5	88.5 $\pm$ 1.4	86.5 $\pm$ 1.2	80.8 $\pm$ 1.9	53
EdgeGraspNet ( Huang et al., 2023 )	92.5 $\pm$ 0.9	94.3 $\pm$ 1.1	91.5 $\pm$ 1.3	92.5 $\pm$ 1.3	843/685

Table 2: Simulated clutter-removal performance with a single random-view camera and Gaussian noise.

N

is the number of sampling rounds used by EquiIGD. We report mean

\pm

standard deviation for GSR and DR. Results for all methods except OrbitGrasp, SE(2)-EquiGrasp, EquiGIGA, and EquiIGD are reproduced from ( Zurbrügg et al., 2024 ) using its software and random seeds. Latency is measured from a depth map to predicted grasps. The best values are shown in blue boldface. Second-best values are shown in pink underlined text only when the best value is unique; tied best values receive the best-value style, and no second-best value is highlighted in that column.

Method	Packed		Pile		Adv
Method	GSR (%, 95% CI)	DR (%, 95% CI)	GSR (%, 95% CI)	DR (%, 95% CI)	GSR (%, 95% CI)	DR (%, 95% CI)
GIGA ( Jiang et al., 2021 )	76.7 (66/86) [66.8, 84.4]	88.0 (66/75) [78.7, 93.6]	61.1 (44/72) [49.6, 71.5]	58.7 (44/75) [47.4, 69.1]	72.5 (66/91) [62.6, 80.6]	88.0 (66/75) [78.7, 93.6]
EdgeGraspNet ( Huang et al., 2023 )	73.4 (58/79) [62.8, 81.9]	77.3 (58/75) [66.7, 85.3]	62.1 (41/66) [50.1, 72.9]	54.7 (41/75) [43.4, 65.4]	72.2 (57/79) [61.4, 80.8]	76.0 (57/75) [65.2, 84.2]
VN-EdgeGraspNet ( Huang et al., 2023 )	71.3 (57/80) [60.5, 80.0]	76.0 (57/75) [65.2, 84.2]	67.7 (44/65) [55.6, 77.8]	58.7 (44/75) [47.4, 69.1]	79.5 (58/73) [68.8, 87.1]	77.3 (58/75) [66.7, 85.3]
IGD ( $N$ =1) ( Song et al., 2024 )	78.0 (64/82) [67.9, 85.6]	85.3 (64/75) [75.6, 91.6]	58.0 (51/88) [47.5, 67.7]	68.0 (51/75) [56.8, 77.5]	78.2 (61/78) [67.8, 85.9]	81.3 (61/75) [71.1, 88.5]
ICGNet ( Zurbrügg et al., 2024 )	72.2 (57/79) [61.4, 80.8]	76.0 (57/75) [65.2, 84.2]	71.1 (54/76) [60.0, 80.0]	72.0 (54/75) [61.0, 80.9]	69.9 (51/73) [58.6, 79.2]	68.0 (51/75) [56.8, 77.5]

Table 3: Real-world clutter-removal performance. Each cell reports the observed percentage and its two-sided 95% Wilson confidence interval in brackets. GSR is followed by the numbers of successful grasps and total trials in parentheses; DR is followed by the number of removed objects out of 75 object instances.

N

is the number of sampling rounds used by IGD and EquiIGD.

Figure 9: Visualization of top-10 grasps in challenging scenes. Grasps whose quality is below a fixed threshold

T

are filtered out. The first row corresponds to EquiGIGA, and the second row corresponds to EquiIGD. Panels (a–c) show pile scenes, and panels (d–f) show packed scenes. Green grippers denote successful grasps, while red grippers denote failed grasps.

Figure 10: Different failure cases of EquiGIGA and EquiIGD. The first row corresponds to EquiGIGA, and the second row corresponds to EquiIGD. Panels (a–c) show pile scenes, and panels (d–f) show packed scenes. Green grippers denote successful grasps, while red grippers denote failed grasps.

No.	Ablated models	GSR (%)	DR (%)	GPU Memory (MB)	GFLOPs	Params (M)	Latency (ms)
1	3D UNet	86.8 $\pm$ 1.1	85.5 $\pm$ 1.5	17226	44.30	2.24	22
2	Eq. 3D UNet	89.9 $\pm$ 1.5	88.8 $\pm$ 2.0	17208	44.30	0.22	22
3	Tri-UNet	84.6 $\pm$ 1.1	85.3 $\pm$ 2.4	3485	1.90	0.84	24
4	Tri-UNet ( $XY$ -separated)	87.0 $\pm$ 1.2	84.8 $\pm$ 1.6	3292	1.90	1.61	24
5	Eq. Tri-UNet (no V2TP)	90.3 $\pm$ 0.6	87.4 $\pm$ 1.7	3860	1.90	0.40	24
6	Tri-UNet (with V2TP)	90.9 $\pm$ 1.4	86.6 $\pm$ 1.3	3405	1.95	1.69	25

Table 4: Ablation of the Equivariant Tri-plane UNet (Eq. Tri-UNet) in the grasp-regression setting. We report mean

\pm

standard deviation for GSR and DR. GFLOPs are measured for the backbone only with a

40\times 40\times 40

input; training GPU memory uses a batch size of 64; and inference latency is measured on an RTX 4070 GPU.

Figure 11: Qualitative comparison of EquiGIGA, GIGA++, and GIGA++A under

0^{\circ}

,

90^{\circ}

,

180^{\circ}

, and

270^{\circ}

planar rotations of the same input scene. We select the top ten query locations whose graspness scores exceed the threshold

T

at

0^{\circ}

and rotate these same locations with the input for the other columns. The top row shows predicted grasps; the remaining rows show normalized Grad-CAM activations for the graspness predictions in the

XY

,

XZ

, and

YZ

feature planes. Star markers indicate the projections of the selected 3D query locations onto the respective feature planes. Compared to GIGA++ and GIGA++A, EquiGIGA produces more consistently transformed grasp predictions and activation patterns.

Figure 12: Sample efficiency experiments on (a–b) EquiGIGA and (c–d) EquiIGD. We train the models with reduced fractions of the available training data and evaluate their performance. Results are averaged across five random seeds, and error bars indicate one standard deviation above and below the mean. For each given training set size, equivariant models (blue) generally outperform non-equivariant ones (green and orange). See text for detailed discussion.

Figure 13: We visualize the curves of four metrics during training in both the training and validation sets: average precision, occupancy loss, graspness loss, and rotation loss. The superior performance of EquiGIGA (blue) compared to GIGA++ and GIGA++A (green and orange) mainly stems from the lower rotation loss, which benefits from the model’s encoded equivariance.

Figure 14: Empirical equivariance consistency error under planar input rotations, measured as the relative

\ell_{2}

discrepancy between predictions from the rotated input and appropriately transformed predictions from the original input. EquiGIGA attains errors at floating-point precision for the non-identity

\textnormal{C}_{4}

rotations in all three heads. GIGA++A exhibits recurring error minima at these rotations for graspness and occupancy, but no corresponding minima for rotation prediction. Rotations outside

\textnormal{C}_{4}

involve TSDF resampling and therefore also introduce interpolation effects.

Figure 15: Robustness to camera viewpoint changes for models trained with a fixed acquisition viewpoint. The camera is moved while the tabletop coordinate frame and initial object arrangement are held fixed, and a new depth observation is integrated into the TSDF. (a,b) GSR and DR under camera azimuth offsets of

0^{\circ}

,

45^{\circ}

,

90^{\circ}

,

135^{\circ}

, and

180^{\circ}

, at a fixed elevation of

30^{\circ}

. (c,d) GSR and DR at camera elevations of

15^{\circ}

,

30^{\circ}

,

45^{\circ}

, and

60^{\circ}

, with the training azimuth fixed. Dashed lines indicate the training viewpoint; points and error bars denote the mean and standard deviation across five random seeds. EquiGIGA achieves the highest mean GSR and DR at every tested viewpoint, although its GSR decreases at intermediate azimuth offsets and both metrics decrease at an elevation of

60^{\circ}

. GIGA++A mitigates the azimuth sensitivity of GIGA++, with GSR recovering at offsets of

90^{\circ}

and

180^{\circ}

.

Figure 20

Figure 16: Ablation study of the number of sampling rounds in EquiIGD. (a) Packed scene. (b) Pile scene. Solid lines denote GSR, and dashed lines denote DR.

Orientation generation method	Graspness	EDA	GC	GSR (%)	DR (%)
FM (10-step)	✓			91.1 $\pm$ 2.5	89.1 $\pm$ 2.1
FM (10-step)	✓	✓		93.5 $\pm$ 1.0	91.1 $\pm$ 0.5
FM (10-step)	✓	✓	✓	97.3 $\pm$ 1.1	92.1 $\pm$ 1.2
DDPM (10-step)	✓	✓	✓	92.6 $\pm$ 0.9	88.1 $\pm$ 1.1
DDPM (100-step)	✓	✓	✓	92.6 $\pm$ 1.4	85.0 $\pm$ 2.2

Table 5: Ablation of EquiIGD’s grasp-orientation model and sampling method. FM denotes flow matching; the Graspness , EDA , and GC columns indicate the use of graspness prediction, Equivariant Deformable Attention, and the Grasp Classifier, respectively. We report mean

\pm

standard deviation for GSR and DR.

Equivariant Volumetric Grasping

Abstract

Figures & tables

Tri-Plane Representations

EquiGQNet: Fast Grasp Quality Evaluation via Shared Equivariant Point Cloud Encoding

VolCo: Volumetric Contact for High-Fidelity Human Grasp Generation

2D and 3D Grasp Planners for the GET Asymmetrical Gripper

Abstract

Tri-Plane Representations

Explore similar work

EquiGQNet: Fast Grasp Quality Evaluation via Shared Equivariant Point Cloud Encoding

VolCo: Volumetric Contact for High-Fidelity Human Grasp Generation

2D and 3D Grasp Planners for the GET Asymmetrical Gripper