GraspLDP: Towards Generalizable Grasping Policy via Latent Diffusion
Authors: Enda Xiang, Haoxiang Ma, Xinzhu Ma, Zicheng Liu, Di Huang
Organizations: State Key Laboratory of Complex and Critical Software Environment, Beihang University, Beijing, 100191, China · School of Computer Science and Engineering, Beihang University, Beijing, 100191, China
This paper focuses on enhancing the grasping precision and generalization of manipulation policies learned via imitation learning. Diffusion-based policy learning methods have recently become the mainstream approach for robotic manipulation tasks. As grasping is a critical subtask in manipulation, the ability of imitation-learned policies to execute precise and generalizable grasps merits particular attention. Existing imitation learning techniques for grasping often suffer from imprecise grasp executions, limited spatial generalization, and poor object generalization. To address these challenges, we incorporate grasp prior knowledge into the diffusion policy framework. In particular, we employ a latent diffusion policy to guide action chunk decoding with grasp pose prior, ensuring that generated motion trajectories adhere closely to feasible grasp configurations. Furthermore, we introduce a self-supervised reconstruction objective during diffusion to embed the graspness prior: at each reverse diffusion step, we reconstruct wrist-camera images back-projected the graspness from the intermediate representations. Both simulation and real robot experiments demonstrate that our approach significantly outperforms baseline methods and exhibits strong dynamic grasping capabilities.
Figures & tables
Figure 2: Framework of proposed GraspLDP. In Action Latent Learning stage action chunks are refined under the guidance of a grasp pose in latent space encoded by a VAE. In Diffusion on Latent Action Space stage the graspness cue is used to condition the diffusion model’s denoising process and to reconstruct for enhancement.
Table 1: Results of evaluation in simulator. In Domain denotes cases where both the objects and their poses were present in the training data; Spatial Generalization measures how well the model handles those training objects placed in unseen poses; Object Generalization assesses performance on entirely novel objects; and Visual Generalization tests robustness under visual disturbances like lighting changes. The † refers to the model that has been fine-tuned on our dataset for fair comparison.
Method
ID
SG
OG
VG
SR ↑
GFE ↓
SR ↑
GFE ↓
SR ↑
GFE ↓
SR ↑
GFE ↓
GraspLDP
80.3
1.33
71.1
1.61
58.2
2.10
64.6
1.92
w/o GC
77.4 (-2.9)
1.49 (+0.16)
67.3 (-3.8)
1.81 (+0.20)
54.2 (-4.0)
2.33 (+0.23)
57.5 (-7.1)
2.30 (+0.38)
w/o LG w/ CG
73.5 (-6.8)
1.58 (+0.25)
62.2 (-8.9)
2.07 (+0.46)
52.3 (-5.9)
2.42 (+0.32)
54.5 (-10.1)
2.35 (+0.43)
w/o LG
60.6 (-19.7)
-
49.8 (-21.3)
-
21.2 (-37.0)
-
19.4 (-45.2)
-
w/o GC & LG
55.1 (-25.2)
-
46.2 (-24.9)
-
16.0 (-42.2)
-
15.7 (-48.9)
-
Table 2: Results of ablation study. ID, SG, OG, and VG denote In Domain, Spatial, Object and Visual Generalization, respectively. GC and LG denotes Graspness Cue and Latent Guidance. CG denotes Condition Guidance used in Ours Baseline.
Figure 4: Inference latency of three methods on an RTX 4090 GPU, with the policy action horizon aligned to 8 for each inference. Results of GraspVLA are after acceleration with torch.compile() .
Figure 5: Qualitative experimental analysis. (a) Grasping trials using objects ”mug”, ”mustard bottle”, and ”thera med” in simulator. (b) Real world grasping trials corresponding to in domain, object generation, and visual generation performance. In particular, we use colored LED strips in low-light conditions to simulate visual interference.
Method
ID
SG
OG
VG
w/ random
66.8 (-13.5)
60.6 (-10.5)
51.7 (-6.5)
54.5 (-10.1)
w/ highest
72.0 (-8.3)
63.1 (-8.0)
55.4 (-2.8)
58.2 (-6.4)
w/ nearest
69.5 (-10.8)
61.8 (-9.3)
53.8 (-4.4)
56.0 (-8.6)
w/ HPS
80.3
71.1
58.2
64.6
Table 3: Ablation study on selection strategy of grasp pose.
Figure 6: Settings of real world evaluation and daily objects used for training and evaluation.
Method
ID&SG
OG
VG
Avg
Diffusion Policy [ 7 ]
65.0
43.0
21.0
43.0
GraspVLA [ 9 ]
29.0
27.0
22.0
26.0
Anygrasp [ 11 ]
82.0
78.0
79.0
79.7
GraspLDP
84.0
75.0
77.0
78.7
Table 4: Results of real world evaluation. Because it’s difficult to ensure identical object poses in the real world, we merge the ID and SG splits into a single evaluation set.
Method
Scene1
Scene2
Scene3
Scene4
Avg
SR ↑
SCR ↑
SR ↑
SCR ↑
SR ↑
SCR ↑
SR ↑
SCR ↑
SR ↑
SCR ↑
Diffusion Policy [ 7 ]
10.0
20.0
0.0
0.0
20.0
28.6
10.0
12.5
10.0
15.4
GraspVLA [ 9 ]
20.0
40.0
0.0
0.0
30.0
42.9
10.0
12.5
15.0
23.1
Anygrasp [ 11 ]
83.3
100.0
66.7
100.0
77.8
100.0
60.0
75.0
70.1
92.3
GraspLDP
83.3
100.0
100.0
100.0
100.0
100.0
60.0
75.0
82.8
92.3
Table 5: Results of cluttered scenarios evaluation in real world.
Figure 7: Detailed results of in domain&spatial generalization and object generalization in the real world.
Method
banana moving
watermelon moving
mug handover
Diffusion Policy [ 7 ]
✗ / ✗ / ✗
✗ / ✗ / ✗
✓ / ✗ / ✗
GraspVLA [ 9 ]
✓ / ✓ / ✗
✓ / ✗ / ✗
✗ / ✗ / ✗
Anygrasp [ 11 ]
✓ / ✓ / ✗
✓ / ✗ / ✗
✓ / ✗ / ✗
GraspLDP
✓ / ✓ / ✗
✓ / ✓ / ✓
✓ / ✓ / ✗
Table 6: Results of dynamic grasp task.
Method
12K
1.2K
120
Avg
Diffusion Policy
62.8
41.5
13.8
39.4
GraspLDP
80.3 (+17.5)
64.6 (+23.1)
43.1 (+29.3)
62.7 (+23.3)
Table 7: Results of In-domain evaluation with few demonstrations.
Method
GSNet
SBG
GraspNet
Grasp Detection
78.5
73.8
70.8
GraspLDP
80.3 (+1.8)
76.9 (+3.1)
75.4 (+4.6)
Table 8: Results of In-domain setting with different grasp detector.
Figure 8: Illustration of the data collection pipeline in simulator.
Figure 9: Language label for each object.
Figure 10: Initial object placements in cluttered scenerios evaluation from scene 1 to scene 4.
Parameter
Value
horizon
16
n_obs_steps
2
n_latent_dims
16
use_conv_encoder
True
conv_latent_dims
64
conv_layer_num
1
Table 9: Training and inference hyperparameters of Action Latent Learning.
Parameter
Value
observation_horizon
2
prediction_horizon
16
action_horizon
8
unet.diffusion_step_embed_dim
256
unet.down_dims
[256,512,1024]
unet.kernel_size
5
Table 10: Training and inference hyperparameters of Diffusion on Latent Action Space.
Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Sweden. · Robotics and Autonomous Systems at University of Turku, Finland.