Recent advances in robotic manipulation have highlighted the effectiveness of learning from demonstration. However, while end-to-end policies excel in expressivity and flexibility, they struggle both in generalizing to novel object geometries and in attaining a high degree of precision. An alternative, object-centric approach frames the task as predicting the placement pose of the target object, providing a modular decomposition of the problem. Building on this goal-prediction paradigm, we propose TAX-DPD, a hierarchical, disentangled point diffusion framework that achieves state-of-the-art performance in placement precision, multi-modal coverage, and generalization to variations in object geometries and scene configurations. We model global scene-level placements through a novel feed-forward Dense Gaussian Mixture Model (GMM) that yields a spatially dense prior over global placements; we then model the local object-level configuration through a novel disentangled point cloud diffusion module that separately diffuses the object geometry and the placement frame, enabling precise local geometric reasoning. Interestingly, we demonstrate that our point cloud diffusion achieves substantially higher accuracy than a prior approach based on SE(3)-diffusion, even in the context of rigid object placement. We validate our approach across a suite of challenging tasks in simulation and in the real-world on high-precision industrial insertion tasks. Furthermore, we present results on a cloth-hanging task in simulation, indicating that our framework can further relax assumptions on object rigidity.
Figures & tables
Fig. 2: Method Overview. ( Left ) Our Global Placement Initialization samples a rough global position using a novel dense GMM-based prediction module, a framework that models highly multi-modal placement distributions at the scene-level. ( Right ) Our Local Configuration Refinement then proceeds with a novel disentangled shape and reference frame diffusion that simultaneously allow precise and dense goal predictions.
Fig. 3: RPDiff Task Environments. ( Top ) Our experiments span various multi-modal placement tasks with significant object and scene variation. ( Middle ) TAX-DPD is able to precisely model goal configuration as point clouds. ( Bottom ) Successful executions of our model’s goal predictions.
Mug/EasyRack
Mug/MedRack
Mug/Multi-MedRack
Book/Shelf
Can/Cabinet
Average
TAX3D [ 12 ]
0.84
0.46
0.32
0.38
0.42
0.48
RPDiff [ 15 ] (w/o classifier-based reranking)
-
-
-
-
-
0.83
RPDiff [ 15 ]
0.92
0.83
0.86
0.94
0.85
0.88
TAX-DPD (Ours) w/o disentangled point diffusion
0.97
0.74
0.61
0.53
0.77
0.72
TAX-DPD (Ours) w/ MLP encodings
0.99
0.84
0.81
0.61
0.64
0.78
TAX-DPD (Ours) w/o GMM
1.00
0.87
0.74
0.75
0.79
0.83
TABLE I: Ablations and Task Success Rates on RPDiff tasks.
OneMug
ManyMugs
TAX-DPD (Ours) w/ SE(3) diffusion
0.97
0.89
TAX-DPD (Ours) (Point diffusion)
0.98
0.95
TABLE II: Additional Ablations Comparing Point Diffusion to SE(3) Diffusion on variations of the Mug Hanging task.
Unimodal
Multimodal
Waterproof
DSUB-25
SSD
Waterproof
TAX-Pose
TAX-DPD (Ours)
TAX-Pose
TAX-DPD (Ours)
TAX-Pose
TAX-DPD (Ours)
TAX-DPD (Ours)
Success Rate
80% (16/20)
100% (20/20)
80% (16/20)
80% (16/20)
0% (0/20)
85% (17/20)
90% (18/20)
Trans. Err. (mm)
1.04
0.72
0.93
1.00
16.18
2.75
-
Rot. Err. ( ∘ )
1.64
1.18
3.16
1.36
13.81
2.77
-
TABLE III: Task Success Rates and Precision Metrics on Real-World Insertion Tasks. For the multimodal Waterproof task, translation and rotation errors are omitted because there is no single canonical target pose.
Fig. 4: Coverage vs. Precision. We further evaluate TAX-DPD and the baselines on coverage and precision with increasing numbers of inference samples on the RPDiff task Book/Shelf .
Fig. 6: DEDO Task Environment. ( Top ) A visualization of the variation in configuration in the HangProcCloth-DH task. ( Middle ) Our method generalizes well to diverse cloth geometries. ( Bottom ) Successful placements with a goal-conditioned placement policy.
SR
Cov. RMSE
Prec. RMSE
TAX3D [ 12 ]
0.50
0.87
1.34
Ours w/o disentangled point diffusion
0.73
0.82
1.16
Ours w/ MLP encodings
0.76
0.96
1.04
Ours w/o GMM
0.69
0.88
0.96
Ours w/o recon. embedding
0.75
0.82
0.81
Ours w/o rot. noise
0.64
0.64
0.95
Appendix
TABLE IV: Ablations and Task Success Rates on modified DEDO HangProcCloth task.
Parameter
Value
Batch size
16
Learning rate
1e-4
Learning rate warmup steps
100
Weight decay
1e-5
Epochs
20,000
Number of DiT blocks
5
Appendix
TABLE V: Training and Model Hyper-parameters.
RPDiff-Mug/EasyRack
number of PO points = 512 number of PS points = 512
RPDiff-Mug/MedRack
number of PO points = 512 number of PS points = 512
RPDiff-Mug/Multi-MedRack
number of PO points = 512 number of PS points = 1024
RPDiff-Book/Shelf
number of PO points = 512 number of PS points = 1024
RPDiff-Can/Cabinet
number of PO points = 256 number of PS points = 1024
DEDO-HangProcCloth
number of PO points = 512 number of PS points = 512
Appendix
TABLE VI: Point Cloud Downsampling per Task.
Fig. 7: Data collection process for the vision-based insertion skill. The number of cycles, pose variation ranges, and setup parameters are defined using a YAML configuration file.