Organizations: College of Computer Science, Sichuan University · Dexmal · School of Information and Communication Engineering, University of Electronic Science and Technology of China
Imitation learning for robotic manipulation has progressed from 2D image policies to 3D representations that explicitly encode geometry. Yet purely geometric policies often lack explicit part-level semantics, which are critical for pose-aware manipulation (e.g., distinguishing a shoe's toe from heel). In this paper, we present HeRO, a diffusion-based policy that couples geometry and semantics via hierarchical semantic fields. HeRO employs dense semantics lifting to fuse discriminative, geometry-sensitive features from DINOv2 with the smooth, globally coherent correspondences from Stable Diffusion, yielding dense features that are both fine-grained and spatially consistent. These features are processed and partitioned to construct a global field and a set of local fields. A hierarchical conditioning module conditions the generative denoiser on global and local fields using permutation-invariant network architecture, thereby avoiding order-sensitive bias and producing a coherent control policy for pose-aware manipulation. In various tests, HeRO establishes a new state-of-the-art, improving success on Place Dual Shoes by 12.3% and averaging 6.5% gains across six challenging pose-aware tasks. Code is available at https://github.com/Chongyang-99/HeRO.
Figures & tables
Fig. 1: Pose-Aware Manipulation with Semantic Understanding. Top: Many manipulation tasks are pose-aware (e.g., placing shoes with toes aligned left), demanding semantic part perception from policies. Bottom Left: Our dense semantic fields are smoother and more consistent than the baseline G3Flow [ 13 ] . Bottom Right: Our method achieves a 12.3% higher success rate on the dual shoe place task and a 6.5% higher average success rate of 6 challenging tasks.
Fig. 2: Comparison of Conditioning Mechanisms. Top: The baseline employs a holistic conditioning approach, encoding the entire object point cloud into a single global vector, which lacks part-level details. Bottom: Our method uses a hierarchical approach. A global encoder captures overall context, while additional encoders extract complementary local features for fine-grained details. The resulting Hierarchical Condition provides both global and local information, enabling more precise manipulation.
Fig. 3: Method Overview. Our framework generates precise, pose-aware actions in a two-stage process. (a) Dense Semantic Lifting: We track object 6D poses from sequential frames and lift fused 2D features from DINOv2 (semantic) and Stable Diffusion (geometric) into an object-centric Dense Semantic Point Cloud . (b) Hierarchical Conditioning Module: This point cloud is abstracted into a hierarchical Object Part Point Cloud , which conditions the diffusion policy via two pathways: as a Global Condition for the action denoiser and as fine-grained local guidance injected by a Permutation-invariant Refine module at each denoising step. This dual mechanism enables precise, pose-aware action generation.
Fig. 4: Hierarchical Conditioning Module Architecture. Our model uses a dual-pathway design to guide the Action Denoiser. The Global Path processes the entire point cloud into a single global condition for high-level context. The Local Path partitions the point cloud into semantic parts, which are encoded by Permutation-invariant Refiners into a set of fine-grained embeddings. These local conditions are injected into the denoiser, enabling actions that are both globally consistent and locally precise.
Fig. 5: Simulated Manipulation Tasks. We evaluate our method on six challenging tasks from the RoboTwin 2.0 benchmark. These tasks necessitate precise pose estimation and a nuanced understanding of object-part semantics to facilitate successful interaction.
Fig. 6: Real-World Experimental Setup. Our experimental setup consists of an AgileX Cobot Magic dual-arm robot equipped with a RealSense D435i head-mounted camera.
Method
Place Dual Shoes
Scan Object
Hanging Mug
Place Phone Stand
Pick Diverse Bottles
Place Object Scale
Average
DP [ 6 ]
7.0 ±3.6
10.3 ±2.1
15.3 ±2.6
37.7 ±6.2
18.3 ±3.1
2.7 ±1.2
15.2
DP3 [ 8 ]
17.7 ±3.4
23.3 ±1.2
23.3 ±6.3
52.0 ±2.9
27.7 ±7.0
10.7 ±3.9
25.8
DP3 w/ color [ 8 ]
16.3 ±2.9
14.6 ±1.2
21.7 ±5.6
41.7 ±6.1
17.0 ±5.1
4.7 ±1.2
19.3
G3Flow [ 13 ]
20.7 ±3.4
22.0 ±3.7
26.7 ±2.5
45.3 ±3.3
32.0 ±8.0
7.7 ±0.9
25.7
Ours
33.0 ±5.7
26.7 ±0.5
31.0 ±1.4
55.3 ±1.9
34.7 ±3.8
11.7 ±1.2
32.3
TABLE I: Comparative Evaluation on the 6 tasks in RoboTwin 2.0 Benchmark with standard setting.
Method
Place Dual Shoes
Scan Object
Hanging Mug
Average
DP [ 6 ]
7.3 ±2.1
4.0 ±2.2
5.0 ±2.2
5.4
DP3 [ 8 ]
11.3 ±3.4
11.3 ±1.2
22.7 ±1.9
15.1
DP3 w/ color [ 8 ]
10.0 ±1.4
8.3 ±1.2
19.7 ±4.2
12.7
G3Flow [ 13 ]
16.3 ±2.9
11.7 ±2.6
25.0 ±3.7
17.7
Ours
28.3 ±5.3
14.3 ±0.5
30.7 ±4.2
24.4
TABLE II: Cross-object generalization results (success rates in %).
Method
Place Dual Shoes
Scan Object
Hanging Mug
Average
DP3 [ 8 ]
5
15
0
6.7
G3Flow [ 13 ]
10
30
10
16.7
Ours
25
35
20
26.7
TABLE III: Real-world experiment results (success rates in %).
Fig. 7: Qualitative comparison of policy execution. For Place Dual Shoes (left), G3Flow executes an erroneous rotation, resulting in task failure, whereas our method achieves correct alignment. For Hanging Mug (right), G3Flow fails to align the handle with the rack. In contrast, our policy’s fine-grained semantic representation enables precise handle orientation and successful task completion, both in simulated environments and on the physical robot.
Fig. 8: Visualization of object semantics field. We visualize the semantics field of G3Flow and our method in simulation versus real-world data. G3Flow exhibit significant noise and geometric inconsistency. In contrast, our method generates smooth, coherent semantic fields that robustly delineate functional parts across both domains, providing a more stable representation for the policy.
Dense Semantic
Global Pose- aware Condition
Part-aware Geometry Refine
Average (%)
23.1
✓
23.7
✓
24.3
✓
27.6
✓
✓
✓
30.2
TABLE IV: Ablation Study of Model Components. We evaluate the contribution of each key module by measuring the average success rate (%) across 3 benchmark tasks.