Generalist robot policies built upon 2D visual representations excel at semantic reasoning but inherently lack the explicit 3D spatial awareness required for high-precision tasks. Existing 3D integration methods struggle to bridge this gap due to the structural irregularity of sparse point clouds and the geometric distortion introduced by multi-view orthographic rendering. To overcome these barriers, we present ReMAP-DP, a novel framework synergizing standardized perspective reprojection with a structure-aware dual-stream diffusion policy. By coupling the re-projected views with pixel-aligned PointMaps, our dual-stream architecture leverages learnable modality embeddings to fuse frozen semantic features and explicit geometric descriptors, ensuring precise implicit patch-level alignment. Extensive experiments across simulation and real-world environments demonstrate ReMAP-DP's superior performance in diverse manipulation tasks. On RoboTwin 2.0, it attains a 59.3% average success rate, outperforming the DP3 baseline by +6.6%. On ManiSkill 3, our method yields a 28% improvement over DP3 on the geometrically challenging Stack Cube task. Furthermore, ReMAP-DP exhibits remarkable real-world robustness, executing high-precision and dynamic manipulations with superior data efficiency from only a handful of demonstrations. Project page is available at: https://icr-lab.github.io/ReMAP-DP/
Figures & tables
Fig. 2: Overall architecture of ReMAP-DP . The method consists of three components: (1) Back projection and re-projection are used to obtain RGB and point maps for workspace alignment and denoising. (2) A dual stream encoder and transformer is used to process RGB and geometric features and fuse them together. (3) An action generation module employs a 1D-UNet Diffusion Policy to predict precise robot actions conditioned on the fused multi-modal features.
Fig. 3: Top: Visualization of ten selected tasks in RoboTwin 2.0 Benchmarks. Bottom: Visualization of five selected tasks in ManiSkill 3 Benchmarks.
Method
Beat Block Hammer
Blocks Ranking RGB
Click Bell
Turn Switch
Place Container Plate
Place Empty Cup
Stack Blocks Two
Stack Blocks Three
Stack Bowls Three
Stack Bowls Two
Avg.
Ours
0.70
0.03
1.00
0.60
0.80
0.68
0.50
0.08
0.72
0.82
0.593
DP3 [ 37 ]
0.72
0.03
0.90
0.46
0.86
0.65
0.24
0.01
0.57
0.83
0.527
ACT [ 39 ]
0.56
0.01
0.58
0.05
0.72
0.61
0.25
0.00
0.48
0.82
0.408
DP [ 4 ]
0.42
0.00
0.54
0.36
0.41
0.37
0.07
0.00
0.63
0.61
0.341
TABLE I: Comparison of different methods across various manipulation tasks. Our method achieves the best average performance.
Fig. 4: Left: Architecture of Geometry Encoder. Mid: Efficacy of Multi-View Projection. Right: Impact of Modality Embeddings.
Method
Push Cube
Stack Cube
Pick Cube
Lift Peg Upright
Peg Insertion Side
Avg.
Ours
0.95
0.28
0.48
0.36
0.01
0.416
DP3
0.92
0.01
0.24
0.75
0.01
0.386
DP
0.88
0.00
0.44
0.03
0.01
0.272
TABLE II: Success Rates on Maniskill 3 Benchmark
Fig. 5: Real-World Experiments . Top: Real-World experiments setup, including cameras’ positions and robotic arms’ settings. Bottom: Objects used in tasks and their sizes.
Task
Task Details
Beat w. Hammer
Strike target object with hammer head
Sweep Table
Sweep all rubbish into dustpan
Open Drawer PnP
Open drawer, insert cube, close drawer
Place Empty Cup
Stable placement in target area
Pouring
Transfer small balls without spilling
Stack Cube
Stable stack of red cube on blue cube
TABLE III: Detailed descriptions for real world tasks.
Task
Ours
DP3
DP
Beat w. Hammer
0.5
0.4
0.2
Sweep Table
0.5
0.3
0.1
Open Drawer PnP
0.3
0.1
0.0
Place Empty Cup
0.6
0.6
0.3
Pouring
0.6
0.4
0.1
Stack Cube
0.3
0.1
0.0
TABLE IV: Success rates (SR) across 10 trials for different methods.