Generalist robot policies built upon 2D visual representations excel at semantic reasoning but inherently lack the explicit 3D spatial awareness required for high-precision tasks. Existing 3D integration methods struggle to bridge this gap due to the structural irregularity of sparse point clouds and the geometric distortion introduced by multi-view orthographic rendering. To overcome these barriers, we present ReMAP-DP, a novel framework synergizing standardized perspective reprojection with a structure-aware dual-stream diffusion policy. By coupling the re-projected views with pixel-aligned PointMaps, our dual-stream architecture leverages learnable modality embeddings to fuse frozen semantic features and explicit geometric descriptors, ensuring precise implicit patch-level alignment. Extensive experiments across simulation and real-world environments demonstrate ReMAP-DP's superior performance in diverse manipulation tasks. On RoboTwin 2.0, it attains a 59.3% average success rate, outperforming the DP3 baseline by +6.6%. On ManiSkill 3, our method yields a 28% improvement over DP3 on the geometrically challenging Stack Cube task. Furthermore, ReMAP-DP exhibits remarkable real-world robustness, executing high-precision and dynamic manipulations with superior data efficiency from only a handful of demonstrations. Project page is available at: https://icr-lab.github.io/ReMAP-DP/
Figures & tables
Fig. 2: Overall architecture of ReMAP-DP . The method consists of three components: (1) Back projection and re-projection are used to obtain RGB and point maps for workspace alignment and denoising. (2) A dual stream encoder and transformer is used to process RGB and geometric features and fuse them together. (3) An action generation module employs a 1D-UNet Diffusion Policy to predict precise robot actions conditioned on the fused multi-modal features.
Fig. 3: Top: Visualization of ten selected tasks in RoboTwin 2.0 Benchmarks. Bottom: Visualization of five selected tasks in ManiSkill 3 Benchmarks.
Method
Beat Block Hammer
Blocks Ranking RGB
Click Bell
Turn Switch
Place Container Plate
Place Empty Cup
Stack Blocks Two
Stack Blocks Three
Stack Bowls Three
Stack Bowls Two
Avg.
Ours
0.70
0.03
1.00
0.60
0.80
0.68
0.50
0.08
0.72
0.82
0.593
DP3 [ 37 ]
0.72
0.03
0.90
0.46
0.86
0.65
0.24
0.01
0.57
0.83
0.527
ACT [ 39 ]
0.56
0.01
0.58
0.05
0.72
0.61
0.25
0.00
0.48
0.82
0.408
DP [ 4 ]
0.42
0.00
0.54
0.36
0.41
0.37
0.07
0.00
0.63
0.61
0.341
TABLE I: Comparison of different methods across various manipulation tasks. Our method achieves the best average performance.
Fig. 4: Left: Architecture of Geometry Encoder. Mid: Efficacy of Multi-View Projection. Right: Impact of Modality Embeddings.
Method
Push Cube
Stack Cube
Pick Cube
Lift Peg Upright
Peg Insertion Side
Avg.
Ours
0.95
0.28
0.48
0.36
0.01
0.416
DP3
0.92
0.01
0.24
0.75
0.01
0.386
DP
0.88
0.00
0.44
0.03
0.01
0.272
TABLE II: Success Rates on Maniskill 3 Benchmark
Fig. 5: Real-World Experiments . Top: Real-World experiments setup, including cameras’ positions and robotic arms’ settings. Bottom: Objects used in tasks and their sizes.
Task
Task Details
Beat w. Hammer
Strike target object with hammer head
Sweep Table
Sweep all rubbish into dustpan
Open Drawer PnP
Open drawer, insert cube, close drawer
Place Empty Cup
Stable placement in target area
Pouring
Transfer small balls without spilling
Stack Cube
Stable stack of red cube on blue cube
TABLE III: Detailed descriptions for real world tasks.
Task
Ours
DP3
DP
Beat w. Hammer
0.5
0.4
0.2
Sweep Table
0.5
0.3
0.1
Open Drawer PnP
0.3
0.1
0.0
Place Empty Cup
0.6
0.6
0.3
Pouring
0.6
0.4
0.1
Stack Cube
0.3
0.1
0.0
TABLE IV: Success rates (SR) across 10 trials for different methods.
Recent advances in robot imitation learning have produced powerful visuomotor policies that manipulate diverse objects from visual inputs. However, monocular observations lack depth information, which is critical for precise manipulation in cluttered or geometrically complex scenes. Explicit depth maps and point clouds are often noisy and fragile in real-world manipulation. We introduce StereoPolicy, a visuomotor policy learning framework that directly leverages synchronized stereo image pairs to improve geometric reasoning without constructing explicit 3D representations. StereoPolicy processes each image with pretrained 2D vision encoders and fuses left-right features through a cross-attention-based Stereo Transformer, capturing spatial correspondence and disparity cues implicitly. The framework integrates with diffusion-based and pretrained vision-language-action (VLA) policies, delivering consistent improvements over RGB, RGB-D, point cloud, and multi-view baselines across three simulation benchmarks and seven real-robot tabletop and bimanual mobile manipulation tasks. Our results show that stereo vision bridges 2D pretrained representations and 3D geometric understanding for robotic manipulation.
Evans Han, Yunfan Jiang, Yingke Wang +6
1Stanford University · 2Northwestern University · 3Lambda, Inc
End-to-end manipulation policies, combined with web-scale pretrained Vision-Language Models (VLMs), show the promise for generalizable and dexterous robotic manipulation. However, they inherit two key limitations from 2D foundation models: 1) the reliance on 2D RGB inputs that ignores the intrinsically 3D nature of manipulation; and 2) the lack of spatial 3D alignment between input-output spaces as well as across diverse robot embodiments, camera setups, and trajectory datasets. In this paper, we present a series of contributions to address these issues. First, we introduce aligned vertex map and vertex spectrum -- a pixel-wise 3D representation that elevates 2D visual inputs to 3D, using camera calibration and optional depth. This novel input representation marries 3D awareness with the generalization of 2D large VLMs. Then, we propose to align the inputs and outputs of manipulation policies by expressing per-pixel 3D information of each camera view and robot actions to a shared coordinate. Based on this, we designate a canonical Bird's-Eye-View (BEV) alignment frame and innovatively propose to construct BEV images, producing a view-invariant representation robust to camera pose variations. To enable training and evaluation at scale, we develop a comprehensive data processing pipeline to perform such alignments; we also introduce a novel temporal alignment scheme for trajectories across diverse robots, human operators, and datasets. These contributions collectively mitigate input and output spatial-temporal misalignments, improving the consistency and generalization for real-world manipulation. Pretrained checkpoint, source code and data processing pipeline are available in https://hnuzhy.github.io/projects/Dex-BEV.
Huayi Zhou, Wei Gao, Dekun Lu +12
1DexForce Technology · 2The Chinese University of Hong Kong, Shenzhen
Imitation learning is a prominent paradigm for robotic manipulation. However, existing visual imitation methods map 2D image observations directly to 3D action outputs, imposing a 2D-3D mismatch that hinders spatial reasoning and degrades robustness. We present VolumeDP, a policy architecture that restores spatial alignment by explicitly reasoning in 3D. VolumeDP first lifts image features into a Volumetric Representation via cross-attention. It then selects task-relevant voxels with a learnable module and converts them into a compact set of spatial tokens, markedly reducing computation while preserving action-critical geometry. Finally, a multi-token decoder conditions on the entire token set to predict actions, thereby avoiding lossy aggregation that collapses multiple spatial tokens into a single descriptor. VolumeDP achieves a state-of-the-art average success rate of 88.8% on the LIBERO simulation benchmark, outperforming the strongest baseline by a substantial 14.8% improvement. It also delivers large performance gains over prior methods on the ManiSkill and LIBERO-Plus benchmarks. Real-world experiments further demonstrate higher success rates and robust generalization to novel spatial layouts, camera viewpoints, and environment backgrounds. Code and videos are available on the project page: https://yzc0731.github.io/VolumeDP/