R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation
Authors: Yuhao Zhang, Wanxi Dong, Yue Shi, Yi Liang, Jingnan Gao, Qiaochu Yang, Yaxing Lyu, Zhixuan Liang, +8 more
Organizations: Shanghai Jiao Tong University · D-Robotics · Southern University of Science and Technology · Xspark AI · Xiamen University Malaysia · The University of Hong Kong · Northeastern University · Wuhan University
Embodied manipulation requires accurate 3D understanding of objects and their spatial relations to plan and execute contact-rich actions. While large-scale 3D vision models provide strong priors, their computational cost incurs prohibitive latency for real-time control. We propose Real-time 3D-aware Policy (R3DP), which integrates powerful 3D priors into manipulation policies without sacrificing real-time performance. A core innovation of R3DP is the asynchronous fast-slow collaboration module, which seamlessly integrates large-scale 3D priors into the policy without compromising real-time performance. The system maintains real-time efficiency by querying the pre-trained slow system (VGGT) only on sparse key frames, while simultaneously employing a lightweight Temporal Feature Prediction Network (TFPNet) to predict features for all intermediate frames. By leveraging historical data to exploit temporal correlations, TFPNet explicitly improves task success rates through consistent feature estimation. Additionally, to enable more effective multi-view fusion, we introduce a Multi-View Feature Fuser (MVFF) that aggregates features across views by explicitly incorporating camera intrinsics and extrinsics. R3DP offers a plug-and-play solution for integrating large models into real-time inference systems. We evaluate R3DP against multiple baselines across different visual configurations. R3DP effectively harnesses large-scale 3D priors to achieve superior results, outperforming single-view and multi-view DP by 32.9% and 51.4% in average success rate, respectively. Furthermore, by decoupling heavy 3D reasoning from policy execution, R3DP achieves a 44.8% reduction in inference time compared to a naive DP+VGGT integration.
Figures & tables
Figure 1: Key modules and performance of R3DP. Our framework explicitly integrates 3D priors from large-scale foundation models (e.g., VGGT) via an asynchronous fast-slow collaboration mechanism. Overall, R3DP achieves real-time, 3D-aware inference, significantly improving both manipulation success rates and processing frequency.
Figure 2: Overview of the R3DP architecture. R3DP serves as a 3D-aware perception module that seamlessly replaces visual encoders in existing imitation learning frameworks. Within the AFSC module ( 4.1 ), sparse keyframes are processed by a 3D foundation model (VGGT), while intermediate frames are handled by our TFPNet ( 4.2 ) for real-time temporal reasoning. MVFF module ( 4.3 ) leverages cross-attention with PRoPE to fuse 2D-3D features into consistent multi-view representations for control.
Figure 3: Architecture and training objective of TFPNet. For clarity, we show the unrolled structure for the first two timesteps; in practice, the network is trained over a sequence of four timesteps. TFPNet leverages historical information to augment current observations, enabling 3D-aware control with real-time inference efficiency.
Task
DP-single
DP-multi
DP3
DP3-DA2
π0
R3DP ( τ =4)
R3DP ( τ =8)
Block Hammer Beat
0%
0%
49%
7%
47%
77 %
77 %
Block Handover
1%
2%
48%
31%
71%
95 %
93%
Bottle Adjust
55%
6%
75 %
37%
46%
62%
56%
Blocks Stack Easy
6%
7%
26%
3%
79 %
69%
62%
Shoe Place
45%
19%
49%
34%
71%
72 %
68%
Container Place
48%
14%
89 %
37%
77%
63%
54%
Table 1: Task success rates on RoboTwin benchmark. We report single-view Diffusion Policy (DP-single), multi-view Diffusion Policy (DP-multi), DP3, DP3+DA2, π0 and Diffusion Policy with R3DP-enhanced feature ( R3DP ). For R3DP, we select τ=4 and τ=8 . Note that τ can serve as a configurable hyperparameter at inference time to achieve a trade-off between inference speed and success rate.
Latency (ms ↓ )
DP+VGGT
DP+VGGT+MVFF
R3DP( τ=4 )
R3DP( τ=8 )
Obs. Encoder
73.1
78.3 ( ↑7.1% )
50.5 ( ↓30.9% )
40.3 ( ↓44.8% )
Action Expert
56.6
57.7
55.3
56.7
Table 2: Average latency from raw multi-view images to fused features, averaged over four tasks (Blocks Stack Easy, Block Hammer Beat, Shoe Place, Diverse Bottles Pick) experimented in the ablation study.
Figure 4: Visualization of depth maps decoded from VGGT features and from our TFPNet-predicted features passed through VGGT’s depth decoder. The close visual agreement indicates that our lightweight TFPNet effectively captures information generated by 3D foundation model in both simulation and real-world experiments.
Task
DP-multi
DP+VGGT
DP+VGGT+MVFF
R3DP
Blocks Stack Easy
7%
30%
51%
69%
Block Hammer Beat
0%
44%
74%
77%
Shoe Place
19%
55%
66%
72%
Diverse Bottles Pick
17%
30%
30%
31%
Average
10.8%
39.8%
55.3%
62.3%
Table 3: Ablation study on the effectiveness of key components. We progressively add the 3D foundation model (VGGT), Multi-View Feature Fusion (MVFF), and AFSC to the baseline (DP-multi). The final model with all components denotes R3DP .
Figure 5: Real-world experimental platforms and point cloud inputs. We evaluate our method on the ArmBot-Y1 bimanual robot for tasks including Place Shoe and Place Glass Cup, and a single-arm robot for Pick Peach and Stack Bowls. Both platforms are equipped with dual RealSense D435 cameras to generate the ground-truth point clouds used by the DP3 baseline.
Task (SR ↑ )
DP
DP3
DP+V+M
R3DP
Place Shoe
46.7%
50.0%
76.7%
86.7 %
Place Glass Cup
23.3%
56.7%
76.7%
83.3 %
Pick Peach(Single Arm)
20%
30%
46.7%
50 %
Stack Bowls(Single Arm)
33.3%
56.7%
66.7 %
66.7 %
Average
30.8%
48.4%
66.7%
71.7%
Latency (ms ↓ )
Inference Breakdown (RTX 4090)
Table 4: Real-world performance and inference latency. V and M denote VGGT and MVFF. For model training, we collect 100 episodes for dual-arm tasks and 50 episodes for single-arm tasks, respectively. All models are subsequently evaluated over 30 trials per task to report the success rates. For DP3, point cloud conversion introduces additional computational overhead.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Max Frames
Description
Bottle Adjust
400
Pick up the bottle on the table headup with the correct arm.
Block Hammer Beat
400
There is a hammer and a block on the table, use the arm to grab the hammer and beat the block.
Shoe Place
450
Use one arm to grab the shoe from the table and place it on the mat.
Container Place
350
Place the container onto the plate.
Dual Shoes Place
600
Use both arms to pick up the two shoes on the table and put them in the shoebox, with the shoe tip pointing to the left.
Diverse Bottles Pick
400
Pick up one bottle with one arm, and pick up another bottle with the other arm.
Appendix
Table A : Descriptions of the RoboTwin manipulation tasks and our additional Tube Insert task. Max Frames denotes the maximum number of frames used to determine whether an evaluation is considered a failure.
Figure A : The head and front camera view of Tube Insert task.
Figure B : Real-world experiment settings and tasks. The first two images correspond to Setting A, while the last two represent Setting B.
Figure C : Real-world experiment data collection settings. Different colors represent different objects. The yellow edge means absolute position, and magenta means relative position or direction. The highlighted points represent the selected placement positions of the objects in the sample images.
Task
Notation
Description
Place Shoe
Success
1 - Successfully placed the shoe inside the red area horizontally.
Failure
0 - Failed to grasp.
2 - Placed the shoe out of the area or in incorrect orientation.
Place Glass Cup
Success
1 - Successfully placed glass cup fully onto the coaster.
Failure
0 - Failed to grasp.
2 - Placed the glass cup out of the coaster.
Appendix
Table B : Detailed categorization of failure modes across four real-world tasks.
Task
DP
DP3
DP+VGGT
R3DP
Place Shoe
1012011100 1110112000 1000011210
0100100110 1100112101 1011020211
1111102121 1011101110 1111111110
1111111111 1112111111 1221111211
Place Glass Cup
1010000011 0000000001 2010010020
1000010100 1111010111 1001011011
1110111110 1111000111 0111110111
1110111111 1112111112 1121110111
Pick Peach
0022212211 0203010021 0220013022
1000110110 2000002001 0001021010
1100211100 2121131210 1100120010
0112210110 2211111201 1101200100
Stack Bowls
1221030321 3221123231 2212211212
3113111001 1313031311 1133113011
1001133111 1311021311 2111131111
1100111333 0111131131 1211111112
Appendix
Table C : Detailed real-world evaluation results
Task
τ=2
τ=4
τ=8
τ=16
τ=32
Average
Block Hammer Beat
60%
77%
77%
87 %
85%
77.2% ± 9.52%
Block Handover
75%
95 %
93%
93%
90%
89.2% ± 7.28%
Bottle Adjust
56%
62 %
56%
53%
53%
56.0% ± 3.29%
Blocks Stack Easy
64%
69%
72 %
67%
69%
68.2% ± 2.64%
Shoe Place
73%
72%
68%
74 %
61%
69.6% ± 4.76%
Container Place
50%
63 %
54%
57%
53%
55.4% ± 4.41%
Appendix
Table D : Performance under different values of the hyperparameter τ .