RTPolicy: Real-Time (32.9 Hz) Multi-View Visuomotor Policy for Robotic Manipulation on Edge Devices
Organizations: School of Future Science and Engineering, Soochow University · D-Robotics · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences
Abstract
Low inference latency is critical for closed-loop robotic manipulation, especially under asynchronous inference, where action chunks must be updated promptly to correct accumulated deviations. However, existing visuomotor policies rely on computationally expensive visual backbones, resulting in high inference latency and deployment overhead. In this paper, we propose RTPolicy, a multi-view visuomotor policy that attains accurate manipulation with high inference efficiency. Its core design combines view-aware feature fusion with visual compression to aggregate complementary information across camera views into a compact token representation. The representation conditions a flow-matching action generator with the robot state and flow time, reducing the cost of action generation. Extensive experiments demonstrate that this design reduces the computational cost of visual representation and action generation while preserving visual information required for manipulation. On the RoboTwin~2.0 single-task clean evaluation across 50 tasks, RTPolicy attains a mean success rate of 52.26%, outperforming the strongest leaderboard baseline (46.4%). We further evaluate RTPolicy under asynchronous streaming inference on a real bimanual platform, where RTPolicy attains a mean success rate of 72.0% and sustains real-time inference at 30.35 ms (32.9 Hz) on the RDK S600 and 149.72 ms (6.7 Hz) on the RDK S100, both in INT16. Project Page: https://d-robotics-ai-lab.github.io/RTPolicy.project.
Figures & tables
| Task | RTPolicy | [ 19 ] | RDT [ 20 ] | ACT [ 1 ] | DP [ 9 ] |
| Adjust Bottle | 97 | 90 | 81 | 97 | 97 |
| Beat Block Hammer | 76 | 43 | 77 | 56 | 42 |
| Blocks Ranking RGB | 5 | 19 | 3 | 1 | 0 |
| Blocks Ranking Size | 3 | 7 | 0 | 0 | 1 |
| Click Alarmclock | 76 | 63 | 61 | 32 | 61 |
| Click Bell | 73 | 44 | 80 | 58 | 54 |
| Method | 50 demos | 100 demos |
|---|---|---|
| DP [ 9 ] | 13 | 38 |
| VO-DP [ 4 ] | 52 | 76 |
| RTPolicy | 64 | 85 |
| Task | Integration Steps | |||||
|---|---|---|---|---|---|---|
| 2 | 3 | 4 | 5 | 6 | 10 | |
| BBH | 75 | 83 | 85 | 87 | 83 | 85 |
| AB | 98 | 99 | 100 | 100 | 100 | 100 |
| CAC | 58 | 56 | 53 | 53 | 51 | 50 |
| CB | 81 | 80 | 79 | 77 | 75 | 69 |
| DBB | 94 | 87 | 90 | 93 | 94 | 89 |
| Backbone | BBH | CAC | SO | Mean | Speedup |
|---|---|---|---|---|---|
| Small | 74 | 69 | 60 | 67.7 | 3.25 |
| Base | 93 | 65 | 58 | 72.0 | 2.15 |
| Large | 94 | 74 | 55 | 74.3 | 1.00 |
| View-aware Feature Fusion | Visual Compression | Task | Mean | ||||
|---|---|---|---|---|---|---|---|
| BBH | AB | CAC | CB | DBB | |||
| 85 | 100 | 53 | 79 | 90 | 81.4 | ||
| 84 | 100 | 83 | 67 | 88 | 84.4 | ||
| 83 | 99 | 64 | 55 | 89 | 78.0 | ||
| 89 | 99 | 49 | 54 | 86 | 75.4 | ||
| Method | Drag Drawer | Stack and Cover | Workpiece Assembly | Mean |
|---|---|---|---|---|
| DP [ 9 ] | 80 | 8 | 0 | 29.3 |
| VO-DP [ 4 ] | 84 | 34 | 4 | 40.7 |
| [ 5 ] | 88 | 56 | 12 | 52.0 |
| RTPolicy | 92 | 94 | 30 | 72.0 |
| Method | Views | Tokens | RTX 4090 | Jetson Orin NX 16GB | RDK S600 | RDK S100 | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| FP32 | FP16 | FP32 | FP16 | INT8 | INT16 | INT8 | INT16 | |||
| DP [ 9 ] | 1 | 1 | 23.15 (43.2) | 8.72 (114.7) | 49.16 (20.3) | 31.25 (32.0) | 16.40 (61.0) | 19.76 (50.6) | 67.23 (14.9) | 77.98 (12.8) |
| VO-DP [ 4 ] | 1 | 1 | 88.16 (11.3) | 25.31 (39.5) | – | 410.34 (2.4) | 79.97 (12.5) | 181.47 (5.5) | 489.0 (2.0) | 1025.38 (1.0) |
| [ 5 ] | 3 | 768 | 109.33 (9.15) | 58.84 (17.0) | – | – | 135.5 (7.4) | 331.6 (3.0) | 778.10 (1.3) | 1491.59 (0.7) |
| RTPolicy | 3 | 12 | 23.80 (42.0) | 10.92 (91.6) | 266.3 (3.8) | 143.2 (7.0) | 18.59 (53.8) | 30.35 (32.9) | 86.56 (11.6) | 149.72 (6.7) |
Visuomotor Policy Learning
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
See all 302 papers