RoTri-Diff: A Spatial Robot-Object Triadic Interaction-Guided Diffusion Model for Bimanual Manipulation
Authors: Zixuan Chen, Nga Teng Chan, Yiwen Hou, Chenrui Tie, Zixuan Liu, Haonan Chen, Junting Chen, Jieqi Shi, +3 more
Organizations: Zixuan Chen and Jing Huo are with the School of Computer Science, Nanjing University, China. · Yiwen Hou, Chenrui Tie, Zixuan Liu, Haonan Chen, Junting Chen and Lin Shao are with the School of Computing, National University of Singapore, Singapore. · Nga Teng Chan is with the Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, China. · Jieqi Shi and Yang Gao are with the School of Intelligence Science and Technology, Nanjing University, China.
Bimanual manipulation is a fundamental robotic skill that requires continuous and precise coordination between two arms. While imitation learning (IL) is the dominant paradigm for acquiring this capability, existing approaches, whether robot-centric or object-centric, often overlook the dynamic geometric relationship among the two arms and the manipulated object. This limitation frequently leads to inter-arm collisions, unstable grasps, and degraded performance in complex tasks. To address this, in this paper we explicitly models the Robot-Object Triadic Interaction (RoTri) representation in bimanual systems, by encoding the relative 6D poses between the two arms and the object to capture their spatial triadic relationship and establish continuous triangular geometric constraints. Building on this, we further introduce RoTri-Diff, a diffusion-based imitation learning framework that combines RoTri constraints with robot keyposes and object motion in a hierarchical diffusion process. This enables the generation of stable, coordinated trajectories and robust execution across different modes of bimanual manipulation. Extensive experiments show that our approach outperforms state-of-the-art baselines by 10.2% on 11 representative RLBench2 tasks and achieves stable performance on 4 challenging real-world bimanual tasks. Project website: https://rotri-diff.github.io/.
Figures & tables
Imitation Learning Guides
Method
Robot Keyposes
Object Movement
Robot–Object Interaction
Prediction Type
ACT [ 17 ]
X
X
X
Continuous Actions
PerAct2 [ 10 ]
✓
X
X
Keyposes
DP3 [ 18 ]
X
X
X
Continuous Actions
AnyBimanual [ 14 ]
✓
X
X
Keyposes
3D Diffuser Actor [ 19 ]
✓
X
X
Keyposes & Continuous Actions
TABLE I : Comparison of representative bimanual imitation learning methods.
Fig. 2 : Overview of RoTri-Diff. (a) Visual Perception and RoTri Modeling: Extracting the initial object point cloud F0 , 3D semantic features St , and the initial RoTri representation R0 from multi-view observations. (b) Imitation Learning Guidance Signals: Three complementary signals used for supervision: Keyposes, Object Pointflow, and the RoTri Relationship. (c) Hierarchical Diffusion Model: The model concurrently predicts object pointflow and autoregressively predicts a future RoTri segment. These predictions then serve as dynamic conditions to guide the denoising and generation of keyposes and continuous actions within a synergistic attention module.
Fig. 3 : Visualization of the 11 RLBench2 tasks.
Fig. 4 : Ablation Study. Comparing Dense with Sparse (5/10 step) variants shows that higher guidance density consistently yields superior task success.
Fig. 5 : Real-world experimental setup.
Fig. 6 : Task process visualizations in 4 real-world tasks.
School of Computer Science and 2Australian Center For Robotics, The University of Sydney, Australia · College of Connected Computing, Vanderbilt University, TN, USA