Enabling a Unified Cross-Domain Representation for Two-Finger Gripper Manipulation via Interaction-Centric Modeling
Authors: Guanlin Li, Shifeng Bao, Yihan Zhao, Haitao Shen, Haoyang Li, Chen Zhao, Tong Yang, Jie Tang, +1 more
Organizations: Key Laboratory of Data Engineering and Knowledge Engineering, MOE, and School of Information, Renmin University of China, China. · Zhipu AI, China. · Department of Computer Science and Technology, Tsinghua University, China.
Achieving robust cross-embodiment generalization in imitation learning demands overcoming a critical representation flaw that inextricably entangles task semantics with hardware-specific visual geometry. We propose an interaction-centric framework that leverages the shared structure of two-finger grippers via a parameterized universal gripper abstraction, yielding a canonical gripper-frame representation. Given language and RGB-D observations, a VLM infers the subtask and grounds an interaction triplet (gripper, held, target), while SAM~2.1 tracks masks to reduce VLM queries. We design concise hybrid features that combine target/collision artificial potential fields for global guidance with segmented gripper-frame point clouds for local geometry, and use a Flow-Matching Transformer to predict smooth 7-DoF action chunks. Experiments in simulation and real-world tasks demonstrate that ours is the first imitation learning approach to simultaneously achieve competitive benchmark scores and extreme cross-embodiment/cross-viewpoint zero-shot sim-to-real transfer to completely distinct, heterogeneous robot platforms.
Figures & tables
Fig. 1: Interaction-centric policy for robust two-finger manipulation. We study universal two-finger manipulation on a real-world Jaka platform. By transforming RGB-D observations into a canonical gripper frame, our policy focuses on the interaction geometry among the gripper, held object, and target object. This formulation achieves stronger OOD generalization.
Fig. 2: Method overview. The Perception Module utilizes a VLM for instruction parsing and SAM 2.1 for real-time triplet mask tracking. The Representation Module lifts 2D masks to 3D to produce disentangled artificial potential fields (APFs) and gripper-frame point clouds (PCDs). The Action Module employs a Flow-Matching Transformer to integrate subtask semantics, spatial features, and proprioception, generating smooth horizon- H 7-DoF action trajectories.
Fig. 3: Artificial Potential Field (APF) visualization. The active gripper (red) is attracted by a target APF (cyan) and repelled by a collision APF (yellow).
Fig. 4: Real-world experiments. We collect data on four tasks and evaluate the model under in-domain and three different out-of-distribution (OOD) settings (Position/Object/Camera OOD). Additionally, we directly deploy the simulation-trained model onto a physical robot with a heterogeneous embodiment relative to the original platform (Embodiment OOD).
Method
Clean
Random
Small Models (Trained from Scratch)
Ours
66.3
46.1
ACT
42.4
3.2
DP3-G
49.0
10.0
DP3-W
73.1
7.5
Large Models (Fine-tuned)
TABLE I: Simulation Experiment. Comparison on 25 tasks from RoboTwin2.0 benchmark. We report success rates (%) under Clean and Random settings. Bold indicates the best and underline indicates the second best.
Fig. 5: We selected six tasks from RoboTwin2.0: Click Alarmclock , Place Phone Stand , Click Bell , Adjust Bottle , Shake Bottle , Shake Bottle Horizontally , and deployed the policy on our robots. Rows from top to bottom: real-world deployment on DOS-W1, on Jaka, and the simulation.
Adjust Bottle
Shake Bottle
Shake Horiz.
Click Alarm.
Click Bell
Place Phone
Sim
96%
97%
92%
78%
80%
32%
DOS-W1
18/30
16/30
15/30
15/30
13/30
6/30
JAKA (near-view)
14/30
15/30
14/30
11/30
9/30
5/30
JAKA (novel-view)
8/30
9/30
7/30
6/30
5/30
2/30
TABLE II: Embodiment OOD results. Sim denotes success rates in RoboTwin2.0. DOS-W1 and JAKA show real-world success counts. Near-view involves minor camera shifts ( ∼ 10cm/25 ∘ ), while Novel-view uses a substantially different perspective.