Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation

Chongyang Xu; Haipeng Li; Shen Cheng; Jingyu Hu; Haoqiang Fan; Ziliang Feng; Shuaicheng Liu

cs.CVFeb 27, 2026

Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation

Authors: Chongyang Xu, Haipeng Li, Shen Cheng, Jingyu Hu, Haoqiang Fan, Ziliang Feng, Shuaicheng Liu

Organizations: College of Computer Science, Sichuan University, China · Dexmal · University of Electronic Science and Technology of China, China · The Chinese University of Hong Kong, HK SAR, China

Abstract

Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions. However, existing methods typically rely on 2D features with limited spatial awareness, or require explicit point clouds that are difficult to obtain reliably in real-world settings. At the same time, recent 3D geometric foundation models show that accurate and diverse 3D structure can be reconstructed directly from RGB images in a fast and robust manner. We leverage this opportunity and propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model. Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap. By explicitly predicting how the 3D scene will evolve together with the action sequence, the policy gains strong spatial understanding and predictive capability using only RGB observations. We evaluate our method both in simulation on the RoboTwin benchmark and in real-world robot executions. Our approach consistently outperforms 2D-based and point-cloud-based baselines, achieving state-of-the-art performance in manipulation success, inter-arm coordination, and 3D spatial prediction accuracy. Code is available at https://github.com/Chongyang-99/GAP.git.

Figures & tables

Figure 1: Paradigm Comparison. 2D-based methods learn implicit 3D representations from multi-view RGB observations, relying purely on 2D cues. 3D-based methods require camera calibration and preset workspaces to crop point clouds, which limits generalization and scalability. In contrast, our approach leverages powerful 2D and 3D pretrained priors to achieve semantic–geometric fusion perception, enabling robust action and geometry joint prediction without strict calibration or workspace constraints.

Figure 2: Overview of our method. Given a sequence of past RGB frames, the current image, and proprioceptive state, our model extracts 3D geometric features, 2D semantic features, and robot state embeddings through three parallel encoders. These signals are fused by a Transformer into a unified semantic and geometric context that conditions a joint denoising process. A conditional diffusion decoder then predicts both a future action chunk and a future 3D latent, which is further decoded into a dense pointmap.

Figure 3: Bimanual tasks in the RoboTwin 2.0 [ 25 ] benchmark.

	Avg. $\uparrow$	Beat Block	Place	Move	Open	Open	Place A2B	Place A2B
Method	$(\%)$	Hammer	Shoe	Can Pot	Laptop	Microwave	Left	Right
ACT [ 47 ]	34.1	59.3 $\pm 6.4$	14.3 $\pm 4.0$	48.0 $\pm 6.0$	59.7 $\pm 2.1$	83.7 $\pm 1.5$	1.3 $\pm 0.6$	0.7 $\pm 0.6$
DP [ 12 ]	44.4	62.0 $\pm 4.0$	43.3 $\pm 6.7$	51.0 $\pm 4.0$	61.3 $\pm 3.5$	53.7 $\pm 6.5$	10.0 $\pm 5.3$	11.7 $\pm 5.0$
DP3 [ 44 ]	61.2	59.3 $\pm 2.3$	57.7 $\pm 2.5$	63.7 $\pm 1.5$	77.7 $\pm 2.1$	70.3 $\pm 2.5$	29.3 $\pm 2.5$	23.0 $\pm 2.6$
G3Flow [ 8 ]	60.7	-	58.3 $\pm 1.5$	-	-	-	25.7 $\pm 1.5$	25.7 $\pm 3.2$
Xu et al. [ 42 ]	58.4	50.0 $\pm 5.6$	58.7 $\pm 9.1$	58.7 $\pm 2.1$	67.7 $\pm 3.2$	-	24.3 $\pm 2.1$	23.0 $\pm 2.6$

Table 1: Comparison on Dominant-select Tasks (16 tasks). Single-arm manipulation tasks requiring appropriate arm selection. We report the mean and standard deviation of success rates averaged over 3 random seeds. Best score in bold , second-best underlined .

	Avg. $\uparrow$	Pick Diverse	Pick Dual	Place Bread	Place Bread	Place Burger	Place Cans	Place Dual	Put Bottles
Method	$(\%)$	Bottles	Bottles	Skillet	Basket	Fries	Plasticbox	Shoes	Dustbin
ACT [ 47 ]	32.4	11.7 $\pm 4.0$	52.7 $\pm 3.1$	12.3 $\pm 2.5$	15.7 $\pm 3.1$	74.3 $\pm 2.5$	29.3 $\pm 3.5$	3.0 $\pm 1.0$	60.0 $\pm 4.6$
DP [ 12 ]	37.1	18.3 $\pm 3.8$	52.3 $\pm 0.6$	25.0 $\pm 4.6$	16.7 $\pm 3.5$	76.7 $\pm 4.2$	62.7 $\pm 3.1$	7.0 $\pm 4.4$	38.3 $\pm 5.1$
DP3 [ 44 ]	45.1	27.7 $\pm 8.6$	59.0 $\pm 3.0$	36.3 $\pm 6.7$	23.7 $\pm 6.8$	60.3 $\pm 3.2$	66.0 $\pm 3.6$	17.7 $\pm 4.2$	70.3 $\pm 2.3$
G3Flow [ 8 ]	45.8	23.0 $\pm 3.6$	58.7 $\pm 1.2$	31.0 $\pm 1.0$	19.3 $\pm 4.2$	73.7 $\pm 5.0$	67.3 $\pm 2.1$	20.7 $\pm 4.2$	73.0 $\pm 4.6$
RDT [ 24 ]	28.8	9.7 $\pm 1.2$	44.3 $\pm 3.8$	21.7 $\pm 2.1$	8.7 $\pm 3.2$	65.7 $\pm 4.0$	15.7 $\pm 3.1$	18.7 $\pm 3.1$	45.7 $\pm 5.9$

Table 2: Comparison on Sync-bimanual Tasks (8 tasks). Bimanual tasks requiring synchronized coordinated operation. We report the mean and standard deviation of success rates averaged over 3 random seeds. Best score in bold , second-best underlined .

	Avg. $\uparrow$	Handover	Hang	Place Object	Scan	Stack	Stack	Stack	Stack
Method	$(\%)$	Block	Mug	Basket	Object	Blocks Two	Blocks Three	Bowls Two	Bowls Three
ACT [ 47 ]	29.4	45.3 $\pm 2.5$	4.3 $\pm 0.6$	6.3 $\pm 0.6$	5.7 $\pm 3.8$	23.3 $\pm 3.5$	0.7 $\pm 1.2$	85.3 $\pm 3.2$	64.3 $\pm 3.2$
DP [ 12 ]	33.6	32.0 $\pm 1.0$	15.3 $\pm 3.2$	30.0 $\pm 3.0$	10.3 $\pm 2.5$	17.3 $\pm 3.8$	1.3 $\pm 1.5$	89.3 $\pm 1.5$	73.3 $\pm 3.1$
DP3 [ 44 ]	36.0	29.7 $\pm 3.1$	23.3 $\pm 7.8$	32.3 $\pm 2.5$	23.3 $\pm 1.5$	31.3 $\pm 3.1$	3.7 $\pm 1.5$	86.7 $\pm 3.8$	58.0 $\pm 6.9$
G3Flow [ 8 ]	46.3	-	26.7 $\pm 3.1$	32.3 $\pm 2.5$	22.0 $\pm 4.6$	-	-	89.7 $\pm 3.1$	61.0 $\pm 2.6$
RDT [ 24 ]	37.5	60.3 $\pm 2.5$	25.3 $\pm 0.6$	37.0 $\pm 1.7$	11.7 $\pm 1.5$	24.3 $\pm 2.1$	1.7 $\pm 0.6$	88.0 $\pm 2.0$	51.3 $\pm 0.6$

Table 3: Comparison on Seq-coordinate Tasks (8 tasks). Sequential coordination tasks requiring multi-step bimanual cooperation. We report the mean and standard deviation of success rates averaged over 3 random seeds. Best score in bold , second-best underlined .

Figure 4: Data Efficiency. Leveraging pre-trained features, our method achieves high data efficiency, outperforming 2D methods in low-data regimes and surpassing the performance of DP3 as more data becomes available.

2D Semantic Module	3D Geometric Module	Geometric Imagination	Success Avg.(%)
✓	✓	✓	25.1
$\times$	✓	✓	24.4
✓	✓	$\times$	23.6
✓	$\times$	$\times$	21.0

Table 4: Ablation study. Impact of removing key components on the average success rate on four tasks.

Task	ACT	DP	Xu et al.	Ours
Place Empty Cup	70	70	75	80
Place Dual Shoes	0	10	15	20
Hanging Mug	0	0	5	20
Scan Object	25	20	35	40
Avg. Success (%) $\uparrow$	23.8	25	32.5	40

Table 5: Real-world experiments. Success rates (%) of different methods on four bimanual manipulation tasks.

Figure 5: Real-World Setting. Our real-world platform featuring the AgileX Cobot Magic bimanual system, equipped with three RealSense D435i cameras to evaluate four challenging tasks.

Explore similar work

JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation
Sep 21, 2026Chuyang Xiao, Peilin Meng, David Held3D Point Tracking Bimanual Robotic Manipulation
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA. · University of Michigan, Ann Arbor, MI 48109, USA.
Geometry-Aware Motion Latents for Learning Robust Manipulation Policies
Jul 6, 2026Yunchao Zhang, Yijia Weng, Ruizhe Liu +3Latent Action Learning Geometric Representation Learning
Department of Computer Science, The University of Hong Kong, Hong Kong, China · Department of Computer Science, Stanford University, Stanford, California, USA · Department of Data Science and AI, Monash University, Melbourne, Australia
Geometric Action Model for Robot Policy Learning
Jun 15, 2026Jisang Han, Seonghu Jeon, Jaewoo Jung +7World Models for Robotics Language-Conditioned Robot Manipulation
KAIST AI · ETH Zurich · ETH AI Center

Sep 21, 2026J/K move · Enter open · S save

JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation

Chuyang Xiao, Peilin Meng, David Held

Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA. · University of Michigan, Ann Arbor, MI 48109, USA.

3D Point Tracking Bimanual Robotic Manipulation Diffusion Policy

Open paper AskPDF arXiv

Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm. Yet most diffusion policies generate actions without explicitly modeling these future geometric consequences, while predictive variants typically use future state only as auxiliary supervision or fixed conditioning. We address this limitation by proposing JAMB, a diffusion policy that jointly denoises bimanual actions and future 3D point tracks. By allowing action and track hypotheses to evolve together within a shared Transformer, each can inform and refine the other throughout denoising. We further ground multimodal representations in a shared spatiotemporal coordinate system to facilitate geometry-aware interaction during joint denoising. We evaluate JAMB on diverse bimanual manipulation tasks in RoboTwin 2.0 and on a real-world robot, comparing it with action-only policies and alternative future-prediction approaches spanning different state representations and learning objectives. Across 16 simulation tasks, JAMB achieves an average success rate of 83.4%, outperforming the strongest baseline by 23.9 percentage points. On three real-world tasks, it outperforms the action-only and auxiliary geometry prediction methods by 50.0 and 21.2 percentage points, respectively. Beyond these performance gains, JAMB shows stronger generalization to cluttered scenes and out-of-distribution backgrounds than the evaluated baselines. Together, these results demonstrate the effectiveness of our joint action-motion modeling framework for coordinated bimanual manipulation. Our project website is available at https://jam-bimanual.github.io/

Abstract

Explore similar work

JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation

Geometry-Aware Motion Latents for Learning Robust Manipulation Policies

Geometric Action Model for Robot Policy Learning