Semantic-Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning

Franziska Herbert; Vignesh Prasad; Han Liu; Dorothea Koert; Georgia Chalvatzaki

cs.ROJan 16, 2026

Semantic-Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning

Authors: Franziska Herbert, Vignesh Prasad, Han Liu, Dorothea Koert, Georgia Chalvatzaki

Organizations: Interactive Robot Perception & Learning (PEARL) Lab, Computer Science Dept., TU Darmstadt, Germany. · Hessian.AI, Darmstadt, Germany. · Robotics Institute Germany (RIG). · Interactive AI Algorithms & Cognitive Models for Human-AI Interaction (IKIDA), Computer Science Dept., TU Darmstadt, Germany. ·

Abstract

Learning structured task representations from human demonstrations is essential for bimanual manipulation, where action ordering, object involvement, and interaction geometry vary significantly across executions. A key challenge lies in jointly capturing the discrete semantic task structure and the temporal evolution of object-centric geometric relations in a form that supports reasoning over task progression. We introduce a semantic-geometric graph-based task representation that jointly encodes object identities, inter-object semantic relations, and per-object motion histories, via a Message Passing Neural Network (MPNN) encoder and a Transformer-based decoder. The encoder operates solely on the temporal scene graph, producing structured representations decoupled from action labels. The decoder then conditions on action-context to forecast future actions, associated objects, and object motions. This decoupling learns task-agnostic representations, enabling encoder reuse across embodiments through decoder-only finetuning on a small robot dataset. Across eleven bimanual tasks from two datasets, we find that the benefit of structured semantic--geometric representations over simpler sequence-based models grows with task variability in action ordering and object involvement. At deployment, a planner couples the action and motion predictions with learned Probabilistic Movement Primitives, achieving full task success on two real-robot bimanual tasks and outperforming graph ablations, Transformer, decoder-only, and finetuned vision-language model baselines.

Figures & tables

Fig. 1 : Model architecture: the graph encoder transforms features into embeddings via the MPNN, and the decoders forecast actions, objects, and motions.

Fig. 2 : A qualitative example from KIT(Bimacs) of the cooking task. Key frames (top row) are aligned with the action and object predictions of our MPNN model over time. The 3D motion predictions are shown below. Solid lines denote model predictions and dashed lines denote the ground-truth.

	Dreher	Lagamtzis	Trans-	Decoder-	MPNN
	[ 8 ]	[ 6 ]	former	Only	(Ours)
Graph-based	✓	✓			✓
Geometric info		✓	✓	✓	✓
Semantic info	✓				✓
Encoder-based	✓	✓	✓		✓

TABLE I : Overview over the different baseline models in this work.

Fig. 3 : Cross validation results on selected tasks from the KIT(Bimacs), from Ours(Bimacs) and a mixed dataset, including also multi-task models. Action prediction accuracies (next

a_{t+1}

, future

a_{F}

and horizon

a_{t+1:t+P}

) are shown on top, object predictions in the middle (next

o_{t+1}

, future

o_{F}

and horizon

o_{t+1:t+P}

) and motion prediction (

\bm{x}_{t+1:t+P}

) RMSE in the bottom row. The baselines are shown in different colors.

	Method	Accuracy $(\uparrow)$						RMSE [m]
	Method	$a_{t+1}$	$a_{F}$	$a_{t+1:t+P}$	$o_{t+1}$	$o_{F}$	$o_{t+1:t+P}$	$(\downarrow)$
Cooking	Robot	0.7460 $\pm$ 0.0501	0.6594 $\pm$ 0.0362	0.6443 $\pm$ 0.0453	0.8452 $\pm$ 0.0317	0.8538 $\pm$ 0.0087	0.7840 $\pm$ 0.0148	0.0547 $\pm$ 0.0013
	Not Finetuned	0.9279 $\pm$ 0.0235	0.9253 $\pm$ 0.0075	0.7661 $\pm$ 0.0108	0.8044 $\pm$ 0.0106	0.9185 $\pm$ 0.0057	0.7950 $\pm$ 0.0062	0.0304 $\pm$ 0.0002
	Finetuned	0.9454 $\pm$ 0.0249	0.9635 $\pm$ 0.0165	0.8815 $\pm$ 0.0166	0.8878 $\pm$ 0.0107	0.9459 $\pm$ 0.0146	0.8664 $\pm$ 0.0059	0.0212 $\pm$ 0.0009
Clear Up	Robot	0.9213 $\pm$ 0.0554	0.8610 $\pm$ 0.0889	0.6784 $\pm$ 0.0797	0.5358 $\pm$ 0.0792	0.2830 $\pm$ 0.0596	0.2052 $\pm$ 0.0612	0.0352 $\pm$ 0.0088
	Not Finetuned	0.9515 $\pm$ 0.0091	0.8873 $\pm$ 0.0188	0.7434 $\pm$ 0.0210	0.6671 $\pm$ 0.0401	0.4508 $\pm$ 0.0232	0.4870 $\pm$ 0.0432	0.0352 $\pm$ 0.0021

TABLE II : Cross-Validation Results of training the MPNN (Ours) Model on robot demonstrations, evaluating mixed models trained on human demonstrations on the robot dataset, and finetuning those models on robot data.

	Method	Success Rate $(\uparrow)$	Infeasibility Rate $(\downarrow)$	RMSE [m] $(\downarrow)$
Cooking	Dreher [ 8 ]	1.0000 $\pm$ 0.0000	0.0429 $\pm$ 0.1020	0.1569 $\pm$ 0.0229
	Lagamtzis [ 6 ]	0.7250 $\pm$ 0.1750	0.0890 $\pm$ 0.1741	0.0854 $\pm$ 0.0036
	Transformer	1.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.0509 $\pm$ 0.0043
	Decoder-Only	1.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.0544 $\pm$ 0.0022
	VLM	0.0000 $\pm$ 0.0000	1.0000 $\pm$ 0.0000	88.1708 $\pm$ 8.4930
	MPNN (Ours)	1.0000 $\pm$ 0.0000	0.0000 $\pm$ 0.0000	0.0469 $\pm$ 0.0022

TABLE III : Results of online action planning on the real-robot cooking and clear up tasks with finetuned models. The table shows the overall success rates, the planner infeasibility rates, and motion prediction RMSE.

Bimanual Robotic Manipulation

Bimanual robotic manipulation coordinates two robot arms to handle objects, tools, and tasks that require collaboration.

See all 189 papers

Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation
Feb 27, 2026Chongyang Xu, Haipeng Li, Shen Cheng +4Bimanual Robotic Manipulation Robotic Manipulation
College of Computer Science, Sichuan University, China · Dexmal · University of Electronic Science and Technology of China, China +1
Unified Learning of Temporal Task Structure and Action Timing for Bimanual Robot Manipulation
Mar 6, 2026Christian Dreher, Patrick Dormanns, Andre Meixner +1Bimanual Robotic Manipulation Temporal Reasoning
Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Karlsruhe, Germany
JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation
Sep 21, 2026Chuyang Xiao, Peilin Meng, David Held3D Point Tracking Bimanual Robotic Manipulation
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA. · University of Michigan, Ann Arbor, MI 48109, USA.

Feb 27, 2026J/K move · Enter open · S save

Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation

Chongyang Xu, Haipeng Li, Shen Cheng, Jingyu Hu+3

College of Computer Science, Sichuan University, China · Dexmal · University of Electronic Science and Technology of China, China+1

Bimanual Robotic Manipulation Robotic Manipulation

Open paper AskPDF arXiv

Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions. However, existing methods typically rely on 2D features with limited spatial awareness, or require explicit point clouds that are difficult to obtain reliably in real-world settings. At the same time, recent 3D geometric foundation models show that accurate and diverse 3D structure can be reconstructed directly from RGB images in a fast and robust manner. We leverage this opportunity and propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model. Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap. By explicitly predicting how the 3D scene will evolve together with the action sequence, the policy gains strong spatial understanding and predictive capability using only RGB observations. We evaluate our method both in simulation on the RoboTwin benchmark and in real-world robot executions. Our approach consistently outperforms 2D-based and point-cloud-based baselines, achieving state-of-the-art performance in manipulation success, inter-arm coordination, and 3D spatial prediction accuracy. Code is available at https://github.com/Chongyang-99/GAP.git.

Abstract

Bimanual Robotic Manipulation

Explore similar work

Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation

Unified Learning of Temporal Task Structure and Action Timing for Bimanual Robot Manipulation

JAMB: Joint Action-Motion Diffusion for Bimanual Manipulation