Learning structured task representations from human demonstrations is essential for bimanual manipulation, where action ordering, object involvement, and interaction geometry vary significantly across executions. A key challenge lies in jointly capturing the discrete semantic task structure and the temporal evolution of object-centric geometric relations in a form that supports reasoning over task progression. We introduce a semantic-geometric graph-based task representation that jointly encodes object identities, inter-object semantic relations, and per-object motion histories, via a Message Passing Neural Network (MPNN) encoder and a Transformer-based decoder. The encoder operates solely on the temporal scene graph, producing structured representations decoupled from action labels. The decoder then conditions on action-context to forecast future actions, associated objects, and object motions. This decoupling learns task-agnostic representations, enabling encoder reuse across embodiments through decoder-only finetuning on a small robot dataset. Across eleven bimanual tasks from two datasets, we find that the benefit of structured semantic--geometric representations over simpler sequence-based models grows with task variability in action ordering and object involvement. At deployment, a planner couples the action and motion predictions with learned Probabilistic Movement Primitives, achieving full task success on two real-robot bimanual tasks and outperforming graph ablations, Transformer, decoder-only, and finetuned vision-language model baselines.
Figures & tables
Fig. 1 : Model architecture: the graph encoder transforms features into embeddings via the MPNN, and the decoders forecast actions, objects, and motions.
Fig. 2 : A qualitative example from KIT(Bimacs) of the cooking task. Key frames (top row) are aligned with the action and object predictions of our MPNN model over time. The 3D motion predictions are shown below. Solid lines denote model predictions and dashed lines denote the ground-truth.
Dreher
Lagamtzis
Trans-
Decoder-
MPNN
[ 8 ]
[ 6 ]
former
Only
(Ours)
Graph-based
✓
✓
✓
Geometric info
✓
✓
✓
✓
Semantic info
✓
✓
Encoder-based
✓
✓
✓
✓
TABLE I : Overview over the different baseline models in this work.
Fig. 3 : Cross validation results on selected tasks from the KIT(Bimacs), from Ours(Bimacs) and a mixed dataset, including also multi-task models. Action prediction accuracies (next at+1 , future aF and horizon at+1:t+P ) are shown on top, object predictions in the middle (next ot+1 , future oF and horizon ot+1:t+P ) and motion prediction ( xt+1:t+P ) RMSE in the bottom row. The baselines are shown in different colors.
Method
Accuracy (↑)
RMSE [m]
at+1
aF
at+1:t+P
ot+1
oF
ot+1:t+P
(↓)
Cooking
Robot
0.7460 ± 0.0501
0.6594 ± 0.0362
0.6443 ± 0.0453
0.8452 ± 0.0317
0.8538 ± 0.0087
0.7840 ± 0.0148
0.0547 ± 0.0013
Not Finetuned
0.9279 ± 0.0235
0.9253 ± 0.0075
0.7661 ± 0.0108
0.8044 ± 0.0106
0.9185 ± 0.0057
0.7950 ± 0.0062
0.0304 ± 0.0002
Finetuned
0.9454 ± 0.0249
0.9635 ± 0.0165
0.8815 ± 0.0166
0.8878 ± 0.0107
0.9459 ± 0.0146
0.8664 ± 0.0059
0.0212 ± 0.0009
Clear Up
Robot
0.9213 ± 0.0554
0.8610 ± 0.0889
0.6784 ± 0.0797
0.5358 ± 0.0792
0.2830 ± 0.0596
0.2052 ± 0.0612
0.0352 ± 0.0088
Not Finetuned
0.9515 ± 0.0091
0.8873 ± 0.0188
0.7434 ± 0.0210
0.6671 ± 0.0401
0.4508 ± 0.0232
0.4870 ± 0.0432
0.0352 ± 0.0021
TABLE II : Cross-Validation Results of training the MPNN (Ours) Model on robot demonstrations, evaluating mixed models trained on human demonstrations on the robot dataset, and finetuning those models on robot data.
Method
Success Rate (↑)
Infeasibility Rate (↓)
RMSE [m] (↓)
Cooking
Dreher [ 8 ]
1.0000 ± 0.0000
0.0429 ± 0.1020
0.1569 ± 0.0229
Lagamtzis [ 6 ]
0.7250 ± 0.1750
0.0890 ± 0.1741
0.0854 ± 0.0036
Transformer
1.0000 ± 0.0000
0.0000 ± 0.0000
0.0509 ± 0.0043
Decoder-Only
1.0000 ± 0.0000
0.0000 ± 0.0000
0.0544 ± 0.0022
VLM
0.0000 ± 0.0000
1.0000 ± 0.0000
88.1708 ± 8.4930
MPNN (Ours)
1.0000 ± 0.0000
0.0000 ± 0.0000
0.0469 ± 0.0022
TABLE III : Results of online action planning on the real-robot cooking and clear up tasks with finetuned models. The table shows the overall success rates, the planner infeasibility rates, and motion prediction RMSE.
Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions. However, existing methods typically rely on 2D features with limited spatial awareness, or require explicit point clouds that are difficult to obtain reliably in real-world settings. At the same time, recent 3D geometric foundation models show that accurate and diverse 3D structure can be reconstructed directly from RGB images in a fast and robust manner. We leverage this opportunity and propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model. Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap. By explicitly predicting how the 3D scene will evolve together with the action sequence, the policy gains strong spatial understanding and predictive capability using only RGB observations. We evaluate our method both in simulation on the RoboTwin benchmark and in real-world robot executions. Our approach consistently outperforms 2D-based and point-cloud-based baselines, achieving state-of-the-art performance in manipulation success, inter-arm coordination, and 3D spatial prediction accuracy. Code is available at https://github.com/Chongyang-99/GAP.git.
Chongyang Xu, Haipeng Li, Shen Cheng +4
College of Computer Science, Sichuan University, China · Dexmal · University of Electronic Science and Technology of China, China +1
Bimanual manipulation requires both temporal task structure - which actions precede or overlap others - and concrete timing - when each action starts and how long it takes. Symbolic relations enable high-level reasoning about task structure and alternative execution sequences, while concrete timing parameters are essential for coordinating two hands at the execution level. Existing approaches address these two levels in isolation, leaving a gap between task planning and movement synchronization. This work presents an approach for learning symbolic and subsymbolic temporal task constraints from human demonstrations and deriving executable, temporally parametrized plans for bimanual manipulation. Specifically, our contributions are (i) a 3-D representation of timings with methods based on multivariate Gaussian Mixture Models to represent temporal relationships between actions on a subsymbolic level, (ii) a method based on the Davis-Putnam-Logemann-Loveland algorithm that finds and ranks all contradiction-free Allen relation assignments of a task, representing its modes, and (iii) an optimization-based planning system combining the identified symbolic and subsymbolic temporal task constraints to derive parametrized plans for robot execution. We quantitatively evaluate our approach and show that the inferred symbolic task constraints are more accurate than those of prior heuristic approaches and that the generated plans are temporally closer to human demonstrations than the most characteristic demonstration. Additionally, we qualitatively demonstrate its applicability through orchestrated task executions in simulation and on real robots.
Christian Dreher, Patrick Dormanns, Andre Meixner +1
Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Karlsruhe, Germany
Coordinated bimanual manipulation is challenging because the motion of either arm can alter the shared 3D scene and thereby affect the other arm. Yet most diffusion policies generate actions without explicitly modeling these future geometric consequences, while predictive variants typically use future state only as auxiliary supervision or fixed conditioning. We address this limitation by proposing JAMB, a diffusion policy that jointly denoises bimanual actions and future 3D point tracks. By allowing action and track hypotheses to evolve together within a shared Transformer, each can inform and refine the other throughout denoising. We further ground multimodal representations in a shared spatiotemporal coordinate system to facilitate geometry-aware interaction during joint denoising. We evaluate JAMB on diverse bimanual manipulation tasks in RoboTwin 2.0 and on a real-world robot, comparing it with action-only policies and alternative future-prediction approaches spanning different state representations and learning objectives. Across 16 simulation tasks, JAMB achieves an average success rate of 83.4%, outperforming the strongest baseline by 23.9 percentage points. On three real-world tasks, it outperforms the action-only and auxiliary geometry prediction methods by 50.0 and 21.2 percentage points, respectively. Beyond these performance gains, JAMB shows stronger generalization to cluttered scenes and out-of-distribution backgrounds than the evaluated baselines. Together, these results demonstrate the effectiveness of our joint action-motion modeling framework for coordinated bimanual manipulation. Our project website is available at https://jam-bimanual.github.io/
Chuyang Xiao, Peilin Meng, David Held
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA. · University of Michigan, Ann Arbor, MI 48109, USA.