It Takes Two to Tangram: Multi-Agent Object Centric Reinforcement Learning for Assembly
Authors: Paul Knaußt, Kay Pompetzki, Nico Bohlinger, Niklas Funk, Cristiana de Farias, Rolf Findeisen, Jan Peters
Organizations: Intelligent Autonomous Systems Lab, Technical University of Darmstadt, Germany. · Control and Cyber-Physical Systems Laboratory, Technical University of Darmstadt · Hessian.AI, Germany. · German Research Center for AI (DFKI), SAIROL, Germany. · Robotics Institute Germany (RIG).
Geometric assembly requires fine spatiotemporal coordination to place parts without causing collisions or deadlocks under tight clearances. To address this, we present an object-centric reinforcement learning framework that treats each part as an agent and learns a shared policy for continuous assembly control. A graph neural network represents the pieces, their target poses, and their spatial relationships, and predicts translational and angular velocities for each component. Per-agent proximal policy optimization with a centralized graph critic directly learns motion and temporal coordination within a single feedback policy. We study this formulation through planar Tangram assembly, where pieces must slide and rotate into a given configuration while remaining on the assembly surface. Procedural generation of target configurations and randomized initial arrangements support training and evaluation across diverse assembly problems. Our experiments isolate the roles of relational structure and multi-agent optimization, and compare learned control with sampling-based planning. We demonstrate concurrent assembly in simulation and transfer the policy to sequential execution on a real robot, as our object-centric formulation decouples learned assembly coordination from low-level physical control. Project page: https://tangrambench.org/
Figures & tables
Fig. 2: TangramBench provides a graded benchmark for planar geometric assembly, ranging from the three-piece TangramBench-Trio to seven-piece TangramBench-Cluster and tightly constrained TangramBench-Puzzle configurations. We provide simulation-ready models, together with a robot-compatible physical counterpart consisting of 3D-printed pieces and an insertion tool for planar manipulation.
Fig. 4: Policy ablations. On the top row, we show the architecture ablation on TangramBench-Trio. Here we report task success, fraction of pieces within 5 cm of their targets, position and yaw errors, and contact rate throughout training. On the bottom, the effect of the reward formulation. We ablate the shared team reward and inter-piece contact cost.
Fig. 5: Success comparison on assemblies containing a different piece count.
Method
Succ.
Err.
Contact
Time
Compute
(%)
(cm)
(%)
(s)
TangramBench-Cluster
GOMA (ours)
75
1.0
4
8
0.68 ms
APF
3
3.8
99
80
≈0
RRT-J
73
1.9
44
20
774 ms
RRT-J +replan
98
0.3
12
21
49 ms
TABLE I: GOMA against the APF controller and joint (RRT-J) and per-piece (RRT-P) RRT planning, planned once or replanned every 5 s, on the held-out TangramBench-Cluster and TangramBench-Puzzle.
Preventing human error in physical assembly tasks requires assistance systems that can intervene before mistakes are made. This is especially challenging in settings where users are free to choose the construction order and correctness depends on the evolving assembly rather than a fixed next step. We present TanCRAFT, a four-stage real-time predictive assistance system that tracks the scene state, reasons about the current partial assembly, forecasts the near-future motion of manipulated pieces, and intervenes when the predicted placement is incompatible with the target. We instantiate our framework and confirm its effectiveness in a controlled tangram assembly task through offline benchmarks and a within-subject user study. Our findings show that the predictive feedback significantly reduces mistakes relative to unguided and reactive sessions, while yielding the lowest mean perceived workload and shortest mean completion time. Overall, these results demonstrate the value of predictive feedback for error prevention and provide a step toward cost-aware assistance for automation and human-robot workcells.
Sassan Mokhtar, Lars Doorenbos, Marius Bock +4
University of Bonn, Germany. · Lamarr Institute, Germany. · University of New Mexico, USA. +1
An interlocking assembly is an assembly in which component parts are connected purely through their geometric arrangement, without relying on external connectors such as glue and nails. Such assemblies have been widely used in a variety of real-world applications due to their structural stability. The problem of generating interlocking assemblies is generally formulated as a shape decomposition problem, where a target 3D object represented as a voxel grid is partitioned into a prescribed number of interlocking pieces. We observe that generating interlocking assemblies is inherently a sequential decision-making problem, where an agent repeatedly decides which piece each voxel should be assigned to. Inspired by the observation, we propose the first reinforcement learning framework RL-Lock for generating interlocking assemblies, without relying on handcrafted search heuristics as existing works did. RL-Lock combines structured action chunking with MCTS-guided policy-value learning to efficiently navigate the large combinatorial search space for interlocking assembly generation. We demonstrate through experiments that RL-Lock allows effective generation of interlocking assemblies, especially for challenging cases in which existing approaches take too long or even fail to find a valid solution.
Xuyang Ma, Chaewoon Kim, Haonan Zhang +3
Singapore University of Technology and Design · The Hong Kong University of Science and Technology · Beijing Normal-Hong Kong Baptist University
A hallmark of human dexterity is the cooperative use of fingers, where different fingers take on distinct yet coordinated roles to accomplish fine manipu- lation, such as capping a pen with the hand that holds it. We study this finger-level coordination through in-hand assembly: mating two rigid objects within a single dexterous hand, with no second arm and no fixture. We present a reinforcement learning formulation to solve this problem in a unified framework, which is driven by a goal relative pose between the two parts. Finger coordination is shaped by a function-based auxiliary reward and regularized toward a single human reference pose, while domain randomization and a fusion of historical proprioception and object observation confer robustness to occlusion-induced estimation noise. The same recipe solves three different assembly tasks (Bottle, Syringe, and Marker). Trained purely in simulation, the policies transfer zero-shot to hardware with a single camera, demonstrating robustness to state-estimation errors caused by oc- clusion. Our experiments also reveal that in-hand assembly places demands on hand morphology and can serve as a benchmark for modern robotic hand systems. Videos and code are available at https://ltbgbird.github.io/in-hand-assembly-page/.
Liuao Pei, Tianyue Wu, Hui Zhang +2
The University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · ETH Zurich +1