Preventing human error in physical assembly tasks requires assistance systems that can intervene before mistakes are made. This is especially challenging in settings where users are free to choose the construction order and correctness depends on the evolving assembly rather than a fixed next step. We present TanCRAFT, a four-stage real-time predictive assistance system that tracks the scene state, reasons about the current partial assembly, forecasts the near-future motion of manipulated pieces, and intervenes when the predicted placement is incompatible with the target. We instantiate our framework and confirm its effectiveness in a controlled tangram assembly task through offline benchmarks and a within-subject user study. Our findings show that the predictive feedback significantly reduces mistakes relative to unguided and reactive sessions, while yielding the lowest mean perceived workload and shortest mean completion time. Overall, these results demonstrate the value of predictive feedback for error prevention and provide a step toward cost-aware assistance for automation and human-robot workcells.
Figures & tables
Fig. 1: TanCRAFT overview. (A) The user assembles the target tangram shape while the system observes the workspace. (B) The system reasons about the evolving assembly state and assigns placed pieces to target-shape subgraphs. (C) As the user moves a piece toward an incompatible location, TanCRAFT issues audio-visual feedback to prevent the mistake.
Fig. 2: Tangram pieces and target shapes used in the study. (A) The seven physical tangram pieces used for assembly. (B) The six target shapes used during data collection and training. In the user study, the rocket was used in the written instructions, the house was used for the warm-up session, and the four animal shapes were used for testing the feedback modes.
Fig. 3: Overview of TanCRAFT. Given an overhead RGB stream, the Track stage estimates hand poses, tracks tangram-piece masks, and estimates object poses using a Kalman filter. The Reason stage infers the evolving assembly state with a Piece-to-Subgraph Classifier. The Forecast stage predicts the near-future motion of active pieces from their recent trajectories and scene context. Finally, the Intervene stage estimates active-piece destinations and checks their compatibility with the remaining subgraph capacities to drive predictive audio-visual warnings or continuous feedback.
Fig. 4: System-level precision, recall, and F1 across thresholds. The dashed line marks the operating point selected before the user study.
Mode
# Mistakes ↓
Overall NASA-TLX ↓
Success Rate ↑
Time (s) ↓
# Feedback
No Feedback
4.90±5.4
3.63±1.18
0.45
193.8±70.9
0.00±0.00
Reactive Feedback
1.95±2.4
3.20±1.12
0.50
172.9±78.4
1.20±1.2
Predictive Feedback
0.90±1.4
3.05±1.09
0.70
148.5±77.5
2.75±1.8
Continuous Feedback
0.75±1.0
3.32±1.09
0.70
170.1±64.7
1.85±1.7
TABLE V: User study results across feedback modes.
Fig. 5: Within-subject comparison of feedback policies. Each panel shows participant-level outcomes for all four modes evaluated. Lines connect results for the same participants. Asterisks mark significant planned model-based contrasts ( p<.05 ).
Interactive assistance systems typically provide feedback after an action has been completed, supporting error recovery but not preventing the error itself. We present TRAFA, a real-time predictive feedback system for procedural tasks that intervenes before errors are committed. TRAFA operationalizes predictive feedback through a Track-Forecast-Act framework that tracks hand and object state, forecasts user motion conditioned on scene context, and triggers feedback when a predicted action is likely to violate task constraints. We instantiate this pipeline in a sequential assembly setting and evaluate it through both technical benchmarking and a controlled user study against conventional reactive feedback. Our results show that predictive feedback improves task accuracy and efficiency while maintaining a comparable number of feedback events. These findings position feedback timing as a key dimension in system design and show how real-time anticipation can be integrated into interactive systems to prevent errors before they occur.
Sassan Mokhtar, Lars Doorenbos, Fatemeh Jabbari +3
University of Bonn · Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence
Geometric assembly requires fine spatiotemporal coordination to place parts without causing collisions or deadlocks under tight clearances. To address this, we present an object-centric reinforcement learning framework that treats each part as an agent and learns a shared policy for continuous assembly control. A graph neural network represents the pieces, their target poses, and their spatial relationships, and predicts translational and angular velocities for each component. Per-agent proximal policy optimization with a centralized graph critic directly learns motion and temporal coordination within a single feedback policy. We study this formulation through planar Tangram assembly, where pieces must slide and rotate into a given configuration while remaining on the assembly surface. Procedural generation of target configurations and randomized initial arrangements support training and evaluation across diverse assembly problems. Our experiments isolate the roles of relational structure and multi-agent optimization, and compare learned control with sampling-based planning. We demonstrate concurrent assembly in simulation and transfer the policy to sequential execution on a real robot, as our object-centric formulation decouples learned assembly coordination from low-level physical control. Project page: https://tangrambench.org/
Paul Knaußt, Kay Pompetzki, Nico Bohlinger +4
Intelligent Autonomous Systems Lab, Technical University of Darmstadt, Germany. · Control and Cyber-Physical Systems Laboratory, Technical University of Darmstadt · Hessian.AI, Germany. +2
Assembling objects from parts requires understanding multimodal instructions, linking them to 3D components, and predicting physically plausible 6-DoF motions for each assembly step. Existing datasets focus on simplified scenarios, overlooking shape complexities and assembly trajectories in industrial assemblies. We introduce AssemblyBench, a synthetic dataset of 2,789 industrial objects with multimodal instruction manuals, corresponding 3D part models, and part assembly trajectories. We also propose a transformer-based model, AssemblyDyno, which uses the instructional manual and the 3D shape of each part to jointly predict assembly order and part assembly trajectories. AssemblyDyno outperforms prior works in both assembly pose estimation and trajectory feasibility, where the latter is evaluated by our physics-based simulations.
Danrui Li, Jiahao Zhang, Bernhard Egger +4
Rutgers, The State University of New Jersey, USA · The Australian National University, Australia · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +1