Preventing human error in physical assembly tasks requires assistance systems that can intervene before mistakes are made. This is especially challenging in settings where users are free to choose the construction order and correctness depends on the evolving assembly rather than a fixed next step. We present TanCRAFT, a four-stage real-time predictive assistance system that tracks the scene state, reasons about the current partial assembly, forecasts the near-future motion of manipulated pieces, and intervenes when the predicted placement is incompatible with the target. We instantiate our framework and confirm its effectiveness in a controlled tangram assembly task through offline benchmarks and a within-subject user study. Our findings show that the predictive feedback significantly reduces mistakes relative to unguided and reactive sessions, while yielding the lowest mean perceived workload and shortest mean completion time. Overall, these results demonstrate the value of predictive feedback for error prevention and provide a step toward cost-aware assistance for automation and human-robot workcells.
Figures & tables
Fig. 1: TanCRAFT overview. (A) The user assembles the target tangram shape while the system observes the workspace. (B) The system reasons about the evolving assembly state and assigns placed pieces to target-shape subgraphs. (C) As the user moves a piece toward an incompatible location, TanCRAFT issues audio-visual feedback to prevent the mistake.
Fig. 2: Tangram pieces and target shapes used in the study. (A) The seven physical tangram pieces used for assembly. (B) The six target shapes used during data collection and training. In the user study, the rocket was used in the written instructions, the house was used for the warm-up session, and the four animal shapes were used for testing the feedback modes.
Fig. 3: Overview of TanCRAFT. Given an overhead RGB stream, the Track stage estimates hand poses, tracks tangram-piece masks, and estimates object poses using a Kalman filter. The Reason stage infers the evolving assembly state with a Piece-to-Subgraph Classifier. The Forecast stage predicts the near-future motion of active pieces from their recent trajectories and scene context. Finally, the Intervene stage estimates active-piece destinations and checks their compatibility with the remaining subgraph capacities to drive predictive audio-visual warnings or continuous feedback.
Fig. 4: System-level precision, recall, and F1 across thresholds. The dashed line marks the operating point selected before the user study.
Mode
# Mistakes ↓
Overall NASA-TLX ↓
Success Rate ↑
Time (s) ↓
# Feedback
No Feedback
4.90±5.4
3.63±1.18
0.45
193.8±70.9
0.00±0.00
Reactive Feedback
1.95±2.4
3.20±1.12
0.50
172.9±78.4
1.20±1.2
Predictive Feedback
0.90±1.4
3.05±1.09
0.70
148.5±77.5
2.75±1.8
Continuous Feedback
0.75±1.0
3.32±1.09
0.70
170.1±64.7
1.85±1.7
TABLE V: User study results across feedback modes.
Fig. 5: Within-subject comparison of feedback policies. Each panel shows participant-level outcomes for all four modes evaluated. Lines connect results for the same participants. Asterisks mark significant planned model-based contrasts ( p<.05 ).
Intelligent Autonomous Systems Lab, Technical University of Darmstadt, Germany. · Control and Cyber-Physical Systems Laboratory, Technical University of Darmstadt · Hessian.AI, Germany. +2
Rutgers, The State University of New Jersey, USA · The Australian National University, Australia · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +1