Joint Prediction of Human Motions and Actions in Human-Robot Collaboration

Alessandra Bulanti; Alessandro Carfì; Fulvio Mastrogiovanni

cs.ROApr 3, 2026

Joint Prediction of Human Motions and Actions in Human-Robot Collaboration

Authors: Alessandra Bulanti, Alessandro Carfì, Fulvio Mastrogiovanni

Organizations: University of Genoa, Italy

Abstract

Fluent human--robot collaboration requires robots to continuously estimate human behaviour and anticipate future intentions. This entails reasoning jointly about continuous movements and discrete actions, which are still largely modelled in isolation. In this paper, we introduce MA-HERP, a hierarchical and recursive probabilistic framework for the joint estimation and prediction of human movements and actions. The model combines: (i) a hierarchical representation in which movements compose into actions through admissible Allen interval relations, (ii) a unified probabilistic factorisation coupling continuous dynamics, discrete labels, and durations, and (iii) a recursive inference scheme inspired by Bayesian filtering, alternating top-down action prediction with bottom-up sensory evidence. We present a preliminary experimental evaluation based on neural models trained on musculoskeletal simulations of reaching movements, showing accurate motion prediction, robust action inference under noise, and computational performance compatible with on-line human--robot collaboration.

Figures & tables

Fig. 1 : An illustrative example of a coffee-making scenario in which the prediction of human motions and actions can improve the perceived collaboration fluency to a great extent.

Fig. 2 : Hierarchical structure of movements and actions MA-HERP . Level $h=0$ encodes continuous movement segments, while levels $h\geq 1$ encode discrete actions. Edges denote temporal relations defined via Allen’s interval algebra.

Panel of Figure 3 — Fig. 3 : A musculoskeletal model executing a reaching movement. Three poses are shown from left to right.

Panel of Figure 4 — Fig. 4 : Predictions (red) in configuration space for $\ell_{1}=\textsf{I}\to\textsf{A}$ obtained with ( 4(a) ) the model trained on D-0 and ( 4(b) ) the model trained on D-0-10-30 , compared against ground truth (black).

D-0
Models	PCC (Avg.) [-1,1]	PCC (Std.) [0,1]	RMSE (Avg.) [rad]	RMSE (Std.) [rad]
$\textsf{I}\to\textsf{A}$	(0.99, 0.99, 0.99)	(0.01, 0.02, 0.06)	(0.01, 0.01, 0.01)	(0.01, 0.01, 0.01)
$\textsf{I}\to\textsf{B}$	(0.98, 0.86, 0.21)	(0.06, 0.32, 0.78)	(0.02, 0.01, 0.01)	(0.01, 0.01, 0.01)
$\textsf{I}\to\textsf{C}$	(0.99, 0.99, 0.99)	(0.01, 0.01, 0.33)	(0.01, 0.01, 0.01)	(0.01, 0.01, 0.01)
$\textsf{A}\to\textsf{B}$	(0.99, 0.98, 0.62)	(0.01, 0.05, 0.60)	(0.01, 0.01, 0.01)	(0.01, 0.01, 0.01)
$\textsf{A}\to\textsf{C}$	(0.99, 0.99, 0.99)	(0.01, 0.01, 0.02)	(0.01, 0.02, 0.01)	(0.01, 0.01, 0.01)

TABLE I : Pearson Correlation Coefficient (PCC) and Root Mean Square Error (RMSE) for all label-conditioned movement predictors over all joint angles (

q_{1},q_{2},q_{3}

), for D-0 and D-0-10-30 . Reported values are rounded to two decimal places.

D-0			D-0-10-30
Model	Avg. [s]	Std. [s]	Model	Avg. [s]	Std. [s]
$\textsf{I}\to\textsf{A}$	0.1363	0.0130	$\textsf{I}\to\textsf{A}$	0.1753	0.0097
$\textsf{I}\to\textsf{B}$	0.1457	0.0013	$\textsf{I}\to\textsf{B}$	0.1783	0.0027
$\textsf{I}\to\textsf{C}$	0.1448	0.0017	$\textsf{I}\to\textsf{C}$	0.1777	0.0019
$\textsf{A}\to\textsf{B}$	0.1359	0.0131	$\textsf{A}\to\textsf{B}$	0.1452	0.0080
$\textsf{A}\to\textsf{C}$	0.1382	0.0133	$\textsf{A}\to\textsf{C}$	0.1433	0.0082

TABLE II : Average and standard deviation inference times for movement prediction (forecasting

P=50

steps) for all label-conditioned models, for D-0 (100 test trajectories) and D-0-10-30 (300 test trajectories).

D-0		D-0-10-30
Model	Accuracy [0,1]	Model	Accuracy [0,1]
$\textsf{I}\to\textsf{A}\textsf{B}\textsf{C}$	0.95	$\textsf{I}\to\textsf{A}\textsf{B}\textsf{C}$	0.66
$\textsf{A}\to\textsf{BC}$	1.00	$\textsf{A}\to\textsf{BC}$	1.00
$\textsf{B}\to\textsf{AC}$	0.89	$\textsf{B}\to\textsf{AC}$	0.50
$\textsf{C}\to\textsf{AB}$	0.61	$\textsf{C}\to\textsf{AB}$	0.80

TABLE III : Classification accuracy (majority vote for sliding-window inference).

D-0			D-0-10-30
Model	Avg. [s]	Std. [s]	Model	Avg. [s]	Std. [s]
$\textsf{I}\to\textsf{A}\textsf{B}\textsf{C}$	0.0007	0.0003	$\textsf{I}\to\textsf{A}\textsf{B}\textsf{C}$	0.0006	0.0000
$\textsf{A}\to\textsf{BC}$	0.0006	0.0000	$\textsf{A}\to\textsf{BC}$	0.0006	0.0000
$\textsf{B}\to\textsf{AC}$	0.0006	0.0000	$\textsf{B}\to\textsf{AC}$	0.0006	0.0000
$\textsf{C}\to\textsf{AB}$	0.0006	0.0000	$\textsf{C}\to\textsf{AB}$	0.0006	0.0000

TABLE IV : Prediction times for action prediction/classification models.

Panel of Figure 9 — Fig. 5 : Confusion matrices for D-0 (top row) and D-0-10-30 (bottom row) datasets. P denotes precision, R the recall, and F1 the Macro F1 score.

Movement	$\alpha(\ell_{k})$	Accuracy	$\ell_{k}$
$\textsf{I}\to\textsf{C}$	reach-c	0.73	reach-c
$\textsf{C}\to\textsf{A}$	reach-a	0.81	reach-a
$\textsf{A}\to\textsf{B}$	reach-b	1.00	reach-b
$\textsf{B}\to\textsf{A}$	reach-a	1.00	reach-a
$\textsf{A}\to\textsf{B}$	reach-b	1.00	reach-b
$\textsf{B}\to\textsf{C}$	reach-c	0.54	reach-c

TABLE V : Predicted versus actual labels for the sequence of

15

reach movements.

Panel of Figure 11 — Fig. 6 : Configuration spaces of the first three reach movements in the sequence, namely $\textsf{I}\to\textsf{C}$ , $\textsf{C}\to\textsf{A}$ , and $\textsf{A}\to\textsf{B}$ , with before relations between consecutive movements. Black: generated motion. Red: predicted trajectories.

Explore similar work

Human Motion Prediction for Human-Robot Collaboration
Sep 26, 2026Placido Falqueto, Elena Basei, Edoardo Lamon +4Human Motion Prediction Human-Robot Collaboration
Dept. of Information Engineering and Computer Science (DISI), University of Trento, Trento, Italy · Department of Industrial Engineering (DII), University of Trento, Trento, Italy
Adaptive Interaction Network for Human Motion Prediction During Human-Robot Collaboration
Oct 12, 2026Mengyuan Liu, Yangting Lin, Qiongjie CuiHuman Motion Prediction Human-Robot Collaboration
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, Shenzhen, Guangdong 518055, China · Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Peking University, Shenzhen Graduate School, Shenzhen, Guangdong 518055, China · National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an, Shaanxi 710049, China
Proactive Motion Planning for Human-Robot Cooperation
Sep 26, 2026Elena Basei, Edoardo Lamon, Matteo Saveriano +2Human-Robot Collaboration Safe Motion Planning
Department of Information Engineering and Computer Science, Universit`a di Trento, Trento, Italy. · Interdepartmental Robotics Labs (IDRA), University of Trento, Trento, Italy · Department of Industrial Engineering, Universit`a di Trento, Trento, Italy.

Abstract

Explore similar work

Human Motion Prediction for Human-Robot Collaboration

Adaptive Interaction Network for Human Motion Prediction During Human-Robot Collaboration

Proactive Motion Planning for Human-Robot Cooperation