Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation

Thanh Nguyen Canh; Thanh-Tuan Tran; Haolan Zhang; Ziyan Gao; Nak Young Chong; Xiem HoangVan

Figure 1: An illustration of Robot Imitation Learning from Human Demonstration.

Figure 2: The proposed Video Understanding architecture consists of two parallel branches: (a) Interacted Object Understanding Module and (b) Action Understanding Module. Initially, the raw input frames

\mathbf{F}=\{\mathbf{f}_{1},\mathbf{f}_{2},\dots,\mathbf{f}_{n}\}

are downsampled to

\tilde{\mathbf{F}}=\{\tilde{\mathbf{f}}_{1},\tilde{\mathbf{f}}_{2},\dots,\tilde{\mathbf{f}}_{m}\}

(where

n>m

) to optimize runtime and match the training data frame rate in Action Understanding Module. (a) The Interacted Object Understanding Module processes

\tilde{\mathbf{F}}

to extract a subset of keyframes

\hat{\mathbf{F}}=\{\hat{\mathbf{f}}_{1},\hat{\mathbf{f}}_{2},\dots,\hat{\mathbf{f}}_{k}\}

, (where

m>k

). These frames are analyzed by our Object Selection algorithm and Vision-Language Models (VLMs) to identify the specific objects involved in the interaction accurately. (b) The Action Understanding Module is implemented based on a CNN architecture with a ResNet-50 He et al. (2016) backbone with Temporal Shift Modules (TSM) that shift feature channels along the temporal dimension to capture fine-grained motion dynamics for action classification.

Figure 3: Pipeline of the proposed Object Selection algorithm. Cropped object regions are tracked across frames to generate trajectory data, which is used to classify objects into Pickable-object and Placeable-object sets based on motion patterns. Each candidate object then undergoes Blur Detection and Overlap Detection to select the highest-quality object instances for subsequent Vision-Language Model processing.

Figure 4: Visualization of Interacted object trajectory tracking across three intersection scenarios. Left: RGB frames showing human hand-object interactions with objects. Right: Corresponding 2D trajectory plots of the tracked objects’ centroid positions over time, where the characteristic parabolic or oscillatory patterns indicate pickable objects, enabling automatic classification of object functional roles (pickable or placeable).

Figure 5: The architecture of our DRL system for robot imitation. processes the environment observation

\mathcal{O}

that consists of agent’s proprioceptive, exteroceptive, relational, and historical data such as: joint angle

\mathbf{j}_{t}

, joint velocity

\mathbf{j}^{\prime}_{t}

, object orientation

\mathbf{R}_{o}

, object coordination and target

\mathbf{p}_{o}

,

\mathbf{p}_{g}

, end effector status

\mathbf{p}_{e}

, we also provide relative observations

\bar{\mathbf{p}}_{eo}

,

\bar{\mathbf{p}}_{og}

with past joint angle

\mathbf{a}_{t-1}

, provided by the Video Understanding framework and outputs the location of the object referenced by

\mathbf{i}_{t}

.

Component	Description	Dimension	Noise
$\mathbf{j}_{t}$	joint positions at time $t$	6D	$\mathcal{U}(\pm 0.005)$
$\mathbf{j}^{\prime}_{t}$	joint velocities at time $t$	6D	$\mathcal{U}(\pm 0.05)$
$\mathbf{R}_{o}$	object orientation	4D	$\mathcal{U}(\pm 0.005)$
$\bar{\mathbf{p}}_{eo}$	end-effector to object vector	3D	$\mathcal{U}(\pm 0.005)$
$\bar{\mathbf{p}}_{og}$	object to goal vector	3D	$\mathcal{U}(\pm 0.005)$
$\mathbf{p}_{o}$	object coordination	3D	$\mathcal{U}(\pm 0.005)$

Table 1: State representation for the DRL agent.

Rewards/Penalties	Description	Weight
$\begin{aligned} r^{e}_{t}=w_{0}\exp\!\left(-\frac{\lVert{}\mathbf{p}_{e}-\mathbf{p}_{o}\rVert}{\lambda_{0}}\right)&+w_{1}\exp\!\left(-\frac{\|\psi_{e}-\pi\|+\|\theta_{e}\|}{\lambda_{1}}\right)\\ &+w_{2}\exp\!\left(-\frac{\lVert{}p^{z}_{e}-p^{z}_{o}\rVert}{\lambda_{2}}\right)\end{aligned}$	safe end-effector approach to the object	$(1.3,0.2,0.15)\rightarrow(0.325,0.2,0.15)$
$\begin{aligned} r^{i}_{t}=&\delta_{c}\left(w_{3}\exp\!\left(-\frac{\lVert{}\mathbf{p}_{o}-\mathbf{p}_{g}\rVert}{\lambda_{3}}\right)+w_{4}\exp\!\left(-\frac{\lVert{}\mathbf{p}_{o}-\mathbf{p}_{g}\rVert}{\lambda_{4}}\right)\right),\\ &\delta_{c}=1\text{ if contact is established},\hskip 17.00024pt\lambda_{3}>>\lambda_{4}\end{aligned}$	object movement toward the target	$(2.0,3.0)$
$r^{a}_{t}=w_{5}\exp\!\left(\frac{\bar{\mathbf{p}}^{\prime}_{o}\cdot\bar{\mathbf{p}}_{og}}{\lambda_{5}}-1\right)\bigg[\bar{\mathbf{p}}^{\prime}_{o}\cdot\bar{\mathbf{p}}_{og}>0\bigg]$	object motion aligned with goal direction	0.8
$c_{c}=\begin{cases}3,&\text{if colliding with ground or itself}\\ 1.5,&\text{if colliding with objects}\\ 0,&\text{otherwise.}\end{cases}$	undesired collisions	1.0
$c_{s}=2$	reaching the max step limit	1.0
$c_{e}=w_{6}\left(\|\psi_{e}-\pi\|-\|\theta_{e}\|\right)$	end-effector pose inclination limitation	1.0

Table 2: Rewards and penalties formula used for training, along with the initial and final weights. ∗ Notation:

\mathbf{p}_{e}

is end-effector position;

\lambda_{\_}

is scale factor;

\theta_{e},\theta_{o},\psi_{e},\psi_{o}

are roll and pitch of end-effector and object respectively;

p_{\_}^{x},p_{\_}^{y},p_{\_}^{z}

are position in x, y, z-axis respectively;

\theta_{o}^{c}

is a constraint;

\mathbf{w}^{c},w_{x}^{c},w_{y}^{c}

are position constraint of workspace in xyz coordinate, only x-axis, and only y-axis respectively.

Figure 6: Distribution of training videos across action categories in the test set

Figure 7: Object categories used for evaluating video understanding generalization. (Left) Standard object set: 12 object categories present in the training dataset, including common household items. (Right) Novel object set: 9 previously unseen object categories used exclusively for testing

Figure 8: Qualitative results of reach action. Left: Human demonstration videos showing reaching motion toward various objects. Right: Corresponding simulated UR5 robot executions, where the robot successfully navigates its end-effector to the target object position across diverse scene configurations with multiple distractor objects.

Figure 9: Qualitative results of the pick action. Left: Human demonstration videos showing grasping motions for objects of varying colors and shapes. Right: Corresponding simulated executions using the suction gripper, with the robot correctly identifying and picking the target object among multiple distractors.

Figure 10: Qualitative results of the move action. Left: Human demonstration videos showing object transportation from one location to another. Right: Corresponding simulated executions, where the robot successfully transports grasped objects along trajectories toward the designated goal positions while maintaining a stable grasp throughout the motion.

Figure 11: Qualitative results of the put action. Left: Human demonstration videos showing object placement into containers (bowls, plates). Right: Corresponding simulated execution demonstrating precise placement of objects at target locations, completing the full pick-and-place manipulation sequence learned from video demonstration.

Hyperparameter	Value
Learning rate (actor)	$4\times 10^{-4}$
Learning rate (critic)	$4\times 10^{-4}$
Discount factor $\gamma$	0.99
Soft update rate $\rho$	0.005
Replay buffer size	$10^{6}$
Batch size	256

Table 3: DRL training hyperparameters.

Test Set	Backbone	Top-1 Acc.	Top-5 Acc.
Test set 1	ResNet-34	82.79	93.50
	ResNet-50	86.79	94.55
	ResNet-101	89.97	96.82
Test set 2	ResNet-34	63.70	94.81
	ResNet-50	68.15	94.81
	ResNet-101	71.11	95.56

Table 4: Action classification accuracy (%) of the Temporal Shift Module (TSM) with varying backbone depths. Test Set 1: standard benchmark split; Test Set 2: self-recorded clips for real-world evaluation. All models use 8-frame sampling with 8-clips

\times

3-crops augmentation using Modified Something-Something V2 during inference.

Backbone	Method	BLEU-1	BLEU-2	BLEU-3	BLEU-4
InceptionV3	Video2Command Nguyen et al. (2018)	0.323	0.196	0.162	0.154
	V2C Nguyen et al. (2019)	0.355	0.227	0.201	0.164
	Watch-and-Act Yang et al. (2023)	0.399	0.287	0.263	0.199
ResNet-50	Video2Command Nguyen et al. (2018)	0.312	0.196	0.173	0.150
	V2C Nguyen et al. (2019)	0.357	0.231	0.201	0.153
	Watch-and-Act Yang et al. (2023)	0.394	0.271	0.248	0.187

Table 5: Video-to-command generation performance (BLEU scores) on standard object sets ( bold and underline are best and second best, respectively).

Backbone	Method	BLEU-1	BLEU-2	BLEU-3	BLEU-4
InceptionV3	Video2Command Nguyen et al. (2018)	0.244	0.132	0.075	0.062
	V2C Nguyen et al. (2019)	0.279	0.117	0.093	0.087
	Watch-and-Act Yang et al. (2023)	0.293	0.141	0.133	0.116
ResNet-50	Video2Command Nguyen et al. (2018)	0.305	0.150	0.090	0.082
	V2C Nguyen et al. (2019)	0.259	0.119	0.092	0.087
	Watch-and-Act Yang et al. (2023)	0.310	0.157	0.134	0.107

Table 6: Video-to-command generation performance (BLEU scores) on novel object sets not seen during training ( bold and underline are best and second best, respectively).

Figure 12: Comparison of DRL algorithm performance across four manipulation actions.

Action	1 cm	2 cm	3 cm	4 cm
Reach	100	100	100	100
Pick	70	80	100	90
Move	90	80	70	90
Put	90	90	70	90

Table 7: Task success rates (%) of our method under varying positional error thresholds.

Figure 13: Box plots comparing average rewards for four manipulation actions.

Figure 14: Box plots comparing average rewards under 4 different positional errors as the success threshold for four manipulation actions.

Figure 15: Real experiment with UF850 manipulator

Action	Success rate	Execution time (s)	Position accuracy (mm)
Reach	100	30 $\pm$ 3.8	25 $\pm$ 6
Pick	80	33 $\pm$ 5.4	20 $\pm$ 4

Table 8: Real-world experiments over 10 trials per action.

Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation

Abstract

Figures & tables

Explore similar work

LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition

KnowDemo: Knowledge-Guided Robot Demonstration Generation from Human Videos

Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video