Organizations: School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · Department of Robotics, Hanyang University, Ansan, 15588, Gyeonggi, Korea.
Learning from Demonstration (LfD) offers a promising paradigm for robot skill acquisition. Recent approaches attempt to extract manipulation commands directly from video demonstrations, yet face two critical challenges: (1) general video captioning models prioritize global scene features over task-relevant objects, producing descriptions unsuitable for precise robotic execution, and (2) end-to-end architectures coupling visual understanding with policy learning require extensive paired datasets and struggle to generalize across objects and scenarios. To address these limitations, we propose a novel ``Human-to-Robot'' imitation learning pipeline that enables robots to acquire manipulation skills directly from unstructured video demonstrations, inspired by the human ability to learn by watching and imitating. Our key innovation is a modular framework that decouples the learning process into two distinct stages: (1) Video Understanding, which combines Temporal Shift Modules (TSM) with Vision-Language Models (VLMs) to extract actions and identify interacted objects, and (2) Robot Imitation, which employs TD3-based deep reinforcement learning to execute the demonstrated manipulations. We validated our approach in PyBullet simulation environments with a UR5e manipulator and in a real-world experiment with a UF850 manipulator across four fundamental actions: reach, pick, move, and put. For video understanding, our method achieves 89.97% action classification accuracy and BLEU-4 scores of 0.351 on standard objects and 0.265 on novel objects, representing improvements of 76.4% and 128.4% over the best baseline, respectively. For robot manipulation, our framework achieves an average success rate of 87.5% across all actions, with 100% success on reaching tasks and up to 90% on complex pick-and-place operations. The project website is available at https://thanhnguyencanh.github.io/LfD4hri.
Figures & tables
Figure 1: An illustration of Robot Imitation Learning from Human Demonstration.
Figure 2: The proposed Video Understanding architecture consists of two parallel branches: (a) Interacted Object Understanding Module and (b) Action Understanding Module. Initially, the raw input frames F={f1,f2,…,fn} are downsampled to F~={f~1,f~2,…,f~m} (where n>m ) to optimize runtime and match the training data frame rate in Action Understanding Module. (a) The Interacted Object Understanding Module processes F~ to extract a subset of keyframes F^={f^1,f^2,…,f^k} , (where m>k ). These frames are analyzed by our Object Selection algorithm and Vision-Language Models (VLMs) to identify the specific objects involved in the interaction accurately. (b) The Action Understanding Module is implemented based on a CNN architecture with a ResNet-50 He et al. (2016) backbone with Temporal Shift Modules (TSM) that shift feature channels along the temporal dimension to capture fine-grained motion dynamics for action classification.
Figure 3: Pipeline of the proposed Object Selection algorithm. Cropped object regions are tracked across frames to generate trajectory data, which is used to classify objects into Pickable-object and Placeable-object sets based on motion patterns. Each candidate object then undergoes Blur Detection and Overlap Detection to select the highest-quality object instances for subsequent Vision-Language Model processing.
Figure 4: Visualization of Interacted object trajectory tracking across three intersection scenarios. Left: RGB frames showing human hand-object interactions with objects. Right: Corresponding 2D trajectory plots of the tracked objects’ centroid positions over time, where the characteristic parabolic or oscillatory patterns indicate pickable objects, enabling automatic classification of object functional roles (pickable or placeable).
Figure 5: The architecture of our DRL system for robot imitation. processes the environment observation O that consists of agent’s proprioceptive, exteroceptive, relational, and historical data such as: joint angle jt , joint velocity jt′ , object orientation Ro , object coordination and target po , pg , end effector status pe , we also provide relative observations pˉeo , pˉog with past joint angle at−1 , provided by the Video Understanding framework and outputs the location of the object referenced by it .
rti=δc(w3exp(−λ3∥po−pg∥)+w4exp(−λ4∥po−pg∥)),δc=1 if contact is established,λ3>>λ4
object movement toward the target
(2.0,3.0)
rta=w5exp(λ5pˉo′⋅pˉog−1)[pˉo′⋅pˉog>0]
object motion aligned with goal direction
0.8
cc=⎩⎨⎧3,1.5,0,if colliding with ground or itselfif colliding with objectsotherwise.
undesired collisions
1.0
cs=2
reaching the max step limit
1.0
ce=w6(∣ψe−π∣−∣θe∣)
end-effector pose inclination limitation
1.0
Table 2: Rewards and penalties formula used for training, along with the initial and final weights. ∗ Notation: pe is end-effector position; λ_ is scale factor; θe,θo,ψe,ψo are roll and pitch of end-effector and object respectively; p_x,p_y,p_z are position in x, y, z-axis respectively; θoc is a constraint; wc,wxc,wyc are position constraint of workspace in xyz coordinate, only x-axis, and only y-axis respectively.
Figure 6: Distribution of training videos across action categories in the test set
Figure 7: Object categories used for evaluating video understanding generalization. (Left) Standard object set: 12 object categories present in the training dataset, including common household items. (Right) Novel object set: 9 previously unseen object categories used exclusively for testing
Figure 8: Qualitative results of reach action. Left: Human demonstration videos showing reaching motion toward various objects. Right: Corresponding simulated UR5 robot executions, where the robot successfully navigates its end-effector to the target object position across diverse scene configurations with multiple distractor objects.
Figure 9: Qualitative results of the pick action. Left: Human demonstration videos showing grasping motions for objects of varying colors and shapes. Right: Corresponding simulated executions using the suction gripper, with the robot correctly identifying and picking the target object among multiple distractors.
Figure 10: Qualitative results of the move action. Left: Human demonstration videos showing object transportation from one location to another. Right: Corresponding simulated executions, where the robot successfully transports grasped objects along trajectories toward the designated goal positions while maintaining a stable grasp throughout the motion.
Figure 11: Qualitative results of the put action. Left: Human demonstration videos showing object placement into containers (bowls, plates). Right: Corresponding simulated execution demonstrating precise placement of objects at target locations, completing the full pick-and-place manipulation sequence learned from video demonstration.
Hyperparameter
Value
Learning rate (actor)
4×10−4
Learning rate (critic)
4×10−4
Discount factor γ
0.99
Soft update rate ρ
0.005
Replay buffer size
106
Batch size
256
Table 3: DRL training hyperparameters.
Test Set
Backbone
Top-1 Acc.
Top-5 Acc.
Test set 1
ResNet-34
82.79
93.50
ResNet-50
86.79
94.55
ResNet-101
89.97
96.82
Test set 2
ResNet-34
63.70
94.81
ResNet-50
68.15
94.81
ResNet-101
71.11
95.56
Table 4: Action classification accuracy (%) of the Temporal Shift Module (TSM) with varying backbone depths. Test Set 1: standard benchmark split; Test Set 2: self-recorded clips for real-world evaluation. All models use 8-frame sampling with 8-clips × 3-crops augmentation using Modified Something-Something V2 during inference.
Backbone
Method
BLEU-1
BLEU-2
BLEU-3
BLEU-4
InceptionV3
Video2Command Nguyen et al. (2018)
0.323
0.196
0.162
0.154
V2C Nguyen et al. (2019)
0.355
0.227
0.201
0.164
Watch-and-Act Yang et al. (2023)
0.399
0.287
0.263
0.199
ResNet-50
Video2Command Nguyen et al. (2018)
0.312
0.196
0.173
0.150
V2C Nguyen et al. (2019)
0.357
0.231
0.201
0.153
Watch-and-Act Yang et al. (2023)
0.394
0.271
0.248
0.187
Table 5: Video-to-command generation performance (BLEU scores) on standard object sets ( bold and underline are best and second best, respectively).
Backbone
Method
BLEU-1
BLEU-2
BLEU-3
BLEU-4
InceptionV3
Video2Command Nguyen et al. (2018)
0.244
0.132
0.075
0.062
V2C Nguyen et al. (2019)
0.279
0.117
0.093
0.087
Watch-and-Act Yang et al. (2023)
0.293
0.141
0.133
0.116
ResNet-50
Video2Command Nguyen et al. (2018)
0.305
0.150
0.090
0.082
V2C Nguyen et al. (2019)
0.259
0.119
0.092
0.087
Watch-and-Act Yang et al. (2023)
0.310
0.157
0.134
0.107
Table 6: Video-to-command generation performance (BLEU scores) on novel object sets not seen during training ( bold and underline are best and second best, respectively).
Figure 12: Comparison of DRL algorithm performance across four manipulation actions.
Action
1 cm
2 cm
3 cm
4 cm
Reach
100
100
100
100
Pick
70
80
100
90
Move
90
80
70
90
Put
90
90
70
90
Table 7: Task success rates (%) of our method under varying positional error thresholds.
Figure 13: Box plots comparing average rewards for four manipulation actions.
Figure 14: Box plots comparing average rewards under 4 different positional errors as the success threshold for four manipulation actions.
Figure 15: Real experiment with UF850 manipulator
Action
Success rate
Execution time (s)
Position accuracy (mm)
Reach
100
30 ± 3.8
25 ± 6
Pick
80
33 ± 5.4
20 ± 4
Table 8: Real-world experiments over 10 trials per action.
The most widely-adopted robot learning pipelines today learn skills from robot demonstrations or structured human data, which are expensive to collect and tied to specific embodiments. In contrast, unstructured human videos provide a scalable alternative. They contain diverse manipulation demonstrations across objects, scenes, and strategies, but are not directly connected to robot action. We propose LUCID, a two-stage framework that learns task intent from unstructured human videos drawn from internet-scale datasets and learns robot control in massively-parallel simulation. The intent model predicts short-horizon intent (what should happen next in the scene) from the current observation in closed loop. An embodiment-specific sensorimotor policy converts this intent into robot actions. The intent interface is shared across controllers, so the same intent model can be applied to different embodiments, from our primary dexterous hand to a parallel-jaw gripper. We evaluate LUCID on five real-world manipulation tasks: stirring, wiping, and binning supervised by only internet video, with zero-shot transfer to novel scenes and object instances; and push-T and cable routing supervised by 1 hr each of self-collected smartphone video. Project page: https://lucid-robot.github.io/.
Harsh Gupta, Guanya Shi, Wenzhen Yuan
University of Illinois Urbana-Champaign · Carnegie Mellon University
Learning robot manipulation policies typically requires substantial demonstration data, which are costly to collect on real robots. Recent methods generate robot demonstrations from human videos by adapting recovered motion and validating the resulting trajectories in simulation. However, methods centered on motion-reference adaptation can limit behavioral diversity by retaining the demonstrated contact strategies and subtask orders, while insufficient understanding of task requirements and scene relations can reduce demonstration generation efficiency by generating invalid candidates. To address these limitations, we propose KnowDemo, a framework that uses structured manipulation knowledge from human videos to generate diverse robot demonstrations for a target workspace. To distinguish task requirements from demonstration-specific choices, we develop a knowledge extraction and reasoning module based on a vision-language model (VLM) that associates object and action descriptions with inferred task conditions, demonstration references, and permissible execution variations. To translate this knowledge into executable demonstrations, we resolve the descriptions against target-scene entities and geometry to guide candidate generation and screening before motion planning and simulation. The resulting demonstrations exhibit multimodal behavior through alternative contact strategies and valid subtask orders, with structured execution labels. Experiments demonstrate additional verified execution modes beyond a reference-only configuration and improved candidate planning success through task-guided grasp sampling. To validate the generated data for policy learning, we fine-tune the pretrained π0.5 model on simulation data, achieving sim-to-real transfer across three tasks. Project page: https://zhiyuan-gao.github.io/knowdemo/
Zhiyuan Gao, Yanxiang Zhan, Mohammad Khoshnazar +2
University of Bremen · Robotics Institute Germany (RIG)
Human manipulation videos are a convenient and intuitive source for robot learning. However, directly transferring human dexterity to robots remains challenging due to perception errors and embodiment gap. To address this, we introduce Video2Sim2Real, a full-stack framework for autonomous skill acquisition from a single human manipulation video. Our framework first uses off-the-shelf foundation models to reconstruct a simulator-ready digital twin and extract robot and object motion priors. Rather than treating the extracted robot motion as a reliable reference throughout execution, our key idea is to recover and leverage the most fundamental sources of supervision from the demonstrated skill: We identify object-centric keyframes to optimize the corresponding robot configurations using object information from the simulator, and use these configurations as anchors that refine the robot motion such that it ultimately has the desired impact on the environment. To bridge the remaining sim-to-real gap, we introduce a sim-to-real strategy that decouples robustness to noisy and incomplete perception from variations in hand-object interaction dynamics. Specifically, we learn to recalibrate robot configurations from noisy real-world point clouds via IL, and leverage residual RL to perform local finger-level adaptations to ensure for robust and effective interactions. Finally, a collision-aware motion planning module enables spatial generalization to novel object configurations. Across several everyday manipulation tasks, Video2Sim2Real improves simulated task success, safety, and trajectory coherence over numerous baselines, and achieves better sim-to-real transfer than existing techniques. These results demonstrate a promising path toward autonomous dexterous skill acquisition from human videos.
Yunhai Han, Jianuo Qiu, Linhao Bai +14
1Georgia Institute of Technology · University of Pennsylvania · 3Toyota Research Institute +1