Robot Learning from Video

Latest papers

67 papers26 papers in the last four weeks, up 271% on the four weeks before. 0.2% of all new papers.

HOC2R: Hand-Object Contact Trajectories for Human-to-Robot Motion Transfer
Nov 6, 2026Hongkun Yu, Francisco M. López, Francisco CruzHand-Object Contact Estimation Robot Learning from Video
School of Computer Science and Engineering, UNSW Sydney, Australia
VREgo: Consumer VR Headsets for Learning Robot Manipulation from Egocentric Demonstrations
Nov 5, 2026Liuchuan YuRobotic Manipulation Robot Imitation Learning
Open-HumanWAM: An Open Recipe for Scaling Human and Robot Experience for Embodied Foundation Models
Nov 4, 2026Juncheng Ma, Jiaxin Li, Jiankai Tu +30Robot Foundation Models Robot Learning from Video
PKU, NUS, MIT, UCSB, NVIDIA
ELA: End-to-End Latent Action Policy Learning from Video Demonstrations
Nov 3, 2026Cheng-Chun Hsu, Peter StoneLatent Action Learning Robot Imitation Learning
Department of Computer Science, The University of Texas at Austin
Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On
Nov 3, 2026Seungyong Lee, Suhyeok Jang, Minyoung Kim +1Synthetic Data Generation for Robotics Robot Learning from Video
KAIST · RLWRLD · Korea University
FloMo: 3D Scene Flow as a World Action Model Intermediate for Learning from Human Video
Nov 3, 2026Jeremy Collins, Namra Patel, Ayush Agarwal +2Robot Learning from Video World Models for VLA Models
Georgia Institute of Technology
Future Representation Distillation for Imagination-free World Action Models
Nov 3, 2026Jaehyeong Jo, Sangwon Jang, Sihyun Yu +5World Action Models Efficient Inference for World Action Models
KAIST
Action Intent as a Bridge: Transferring Manipulation Knowledge from Human-Centric Videos to Robots
Nov 3, 2026Zhiqian Xia, Lianyang Ma, Haifeng Xia +3Robot Learning from Video Cross-Embodiment Robot Learning
Southeast University, Nanjing, China · Astribot, Shenzhen, China · Sun Yat-sen University, Shenzhen, China
D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models
Nov 2, 2026Marius Memmel, Shawn Yin, Urvi Rutia +9Robot Task Progress Estimation VLMs for Robotics
University of Washington · NVIDIA
Gnomon4D: Learning Where and When to Focus for Geometric Motion-Grounded World-Action Modeling
Nov 2, 2026Lingwei Dang, Yan Zhou, Juntong Li +10World Action Models Robot Learning from Video
South China University of Technology · Tsinghua University · KlingAI Research +3
OpenNWM: A Generalist Navigation World Model with Latent Action Pretraining
Nov 1, 2026Dujun Nie, Ruiqi Song, Zixuan Wang +6World Model Learning Robot Navigation
CASIA · UCAS · Tongji University +3
Nexus-VLA: Bridging Human and Robot Experiences via a Shared Hierarchical Semantic Latent Space
Oct 30, 2026Biaoliang Guan, Tingyu Yuan, Wen Ye +5Robot Policy Learning Cross-Embodiment Robot Learning
XJTU · CASIA · UCAS +1
HandMimicX: Learning Human-Like Dexterous Manipulation via an Object-Centric Policy from Egocentric Videos
Oct 30, 2026Zihan Gao, Yufei Liu, Yuhong Zhang +7Robotic Manipulation Robot Learning from Video
Tsinghua University · TsingTec · MeiTuan +2
WorldBridge: Bridging Video and Action for Task Generalization in World Action Models
Oct 29, 2026Jun Cen, Yubo Guo, Hao Li +9World Models for Robotics Robot Policy Generalization
WorldBridge Team, Joy Future Academy
ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning
Oct 27, 2026Chaehyun Kim, Chi-Pin Huang, Jisoo Kim +4Reasoning for VLA Models Robot Learning from Video
NVIDIA · KAIST
Ego-CEVR: Learning Cross-Embodiment Visual Representations from Human Egocentric Videos
Oct 27, 2026Keru Zhou, Zhirui Fang, Xinrun Xu +19Visual Representation Learning Robot Learning from Video
Maniformer · Tsinghua University · Nanyang Technological University +4
TAPESTRY: A Single Backbone for Video Generation and Robot Control
Oct 27, 2026Jeremy A. Collins, Seungjae Lee, Rachanon Wachakorn +4Vision-Based Robot Control Diffusion Policy
Georgia Institute of Technology. · Toyota Research Institute. · University of Maryland.
Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction
Oct 27, 2026Taeyoung Kim, Junyoung Sung, Heeseung Kwon +2World Action Models Vision-Based Robot Control
KAIST · RLWRLD
RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions
Oct 25, 2026Haoyu Zhao, Xingyue Zhao, Zhian Su +11VLA Model Training Robot Foundation Models
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +3
HuGe: Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience
Oct 25, 2026Yang Liu, Jiacheng Liu, Hengtao Li +14VLA Model Training Vision-Language-Action Models
Westlake University · Zhejiang University · Shanghai Innovation Institute +7
X-World: Transferring Video Priors to Variable-View Robot Control
Oct 25, 2026Chun Gu, Xiaofei Wei, Yiming Chen +2Robotic Control Vision-Based Robot Control
School of Data Science, Fudan University · Shanghai Innovation Institute · Logos Intelligence Lab
EXE: Egocentric Cross-Embodiment Pretraining via Factorized Latent Actions for VLA Models
Oct 24, 2026Qingyuan Cai, Bo Zhang, Yongzhen Huang +1Cross-Embodiment Robot Learning Robot Learning from Video
ROBOTERA. · Beijing Normal University. · Tsinghua University.
ViT-H: Generative Video-To-Humanoid Control with Motion Tracking Feedback
Oct 23, 2026Zimo He, Tianyi Bian, Zizhong Yang +7Robot Learning from Video Human-to-Robot Motion Imitation
School of Computer Science, Peking University · Beijing Institute for General Artificial Intelligence · School of Psychological and Cognitive Sciences, Peking University +6
EgoForesight: Effectively Scaling Policy Learning with Noisy Egocentric Actions
Oct 22, 2026Chubin Zhang, Changyuan Wang, Yirui Wang +7Robot Imitation Learning Robot Learning from Video
Tsinghua University · JD Future Academy · Loopeva +2
Self-Distilling Future Video Representations for World Action Models
Oct 21, 2026Huiwon Jang, Suhyeok Jang, Hyeonmo Kang +3World Action Models Robot Learning from Video
KAIST · RLWRLD
Real2Sem2Real: 3D Semantic Augmentations for Robot Learning from a Single Human RGB Demonstration
Oct 20, 2026Matthew Strong, Ankush K. Dhawan, Julien Poffet +3Robot Learning from Video Synthetic Data Generation for Robotics
Department of Computer Science, Stanford University, Stanford, CA, USA · Department of Mechanical Engineering, Stanford University, Stanford, CA, USA · D-MAVT, ETH Zurich, Switzerland; work done while visiting Stanford University
Multi-View Latent Action Pretraining for Cross-Embodiment Robot Learning from Videos
Oct 20, 2026Oliver Hausdörfer, Felix Minzenmay, Mani Majd +3Robot Learning from Video Latent Action Learning
Learning Systems and Robotics Lab, Technical University of Munich; and Munich Institute of Robotics and Machine Intelligence · University of Toronto
A Head Camera Is (Almost) All You Need: An Electronics-Free UMI Glove Labelled and Observed by One Moving Camera
Oct 20, 2026Yosub Shin, Yuan Liu, Igor MolybogRobotic Data Collection Data-Efficient Robot Learning
University of Hawaii at Manoa · Independent
Ropedia W0: Scaling Loco-Manipulation World Models across Embodiments with Human Experience
Oct 19, 2026Tianqi Liu, Hanfei Zhao, Qifeng Liang +12Robotic Loco-Manipulation Embodied World Models
Nanyang Technological University · Ropedia · Duke University +2
ScaleNav: Scalable Internet Video Pretraining for Language-Conditioned Local Navigation
Oct 19, 2026Yihui Yang, Yuanhong Yu, Jiawei Wang +7Vision-Language Navigation Robot Learning from Video
Zhejiang University · Li Auto Inc. · Shenzhen University

Nov 6, 2026J/K move · Enter open · S save

HOC2R: Hand-Object Contact Trajectories for Human-to-Robot Motion Transfer

Hongkun Yu, Francisco M. López, Francisco Cruz

School of Computer Science and Engineering, UNSW Sydney, Australia

Hand-Object Contact Estimation Robot Learning from Video Hand-Object Interaction Detection Hand-Object Interaction Human-to-Robot Motion Imitation Robotic Manipulation

Open paper AskPDF arXiv

Human videos contain abundant object-manipulation demonstrations, but they do not directly provide motion cues that a robot can follow. We present HOC2R, a Hand-Object Contact trajectory extractor for Robot motion that combines frozen DINOv3 and SAM 2.1 features with learned heads for hand-object association and contact localization. HOC2R supports two modes: count-aware automation that detects, counts, and localizes visible contacts in untrimmed videos, and fixed-two localization for two-finger grippers when contact frames are known. On a 30-task point-annotated benchmark, count-aware automation improves contact-count accuracy from 58.1% to 71.4% and reduces mean localization distance from 19.81 to 10.46 pixels relative to an adapted VRB2 baseline. On seven unseen RGB-D rollouts, a five-task fixed-two checkpoint reduces mean localization distance from 67.67 to 27.02 pixels. Across six robot evaluation sequences, the 30-task fixed-two checkpoint produces four feasible trajectories and succeeds on three tasks, compared with two feasible trajectories and one successful task for each adapted baseline. HOC2R is competitive in the evaluated single-hand scenes, robust to the hand-color shift, and strongest in the evaluated multi-hand scenes. These results support HOC2R as a practical pseudo-labeling method for extracting robot-usable contact trajectories from human demonstrations.