Hand-Object Interaction Reconstruction

Latest papers

17 papers5 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

EgoMatrix: Physically Faithful Dexterous Robot Demonstrations from Egocentric Human Videos
Oct 13, 2026Ziheng Wang, Yanming Shao, Ting Mao +4Hand-Object Interaction Reconstruction Robotic Data Collection
Shanghai Jiao Tong University · The University of Hong Kong · Zhejiang University +1
Hand2Robot: Object-Centric 4D Hand Reconstruction from Human Demonstrations for Robotics
Oct 11, 2026Zetao Du, Weiqing Wang, Hanbing Zhang +43D Hand Reconstruction Hand-Object Interaction Reconstruction
ShanghaiTech University · InstAdapt · Institute of Automation, Chinese Academy of Sciences
OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video
Oct 7, 2026Ting Mao, Yanming Shao, Ziheng Wang +4Hand-Object Interaction Hand-Object Interaction Reconstruction
Zhejiang University · The University of Hong Kong · Shanghai Jiao Tong University +1
4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction
Oct 6, 2026Shiqi Li, Sean Cho, Yijie Li +3Physics-Informed Generative Modeling 4D Reconstruction
Texas A&M University · NVIDIA
DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction
Sep 29, 2026Wenliang Guo, Zhanbo Huang, Yu Kong3D Reconstruction Monocular 3D Reconstruction
Michigan State University
Track, Articulate, Act: Generating Articulation from Casual Human Videos
Sep 16, 2026Jiaming Zhang, Homanga BharadhwajArticulated Object Reconstruction Articulated Object Manipulation
Department of Computer Science, Johns Hopkins University
GOLF: Global Observation with Local Focus for Calibration-Aware Stereo Interaction Field Estimation
Sep 8, 2026Minqiang Zou, Riqiang Jin, Zhi Lv +7Egocentric Video Understanding Hand-Object Interaction Reconstruction
JIIOV Technology
GraspHOI: Full-Body 3D Human-Object Reconstruction with Finger-Level Grasps from a Single In-the-Wild Image
Aug 28, 2026Semin Kim, Haechan Shin, Jongyoo Kim3D Hand Pose Estimation 3D Reconstruction
Yonsei University
C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video
Aug 7, 2026Jie Ren, Zhehao Jiang, Yinhong Yang +9Contact-Rich Robotic Manipulation 3D Reconstruction
1Nanjing University, Nanjing, China. · 2China Mobile Research Institute, Beijing, China.
EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning
Jun 16, 2026Gaotian Wang, Kejia Ren, Andrew Morgan +4Robotic Data Collection Hand-Object Interaction Reconstruction
Rice University · Robotics and AI Institute
Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction
May 20, 2026Hao Xu, Yilin Liu, Yinqiao Wang +2Human-Object Interaction 3D Reconstruction
The Chinese University of Hong Kong, China · University College London, United Kingdom · Adobe Research, United Kingdom
PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions
May 10, 2026Jihyun Lee, Changmin Lee, Donghwan Kim +13D Reconstruction Hand-Object Interaction Reconstruction
School of Computing, KAIST, Daejeon, South Korea.
Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions
Apr 14, 2026Ayce Idil Aytekin, Xu Chen, Zhengyang Shen +4Hand-Object Interaction Reconstruction 3D Reconstruction
MPI for Informatics, SIC & VIA Research Center, Germany · Google, Switzerland · Google, USA +1
ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions
Mar 26, 2026Zikai Wang, Zhilu Zhang, Yiqing Wang +2Hand-Object Interaction Reconstruction 4D Reconstruction
Harbin Institute of Technology · Shanghai Jiao Tong University
GHOST: Fast Category-agnostic Hand-Object Interaction Reconstruction from RGB Videos using Gaussian Splatting
Mar 19, 2026Ahmed Tawfik Aboukhadra, Marcel Rogge, Nadia Robertini +4Hand-Object Interaction Reconstruction Hand-Object Interaction
RPTU · DFKI-AV Kaiserslautern · NUST-SEECS Pakistan +1
WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos
Feb 25, 2026Yufei Ye, Jiaman Li, Ryan Rong +1Hand-Object Interaction Reconstruction Hand-Object Interaction
Stanford University · Amazon FAR (Frontier AI & Robotics)
Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction
Nov 18, 2025Hao Tian, Chenyangguang Zhang, Rui Liu +23D Gaussian Splatting 3D Reconstruction
Chengdu Institute of Computer Applications, Chinese Academy of Sciences, Chengdu, China · PetroChina (Beijing) Digital Intelligence Research Institute Co., Ltd., Beijing, China · University of Chinese Academy of Sciences, Beijing, China +2

Oct 13, 2026J/K move · Enter open · S save

EgoMatrix: Physically Faithful Dexterous Robot Demonstrations from Egocentric Human Videos

Ziheng Wang, Yanming Shao, Ting Mao, Kaihan Chen+3

Shanghai Jiao Tong University · The University of Hong Kong · Zhejiang University+1

Hand-Object Interaction Reconstruction Robotic Data Collection Synthetic Data Generation for Robotics Bimanual Robotic Manipulation Egocentric Video Understanding 3D Reconstruction

Open paper AskPDF arXiv

Egocentric human videos offer a scalable alternative to robot teleoperation for collecting dexterous manipulation data. However, two challenges limit their use as reliable training data: accurate hand-object interaction (HOI) reconstruction from monocular RGB and high-quality robot demonstrations, especially visual observations. We present EgoMatrix, an end-to-end data engine that converts monocular egocentric human videos into bimanual dexterous trajectories and paired robot observation videos, with the trajectories directly replayable on real robots. It reconstructs HOI with emphasis on relative geometry and contact rather than exact metric scene recovery, then uses retargeting and reinforcement learning (RL) to reproduce the demonstrated tasks in simulation. We jointly render robot hands and objects from these simulated trajectories to produce physically faithful observations synchronized with state and control records. Rendering the same trajectories in diverse, task-compatible backgrounds expands paired training data while preserving robot-object motion and contact. Across TACO, OakInk2, and self-collected data, EgoMatrix achieves 54.84% end-to-end success in simulation versus 14.52% for the strongest baseline. Reconstruction reaches state-of-the-art performance on most metrics of the two public benchmarks. Qualitative results show more faithful hand-object interactions than prior methods, while feature-space analysis shows closer alignment with real-robot observations than GPT-6 Astra. Our project page is available at https://spikew726.github.io/EgoMatrix/.