Human-Object Interaction

Also known as HOI

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 75

All topics

Do as I Do: Dexterous Manipulation Data from Everyday Human Videos
Jun 17, 2026Bhawna Paliwal, Haritheja Etukuru, William Liang +3Robotic Data Collection Cross-Embodiment Robot Learning
UC Berkeley
MOCHI: Motion Enhancement of Collaborative Human-object Interactions
Jun 16, 2026Jiye Lee, Yonghun Choi, Jungdam WonHOI Generation Human Motion Generation
Department of Computer Science and Engineering, Seoul National University, South Korea
Improving and Evaluating Hand-Object Interaction Detection
Jun 16, 2026Ahmad Darkhalil, Dima Damen, David FouheyDetection Transformer Human-Object Interaction
School of Computer Science University of Bristol, Bristol, UK · Computer Science and Electrical and Computer Engineering New York University, NY, US
GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training
Jun 11, 2026Zhihai Bi, Qiang Zhang, Guoyang Zhao +8Contact-Rich Robotic Manipulation Robot Motion Generation
The Hong Kong University of Science and Technology (Guangzhou) · Artificial General Intelligence Institute, University of Science and Technology of China · The University of Hong Kong +1
Hand-centric Human-to-Robot Trajectory Transfer from Video Demonstrations via Open-World Contact Localization
Jun 9, 2026Yitian Shi, Di Wen, Zhengqi Han +6Contact-Rich Robotic Manipulation Cross-Embodiment Robot Learning
Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
SynManDex: Synthesizing Human-like Dexterous Grasps from Synthetic Human Pre-Grasps
Jun 8, 2026Yanming Shao, Zanxin Chen, Wenwei Lin +5Robotic Grasping Dexterous Manipulation
1Shanghai AI Lab · 2Shanghai Jiaotong University · 3Shenzhen University +3
VAIC: Vision-Guided Humanoid Agile Object Interaction Control via Decoupled Commands
Jun 8, 2026Dongting Li, Qianyang Wu, Xingyu Chen +9Humanoid Robot Control Robotic Manipulation
Tsinghua University · Xiaomi Robotics Lab · HKUST(Guangzhou)
LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
Jun 3, 2026Shahrzad Esmat, Chaunte W. Lacewell, Sameh Gobriel +2Large Language Model-Guided Optimization Hyperparameter Optimization
Iowa State University · Intel Corporation
Recovering Physically Plausible Human-Object Interactions from Monocular Videos
Jun 3, 2026Dingbang Huang, Etienne Vouga, Qixing Huang +13D Reconstruction Monocular 3D Reconstruction
University of Texas at Austin
GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors
Jun 3, 2026Tianyi Xie, Haotian Zhang, Jinhyung Park +17HOI Generation Humanoid Loco-Manipulation
NVIDIA · UCLA
See Better, Foresee Better, Act Wiser: Physically Grounded Proactive Modeling and Decision Making
Jun 2, 2026Honghui Zhang, Anna Min, Chenmeinian Guo +9Counterfactual Prediction Video Reasoning
Sichuan University · New York University · University of Alberta +6
Diagnosis of Human Object Interaction Detectors for Real World Educational Applications
Jun 1, 2026Divya Mereddy, Ashwin Tudur Sadashiva, Marcos Quinones-Grueiro +1Domain Adaptation AI in Education
Vanderbilt University Nashville, TN, USA
Dex2HOI: Dexterous Bimanual Two-Object Interaction Generation
May 28, 2026Chrysa Pratikaki, Pablo Ruiz-Ponce, Jiankang Deng +2HOI Generation Human Motion Generation
Imperial College London, UK · University of Alicante, Spain
PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions
May 28, 2026Omer Benishu, Gal Fiebelman, Sagie BenaimHOI Generation Human Motion Generation
Hebrew University of Jerusalem
HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
May 24, 2026Zhi Wang, Botao He, Kelin Yu +4Zero-Shot Learning Zero-Shot Robotic Manipulation
University of Maryland
Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors
May 21, 2026Jiahe Chen, ZiRui Wang, Feiyu Jia +7Robot Motion Generation Robotic Manipulation
Zhejiang University · Shanghai AI Laboratory · The Chinese University of Hong Kong
Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction
May 20, 2026Hao Xu, Yilin Liu, Yinqiao Wang +23D Hand Pose Estimation 3D Reconstruction
The Chinese University of Hong Kong, China · University College London, United Kingdom · Adobe Research, United Kingdom
RHINO: Reconstructing Human Interactions with Novel Objects from Monocular Videos
May 16, 2026Lixin Xue, Chengwei Zheng, Georgios Paschalidis +43D Reconstruction Monocular 3D Reconstruction
1ETH Zürich · 2The University of Tokyo · University of Amsterdam +1
Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos
May 14, 2026Yubo Zhao, Yujin Chai, Yunao Dong +43D Reconstruction Monocular 3D Reconstruction
The Hong Kong University of Science and Technology · Tencent IEG
Dynamic Full-body Motion Agent with Object Interaction via Blending Pre-trained Modular Controllers
May 12, 2026Sanghyeok Nam, Byoungjun Kim, Daehyung Park +1HOI Generation Human Motion Generation
KAIST
PhysHanDI: Physics-Based Reconstruction of Hand-Deformable Object Interactions
May 10, 2026Jihyun Lee, Changmin Lee, Donghwan Kim +13D Reconstruction Deformable Shape Reconstruction
School of Computing, KAIST, Daejeon, South Korea.
DexSynRefine: Synthesizing and Refining Human-Object Interaction Motion for Physically Feasible Dexterous Robot Actions
May 7, 2026Hyesung Lee, Hyunwoo Jung, Si-Hwan Heo +1HOI Generation Dexterous Manipulation
Korea Institute of Science and Technology · KAIST · Hanyang University
MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation
May 7, 2026Hao Wang, Shiqi Wang, Qi LiuHOI Generation Human Motion Generation
School of Future Technology, South China University of Technology, Guangzhou, Guangdong, China.
ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detection
May 6, 2026Minh Anh Nguyen, Quang Huy Tran, Bao Ngoc Le +3Counterfactual Learning Human-Object Interaction
Phenikaa University
Active Contact Sensing for Robust Robot-to-Human Object Handover
May 6, 2026Linfeng Li, Lin Shao, David HsuActive Sensing Robotic Manipulation
Deco: Extending Cherished Physical Objects into AI Companion Agents through Dual Embodiment
May 5, 2026Zhihan Jiang, Mengyuan Millie Wu, Ruishi Zou +14Human-AI Interaction Human-Centered AI
Columbia University, United States · University of Notre Dame, United States
MHPR: Multidimensional Human Perception and Reasoning Benchmark for Large Vision-Languate Models
May 5, 2026Kangkang Wang, Qinting Jiang, Wanping Zhang +2VLM Evaluation Vision-Language Models
AI Compute Group, Baidu · Tsinghua University
IMPACT-HOI: Supervisory Control for Onset-Anchored Partial HOI Event Construction
May 3, 2026Haoshen Zhang, Di Wen, Kunyu Peng +12Human-in-the-Loop Annotation Egocentric Video Understanding
Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany · INSAIT, Sofia University “St. Kliment Ohridski”, 1784 Sofia, Bulgaria · ETH Zurich, 8092 Zurich, Switzerland +1
InterPhys: Physics-aware Human Motion Synthesis in a Dynamic Scene
May 1, 2026Chaoyue Xing, Wei Mao, Miaomiao LiuHOI Generation Human Motion Generation
Australian National University, Canberra, Australia
HOI-aware Adaptive Network for Weakly-supervised Action Segmentation
Apr 29, 2026Runzhong Zhang, Suchen Wang, Yueqi Duan +3Temporal Action Segmentation Human Activity Recognition
1Nanyang Technological University · 2Tsinghua University · 3Beijing Jiaotong University

Jun 17, 2026, cs.ROJ/K move · Enter open · S save

Do as I Do: Dexterous Manipulation Data from Everyday Human Videos

Bhawna Paliwal, Haritheja Etukuru, William Liang, Pieter Abbeel+2

UC Berkeley

Robotic Data Collection Cross-Embodiment Robot Learning Synthetic Data Generation for Robotics Robotic Manipulation Human-Object Interaction

Open paper AskPDF arXiv

How can we scalably generate data for robotic manipulation, especially on human-like platforms such as dexterous multi-fingered hands? Learning from human videos has recently emerged as a likely answer to this question. However, difficulties in estimating hand-object interaction and crossing the human-to-robot embodiment gap have hindered the adoption of abundant monocular RGB-only human videos as the primary source of robot manipulation data. In this work, we present DO AS I DO, an algorithm to reconstruct and retarget monocular RGB human videos to multi-fingered dexterous robotic hands. DO AS I DO reconstructs hand-object interactions from various egocentric and exocentric in-the-wild video sources. The algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world, yielding robot-complete manipulation data from disparate human videos. Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB videos, as we show in experiments on datasets with ground truths and on a dataset of video clips collected online. Our experiments enable us to propose an efficacy playbook for practitioners collecting human data for manipulation.

Momentum

Related topics

Latest papers 75