Hand-Object Interaction

Latest papers

43 papers11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

PiHand: Latent Hand-Object Interaction Priors for Physics-Based Policy Learning
Oct 22, 2026Niklas Berndt, Xianghui Xie, Enes Duran +2Hand-Object Interaction Robotic Manipulation
University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany +1
Touch-JEPA: Tactile Representation Learning from Hand-Object Video with Limited Tactile Supervision
Oct 20, 2026Minseong Kweon, Qiao Feng, Yiming Huang +1Visuo-Tactile Representation Learning Hand-Object Interaction
University of Minnesota · University of Pennsylvania
Decoupling Action from Egocentric Observation for World Simulation
Oct 20, 2026Yue Ma, Pengjie Song, Xinyu Wang +10Action-Conditioned Video Generation Hand-Object Interaction
HKUST · Tsinghua University · Stanford University +2
Gaze2HOI: Gaze-Aware Intention Disambiguation for Egocentric Hand-Object Interaction Forecasting
Oct 20, 2026Chuanyang Ma, Chenye Wang, Yixuan Wang +6Hand-Object Interaction Human Motion Prediction
Beijing University of Posts and Telecommunications, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · North China Electric Power University, Beijing, China +2
DexOAK: Object-Aware Kinematic Retargeting
Oct 19, 2026Federica Spinola, Robert Katzschmann, Cordelia SchmidMotion Retargeting Human-to-Robot Motion Imitation
Inria and the Departement d’Informatique of Ecole Normale Supérieure, PSL University, France · Soft Robotics Lab, ETH Zürich, Switzerland
DenseHap: Dense Hand Haptic Estimation from Monocular Images
Oct 19, 2026Junhan Xue, Rolandos Alexandros PotamiasHand-Object Interaction Detection Hand-Object Interaction
Imperial College London
SituHOI: Scalable Capture of Scene-Situated Full-Body Human-Object Interaction
Oct 13, 2026Qifan Wang, Qihang Fang, Boao Shi +4Human-Object Interaction Hand-Object Interaction
The University of Hong Kong
OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting
Oct 9, 2026Solvin Sigurdson, Prasanna Sriganesh, Juan Borbon +3Motion Retargeting Robotic Manipulation
Multiply Labs, San Francisco, CA 94107, USA · Department of Control and Dynamical Systems, California Institute of Technology, Pasadena, CA 91125, USA · Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
Human2Dex: Dexterous Robot Learning from Wrist-Centric Human Demonstrations
Oct 9, 2026Jiachen Zhang, Yuanyuan Li, Shaqi LuoRobotic Manipulation Cross-Embodiment Robot Learning
Beijing Academy of Artificial Intelligence, Beijing, China.
OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video
Oct 7, 2026Ting Mao, Yanming Shao, Ziheng Wang +4Hand-Object Interaction Hand-Object Interaction Reconstruction
Zhejiang University · The University of Hong Kong · Shanghai Jiao Tong University +1
VolCo: Volumetric Contact for High-Fidelity Human Grasp Generation
Oct 7, 2026Zhuo Chen, Yihua Cheng, Aleš Leonardis +1Hand-Object Interaction HOI Generation
University of Birmingham · Beijing Institute of Technology
PatternDex: Learning Interaction Patterns to Guide Reinforcement Learning of Bimanual Dexterous Manipulation of Articulated Objects
Oct 3, 2026David Minkwan Kim, Runfa Blark Li, Beckham Po-Ju Lee +2Robotic Manipulation Bimanual Robotic Manipulation
University of California, San Diego
GOTT: Object-centric Dexterous Manipulation with a Reusable Cross-Embodiment Primitive
Oct 2, 2026Yulin Liu, Lai Wei, Yen-Jen Wang +4Dexterous Manipulation Contact-Rich Robotic Manipulation
Amazon FAR · UC San Diego · University of Chicago +1
EgoFound3R: End-to-End Egocentric Hand Reconstruction in World Space with Point-Wise Interaction Attributes
Oct 1, 2026Hongming Fu, Jingcheng Shi, Wenjia Wang +23D Hand Pose Estimation 3D Reconstruction
School of AI, Shanghai Jiao Tong University · Rutgers University · The University of Hong Kong +1
DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction
Sep 29, 2026Wenliang Guo, Zhanbo Huang, Yu Kong3D Reconstruction Monocular 3D Reconstruction
Michigan State University
Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy
Sep 23, 2026Tara Sadjadpour, Siming He, C. K. Wolfe +5Robot Imitation Learning Motion Retargeting
Department of Electrical Engineering and Computer Science at the University of California, Berkeley, CA, USA
Touch2Robot: Robot Touch in the Human Demonstration Loop
Sep 21, 2026Shengcheng Luo, Xiaoyang Cheng, Hong Ying +6Tactile Sensing for Robotic Manipulation Robotic Data Collection
ShanghaiTech University · Beijing Institute for General Artificial Intelligence (BIGAI) · Shanghai Jiao Tong University +1
Track, Articulate, Act: Generating Articulation from Casual Human Videos
Sep 16, 2026Jiaming Zhang, Homanga BharadhwajArticulated Object Reconstruction Articulated Object Manipulation
Department of Computer Science, Johns Hopkins University
Learning In-Hand Object Reaching to General 6D Poses
Sep 12, 2026Junxiao Lin, Tianyue Wu, Jie Yin +3In-Hand Manipulation Robotic Manipulation
Sharpa. · School of Computer Science, The University of Sydney, Australia. · School of Computing and Data Science, The University of Hong Kong, HKSAR.
Single-Query Person-Centric Bimanual Hand-Object Interaction Detection
Sep 10, 2026Jonghyun Kim, Junho Roh, Yubin Yoon +5Hand-Object Interaction Detection Human-Object Interaction
AI Lab, CTO Division, LG Electronics, South Korea
Assembling Two Parts in One Hand
Sep 9, 2026Liuao Pei, Tianyue Wu, Hui Zhang +2RL for Robotics Dexterous Manipulation
The University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · ETH Zurich +1
Joint-Conditioned Stereo Surface Reasoning for Interaction Field Estimation
Sep 7, 2026Yanlin Jin, Yifan Yang, Bowen Yang +1Hand-Object Interaction Robotic Manipulation
Ant Digital Technologies, Ant Group · Institute of Automation, Chinese Academy of Sciences (CASIA)
HOPE: Hand-Object Pressure Estimation from Monocular Videos
Aug 6, 2026Subin Jeon, Byungjun Kim, Hanbyul JooTactile Sensing Hand-Object Interaction
Seoul National University · RLWRLD
PhotoHOI: Synthesizing 3D Hand-Object Interactions from a Single RGB Photograph
Aug 3, 2026Zhenhao Zhang, Jiajun Zhang, Wei Min +1HOI Generation Dexterous Manipulation
Tsinghua University · Shadow AI
GraspGraphNet: Graph-Structured Multi-Embodiment Dexterous Grasp Generation
Jul 13, 2026Yeonseo Lee, Taeyeop Lee, Hyosup Shin +2Robotic Grasping Graph Neural Networks
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea.
JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generation
Jul 2, 2026Mingyeong Song, Jungbin Cho, Jisoo Kim +5HOI Generation Diffusion Models
Ewha Womans University, Seoul, Korea · Yonsei University, Seoul, Korea · Carnegie Mellon University, Pittsburgh PA, USA +1
Do as I Do: Dexterous Manipulation Data from Everyday Human Videos
Jun 17, 2026Bhawna Paliwal, Haritheja Etukuru, William Liang +3Robotic Data Collection Cross-Embodiment Robot Learning
UC Berkeley
Improving and Evaluating Hand-Object Interaction Detection
Jun 16, 2026Ahmad Darkhalil, Dima Damen, David FouheyDetection Transformer Hand-Object Interaction Detection
School of Computer Science University of Bristol, Bristol, UK · Computer Science and Electrical and Computer Engineering New York University, NY, US
TopoRetarget: Interaction-Preserving Retargeting for Dexterous Manipulation
Jun 15, 2026Jielin Wu, Shenzhe Yao, Guanqi He +6Robot Skill Learning Contact-Rich Robotic Manipulation
IIIS, Tsinghua University
RealDexUMI: A Wearable Universal Manipulation Interface for Dexterous Robot Learning
Jun 4, 2026Chaoyi Xu, Yixuan Jiang, Jiahui Huan +7Dexterous Manipulation In-Hand Manipulation
Peking University · BeingBeyond · Beihang University +2

Oct 22, 2026J/K move · Enter open · S save

PiHand: Latent Hand-Object Interaction Priors for Physics-Based Policy Learning

Niklas Berndt, Xianghui Xie, Enes Duran, Yixiao Wang+1

University of Tübingen, Germany · Tübingen AI Center, Germany · Max Planck Institute for Informatics, Saarland Informatics Campus, Germany+1

Hand-Object Interaction Robotic Manipulation Goal-Conditioned Robotic Manipulation Motion Representation Learning Robot Imitation Learning

Open paper AskPDF arXiv

Synthesizing physically grounded dexterous hand-object interactions is important for VR/AR, gaming, and robotics, yet remains challenging because subtle contact dynamics, diverse object geometries, and target trajectories must be jointly respected while maintaining natural grasps. Existing methods either track mocap trajectories with limited transfer to novel objects, or rely on hand-only generative priors that fail to capture coupled hand-object motion, sacrificing contact fidelity and sample efficiency. To tackle these challenges, we introduce PiHand, a Physically grounded Interaction-aware Hand motion synthesis framework. PiHand first trains expert policies that track mocap demonstrations in simulation, distills them into a unified hand-object interaction prior using a goal-conditioned VAE, and finally solve diverse dexterous manipulation tasks through bounded residual exploration in its latent space. With only a single reward and no complex curriculum training, this yields general tabletop manipulation policies that generalize to unseen objects and trajectories on GRAB and DexYCB. The same prior transfers to object catching and bimanual handover, two tasks that are rarely seen in its training data. Across all tasks, PiHand consistently outperforms baselines in all metrics. We will release the processed data and model checkpoints.