We present OCRA, an Object-Centric framework for video-based human-to-Robot Action transfer that learns directly from human demonstration videos to enable robust manipulation. Object-centric learning emphasizes task-relevant objects and their interactions while filtering out irrelevant background, providing a natural and scalable way to teach robots. OCRA leverages multi-view RGB videos, the state-of-the-art 3D foundation model VGGT, and advanced detection and segmentation models to reconstruct object-centric 3D point clouds, capturing rich interactions between objects. To handle properties not easily perceived by vision alone, we incorporate tactile priors via a large-scale dataset of over one million tactile images. These 3D and tactile priors are fused through a multimodal module (ResFiLM) and fed into a Diffusion Policy to generate robust manipulation actions. Extensive experiments on both vision-only and visuo-tactile tasks show that OCRA significantly outperforms existing baselines and ablations, demonstrating its effectiveness for learning from human demonstration videos.
Figures & tables
Fig. 1: Overview of OCRA . OCRA leverages object-centric learning from multi-view human demonstration videos and tactile sensing to enable robust execution of diverse manipulation tasks transferred from humans to robots.
Fig. 2: Framework. The left column illustrates our human demonstration collection system. Two RGB cameras capture demonstration videos, while the blue box highlights a portable tactile gripper for tactile data collection, which also records fingertip tactile images used to build our large-scale tactile dataset (shown at the bottom). The first row depicts how OCRA processes multi-view RGB inputs to obtain object-centric 3D priors. We first reconstruct the 3D scene using VGGT, followed by bi-view metric depth prediction for world-scale alignment. GroundingDINO and SAM2 then provide object segmentation masks, divided into a Manipulable Object Mask (for target objects) and a Context Object Mask (for surrounding objects). These are used to extract visual object-centric representations across modalities (segmentation, point cloud). The middle of the second row shows tactile-prior extraction via Tactile Encoder pretraining under a Masked Autoencoder paradigm. The right of the second row presents policy deployment. Multi-view RGB and tactile images are encoded into geometric and tactile features, which are fused by ResFiLM and passed to a Diffusion Policy. The policy predicts actions through iterative denoising of noisy action samples.
Fig. 3: Experiments Visualization . The first row shows a human demonstrator using either a hand or a portable gripper to collect demonstrations. The left four tasks are vision-only, while the right three are visuo-tactile. The following rows show the robot’s execution of our policy. The last two columns illustrate visuo-tactile tasks under different input conditions: in the Robot Deployment subfigures, the left and right insets correspond to different execution attempts, with arrows of different colors indicating motion directions. In the Weight Sorting task, cups are guided to distinct target locations based on mass; in the Texture Sorting task, objects are sorted by surface texture. For the Texture Sorting column (column 7, inset 3), we additionally show tactile images from grasps on different textures. These results demonstrate that vision alone is insufficient for reliable discrimination, highlighting the critical role of tactile perception in accurate decision-making.
Fig. 4: Experimental setup for our system . For the right part, the top half shows the robot and portable gripper, which uses the same tactile device. The bottom half shows our experimental objects. We display the camera setting in Fig. 2 . For the left part, the image shows a subset of the objects we used to collect our large-scale tactile image dataset.
Task
Method
Success
Proc. Fail.
Out. Fail.
Stack
Ours
100%
0%
0%
DP (RGB)
0%
100%
0%
DP (w/o OC)
10%
50%
40%
Scoop
Ours
70%
30%
0%
DP (RGB)
0%
70%
30%
DP (w/o OC)
0%
100%
0%
TABLE I: Real-world manipulation task evaluation. Outcomes are categorized into Success, Process Failure (Proc. Fail.), and Outcome Failure (Out. Fail.).
Method
Fragile Cup
Weight Sort
Texture Sort
Average
Ours
90%
90%
90%
90%
Ours(W/o Tactile)
60%
30%
30%
40%
TABLE II: Success rates of different methods on three evaluated real-world manipulation tasks
Method
Success
Proc. Fail.
Out. Fail.
Ours
90%
0%
10%
Ours (w/o Pretrain)
30%
20%
50%
Ours (w/o Fusion)
30%
40%
30%
TABLE III: Ablations on the Texture Sort task. Outcomes categorized as Success, Process Failure (Proc. Fail.), and Outcome Failure (Out. Fail.).
Fig. 5: Visualization of camera views . We use the two cameras marked by green boxes during both the demonstration collection and testing. To evaluate the view generalization ability of OCRA, the left green-boxed camera remains fixed, while the camera in both green and yellow boxes is replaced according to the arrow direction.
Method
Stack
Sweep
Scoop
Pour
Tactile Tasks
Ours
8
8
8
8
5
Teleoperation
20
20
N/A
N/A
N/A
TABLE IV: Demonstration collection time (seconds per task). “N/A” denotes tasks that are not applicable or cannot be performed by the method. Our approach achieves substantially lower demonstration time and greater adaptability compared to teleoperation.
Fig. 6: Visualization of depth reconstruction quality. We compare the depth map generated by VGGT with the camera ground-truth (GT) depths from the RGBD cameras on the Texture Sort task. The GT depth contains numerous holes and often produces misaligned point clouds due to measurement instability (see red points in the figure), whereas VGGT yields smoother depth maps and more consistent, regular point clouds.
Task
Depth Setting
Success Rate
Stack
Ours
100%
Single-View GT Depth
60%
Texture Sort
Ours
90%
Bi-view GT Depth
50%
TABLE V: Depth analysis across tasks . Success rates for our reconstruction compared with the camera’s ground-truth (GT) depth baselines.
Video demonstrations offer a scalable alternative to costly robot data for learning manipulation, yet existing reconstruction-based approaches often rely on constrained camera viewpoints or human-to-robot retargeting, while the reconstructed trajectories are difficult to adapt to new objects configurations without distorting the trajectory shape. Another key limitation is that the resulting policies often lack precise object-level 3D geometry awareness, limiting object grounding and object shape awareness critical for precise manipulation. To bridge these gaps, we propose VidAct, an efficient video-to-robot framework that learns object-centric, 3D-aware manipulation policies from a single monocular video per task and enables zero-shot real-world deployment. VidAct consists of three key components. First, VidAct reconstructs object meshes and motion from arbitrary demo videos and canonicalizes the motion in the static object frame, avoiding embodiment-specific retargeting and accommodating diverse camera viewpoints. Second, VidAct employ residual trajectory transfer for adapting the reconstructed motion to novel object configurations while preserving its motion shape. Finally, as the key policy-learning component, VidAct predicts simulation-provided privileged complete-object point clouds at each frame as an auxiliary task while retaining RGB-only deployment, providing dense object-centric supervision over both object pose and 3D geometry. Experiments on human, robot, generated, and internet videos demonstrate broad video applicability and zero-shot deployment. Per-frame complete-object 3D supervision improves policy generalization and sim-to-real success, while residual trajectory transfer enables reliable trajectory adaptation with better shape preservation.
Hang Li, Mingxin Zhang, Zihan Wu +9
Technical University of Munich · Peking University · Huawei Heisenberg Research Center +2
As an essential modality for dexterous and contact-rich tasks, tactile sensing provides precise force feedback that cannot be reliably inferred from vision. However, limited by hardware and data collection systems, existing datasets with tactility remain small in scale and narrow in contact coverage. Meanwhile, Vision-Language-Action (VLA) models with tactile modality are constrained on dynamics-agnostic post-training, which limits the performance ceiling on downstream tasks. In this paper, we present H-Tac, a large-scale tactile-action dataset with 160-hour egocentric human videos containing more than 300 tasks and 135k episodes. Building upon this, we propose Transferable Tactile Pre-Training (TTP), a system of tactile-based pre-training on human data for fine-grained robotic tasks. To bridge the gap between humans and robots, we use unified tactile and action spaces throughout the pre-training and post-training phases, preserving prior knowledge during human-to-robot transfer. By leveraging a tactile expert for future tactile prediction, our framework explicitly models the contact dynamics and precise physical interactions. Extensive experiments in simulation and on real robots demonstrate that our model achieves superior performance, exhibiting robust generalization and fine-grained manipulation capabilities. TTP paves the way for scalable tactile pre-training via human-to-robot transfer.
Chi Zhang, Penglin Cai, Ziheng Xi +6
1Peking University · 2BeingBeyond · 3Tsinghua University
Human videos are a scalable source of supervision for robot manipulation, as they are abundant and naturally capture rich object interactions. However, transferring human demonstrations to robots remains challenging due to embodiment mismatch, scene variation, and robot-specific feasibility constraints. We present Human2Any, a framework for learning reusable object-centric interaction priors from human videos without requiring real-world robot demonstrations in the target task contexts. Human2Any represents manipulation through object-object interaction motion, capturing task-relevant scene changes while abstracting away embodiment-specific details. It composes learned interaction priors with robot-side feasibility reasoning and motion planning, allowing the same human-derived knowledge to adapt to different embodiments, scene geometries, and task contexts. We validate Human2Any across diverse manipulation settings, including real-world experiments on a Franka tabletop setup and an RBY-1 humanoid mobile robot, demonstrating robust interaction-centric manipulation without real-world robot training data. Project website: https://human2any.github.io/.
Shuo Cheng, Chuye Zhang, Alfred Cueva +3
Georgia Institute of Technology · NVIDIA Corporation