We present Actron3D, a framework that enables robots to acquire transferable 6-DoF manipulation skills from monocular, uncalibrated, RGB-only human demonstration videos. Our key idea is to represent manipulation knowledge within a video as a continuous neural function over object space. At the core of Actron3D lies the Neural Affordance Function, which distills geometry, visual features, contact priors, and action flows from diverse demonstration videos into a compact 3D neural representation. During deployment, we adopt a hierarchical pipeline that retrieves the matched affordance function and transfers encoded manipulation knowledge to novel objects through coarse-to-fine differentiable optimization. Leveraging the continuous nature of Neural Affordance Function, the framework performs spatial queries over multimodal features to align demonstrations with observations and generates precise 6-DoF manipulation policy. Experiments in both simulation and the real-world demonstrate that Actron3D significantly outperforms prior methods, achieving a 14.9 percentage point improvement in the average success rate across 13 tasks while requiring only 2-3 demonstration videos per task.
Figures & tables
Fig. 1 : Overview of Actron3D framework. In the distillation phase, multimodal knowledge from RGB videos is encoded into Neural Affordance Function s , and forming a sparse function bank M . At inference time, Gtrans retrieves the best matched NAF, aligns it to the target object using a coarse-to-fine optimization module and produces 6-DoF EEF actions (Sect. III-D ), in a zero-shot manner.
Fig. 2 : Multimodal information and robot actions synthesized from novel views of NAF s fit to diverse videos.
T01
T02
T03
T04
T05
T06
T07
T08
T09
T10
T11
T12
T13
Avg.
Where2Act [ 12 ]
60.0
40.0
33.3
53.3
100.0
100.0
40.0
6.7
33.3
40.0
26.7
100.0
100.0
56.4
VRB † [ 2 ]
60.0
60.0
40.0
40.0
13.3
100.0
0.0
0.0
53.3
10.0
53.3
93.3
100.0
42.6
RAM [ 4 ]
46.7
60.0
33.3
53.3
13.3
93.3
33.3
93.3∗
66.7∗
33.3∗
53.3∗
80.0
80.0
58.5
GFlow [ 10 ]
6.7
13.3
20.0
33.3
20.0
100.0
93.3
0.0
13.3
0.0
20.0
100.0
93.3
39.6
VidBot [ 7 ]
46.7
100.0
73.3
100.0
26.7
100.0
33.3
86.7
73.3
93.3
46.7
100.0
100.0
75.9
Ours
73.3
100.0
80.0
100.0
80.0
93.3
100.0
93.3
86.7
93.3
73.3
100.0
100.0
90.8
TABLE I: Quantitative results on tasks evaluated in simulators on success rate (%). T01 : Open drawer T02 : Close drawer, T03 : Open microwave, T04 : Close microwave, T05 : Open hinge cabinet, T06 : Close hinge cabinet, T07 : Open dishwasher, T08 : Open slide cabinet, T09 : Close slide cabinet, T10 : Close laptop lid, T11 Put down toilet seat: T12 : Pick up cup, T13 : Pick up bottle. * uses self-collected data for novel tasks. † uses strategy from [ 4 ] to lift affordance to 3D.
RT01
RT02
RT03
RT04
RT05
Avg.
RAM
40.0
20.0
60.0
0.0∗
40.0∗
32.0
VidBot
60.0
40.0
100.0
0.0
50.0
50.0
Ours
70.0
70.0
90.0
80.0
70.0
76.0
TABLE II: Real-world experiment results on selected tasks. RT01 : Open drawer, RT02 : Open microwave, RT03 : Close microwave, RT04 : Pour water, RT05 : Wiping table with sponge. * : Use self-collected data for retrieval.
Fig. 3 : Inferred robot gripper actions by different methods.
AT01
AT02
AT03
AT04
AT05
AT06
Avg.
Ours [Full Model]
73.3
100.0
80.0
100.0
93.3
73.3
86.7
w/o Neural Function [V1]
66.7
26.7
73.3
0.0
6.7
6.7
30.0
w/o Contact Sampling [V2]
53.3
13.3
26.7
73.3
100.0
73.3
56.6
w/o Viewpoint Ranking [V3]
66.7
33.3
0.0
100.0
100.0
46.7
57.8
w/o Afford. Optimization [V4]
73.3
0.0
0.0
26.7
20.0
6.7
21.1
w/o Contact Refinement [V5]
66.7
66.7
33.3
66.7
100.0
66.7
66.7
TABLE III: Ablation results on 6 selected tasks. AT01 : Open drawer, AT02 : Close drawer, AT03 : Open microwave, AT04 : Close microwave, AT05 : Open slide cabinet. AT06 : Put down toilet seat. † : Use strategy from [ 4 ] to lift affordance to 3D.
Fig. 4 : Loss curves of the action head when trained from scratch vs. initialized with pretrained weights of the retrieved NAF model.
Video demonstrations offer a scalable alternative to costly robot data for learning manipulation, yet existing reconstruction-based approaches often rely on constrained camera viewpoints or human-to-robot retargeting, while the reconstructed trajectories are difficult to adapt to new objects configurations without distorting the trajectory shape. Another key limitation is that the resulting policies often lack precise object-level 3D geometry awareness, limiting object grounding and object shape awareness critical for precise manipulation. To bridge these gaps, we propose VidAct, an efficient video-to-robot framework that learns object-centric, 3D-aware manipulation policies from a single monocular video per task and enables zero-shot real-world deployment. VidAct consists of three key components. First, VidAct reconstructs object meshes and motion from arbitrary demo videos and canonicalizes the motion in the static object frame, avoiding embodiment-specific retargeting and accommodating diverse camera viewpoints. Second, VidAct employ residual trajectory transfer for adapting the reconstructed motion to novel object configurations while preserving its motion shape. Finally, as the key policy-learning component, VidAct predicts simulation-provided privileged complete-object point clouds at each frame as an auxiliary task while retaining RGB-only deployment, providing dense object-centric supervision over both object pose and 3D geometry. Experiments on human, robot, generated, and internet videos demonstrate broad video applicability and zero-shot deployment. Per-frame complete-object 3D supervision improves policy generalization and sim-to-real success, while residual trajectory transfer enables reliable trajectory adaptation with better shape preservation.
Hang Li, Mingxin Zhang, Zihan Wu +9
Technical University of Munich · Peking University · Huawei Heisenberg Research Center +2
Learning robot manipulation from human videos is appealing due to the scale and diversity of human demonstrations, but transferring such demonstrations to executable robot behavior remains challenging. Prior work either relies on robot data for downstream adaptation or learns affordance representations that remain at the perception level and do not directly support real-world execution. We present BridgeACT, an affordance-driven framework that learns robotic manipulation directly from human videos without requiring any robot demonstration data. Our key idea is to model affordance as an embodiment-agnostic intermediate representation that bridges human demonstrations and robot actions. BridgeACT decomposes manipulation into two complementary problems: where to grasp and how to move. To this end, BridgeACT first grounds task-relevant affordance regions in the current scene, and then predicts task-conditioned 3D motion affordances from human demonstrations. The resulting affordances are mapped to robot actions through a grasping module and a lightweight closed-loop motion controller, enabling direct deployment on real robots. In addition, we represent complex manipulation tasks as compositions of affordance operations, which allows a unified treatment of diverse tasks and object-to-object interactions. Experiments on real-world manipulation tasks show that BridgeACT outperforms prior baselines and generalizes to unseen objects, scenes, and viewpoints.
Yifan Han, Jianxiang Liu, Haoyu Zhang +3
Institute of Automation, Chinese Academy of Sciences. · School of Artificial Intelligence, Shanghai Jiao Tong University. · 3South China University of Technology.
We present OCRA, an Object-Centric framework for video-based human-to-Robot Action transfer that learns directly from human demonstration videos to enable robust manipulation. Object-centric learning emphasizes task-relevant objects and their interactions while filtering out irrelevant background, providing a natural and scalable way to teach robots. OCRA leverages multi-view RGB videos, the state-of-the-art 3D foundation model VGGT, and advanced detection and segmentation models to reconstruct object-centric 3D point clouds, capturing rich interactions between objects. To handle properties not easily perceived by vision alone, we incorporate tactile priors via a large-scale dataset of over one million tactile images. These 3D and tactile priors are fused through a multimodal module (ResFiLM) and fed into a Diffusion Policy to generate robust manipulation actions. Extensive experiments on both vision-only and visuo-tactile tasks show that OCRA significantly outperforms existing baselines and ablations, demonstrating its effectiveness for learning from human demonstration videos.
Kuanning Wang, Ke Fan, Yuqian Fu +6
Fudan University, China. · Thomas Lord Department of Computer Science, University of Southern California, USA.