EgoAVFlow: Robot Policy Learning with Active Vision from Human Egocentric Videos via 3D Flow
Authors: Daesol Cho, Youngseok Jang, Danfei Xu, Sehoon Ha
Organizations: School of Interactive Computing, Georgia Institute of Technology, Georgia, USA · InnoCORE AI-Transformed Aerospace Research Center, KAIST, Daejeon, Republic of Korea
Egocentric human videos provide a scalable source of manipulation demonstrations; however, deploying them on robots requires active viewpoint control to maintain task-critical visibility, which human viewpoint imitation often fails to provide due to human-specific priors. We propose EgoAVFlow, which learns manipulation and active vision from egocentric videos through a shared 3D flow representation that supports geometric visibility reasoning and transfers without robot demonstrations. EgoAVFlow uses diffusion models to predict robot actions, future 3D flow, and camera trajectories, and refines viewpoints at test time with reward-maximizing denoising under a visibility-aware reward computed from predicted motion and scene geometry. Real-world experiments under actively changing viewpoints show that EgoAVFlow consistently outperforms prior human-demo-based baselines, demonstrating effective visibility maintenance and robust manipulation without robot demonstrations.
Figures & tables
Fig. 1 : EgoAVFlow learns manipulation and active viewpoint control from egocentric human videos by predicting future 3D flow and optimizing camera viewpoints for visibility, yielding viewpoint-robust robot execution without robot demonstrations.
Fig. 2 : Method overview. EgoAVFlow consists of three diffusion models. The robot policy πr produces future robot action sequences. The flow generation model f predicts future 3D flows from the outputs of πr . The view policy πv produces future camera viewpoints from the outputs of πr , f , and reconstructed mesh surfaces through a visibility-aware reward-maximizing denoising process. Viewpoints (A) represent that most query points are invisible ( Red LOS ) due to the table’s mesh surface or out of FoV, whereas in viewpoints (B) these points are visible ( Green LOS ), yielding a higher visibility reward.
Fig. 3 : Tasks. Each task requires appropriate viewpoint adjustments. Otherwise, the object is occluded by the robot or elements in the environment, such as a table or drawer.
Fig. 4 : Visibility comparison (best viewed in the digital version). The visibility is computed from each different fixed viewpoint. No single viewpoint can maintain full visibility throughout the execution, indicating that the viewpoint must be continuously adjusted online to maximize visibility.
Fig. 5 : Visibility reward. For all tasks, EgoAVFlow achieves higher average visibility rewards Rvis than HVI , demonstrating our method’s visibility maintenance capability. The error bars represent 1 standard error.
Fig. 6 : Qualitative comparison. Due to the visibility-maximizing viewpoint adjustments, EgoAVFlow maintains visibility of the query points and their predicted future flows, whereas HVI fails to keep them in view, causing the query points to move out of the FoV. All experimental figures and videos in Section V are best viewed in the supplementary video.
Method
Spray
Doll
Toilet Paper
Towel
V-B
HVI
5/25
10/25
7/25
5/25
V-C
AMPLIFY
4/25
6/25
9/25
9/25
EgoZero
10/25
7/25
7/25
9/25
Phantom
5/25
7/25
6/25
8/25
EgoAVFlow
20/25
18/25
17/25
18/25
( ×2.0 )
( ×2.5 )
( ×1.8 )
( ×2.0 )
TABLE I : Success rates of EgoAVFlow, HVI (Human Viewpoint Imitation), and robot policy learning baselines. All methods are trained on the same egocentric human video dataset (no robot data). Relative improvements compared to the second-best robot policy learning baseline are shown in parentheses .
Fig. 7 : Failure analysis. Top: Method-wise composition of failures for each category. Bottom: Example frames of a grasping failure. The breakdown suggests that manipulation-related failures in robot policy baselines are driven by a mismatch between their learned representations/assumptions, whereas human viewpoint imitation fails primarily due to the lack of visibility-aware viewpoint optimization.
Dexterous manipulation is limited by the cost of collecting large-scale robot demonstrations. Egocentric human videos offer a scalable source of diverse manipulation behaviors, but directly using them for robot learning requires bridging two gaps: the visual gap between human and robot observations, and the action gap between human motion and robot-executable action. We propose EgoEngine, a scalable framework for transforming egocentric human manipulation videos into high-fidelity robot data. Given an egocentric RGB video, EgoEngine produces: (i) a high-fidelity robot observation video replacing human with robot while preserving scene context and temporal alignment, and (ii) a task-aligned, executable robot action trajectory under feasibility constraints. Experiments in simulation and on real robots show that EgoEngine enables scalable conversion of human videos into robot data and, to our knowledge, demonstrates the first zero-shot visuomotor dexterous policy learning from egocentric human videos without real-robot demonstrations. Project website: https://egoengine.github.io.
Yangcen Liu, Shuo Cheng, Xinchen Yin +6
Georgia Institute of Technology · Tsinghua University
Human egocentric video captures rich manipulation demonstrations without any robot hardware, yet transferring these skills to robots remains challenging due to the embodiment gap between human and robot in both visual appearance and kinematics. We present HumanEgo, a framework that bridges the embodiment gap by lifting each human demonstration to an entity-level representation of hand-object interaction, and training a flow matching policy with dense auxiliary objectives that amplify supervision from every trajectory. HumanEgo is robot-data-free, hardware-agnostic, data-efficient, and zero-shot human-to-robot transferable. With only 30 minutes of human videos per task, HumanEgo achieves 92.5% average success across four real-world tasks (75% with just 15 minutes), outperforms matched-time robot teleoperation by 41%, and robustly transfers zero-shot across novel robots, cameras, and environments. We release HumanEgo as an easy-to-use, open-source framework for learning robot policies directly from human data: https://github.com/TX-Leo/HumanEgo
Egocentric human video offers a scalable alternative to robot data for pretraining, yet models pretrained on such video consistently underperform those pretrained on robot data. We attribute this gap to a missing signal, the active perception behavior in egocentric videos, where humans continuously reposition their viewpoint during manipulation, inducing camera motion that standard pipelines treat as noise. To address this, we present ActiveMimic, a pretraining framework that recovers synchronized camera and wrist trajectories from a single body-worn RGB camera, models camera motion as a viewpoint action, and jointly learns active perception and manipulation from in-the-wild egocentric human video before adapting to a target robot. Empirically, real-world experiments across tasks with diverse active perception demands show that ActiveMimic consistently surpasses baselines pretrained on human video and matches state-of-the-art models pretrained on robot data. Further analysis provides evidence that active perception capability originates from egocentric human video pretraining rather than robot-specific fine-tuning, confirming active perception as the key to unlocking egocentric human video for robot pretraining.
Xingyao Lin, Guojin Zhong, Tianyi Lu +4
Fudan University · Shanghai Innovation Institute · Current Robotics +1