Large-scale real-world robot data collection is a prerequisite for bringing robots into everyday deployment. However, existing pipelines often rely on specialized handheld devices to bridge the embodiment gap, which not only increases operator burden and limits scalability, but also makes it difficult to capture the naturally coordinated perception-manipulation behaviors of human daily interaction. This challenge calls for a more natural system that can faithfully capture human manipulation and perception behaviors while enabling zero-shot transfer to robotic platforms. We introduce ActiveGlasses, a system for learning robot manipulation from ego-centric human demonstrations with active vision. A stereo camera mounted on smart glasses serves as the sole perception device for both data collection and policy inference: the operator wears it during bare-hand demonstrations, and the same camera is mounted on a 6-DoF perception arm during deployment to reproduce human active vision. To enable zero-transfer, we extract object trajectories from demonstrations and use an object-centric point-cloud policy to jointly predict manipulation and head movement. Across several challenging tasks involving occlusion and precise interaction, ActiveGlasses achieves zero-shot transfer with active vision, consistently outperforms strong baselines under the same hardware setup, and generalizes across two robot platforms.
Figures & tables
Fig. 1 : ActiveGlasses enables operators to collect manipulation demonstrations with bare hands. The head-mounted glasses record the stereo observations of the current task and the operator’s head movement, and realize zero-shot transfer of manipulation with active vision to robotic platforms. Active vision allows the robot to complete tasks, including occluded scenarios, with only a head camera input.
Fig. 2 : Our system uses XREAL Glasses combined with a ZED Mini stereo camera, enabling egocentric stereo video and 6DoF head movement data collection. During demonstration, the operator actively explores task-relevant regions in the environment and complete manipulation tasks without any hand-held devices. We proposed an object-centric 3D Policy modified from RISE [ 13 ] , which predicts future 6-DOF object trajectory in the task space. After training, the policy is deployed zero-shot on a real-world robot. A 6-DoF robotic arm synchronously executes the operator’s head motions during inference, allowing the robot to reproduce active vision.
Fig. 3 : (a) Perception device. A ZED Mini stereo camera is mounted on smart glasses. No additional external or wrist-mounted cameras are used. Considering the gap between the user’s and the ZED camera’s field of view (FOV), we added a fixed canvas to the Glass UI to indicate the bottom of the current camera view. (b) Comparison of recorded trajectories. Several numerical jumps are observed in the ZED trajectory; therefore, we choose to adopt tracking data from the glasses in training and inference.
Fig. 4 : Task setting. We introduce three tabletop manipulation tasks that require active viewpoint adjustment. In Book Placement , the camera initially faces the side of the bookshelf and must move closer and rotate to observe the empty slot before placing the book. In Occluded Distant Pour Water , the target cup is occluded by a screen, requiring the camera to adjust its viewpoint to perceive the pouring target. In Bread Insertion , the camera must tilt and reorient to observe the toaster slot before accurately inserting the bread. To reflect the operator’s torso movement during data collection, the perception arm is also mounted on a movable wheeled table, and its base position is randomized within a small range at start of each rollout when deployment.
Fig. 5 : Data collection performance. Sigma is reported as 0 g because it operates in a zero-gravity mode.
Book placement
Bread insertion
Occluded distant water pouring
Stage 1
Stage 2
Stage 3
Stage 1
Stage 2
Stage 3
Stage 1
Stage 2
Stage 3
ActiveGlasses
20/20
16/20
14/20
20/20
15/20
11/20
20/20
15/20
10/20
w/o active vision
20/20
8/20
7/20
11/20
1/20
0/20
18/20
10/20
4/20
Pi05
20/20
9/20
7/20
20/20
18/20
6/20
20/20
12/20
4/20
TABLE I: We compare ActiveGlasses with two baselines across three tasks: a variant without active perception and Pi0.5 [ 4 ] . The Book Placement , Bread Insertion , and Occluded Distant Water Pouring tasks are trained using 200, 100, and 100 demonstrations, respectively. During evaluation, the poses of task-related objects are randomized within predefined ranges, including the bookshelf, empty slot location, the cup and screen, and the toaster.
Fig. 6 : Policy Design Ablation. We predict actions of the manipulation arm and the perception head using separate diffusion heads. For the manipulation head, we investigate whether the current object pose should be provided as an additional conditioning signal to the diffusion policy. For the predicted object trajectory, we do ablation study on the choice of action representation (i.e. absolute or relative output).
Book Placement
w/o curr pose
w curr pose
absolute
14/20
3/20
relative
–
10/20
TABLE II : We evaluate four policy designs on the Book Placement task. An episode is considered successful if the placement is completed without any collision.
Book Placement
Stage 1
Stage 2
Stage 3
Flexiv Rizon 4
20/20
16/20
14/20
UR5
20/20
16/20
11/20
TABLE III : We deploy our policy across two robotic arms and evaluate the policy performance on the Book Placement task.
Inspired by human vision, we introduce a framework using active gaze to enable fine-grained bimanual manipulation with only a single stereo camera. EyeRobot 2.0 physically attends to a 3D fixation point in the scene by swiveling two eye viewpoints to center their gaze on it. The resulting images are processed foveally by allocating more visual tokens to the image centers, focusing computation on task-relevant features. Such Active Visual Fixation (AVF) requires carefully coordinated gaze during task execution, which we accomplish hierarchically by first training a low-level gaze servoing policy conditioned on a goal object, then training a target selector which emits fixation goals based on task progress. Both modules are trained with RL on real-world data: the first is trained with a dense geometric reward and the second co-trains with the BC gripper policy which allows it to discover fixation sequences that can resemble a human's fixation sequence while performing the task. EyeRobot 2.0 further takes advantage of fixation by canonicalizing gripper information into a fixation-relative SE(3) frame, which compacts the size of the action distribution to learn. We collect teleoperation data for 7 real-world and 6 simulated tasks, and conduct over 1000 physical and 1800 simulated robot trials comparing EyeRobot 2.0 against passive stereo and ego + wrist camera policies trained on the same data. Removing wrist cameras is costly for standard policies: with only passive stereo, real-world success drops from 52% to 27%. EyeRobot 2.0 closes this gap with only stereo, outperforming passive stereo by 40% in real and 20% in sim. It matches ego + wrist policies when their wrist views are clear (69% vs. 64%), and more than doubles their success when grasped objects occlude the wrist cameras (48% vs. 22%)
Active perception is essential for robotic manipulation when fixed viewpoints leave task-relevant information occluded or unobserved. However, enabling vision-language-action (VLA) models to reason across changing viewpoints and actively acquire informative observations remains challenging. We present ActiveScale, a framework that advances active perception through coordinated model, data, and hardware designs. Our model augments a VLA with historical video observations and explicit camera-pose supervision, using per-frame pose tokens and a lightweight prediction head to associate observations across viewpoints and support a coherent understanding of the scene. To learn from the camera motion naturally present in human activity, we introduce a scalable human--robot mid-training recipe using 1000 hours of egocentric and robotic data, adapting the model to temporal inputs and pose supervision. We further introduce Active-perception Mobile-manipulation Platform (AMP), a robotic platform that supports active perception and mobile manipulation through single-operator teleoperation, enabling scalable collection of demonstrations that coordinate viewpoint changes and manipulation. Experiments demonstrate improved success rates on active-perception tasks, while ablation studies validate the contributions of camera-pose-aware modeling and egocentric mid-training. Together, these components provide an integrated foundation for studying and developing active perception in robotic manipulation.
Shuai Zhou, Kaisheng Pang, Wenxuan Song +3
Robotics Institute, Carnegie Mellon University · The Hong Kong University of Science and Technology (Guangzhou)
Egocentric human videos provide a scalable source of manipulation demonstrations; however, deploying them on robots requires active viewpoint control to maintain task-critical visibility, which human viewpoint imitation often fails to provide due to human-specific priors. We propose EgoAVFlow, which learns manipulation and active vision from egocentric videos through a shared 3D flow representation that supports geometric visibility reasoning and transfers without robot demonstrations. EgoAVFlow uses diffusion models to predict robot actions, future 3D flow, and camera trajectories, and refines viewpoints at test time with reward-maximizing denoising under a visibility-aware reward computed from predicted motion and scene geometry. Real-world experiments under actively changing viewpoints show that EgoAVFlow consistently outperforms prior human-demo-based baselines, demonstrating effective visibility maintenance and robust manipulation without robot demonstrations.
Daesol Cho, Youngseok Jang, Danfei Xu +1
School of Interactive Computing, Georgia Institute of Technology, Georgia, USA · InnoCORE AI-Transformed Aerospace Research Center, KAIST, Daejeon, Republic of Korea