cs.RONov 5, 2026

VREgo: Consumer VR Headsets for Learning Robot Manipulation from Egocentric Demonstrations

Authors: Liuchuan Yu

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

VREgo trains manipulation policies from demonstrations recorded on consumer VR headsets; we instantiate it with EgoKit on one such headset, the PICO 4 Ultra. On-device head and hand tracking and stereo object localisation allow the operator to begin a demonstration at the table. We adapt HumanEgo's preprocessing to estimate grasp thresholds per recording, update object positions after release, and propagate segmentation masks between keyframes. We evaluate policies in simulation on two tabletop tasks recorded by one operator. For cube placement, we compare PICO and Project Aria recordings of the same 41 layouts. With 40 training demonstrations, the PICO policies placed the cube on the tray in 67, 74, and 60 of 100 evaluation layouts across three training seeds. The corresponding Aria results were 73, 78, and 0. For a two-stage stacking task recorded with the PICO alone, the baseline placed the block on the tray in 94 of 100 layouts and met the end-of-episode criterion for both placements in two. In one diagnostic episode, repeated replanning deferred release beyond the eight executed steps. Demonstration replays also showed that the open gripper could displace the cube before grasping it. After adapting the training trajectories and adding task-specific execution rules that use simulator object positions, the system met both placement criteria in 93.8% of episodes across four evaluation noise seeds. The same 100 layouts informed rule development and evaluation. VREgo is available at https://vrego.chuange.org/.

Figures & tables

 Project Aria Gen 1PICO 4 Ultra (VREgo)
Cameras usedone RGB, fisheye, 1408 ×\times 1408two see-through, 1280 ×\times 960 each, rectified, 64 mm apart
Image for the pipelineundistorted, 640 ×\times 480, 90 ∘ wideleft eye, 640 ×\times 480, 76 ∘ wide
Frame rate30 fps30 fps
Head poseMPS SLAM, in the cloudheadset runtime, on the device
HandMPS, 21 landmarks, in the cloudOpenXR, 26 joints, on the device
Objects located byhead motion, 2 s of the walk-upstereo, 1 s; walk-up optional
TABLE 1: The two recording devices: the reference’s Project Aria glasses and VREgo’s PICO 4 Ultra. Sizes are for the 41 takes of task 1, trimmed to the task.
Fig. 2: One take of each recording protocol: the first frame, the reference frame of the object window (orange border) and the first grasp (blue border), and the head’s speed over the take. (a, b) Task 1 begins with a walk-up (gray), seen from across the room in the first frame. The object window (orange) is the last 2 s of the walk-up on the Aria and its last second on the PICO. (c) Task 2 begins with the wearer at the table: the object window is the take’s first second, in which the head barely moves, and the stereo pair supplies the parallax. Blue bars mark where the grasp label is closed: once in task 1, once per move in task 2. Over all 41 takes of each set, the head moved a median 36.5 cm (Aria) and 6.0 cm (PICO, task 1) between the first and last frames of the object window, against 0.8 cm in task 2. Everything but the countertop is pixelated.
ChangeWhy it was neededMeasured effect
Objects located by stereo, with keypoints on long edges placed on a plane or an edge (Section 3.3 )One camera needs head motion; an edge along the stereo baseline cannot be matchedStatic points repeat to 1.1–4.3 mm (9–925 mm from head motion); no keypoint of the 82 PICO takes located from head motion
Stationary start (Section 3.4 )Monocular triangulation requires viewpoint displacementHead moves 0.8 cm in the object window; task 2’s separation error ++ 0.22 ±\pm 0.65 cm (task 1: ++ 0.25 ±\pm 0.49)
Grasp threshold per take; hand open until it is in the imageThe reference’s hand is closed below 10.5 cm, set by its operators, whose open hand reads 14 cm; ours read 7.6–9.6 cm on the PICO, and a hand tracked before it enters the image hangs relaxed and reads closedLabel closes once per move in 11 of 11 pilot takes (7 of 9 with a fixed threshold) and in all 123 bulk takes
Objects drawn from their recordAn object waiting for its turn loses its point tracks near the image border or under the hand, and they are not recoveredPilot task 2: the cube’s drawn keypoints off the cube in 51–80 % of frames before, 7–43 % after; bulk task 2: median 0 %
Carried objects located again after each release, in the data and in the simulatorA carried object is put down where the hand’s record says, which depends on the label’s timing; 12 pilot holds ended 0.3–2.8 cm (median 1.5) from where the camera saw themOn two pilot takes, 0.1–0.5 cm from the point tracks after release (2.3–2.7 before); bulk task 2: median 0.3 cm (block) and 0.2 cm (cube) from the camera’s view
One detection box per object; isolated keypoints dropped; label runs under 20 frames removedAll detections above the threshold are merged into one mask, and the cube’s prompt also boxed the block next to it; two stray keypoints moved a cube by 4 cmThe cube located in the pilot take where it had not been; the stray keypoints removed
TABLE 2: Where VREgo departs from the reference pipeline (HumanEgo), why, and what it changed. “Pilot” refers to the 14 takes recorded before the experiments’ data (Section 4 ), “bulk” to the 123 takes of the experiments.
 Per frameSAM 2 videoSAM 3.1
Segmentation, task-2 take 000 (s)455114488
Objects, gates, replaysreferencesame 1differ 2
False-arm frames, 38 takes1,7451,885 3108
Arm masks unlike the default’s 4–none9 of 38
TABLE 3: Segmentation backends on bulk takes. The per-frame backend is the reference’s; VREgo uses SAM 2 as video.
Fig. 3: A task-2 take (layout 16) through VREgo’s preprocessing, at five moments. Top: the left eye as the pipeline uses it. Middle: the detection masks of the arm (red), the tray (cyan), the block (green) and the cube (magenta). Bottom: the training image, with the arm removed, the virtual gripper drawn at the tracked hand (closed: white to magenta; open: yellow to red), and each object’s keypoints drawn where its record has it. The block’s keypoints follow it onto the tray, and the cube’s onto the block, because each carried object was located again by the camera after its release. The arm’s shadow is not part of the arm mask, so it stays in the training images.
 Task 1Task 2
 AriaPICOPICO
Data
Walk-upyesyesno
Take length, median (s)20.021.523.7
Training frames per take, median363377616
Preprocessing per take, median (min)9.08.312.8
TABLE 4: Data and policies of the two tasks, 41 takes per column. Policies use seed 7.
Fig. 4: Layouts, of 100 that no take recorded, in which the policy set its object down, against the number of training takes. Lines are seed 7. The bars at 40 takes span task 1’s three training seeds (for the Aria, two: its seed-9 policy declares the task finished at its first step and never moves).
Fig. 5: The 40-take task-2 policy before the changes of Section 5.4 , in simulation, on two of the 100 unseen layouts. The first column shows the policy input. The remaining columns show the episode from an external viewpoint. (a) One of the two layouts in which both moves succeed. The policy carries the block onto the tray, lets go, picks up the cube, and sets it on the studs 2.2 cm from the block’s centre, at the edge of what the studs allow. It never opens the gripper again; when the evaluation loop ends and its clean-up opens the gripper, the cube tips off. (b) The most common outcome: the block is set down on the tray and held until the loop ends, while the gripper does not reach the cube. The episodes were re-rendered on another GPU: (b) repeats the evaluated episode exactly, and (a) departs from it by 0.1 mm at step 42 and ends the same way.
Training takes102040
Block not picked up69296
Block picked up, not set down on the tray310
Block set down; gripper never opens again236549
Gripper opens, never closes again4022
Gripper closes again, cube not picked up0111
Cube picked up, not set down on the block1110
TABLE 5: How the task-2 policies’ episodes ended on the 100 unseen layouts, before the changes of Section 5.4 .
Fig. 6: Number of development layouts (out of 100) in which the 40-take task-2 policy set the block on the tray and completed both moves: the original policy (hollow markers; one noise seed), then the policy retrained on the rewritten records, with the rules for executing its output added one at a time (Table 6 ). Points are the mean over four noise seeds; bars span the seeds.
ExecutionBoth moves (SD)(a)(b)(c)(d)
Loop unchanged18.0 (2.4)5.549.38.017.5
+ Commitment24.8 (3.3)012.544.517.0
+ Settled, snapped grasps33.0 (3.7)09.014.842.3
+ Placement71.0 (3.2)09.09.59.5
+ Reach78.0 (0.8)06.56.58.0
+ From above93.8 (1.9)02.52.31.0
TABLE 6: The 40-take task-2 policy retrained on the rewritten records, with the rules of Section 5.4 added one at a time: outcomes on the 100 development layouts, averaged over four noise seeds. The original policy completed both moves in 2.
More in this topic

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,765 papers

Explore similar work

CardsList
Jun 6, 2026

EgoAERO: Learning Dexterous Manipulation from a Single Egocentric Video without Object Assets

Egocentric RGB-D videos offer a natural source of human dexterous manipulation demonstrations, but existing data is difficult to use for robot learning because object pose, geometry, and contact information are often missing or require pre-scanned object assets. We present EgoAERO, the first framework that learns dexterous manipulation from a single egocentric RGB-D human demonstration without object assets. EgoAERO reconstructs contact-consistent hand-object trajectories through asset-free object tracking and reconstruction, ego motion compensation, and adaptive contact optimization, then converts them into robot policies using two-stage residual learning. We further introduce an online quality assessment mechanism and construct EgoDex-R, a large-scale egocentric dataset with 4.3M RGB-D frames for dexterous policy learning. Simulation and real-world experiments show that EgoAERO enables single-demonstration dexterous manipulation and achieves downstream performance close to CAD-based reconstructions on HOI4D.

Yichen Niu, Haoran Lv, Xinrui Zhang +12

School of Astronautics, Harbin Institute of Technology · 2Lumos Robotic · 3Suzhou Research Institute, Harbin Institute of Technology +5

Contact-Rich Robotic ManipulationEgocentric Video UnderstandingRobotic ManipulationRobot Imitation LearningRobot Learning
AskPDF
Oct 9, 2026

EgoReveal: Learning Active Vision and Object Manipulation under Partial Occlusions from Egocentric Demonstrations

Learning from Demonstrations (LfD) enables robots to learn object manipulation directly from human demonstrations. In contrast to resource-intensive and labor-intensive teleoperation, egocentric demonstrations are easy to collect, enabling scalable data collection. Manipulated objects are often partially occluded in everyday environments, yet prior egocentric demonstration-based methods that require no robots or specialized teaching devices do not explicitly address occlusions. To fill this gap, we propose EgoReveal, a framework that learns viewpoint-repositioning together with manipulation actions from egocentric human demonstrations to reposition the viewpoint and obtain target visibility under partial occlusions. EgoReveal instantiates this as two imitation policies: one predicts the future camera-arm pose sequence from the current view and camera-arm pose, and the other predicts the future control commands from object keypoints and a history of control commands. EgoReveal switches between the two based on target visibility. In experiments where the manipulated object is partially occluded initially, EgoReveal repositions the camera to obtain visibility and completes the task more reliably than fixed-camera baselines.

Hiromu Kurihara, Kazutoshi Tanaka, Masashi Hatano +3

Keio University, Yokohama, Kanagawa 223-8522, Japan · OMRON SINIC X Corporation, Bunkyo-ku, Tokyo 113-0033, Japan

Active Vision for Robot ManipulationRobotic ManipulationLearning from DemonstrationRobot Imitation Learning
AskPDF
Oct 13, 2026

EgoCPT: Learning Dexterous Manipulation from a Single Ego-Centric Video with Contact-Aware Optimization

While expert demonstrations on real robots are difficult to collect, human ego-centric videos are easy to collect and abundant on the internet. However, learning dexterous manipulation policies from human videos remains challenging---one of the most important challenges being the difficulty of reconstructing physically grounded robot trajectories due to the large embodiment gap. In this paper, we propose EgoCPT, a novel paradigm for learning high-quality retargeted dexterous hand trajectories directly from an ego-centric video, which can then be deployed directly on real robots. The only inputs to our method are an ego-centric RGB video and a reference hand pose trajectory, which can be easily obtained through off-the-shelf software or specialized cameras during recording. Our method is particularly robust to noisy hand poses attributed to our novel contact-aware optimization framework, which iteratively optimizes the contact points between the robot hands and objects. Across five manipulation tasks, our method achieves overall success rates of 67.3% on noisy EgoDex hand-pose reconstructions and 92.0% on high-quality TACO reconstructions collected using motion-capture hardware, outperforming multiple baselines on both benchmarks.

Zhixin Sun, Kairui Shi, Tianao Ren +3

Ant Group · Computer Science and Engineering, University at Buffalo · Mechanical Engineering, Stanford University

Robotic ManipulationRobot Imitation LearningCross-Embodiment Robot LearningMotion RetargetingContact-Rich Robotic Manipulation
AskPDF