Scaling policy learning requires broad experience and reliable supervision. Robot demonstrations provide high-quality trajectory supervision but are costly to collect. Egocentric videos broaden task and environment coverage, yet motion reconstruction and retargeting introduce noisy action labels. To address this challenge, we introduce EgoForesight, a vision-language-action framework that learns jointly from both sources by reducing the influence of unreliable action labels and exploiting demonstrated outcomes as complementary supervision. A joint vision-action flow couples future visual representation prediction with action generation, unifying reliability-aware action learning with outcome-informed representation transfer. Specifically, reliability-conditioned corruption allocates action supervision across noise levels according to source reliability, while reliability-weighted regression regulates each source's contribution. Complementing this treatment of noisy labels, privileged-future representation distillation draws supervision from demonstrated outcomes. An exponential-moving-average teacher observes the current scene and clean future to encode action-relevant structure from the resulting scene change. Its deeper representations supervise an earlier student layer receiving a corrupted future, providing outcome-informed structure beyond noisy action annotations. Together, these mechanisms support effective scaling with noisy egocentric actions. Over the tested range, EgoForesight exhibits steeper scaling slopes than the baseline, reducing real-world open-loop action MSE by 20.5% and improving RoboCasa few-shot success by 8.1 percentage points at 30,000 hours of pretraining data. After post-training, it also achieves strong absolute performance in both simulated and real-world dexterous manipulation.
Figures & tables
Figure 1: Overview of EgoForesight. Large-scale human videos expand training data but have less reliable action labels. Reliability-conditioned corruption samples stronger action corruption as reliability falls. The vision–action flow jointly predicts future visual representations and action chunks; during training, an observed-future teacher distills features into the student.
Figure 2: Two-stage pretraining data. Left: examples from AgiBot World, EgoDex, InternData-A1, and T-Rex used in S1. Center: examples from the Internal Ego-Centric Dataset, which adds 20,000 hours to S2 alongside all S1 sources. Right: task terms sized by frequency in 8,192 sampled instructions from this dataset, excluding stopwords and generic descriptors.
Full data (1,000 demonstrations / task)
Few-shot (100 demonstrations / task)
Method
PnP
Art.
Avg.
Method
PnP
Art.
Avg.
GR00T N1.6
55.4
24.2
47.6
GR00T-Qwen2.5
17.1
36.0
21.8
LDA-1B
55.1
56.3
55.4
GR00T-Qwen2.5-FT
26.9
41.7
30.6
RLDX-1
59.0
57.7
58.7
+ FLARE
47.7
64.7
51.9
JoyAI-RA
60.3
71.7
63.2
DIAL (robot-only)
56.0
65.3
58.3
DIAL (robot-only)
68.9
74.3
70.2
DIAL + EgoDex
60.8
62.0
61.1
Table 1: RoboCasa evaluation. Success rates (%) over 18 pick-and-place (PnP), 6 articulated-object (Art.), and all 24 tasks (Avg.). Blue and green denote full-data and few-shot training; dark and light shading mark the best and second-best score per column within each setting. Baseline variants and sources are detailed in Appendix D.1 .
Figure 3: Real-world dexterous manipulation. Each task pairs a head-camera image with success rates (%, solid bars) and progress scores (0–100, dashed borders) on the R1 Pro with Sharpa Wave hands. The right column reports the equal-weight mean across the four tasks. The evaluation protocol uses 100 demonstrations and 20 trials per task.
Table 5
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Retained hours
Supervision used in this work
Stage 1: Higher-quality initialization
AgiBot World (2025) a
2,392
EEF, arm joints, gripper, head/waist, selected base commands
EgoDex b
715
Wrist poses, fingertips, and retargeted gripper/hand targets
InternData-A1 c
7,330
Simulated single-/dual-arm EEF, joint, and gripper targets
T-Rex d
50
Bimanual and dexterous-hand joints
Stage 2: Human-video expansion
Appendix
Table 4: Pretraining sources and data allocation. Hours denote the retained manipulation data after filtering.
Figure 6: Shared input format. Input format across data sources, illustrated with a T-Rex example. Left: the primary view occupies the upper tile, with left and right wrist views below; each view is resized with its aspect ratio preserved and centered on black padding. Dimensions are height × width. Right: the VLM user message contains the composite image followed by task instructions and available metadata. Image labels and borders are explanatory overlays; prompt line breaks are added for readability.
Field
Indices
Target convention
Arm joints
L: 0–6; R: 29–35
Up to 7 absolute joint positions per arm
EEF motion
L: 7–12; R: 36–41
3D translation and 3D rotation vector, relative to the current pose in camera coordinates
Gripper
L: 13; R: 42
One opening/control scalar per gripper
Fingertips
L: 14–28; R: 43–57
Five 3D positions per hand: thumb, index, middle, ring, little finger
Base velocity
58, 63
Forward velocity and yaw rate for selected AgiBot tasks
Head / waist
64–65 / 68–69
Two absolute coordinates each, in source order
Appendix
Table 5: Canonical action space. Used coordinates in the 130-dimensional canonical action space. Indices are zero-based and ranges are inclusive. Availability is source-specific; unused slots and unused capacity within a slot are omitted.
Figure 7: Retargeting mismatch and tracking errors. (a) Human–robot morphological differences lead to residual geometric mismatch when retargeting human motions to Sharpa Wave. (b) Tracking annotations can deviate from visible hand poses in both EgoDex and Internal Ego. Both effects introduce noise into egocentric action supervision.
Setting
S1
S2
Optimizer updates
100k
200k
Global batch size
2,048
2,048
Learning-rate warmup
5k
5k
Peak base LR: VLM / coupled module
10−5 / 5×10−5
10−5 / 5×10−5
Flow-time sampling
Synchronized Beta(1.5, 1)
Reliability-conditioned
Action reliability qd
1 for all sources
Robot: 1; human: 0.2
Appendix
Table 6: Pretraining settings. Settings for S1 initialization and S2 expansion. Global batch sizes count manipulation samples.
Setting
Few-shot
Full data
Demonstrations per task
100
1,000
Total robot trajectories
2,400
24,000
Global batch size
128
512
Training steps
70,000
200,000
Feature distillation
Enabled
Enabled
Evaluation episodes per task
50
50
Appendix
Table 7: RoboCasa post-training settings. Settings for EgoForesight on RoboCasa GR1.
Task
Full data
Few-shot
Pick-and-place (18 tasks)
Cutting board → Basket
98.0
88.0
Cutting board → Cardboard box
62.0
52.0
Cutting board → Pan
90.0
84.0
Cutting board → Pot
90.0
90.0
Cutting board → Tiered basket
52.0
48.0
Appendix
Table 8: Per-task results on RoboCasa. Success rates (%) of EgoForesight on RoboCasa GR1. Each task is evaluated over 50 episodes in each setting.
Figure 8: Real-world hardware and teleoperation setup. (a) Galaxea R1 Pro with two Sharpa Wave hands, a RealSense L515 head camera, and OAK-1-W wrist cameras. (b) MANUS gloves with a VIVE tracker on the back of each hand. The operator’s face and clothing logo are pixelated.
Figure 9: Real-world task demonstrations. Each row presents a task from its initial configuration to completion, ordered from left to right.
Figure 10: Future prediction and IDM outputs. (a) Current/future RGB and RAE reconstructions of pooled GT versus predicted DINO features, with identical decoding and cropping. (b) Relatively smooth IDM outputs (dashed) and GT (solid).