Scaling policy learning requires broad experience and reliable supervision. Robot demonstrations provide high-quality trajectory supervision but are costly to collect. Egocentric videos broaden task and environment coverage, yet motion reconstruction and retargeting introduce noisy action labels. To address this challenge, we introduce EgoForesight, a vision-language-action framework that learns jointly from both sources by reducing the influence of unreliable action labels and exploiting demonstrated outcomes as complementary supervision. A joint vision-action flow couples future visual representation prediction with action generation, unifying reliability-aware action learning with outcome-informed representation transfer. Specifically, reliability-conditioned corruption allocates action supervision across noise levels according to source reliability, while reliability-weighted regression regulates each source's contribution. Complementing this treatment of noisy labels, privileged-future representation distillation draws supervision from demonstrated outcomes. An exponential-moving-average teacher observes the current scene and clean future to encode action-relevant structure from the resulting scene change. Its deeper representations supervise an earlier student layer receiving a corrupted future, providing outcome-informed structure beyond noisy action annotations. Together, these mechanisms support effective scaling with noisy egocentric actions. Over the tested range, EgoForesight exhibits steeper scaling slopes than the baseline, reducing real-world open-loop action MSE by 20.5% and improving RoboCasa few-shot success by 8.1 percentage points at 30,000 hours of pretraining data. After post-training, it also achieves strong absolute performance in both simulated and real-world dexterous manipulation.
Figures & tables
Figure 1: Overview of EgoForesight. Large-scale human videos expand training data but have less reliable action labels. Reliability-conditioned corruption samples stronger action corruption as reliability falls. The vision–action flow jointly predicts future visual representations and action chunks; during training, an observed-future teacher distills features into the student.
Figure 2: Two-stage pretraining data. Left: examples from AgiBot World, EgoDex, InternData-A1, and T-Rex used in S1. Center: examples from the Internal Ego-Centric Dataset, which adds 20,000 hours to S2 alongside all S1 sources. Right: task terms sized by frequency in 8,192 sampled instructions from this dataset, excluding stopwords and generic descriptors.
Full data (1,000 demonstrations / task)
Few-shot (100 demonstrations / task)
Method
PnP
Art.
Avg.
Method
PnP
Art.
Avg.
GR00T N1.6
55.4
24.2
47.6
GR00T-Qwen2.5
17.1
36.0
21.8
LDA-1B
55.1
56.3
55.4
GR00T-Qwen2.5-FT
26.9
41.7
30.6
RLDX-1
59.0
57.7
58.7
+ FLARE
47.7
64.7
51.9
JoyAI-RA
60.3
71.7
63.2
DIAL (robot-only)
56.0
65.3
58.3
DIAL (robot-only)
68.9
74.3
70.2
DIAL + EgoDex
60.8
62.0
61.1
Table 1: RoboCasa evaluation. Success rates (%) over 18 pick-and-place (PnP), 6 articulated-object (Art.), and all 24 tasks (Avg.). Blue and green denote full-data and few-shot training; dark and light shading mark the best and second-best score per column within each setting. Baseline variants and sources are detailed in Appendix D.1 .
Figure 3: Real-world dexterous manipulation. Each task pairs a head-camera image with success rates (%, solid bars) and progress scores (0–100, dashed borders) on the R1 Pro with Sharpa Wave hands. The right column reports the equal-weight mean across the four tasks. The evaluation protocol uses 100 demonstrations and 20 trials per task.
Table 5
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Retained hours
Supervision used in this work
Stage 1: Higher-quality initialization
AgiBot World (2025) a
2,392
EEF, arm joints, gripper, head/waist, selected base commands
EgoDex b
715
Wrist poses, fingertips, and retargeted gripper/hand targets
InternData-A1 c
7,330
Simulated single-/dual-arm EEF, joint, and gripper targets
T-Rex d
50
Bimanual and dexterous-hand joints
Stage 2: Human-video expansion
Appendix
Table 4: Pretraining sources and data allocation. Hours denote the retained manipulation data after filtering.
Figure 6: Shared input format. Input format across data sources, illustrated with a T-Rex example. Left: the primary view occupies the upper tile, with left and right wrist views below; each view is resized with its aspect ratio preserved and centered on black padding. Dimensions are height × width. Right: the VLM user message contains the composite image followed by task instructions and available metadata. Image labels and borders are explanatory overlays; prompt line breaks are added for readability.
Field
Indices
Target convention
Arm joints
L: 0–6; R: 29–35
Up to 7 absolute joint positions per arm
EEF motion
L: 7–12; R: 36–41
3D translation and 3D rotation vector, relative to the current pose in camera coordinates
Gripper
L: 13; R: 42
One opening/control scalar per gripper
Fingertips
L: 14–28; R: 43–57
Five 3D positions per hand: thumb, index, middle, ring, little finger
Base velocity
58, 63
Forward velocity and yaw rate for selected AgiBot tasks
Head / waist
64–65 / 68–69
Two absolute coordinates each, in source order
Appendix
Table 5: Canonical action space. Used coordinates in the 130-dimensional canonical action space. Indices are zero-based and ranges are inclusive. Availability is source-specific; unused slots and unused capacity within a slot are omitted.
Figure 7: Retargeting mismatch and tracking errors. (a) Human–robot morphological differences lead to residual geometric mismatch when retargeting human motions to Sharpa Wave. (b) Tracking annotations can deviate from visible hand poses in both EgoDex and Internal Ego. Both effects introduce noise into egocentric action supervision.
Setting
S1
S2
Optimizer updates
100k
200k
Global batch size
2,048
2,048
Learning-rate warmup
5k
5k
Peak base LR: VLM / coupled module
10−5 / 5×10−5
10−5 / 5×10−5
Flow-time sampling
Synchronized Beta(1.5, 1)
Reliability-conditioned
Action reliability qd
1 for all sources
Robot: 1; human: 0.2
Appendix
Table 6: Pretraining settings. Settings for S1 initialization and S2 expansion. Global batch sizes count manipulation samples.
Setting
Few-shot
Full data
Demonstrations per task
100
1,000
Total robot trajectories
2,400
24,000
Global batch size
128
512
Training steps
70,000
200,000
Feature distillation
Enabled
Enabled
Evaluation episodes per task
50
50
Appendix
Table 7: RoboCasa post-training settings. Settings for EgoForesight on RoboCasa GR1.
Task
Full data
Few-shot
Pick-and-place (18 tasks)
Cutting board → Basket
98.0
88.0
Cutting board → Cardboard box
62.0
52.0
Cutting board → Pan
90.0
84.0
Cutting board → Pot
90.0
90.0
Cutting board → Tiered basket
52.0
48.0
Appendix
Table 8: Per-task results on RoboCasa. Success rates (%) of EgoForesight on RoboCasa GR1. Each task is evaluated over 50 episodes in each setting.
Figure 8: Real-world hardware and teleoperation setup. (a) Galaxea R1 Pro with two Sharpa Wave hands, a RealSense L515 head camera, and OAK-1-W wrist cameras. (b) MANUS gloves with a VIVE tracker on the back of each hand. The operator’s face and clothing logo are pixelated.
Figure 9: Real-world task demonstrations. Each row presents a task from its initial configuration to completion, ordered from left to right.
Figure 10: Future prediction and IDM outputs. (a) Current/future RGB and RAE reconstructions of pooled GT versus predicted DINO features, with identical decoding and cropping. (b) Relatively smooth IDM outputs (dashed) and GT (solid).
Human videos provide rich information about object manipulation at a scale difficult to reproduce with robots, but often lack action annotations that can directly supervise robot policies. Realizing this potential requires distinguishing interaction-relevant changes from static appearance and connecting the learned interaction knowledge to executable robot actions. We present WALA, a framework named for World- and Action-supervised Latent Actions. Its semantic-geometric latent action model (LAM) learns latent actions by encoding and predicting semantic and geometric changes, emphasizing interactions over static appearance while retaining spatial detail. During robot policy training, the LAM encoder and decoder provide guidance through distillation and visual prediction, respectively. Together with robot action supervision, these signals shape policy-generated latent actions to support world prediction and capture the information needed to directly generate executable robot actions. Because LAM supervision requires no action labels, WALA enables co-training on action-labeled robot demonstrations and task-relevant action-free videos. In simulation experiments, our method achieves 92.33% average success on RoboTwin and 75.2% on RoboCasa-GR1-Tabletop, exceeding the previous state of the art on RoboCasa by 5.0 percentage points. WALA also shows strong real-robot performance, further enhanced by task-relevant action-free human videos that improve data efficiency and support target-task zero-shot and few-shot transfer.
Scaling generalist robotic policies is limited by the high cost of collecting embodied interaction data. Large-scale egocentric human videos offer a rich alternative, but they lack robot action labels, and existing action annotation methods are often noisy under real-world visual complexity and diverse manipulation dynamics. We introduce LAPT-Scale, a scalable latent action pre-training paradigm with two components: LAPT-Scale-LAM learns continuous latent actions by explaining frame-to-frame visual transitions in human videos and uses them as human-derived supervision, while LAPT-Scale-Policy trains a VLA policy through human-video latent-action pre-training, target-robot embodiment alignment, and task-specific post-training. Unlike prior vector-quantized LAMs, LAPT-Scale-LAM uses a continuous latent space to preserve fine-grained motion and interaction dynamics. We further combine perceptual, semantic, and optical-flow alignment signals to improve transition-level supervision beyond pixel reconstruction. Trained on 23.3K hours of egocentric video, LAPT-Scale-LAM provides implicit action supervision for policy pre-training, yielding consistent scaling gains and improving real-robot success from 63.3% to 85.0% as annotated human video scales from 1K to 5K hours. The learned policy also generalizes to unseen visual conditions.
Zhiyuan Xiang, Peidong Liu, Mingyang Li +4
Joy Future Academy, JD · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Tsinghua SIGS, Tsinghua University +1
Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.
Haoyu Zhao, Xingyue Zhao, Zhian Su +11
DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK +3