In this paper, we investigate how human egocentric videos can benefit robot policy learning without action supervision. Building on a pretrained VLM, we propose Shared Prediction on Action and Representation (SPAR), an MoT architecture in which future visual state prediction can provide independent and complementary learning signals alongside action prediction. We find that representations learned from egocentric videos improve downstream performance on out-of-distribution robot tasks, with further gains as video data scales. We also study the effects of objective choices and data curation, providing insights into how to leverage action-free videos for scalable robot learning.
Figures & tables
Figure 1: Architecture Overview of SPAR. Starting from a common VLA [ 8 , 9 ] , SPAR extends it with a generation expert that mirrors the action expert design, and predicts the representation of the future visual state simultaneously. Robot data supervises both experts, while the egocentric videos supervise only the generation expert. The VLM backbone can therefore be optimized with heterogeneous data composition. The generation expert is removed at inference.
Figure 2: Success rates across seven distribution shifts. (a) Diffusion and (b) regression prediction of one future state frame with six visual targets; the two panels share a target legend. (c) Diffusion prediction of one versus four future frames with SigLIP2 in a separate frame-count experiment.
Figure 3: Ego4D curation and corresponding OOD task success rates. Left: hours retained after the motion-quality and visible-hand gates. Right: robot-only training compared with co-training on raw Ego4D, Gate-1 clips, or clips passing both gates, across fine-tuning steps.
Figure 4: Egocentric video data ablations. (a) In-domain and OOD success with 0, 150, 450, and 1,500 hours of Open-AoE video co-training. (b) LIBERO-Plus success as scene, object, or interaction diversity varies within a fixed budget of 200K clip–description pairs.
Figure 5: Real-world evaluation setup. Left: The hardware setup; objects from several tasks are placed together for illustration only. Right: Five sequential snapshots per task.
Cucumber
Flower
Toaster
Cup
Bottle
Total
Transfer
Handover
Loading
Hanging
Uncapping
# Demos
220
160
150
139
57
–
π0.5
9/10
9/10
2/10
1/10
1/10
22/50
Ours
9/10
8/10
2/10
0/10
1/10
20/50
Table 1: Real-world Results . # Demos denotes the number of tele-operation demonstration collected for each task. We report the success rate out of 10 trials for each task.
Figure 6: Real-world generalization evaluation setting under visual perturbations.
Cucumber Transfer
Flower Handover
Method
Default
Illum.
Backgr.
Distr.
Default
Illum.
Backgr.
Distr.
π0.5
9
7
7
4
9
5
2
3
Ours
9
9
8
5
8
2
2
0
Table 2: Success under visual perturbations out of 10 trials for each task. Default: unperturbed setting; Illum.: illumination changes; Backgr.: background variations; Distr.: static distractors.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Target
Encoder
Pretraining objective
Res.
N
d
Qwen-ViT
Qwen3.5-2B vision encoder [ 12 ]
vision–language (VLM)
224
49
2048
SigLIP2
So400m/16 [ 32 ]
language–image
256
256
1152
DINOv2
ViT-g/14 [ 33 ]
self-distillation
224
257
1536
LingBot
LingBot-Vision ViT-g/16 [ 29 ]
masked boundary modeling
512
1024
1536
V-JEPA2 †
ViT-g/16 [ 26 ]
latent video prediction
256
256
1408
Wan-VAE
Wan2.2 VAE [ 28 ]
pixel reconstruction
256
256
48
Appendix
Table 3: Target encoders. N and d are the number and dimension of target tokens per frame.
Optimizer
AdamW, β=(0.9,0.95) , ϵ=10−8 , weight decay 10−8
Learning rate
LLM 2.83×10−5 ; experts 2.83×10−4 ; other 1.41×10−4
Scaling policy learning requires broad experience and reliable supervision. Robot demonstrations provide high-quality trajectory supervision but are costly to collect. Egocentric videos broaden task and environment coverage, yet motion reconstruction and retargeting introduce noisy action labels. To address this challenge, we introduce EgoForesight, a vision-language-action framework that learns jointly from both sources by reducing the influence of unreliable action labels and exploiting demonstrated outcomes as complementary supervision. A joint vision-action flow couples future visual representation prediction with action generation, unifying reliability-aware action learning with outcome-informed representation transfer. Specifically, reliability-conditioned corruption allocates action supervision across noise levels according to source reliability, while reliability-weighted regression regulates each source's contribution. Complementing this treatment of noisy labels, privileged-future representation distillation draws supervision from demonstrated outcomes. An exponential-moving-average teacher observes the current scene and clean future to encode action-relevant structure from the resulting scene change. Its deeper representations supervise an earlier student layer receiving a corrupted future, providing outcome-informed structure beyond noisy action annotations. Together, these mechanisms support effective scaling with noisy egocentric actions. Over the tested range, EgoForesight exhibits steeper scaling slopes than the baseline, reducing real-world open-loop action MSE by 20.5% and improving RoboCasa few-shot success by 8.1 percentage points at 30,000 hours of pretraining data. After post-training, it also achieves strong absolute performance in both simulated and real-world dexterous manipulation.
Chubin Zhang, Changyuan Wang, Yirui Wang +7
Tsinghua University · JD Future Academy · Loopeva +2
Vision-Language-Action (VLA) models are bottlenecked by the scarcity of robot data. Egocentric human manipulation videos offer an abundant, diverse, and semantically-rich alternative that inherently captures manipulation intent. However, the embodiment gap makes it difficult to extract action-relevant representations from these videos. From the perspective of action execution, human hands and robots share analogous manipulation primitives---reaching, grasping, placing---yet exhibit distinct kinematics tied to their embodiments. This might motivate our key approach: egocentric video pretraining can yield shared action representations capturing transferable manipulation skills, while human-specific motion patterns should be disentangled to avoid interfering with robot policy learning. To this end, we propose EXE, an Egocentric Cross(X)-Embodiment pretraining framework that learns a factorized latent action representation from egocentric human videos, separating shared manipulation priors from embodiment-specific signals. By leveraging the shared action space, the robot policy acquires generalizable manipulation semantics from diverse human demonstrations without being biased by human-specific kinematics. This shared knowledge transfers across embodiments with different kinematic structures, enabling the same pretraining to benefit both parallel grippers and dexterous hands. EXE yields consistent gains over baseline without pretraining on RoboTwin 2.0, RoboCasa GR1, and real-world manipulation tasks.
Qingyuan Cai, Bo Zhang, Yongzhen Huang +1
ROBOTERA. · Beijing Normal University. · Tsinghua University.
Video-based policy learning is particularly promising, as it illustrates target behaviors without requiring action annotations or embodiment-matched demonstrations. A central challenge is deciding what information should be transferred from the video to the robot. Existing approaches commonly convert visual observations into scalar similarity or value signals, or ask foundation models to directly generate reward code. These approaches can make the temporal structure of a task difficult to inspect, ground, and reuse. We present Video2STL, a framework that converts observation-only videos into parametric Signal Temporal Logic (STL) specifications and uses the resulting formal representation for robot learning. A vision-language model extracts an embodiment-independent semantic event trace and constructs a bank of symbolic temporal specifications. The model determines the task structure, while numerical predicate thresholds and temporal bounds are grounded from successful robot trajectories. For policy learning, we separate short- and long-timescale temporal information: short-horizon specifications provide dense rewards through rolling-window quantitative robustness, while a causal monitor over a retained long-horizon specification provides one-time progress rewards for valid temporal prefixes. The same representation supports cross-embodiment transfer from human or animal videos to robot control. Across four manipulation tasks, Video2STL achieves 85.8% average success-once and 67.0% success-at-end, compared with 81.5%/59.5% for native dense PPO and 65.0%/42.3% for Text2Reward; in quadruped locomotion, Qwen-3.8 and GPT-5.6-based Video2STL policies achieve 100% success across velocities from 0.3 to 2.1m/s while remaining competitive in high-speed energy efficiency. Project webpage: video2stl.
Merve Atasever, Keyan Azbijari, Cagan Bakirci +6
Department of Computer Science, University of Southern California Los Angeles, CA, USA · University of Florida Gainesville, FL, USA