EgoLAP: Learning from Egocentric Human Data through Language-Action Reasoning

Lihan Zha; Shresth Grover; Tenny Yin; Samuel M. Bateman; Hengkai Pan; Mengchao Zhang; Aykut Onol; Allen Z. Ren

cs.ROOct 6, 2026

EgoLAP: Learning from Egocentric Human Data through Language-Action Reasoning

Authors: Lihan Zha, Shresth Grover, Tenny Yin, Samuel M. Bateman, Hengkai Pan, Mengchao Zhang, Aykut Onol, Allen Z. Ren, +2 more

Organizations: Princeton University · Toyota Research Institute · Physical Intelligence

Abstract

Egocentric human data offer a path to scaling robot learning beyond costly robot demonstrations, yet the embodiment gap makes raw human trajectories a poor supervisory target for control. Our key insight is that, although low-level actions are embodiment-specific, their underlying motion intent can capture task-relevant structure that transfers across humans and robots. We introduce EgoLAP, a VLA pre-training framework that jointly learns from human and robot trajectories through a shared language-based action chain-of-thought. EgoLAP expresses motion intent as structured, temporally abstracted language actions and pairs them with motion-level reasoning grounded in scene geometry, physics, and object affordances. Across extensive real-world and simulated experiments, EgoLAP transfers human experience to robot control more effectively than alternative action representations and reaches 80.1% mean real-world task progress, a 2.3x performance gain over alternative action representations. Motion-level reasoning also outperforms a composite reasoning format that combines subtask, object-box, and visual-trace reasoning.

Figures & tables

Figure 1: EgoLAP turns egocentric human experience into transferable robot control. Human hands and robots differ in embodiment and low-level execution, yet share the same underlying motion intent when performing a task. EgoLAP captures this intent through motion-level reasoning and language actions, providing a shared supervision target across both data sources and leading to a $2.3\times$ performance gain over alternative action representations.

Figure 2: EgoLAP places human and robot motion in a shared, physically grounded language space. By capturing the physical relationship between the current state and the motion needed to produce the intended effect, motion-level reasoning provides supervision that connects directly to low-level actions that transfers across embodiments.

Figure 3: Attention and gradient structure. Orange marks allowed attention, gray blocked attention, and blue attention whose gradients are stopped at the VLM boundary. Reasoning and language action tokens are causally masked and condition on the observation prefix; action-expert tokens never attend to them, so control requires no text generation.

Figure 4: Zero-shot evaluation on bimanual YAM robots. EgoLAP achieves 80.1% mean progress on real-world tasks—a $2.3\times$ gain over alternative action representations—and a 38.3% success rate on simulated tasks, outperforming ABC-VLA [ 1 ] by $1.8\times$ . When paired with language actions, motion-level reasoning yields a 35.6-point improvement, substantially greater than the gains achieved with other action representations.

Figure 5: Human data helps with EgoLAP. Adding egocentric human data more than doubles simulation success for the same language action policy.

Training data	No reasoning	Reasoning
Robot only	7.2	18.4
Robot + human	16.4	38.3

Table 1: Human data strengthens reasoning. Zero-shot success (%) on the 14-task simulation suite.

Figure 6: Motion-level reasoning is what carries the gain. Motion-level reasoning outperforms ECoT-style reasoning [ 54 ] , and augments it by a large margin.

Figure 7: GRPO over language actions over simulation tasks.

Retrieval	w/o	EgoLAP	$\Delta$
R $\rightarrow$ H $\cdot$ 3w R@1	51	76	$\mathbf{+25}$
H $\rightarrow$ R $\cdot$ 3w R@1	42	63	$\mathbf{+21}$
R $\rightarrow$ H $\cdot$ 2w R@1	69	92	$\mathbf{+23}$
R $\rightarrow$ H $\cdot$ 3w MRR	61	80	$\mathbf{+19}$

Table 2: Motion-level reasoning improves cross-embodiment skill retrieval. All values are percentages.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Detailed EgoLAP architecture. A shared SigLIP encoder maps each camera view to visual tokens. The 2B VLM branch predicts motion-level reasoning and the language action; the 300M action expert maps a noisy action chunk to its flow velocity. The two branches hold separate parameters but join the same masked-attention operation at every transformer layer. Action tokens attend to the observation, instruction, and discretized state, never to the textual targets, and gradients through the VLM keys and values are stopped.

Component	Positive direction	Negative direction
Translation $x$	move forward $n$ cm	move back $n$ cm
Translation $z$	move up $n$ cm	move down $n$ cm
Translation $y$	move left $n$ cm	move right $n$ cm
Roll	tilt left $\theta$ degrees	tilt right $\theta$ degrees
Pitch	tilt back $\theta$ degrees	tilt forward $\theta$ degrees
Yaw	rotate counterclockwise $\theta$ degrees	rotate clockwise $\theta$ degrees

Appendix

Table 3: Canonical component vocabulary for language actions. Here

n

is an integer number of centimeters and

\theta

is an integer multiple of

5

degrees. A component is omitted when its rounded magnitude is zero.

Hyperparameter	Value
Batch size	2048
Hardware	64 $\times$ TPU v6e
Training steps	40k
Learning-rate schedule	2500-step linear warmup, then constant
Peak learning rate	$5\times 10^{-5}$
Optimizer	Adam ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.95$ , $\epsilon{=}10^{-8}$ )

Appendix

Table 4: Shared policy pre-training hyperparameters.

Figure 9: Training data mixture. Inner ring: group weights; outer ring: individual dataset contributions within each group. Only MECKA ( $20\%$ ) and DROID ( $12\%$ ) carry motion-level reasoning labels; the remaining samples are supervised with language actions and continuous actions alone.

Setting	Value
Observations per update	32
Completions per observation	64 (2,048 responses per update)
Gradient accumulation	16 microbatches of 128 responses, one step per rollout bank
Optimizer	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.95$ , $\epsilon{=}10^{-8}$ ), weight decay 0
Learning rate	linear warmup $5\times 10^{-8}\!\rightarrow\!10^{-6}$ over 20 updates, then constant
Gradient clipping	global norm 1.0, after accumulation

Appendix

Table 5: Shared language-action GRPO settings.

Figure 10: Masking the action-expert loss on human samples hurts. Simulation success over the 14-task suite for EgoLAP (“no human mask”, the main model) and for a matched variant that withholds the flow-matching loss on human samples (“use human mask”). Both variants receive identical language action and motion-level reasoning supervision; they differ only in whether the continuous action expert is trained on human motion. We compare the 25k step checkpoint for both models. Error bars are one binomial standard error.

Figure 11: Per-task zero-shot simulation success. Rows expand the same 14-task suite summarized in Fig. 4 (a), grouped as Pick, Place, Color, and Fold. Columns show ABC-VLA and the reasoning and no-reasoning variants of EgoLAP, EgoFAST, and EgoOAT. Each cell reports the empirical task success rate, with darker shading indicating higher success.

Representation	Reasoning	Robot only	Robot + human	Human-data gain
Language	No	7.2%	16.4%	+9.2 pp
Language	Yes	18.4%	38.3%	+19.9 pp
FAST	Yes	1.4%	8.7%	+7.3 pp

Appendix

Table 6: Language actions benefit more from human data under motion-level reasoning supervision. Zero-shot success on the 14-task simulation suite. Human-data gains are absolute percentage-point differences. Reasoning denotes motion-level reasoning supervision.

Explore similar work

ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining
Jun 15, 2026Hao Li, Ganlong Zhao, Yufei Liu +8Robotic Data Acquisition Egocentric Video
ACE Robotics · CUHK MMLab · SJTU +2
HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos
May 24, 2026Zhi Wang, Botao He, Kelin Yu +4Human-To-Robot Transfer Egocentric Video
University of Maryland
Ego-Pi: VLA Fine-Tuning for Ego-Centric Human and Robot Data
Jun 6, 2026Ji Woong Kim, Ke Wang, Zipeng Fu +4Egocentric Dataset Humanoid
1Stanford University · 2Meta

Jun 15, 2026, cs.ROJ/K move · Enter open · S save

ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining

Hao Li, Ganlong Zhao, Yufei Liu, Haotian Hou+7

ACE Robotics · CUHK MMLab · SJTU+2

Robotic Data Acquisition Egocentric Video Pretraining

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.

Abstract

Explore similar work

ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining

HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos

Ego-Pi: VLA Fine-Tuning for Ego-Centric Human and Robot Data