LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos

Zhiyuan Xiang; Peidong Liu; Mingyang Li; Jiawei Li; Wenhow Li; Wenbo Ding; Xiang Deng

cs.CVOct 9, 2026

LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos

Authors: Zhiyuan Xiang, Peidong Liu, Mingyang Li, Jiawei Li, Wenhow Li, Wenbo Ding, Xiang Deng

Organizations: Joy Future Academy, JD · School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Tsinghua SIGS, Tsinghua University · Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Scaling generalist robotic policies is limited by the high cost of collecting embodied interaction data. Large-scale egocentric human videos offer a rich alternative, but they lack robot action labels, and existing action annotation methods are often noisy under real-world visual complexity and diverse manipulation dynamics. We introduce LAPT-Scale, a scalable latent action pre-training paradigm with two components: LAPT-Scale-LAM learns continuous latent actions by explaining frame-to-frame visual transitions in human videos and uses them as human-derived supervision, while LAPT-Scale-Policy trains a VLA policy through human-video latent-action pre-training, target-robot embodiment alignment, and task-specific post-training. Unlike prior vector-quantized LAMs, LAPT-Scale-LAM uses a continuous latent space to preserve fine-grained motion and interaction dynamics. We further combine perceptual, semantic, and optical-flow alignment signals to improve transition-level supervision beyond pixel reconstruction. Trained on 23.3K hours of egocentric video, LAPT-Scale-LAM provides implicit action supervision for policy pre-training, yielding consistent scaling gains and improving real-robot success from 63.3% to 85.0% as annotated human video scales from 1K to 5K hours. The learned policy also generalizes to unseen visual conditions.

Figures & tables

Figure 1: Overview of LAPT-Scale. LAPT-Scale-LAM annotates egocentric human videos with continuous implicit actions. LAPT-Scale-Policy uses these annotations for human-video pre-training, followed by target-robot embodiment alignment and downstream post-training.

Figure 2: Human-video data overview. The corpus spans everyday egocentric manipulation across varied environments and interactions, providing broad coverage for latent-action learning.

Figure 3: LAPT-Scale-LAM training alignments. The latent action explains the transition from the current frame to the next frame. Multi-level alignments encourage it to capture hand-object motion rather than static appearance.

Method	$\pi_{0}$ [ 14 ]		$\pi_{0.5}$ [ 15 ]		Motus [ 7 ]		LingBot-VLA [ 18 ]		JoyAI-RA [ 44 ]		LAPT-Scale
	Easy	Hard	Easy	Hard	Easy	Hard	Easy	Hard	Easy	Hard	Easy	Hard
Success Rate (%)	65.9	58.4	82.7	76.8	88.7	87.0	88.6	86.7	88.6	87.2	90.0	89.3

Table 1: Main downstream policy results on RoboTwin 2.0. Results are average success rates in the Easy and Hard settings.

Method	GR00T-N1.6 [ 16 ]	Qwen3PI [ 48 ]	ABot-M0 [ 19 ]	JoyAI-RA [ 44 ]	LAPT-Scale
Success Rate (%)	47.6	43.9	58.3	63.2	64.8

Table 2: Main downstream policy results on RoboCasa-GR1. Results are average success rates.

Diffusion Policy [ 49 ] : Mouse 50.0, Head. 40.0, Cup 45.0, Avg. 45.0.
Method	$\pi_{0.5}$ [ 15 ]				JoyAI-RA [ 44 ]				LAPT-Scale
	Mouse	Head.	Cup	Avg.	Mouse	Head.	Cup	Avg.	Mouse	Head.	Cup	Avg.
Success Rate (%)	55.0	40.0	75.0	56.7	80.0	95.0	65.0	80.0	85.0	95.0	75.0	85.0

Table 3: Main downstream policy results on the three original real-robot tasks, with 20 trials per task. Diffusion Policy uses the same 250-hour robot corpus.

Figure 4: Scaling behavior of LAPT-Scale. Left: policy pre-training loss for different amounts of human-video pre-training data. Right: real-robot average success rate as the amount of human-video pre-training data increases, with all robot data held fixed.

Stage-1 supervision	Stage-2 training	Success (%)
None	Standard	55.0
None	Extended, compute-matched	53.3
LAPT-Scale, 5K hours	Standard	85.0

Table 4: Stage-1 controls: average success on the three original real-robot tasks.

LAM objective	PSNR	LARY	RT-H (%)
MSE-only (Recon.+KL)	33.9844	0.675	74.2
Full w/o LPIPS	34.2105	0.691	82.5
Full w/o DINO	34.1500	0.698	84.1
Full w/o flow	34.3012	0.685	77.8
Full objective	34.4217	0.713	89.3

Table 5: Controlled LAM alignment ablation. RT-H is RoboTwin Hard success.

Method	Seen	Position	Illumination	Background	Average
Diffusion Policy	60.0	20.0	15.0	15.0	27.5
$\pi_{0.5}$	85.0	50.0	45.0	40.0	55.0
LAPT-Scale	85.0	75.0	70.0	70.0	75.0

Table 6: Four-task generalization success (%); Average includes all four domains.

LAM Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	LARY Acc. $\uparrow$
VQ-LAM	18.3416	0.6741	0.3443	0.232
MSE-only LAM	33.9844	0.9234	0.0769	0.675
LAPT-Scale-LAM	34.4217	0.9326	0.0546	0.713

Table 7: Human-video LAM evaluation. Reconstruction metrics are measured on human-video transitions; LARY reports first-person human-action classification accuracy from frozen latent actions.

Figure 5: Human-video transition reconstruction. Given the previous frame, VQ-LAM, MSE-only LAM, and LAPT-Scale-LAM predict the next frame, shown against the ground truth. Additional visualizations are provided in the appendix.

Figure 6: Human ego-view latent-action replacement across three donor-target pairs. Rows show donor self-rollout frames, the target base prediction, and the target prediction after replacing the target latent action with the donor latent action. Across all three examples, the replacement changes the short-horizon motion pattern while preserving the overall target scene layout.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Primary Content	Role in Corpus	Raw Hours	Cleaned Stage-1 Hours
Egocentric-10K [ 30 ]	Filtered egocentric activity clips	Additional LAM coverage; excluded from Stage-1 scaling because it lacks hand-pose annotations needed by the hand-motion baseline	10.0K	0.0K
EgoDex [ 32 ]	Dexterous egocentric manipulation	Fine-grained hand-object motion	0.7K	0.5K
Ego4D filtered by EgoVid [ 1 , 34 ]	Ego4D clips retained by EgoVid-style filtering	Broad daily activity with stronger manipulation relevance	1.8K	0.8K
Xperience [ 31 ]	Large-scale egocentric multimodal activity data	Scene, object, and motion diversity	1.1K	0.4K
EgoVerse [ 33 ]	Diverse egocentric robot-learning videos	Cross-scene and cross-task diversity	1.2K	0.5K
Self-collected human videos	Egocentric human manipulation videos collected by us	Target-relevant activities and camera viewpoints	8.5K	2.9K

Appendix

Table 8: Human-video source composition. Raw hours denote the source mixture used to form the 23.3K-hour LAM training corpus. The cleaned subset column denotes the portion retained after manipulation-focused filtering for Stage-1 policy pre-training.

Component	Item	Value
Human-video corpus	LAM training data	23.3K hours
Human-video corpus	Cleaned Stage-1 subset	5.1K hours
Human-video corpus	Scaling subsets	1K, 3K, and 5K hours
LAM	Latent action dimension	32
LAM	Latent type	Continuous Gaussian posterior
LAM	Prediction target	Next-frame reconstruction

Appendix

Table 9: Implementation, training, and compute details for LAPT-Scale-LAM training, offline annotation, and policy training.

Stage	Frozen modules	Batch size	Base LR	Qwen-VL LR	Action LR	Time
Stage 1	None	1024 (32/GPU $\times$ 32)	$2.5{\times}10^{-5}$	$1{\times}10^{-5}$	$1{\times}10^{-4}$	43h
Stage 2	Qwen-VL LM backbone	1024 (32/GPU $\times$ 32)	$2.5{\times}10^{-5}$	$1{\times}10^{-5}$	$1{\times}10^{-4}$	50h

Appendix

Table 10: Policy pre-training and embodiment-alignment configuration. Learning rates are reported for the base, Qwen-VL interface, and action-model parameter groups.

Task	Batch size	Base LR	Qwen-VL LR	Action LR	Time
Mouse Pad	64 (8/GPU $\times$ 8)	$3{\times}10^{-5}$	$1{\times}10^{-5}$	$1{\times}10^{-4}$	7h
Headphones	64 (8/GPU $\times$ 8)	$3{\times}10^{-5}$	$1{\times}10^{-5}$	$1{\times}10^{-4}$	7h
Cup	64 (8/GPU $\times$ 8)	$3{\times}10^{-5}$	$1{\times}10^{-5}$	$1{\times}10^{-4}$	8h

Appendix

Table 11: Real-robot post-training configuration. All three tasks use full fine-tuning, task-specific robot demonstrations, and 30K training steps.

Method	Target / Latent Type	Training Signal	Used For
Hand-motion annotation	Reconstructed hand trajectory; explicit pseudo-action with hand pose	Hand-pose trajectory fitting	Stage-1 policy scaling
DreamDojo LAM	Implicit transition action; latent action without hand pose	Pixel-level future-frame reconstruction	Stage-1 policy scaling
VQ-LAM	Implicit transition action; discrete codebook without hand pose	Future-frame reconstruction with vector quantization	LAM diagnostic baseline
MSE-only LAM	Implicit transition action; continuous Gaussian without hand pose	MSE future-frame reconstruction and KL	LAM diagnostic baseline
LAPT-Scale-LAM	Implicit transition action; continuous Gaussian without hand pose	MSE, LPIPS, DINO, optical-flow, and KL alignment	Stage-1 policy scaling and LAM diagnostics

Appendix

Table 12: Implementation comparison of human-video supervision baselines. All policy baselines share the same VLA backbone, action expert, robot alignment data, and downstream training schedule.

Benchmark	Setting	Metric	Evaluation scope	Success Criterion
RoboTwin 2.0	Easy	Success rate	Official task suite	Benchmark detector
RoboTwin 2.0	Hard	Success rate	Official task suite	Benchmark detector
RoboCasa-GR1	Tabletop	Success rate	24 tasks $\times$ 50 rollouts	Benchmark detector
Real robot	Mouse Pad	Success rate	20	Task completion
Real robot	Headphones	Success rate	20	Task completion
Real robot	Cup	Success rate	20	Task completion

Appendix

Table 13: Summary of downstream evaluation protocols.

Stage-1 supervision	0 hours	1K hours	3K hours	5K hours
Hand motion	55.0	58.3	75.0	55.0
DreamDojo LAM	55.0	61.7	75.0	76.7
LAPT-Scale-LAM	55.0	63.3	80.0	85.0

Appendix

Table 14: Stage-1 scaling on the original real-robot tasks. Each entry is an average success rate (%) over three tasks with 20 trials per task. The shared 0-hour column is the same robot-only control.

Human Supervision	Human Video	Mouse Pad	Headphones	Cup Task	Avg.
Hand-motion annotation	1K hours	45.0	80.0	50.0	58.3
Hand-motion annotation	3K hours	80.0	45.0	100.0	75.0
Hand-motion annotation	5K hours	75.0	50.0	40.0	55.0
DreamDojo LAM	1K hours	60.0	55.0	70.0	61.7
DreamDojo LAM	3K hours	80.0	85.0	60.0	75.0
DreamDojo LAM	5K hours	65.0	75.0	90.0	76.7

Appendix

Table 15: Per-task breakdown for the real-robot scaling study. Human-video scale denotes the amount of annotated human video used for VLA pre-training; all robot data is held fixed. Each task is evaluated over 20 trials.

Stage-1 supervision	Stage-2 training	Average success (%)
None (robot-only)	Standard	55.0
None (robot-only)	Extended, compute-matched	53.3
LAPT-Scale, 5K hours	Standard	85.0

Appendix

Table 16: Stage-1 removal and compute-matched controls on real-robot tasks.

Protocol	LAM Variant	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Teacher-forced	LAPA	18.3416	0.6741	0.3443
Teacher-forced	VILLA-X	26.9595	0.7973	0.1685
Teacher-forced	MSE-only LAM	33.9844	0.9234	0.0769
Teacher-forced	LAPT-Scale-LAM	34.4217	0.9326	0.0546
Autoregressive	LAPA	15.8607	0.5551	0.4593
Autoregressive	VILLA-X	22.5951	0.7008	0.3228

Appendix

Table 17: LAM reconstruction comparison under teacher-forced and autoregressive rollout protocols. Teacher-forced evaluation conditions each prediction on the ground-truth previous frame, while autoregressive evaluation starts from the ground-truth first frame and then conditions on the model’s previous prediction. Higher PSNR and SSIM are better; lower LPIPS is better.

Figure 7: Teacher-forced LAM reconstruction comparison. Each prediction conditions on the ground-truth previous frame. LAPA and VILLA-X show weaker local transition fidelity, MSE-only LAM is smoother around hand contacts, and LAPT-Scale-LAM preserves sharper hand-object structure.

Figure 8: Autoregressive LAM rollout comparison. Later predictions condition on the model’s own generated frames. Error accumulation makes the differences more visible: MSE-only LAM gradually loses fine hand structure, while LAPT-Scale-LAM preserves a visible hand region and more coherent short-horizon object motion.

Panel of Figure 9 — Figure 9: Latent-action clusters with shared motion patterns across different data sources. Each panel shows examples grouped by the learned latent-action representation. Clusters are computed from 50K motion-filtered 16-frame windows after PCA reduction to 64 dimensions and 50-way clustering. Although the visual scenes and objects vary, samples within a cluster exhibit similar short-horizon motion patterns, suggesting that LAPT-Scale-LAM captures transition-level structure beyond dataset-specific appearance.

Rollout Target	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
VLA-predicted latent rollout	25.5371	0.8137	0.3033
Ground-truth latent rollout	30.7705	0.8982	0.2271

Appendix

Table 18: Stage-1 latent-action rollout quality on human videos. Higher PSNR and SSIM are better; lower LPIPS is better.

Figure 10: Stage-1 latent-action rollout on human videos: dice rolling. Frames are sampled every six video frames and arranged from left to right within each row. The first row shows the ground-truth video, the second row shows decoder rollout using the ground-truth latent action encoded by LAPT-Scale-LAM, and the third row shows decoder rollout using the latent action predicted by the Stage-1 VLA policy.

Figure 11: Stage-1 latent-action rollout on human videos: sandwich making. Frames are sampled every six video frames and arranged from left to right within each row. The Stage-1 predictions produce rollouts with motion patterns close to the ground-truth-latent rollout and the observed video, indicating that human-video pre-training learns meaningful human motion representations rather than only static visual appearance.

Figure 12: Original evaluation domain and the three unseen generalization shifts.

Method	Seen	Position	Illumination	Background	Average
Diffusion Policy	60.0	20.0	15.0	15.0	27.5
$\pi_{0.5}$	85.0	50.0	45.0	40.0	55.0
LAPT-Scale	85.0	75.0	70.0	70.0	75.0

Appendix

Table 19: Four-task real-robot generalization (20 trials per task per domain).

Explore similar work

WLA $^3$ : World Latent Action Modeling for Semantics, Dynamics, and Kinematics
Sep 14, 2026Peidong Liu, Zhiyuan Xiang, Mingyang Li +4Latent Action Learning World Models for Robotics
Joy Future Academy, JD Group
Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos
Jun 17, 2026Runze Xu, Yiluo Zhang, Jian Wang +2Latent Action Learning VLM Adaptation
Department of Electronic Engineering, Tsinghua University · Tianfu Jiangxi Laboratory
From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data
May 18, 2026Zhiyuan Feng, Qixiu Li, Huizhi Liang +12Language-Conditioned Robot Manipulation Cross-Embodiment Robot Learning
1Tsinghua University · 5Microsoft Research Asia · 2HKUST +4

Sep 14, 2026J/K move · Enter open · S save

WLA $^3$ : World Latent Action Modeling for Semantics, Dynamics, and Kinematics

Peidong Liu, Zhiyuan Xiang, Mingyang Li, Wenhao Li+3

Joy Future Academy, JD Group

Latent Action Learning World Models for Robotics Cross-Embodiment Robot Learning Vision-Language-Action Models

Open paper AskPDF arXiv

Scaling generalist policy models with heterogeneous data is limited by the lack of unified, low-noise action supervision. Human egocentric videos are abundant, but only a small fraction comes with high-quality hand-action labels. Observed world transitions offer a common source of action-related supervision across data sources. We introduce WLA

^3

(World Latent Action Modeling for Semantics, Dynamics, and Kinematics), a unified generalist policy model framework built around representations learned by a World Latent Action Model (WLAM). WLAM first learns how multimodal world states change over a local interval, encoding synchronized camera views and available embodiment-state changes into a compact local latent action and a richer transition feature. Reconstruction from partial modalities and consistency across overlapping windows encourage robust transition representations. WLA

^3

reuses them across semantics, dynamics, and kinematics: local latent actions support action-sensitive physical-dynamics modeling, segment-level features directly supervise the VLM through a Semantic Latent Aggregate (SLA), and an action expert jointly predicts latent actions together with embodiment-specific robot controls. Human videos provide scalable transition supervision, while robot trajectories ground the shared representation in executable native controls. On LARYBench, the final 32D latent action reaches 67.89% average classification accuracy. WLA

^3

achieves 81.9% average success across six real-robot tasks versus 66.2% for

π_{0.5}

. Performance improves as generalist policy model mid-training data scales, and human videos support human-to-robot transfer. Project page can be found at https://wla-3.github.io/.

Abstract

Explore similar work

WLA3^33: World Latent Action Modeling for Semantics, Dynamics, and Kinematics

Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos

From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data

WLA $^3$ : World Latent Action Modeling for Semantics, Dynamics, and Kinematics