Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wild

Hao Luo; Ye Wang; Wanpeng Zhang; Haoqi Yuan; Yicheng Feng; Haiweng Xu; Sipeng Zheng; Zongqing Lu

cs.ROFeb 25, 2026

Joint-Aligned Latent Action: Towards Scalable VLA Pretraining in the Wild

Authors: Hao Luo, Ye Wang, Wanpeng Zhang, Haoqi Yuan, Yicheng Feng, Haiweng Xu, Sipeng Zheng, Zongqing Lu

Organizations: Peking University · BeingBeyond · Renmin University of China

Abstract

Despite progress, Vision-Language-Action models (VLAs) are limited by a scarcity of large-scale, diverse robot data. While human manipulation videos offer a rich alternative, existing methods are forced to choose between small, precisely-labeled datasets and vast in-the-wild footage with unreliable hand tracking labels. We present JALA, a pretraining framework that learns Jointly-Aligned Latent Actions. JALA bypasses full visual dynamic reconstruction, instead learns a predictive action embedding aligned with both inverse dynamics and real actions. This yields a transition-aware, behavior-centric latent space for learning from heterogeneous human data. We scale this approach with UniHand-Mix, a 7.5M video corpus (>2,000 hours) blending laboratory and in-the-wild footage. Experiments demonstrate that JALA generates more realistic hand motions in both controlled and unconstrained scenarios, significantly improving downstream robot manipulation performance in both simulation and real-world tasks. These results indicate that jointly-aligned latent actions offer a scalable pathway for VLA pretraining from human data.

Figures & tables

Figure 1: Comparison of VLA’s latent action paradigms with human videos. (left) Prior reconstruction-based methods like LAPA [ 19 ] rely on multi-stage pipelines extracting latent actions via dynamics reconstruction as pseudo-labels. (middle) Our JALA introduces predictive embeddings aligned with latent actions. (right) Transformer-based JALA implementation where intermediate hidden states serve as the predictive embeddings to align with latent actions, while output tokens use available action labels as supervision.

Figure 2: The JALA framework . Pre-training (left) : Hidden states of masked motion chunks serve as predictive embeddings to align with latent actions from boundary frames. The Latent Action Perceiver (LAP) maps boundary frames to latent action space, providing supervision without action labels. A parameter-shared Latent State Perceiver (LSP) injects initial frame context, with LAP and LSP linked via decoupled EMA update for stability. Post-training (right) : The predictive embeddings are fed into a flow-matching head for robot task transfer.

Figure 3: Dataset statistics of UniHand-Mix. Top-left: distribution of data types (motion generation, video-only, motion description, and motion continuation). Bottom-left: distribution of clip lengths (1–10 seconds). Center: data source distribution across 8 data sources with a donut percentage chart on the right.

Model	MPJPE $\downarrow$		PA-MPJPE $\downarrow$		MWTE $\downarrow$		MDE $\downarrow$
	Lab	Wild	Lab	Wild	Lab	Wild	Lab	Wild
# Next Token Prediction
Being-H0	7.61	16.91	1.34	3.81	6.03	14.54	7.16	18.33
Being-H0+dino	7.54	15.14	0.90	2.78	5.85	13.65	6.95	17.17
# Masked Chunk Prediction
JALA w/o align	7.72	15.73	0.89	2.34	6.18	14.02	8.09	16.23

Table 1: Comparison of hand motion generation and prediction tasks on both Lab and Wild splits.

Figure 4: t-SNE of predictive embeddings

h

and latent actions

z

across Lab and Wild. The two spaces cluster in closely aligned regions, and Wild samples largely expand the Lab manifold.

Figure 5: Qualitative hand-motion generation on lab (left column) and wild (right column) scenes. Colored overlays denote generated hand poses.

Model	Spatial	Object	Goal	Long	Average
LAPA [ 19 ]	83.4	87.6	78.2	68.8	79.5
MolmoAct [ 63 ]	87.0	95.4	87.6	77.2	86.6
$\pi$ 0-FAST [ 32 ]	96.4	96.8	88.6	60.2	85.5
GR00T N1.5 [ 3 ]	94.4	97.6	93.0	90.6	93.9
$\pi$ 0 [ 2 ]	96.8	98.8	95.8	85.2	94.2
UniVLA [ 39 ]	95.4	98.8	93.6	94.0	95.5

Table 2: Two-view LIBERO results showing success rates (%) across task categories and overall average. Blue rows denote models trained on the action-available subset only, while green rows denote models pretrained on the full UniHand-Mix. LAPA † is LAPA retrained on our data with the JALA backbone.

Model	Spatial	Object	Goal	Long	Average
# $>$ 3B Backbones
LAPA ⋆ [ 19 ]	73.8	74.6	58.8	55.4	65.7
OpenVLA [ 8 ]	84.7	88.4	79.2	53.7	76.5
TriVLA [ 65 ]	91.2	93.8	89.8	73.2	87.0
4D-VLA [ 66 ]	88.9	95.2	90.9	79.1	88.5
UniVLA-human † [ 39 ]	91.2	94.2	90.2	79.4	88.7

Table 3: Single-view LIBERO results showing success rates (%) across task categories and overall average. Models are grouped by size:

>

3B parameters (top) and

\leq

3B parameters (bottom). The results of LAPA ⋆ are reported in UniVLA. UniVLA-human † uses only human pretraining data. UniVLA-full †† incorporates Bridge-V2 [ 64 ] for pretraining.

Model	RoboCasa Syn.	RoboCasa Human	GR1 Tabletop
GR00T N1.5	20.83	35.17	20.41
LAPA	16.25	22.42	11.42
Being-H0	23.83	31.33	12.91
JALA-act	24.92	32.42	20.25
JALA w/o dec.	14.25	19.33	9.25
LAPA †	20.25	27.33	13.50

Table 4: Results on the RoboCasa benchmark and GR1 tabletop tasks. We report success rates (%). Blue rows denote models trained on the action-available subset only, while green rows denote models pretrained on the full UniHand-Mix.

Figure 6: Ablation studies on JALA-dino evaluated on LIBERO. Left: performance across various the proportion of in-the-wild data used during pretraining (0%, 25%, 50%, 100%). Right: performance when feeding hidden states from different backbone layers (14, 19, 24, 28) into the flow-matching head during adaptation.

Figure 7: Teleoperated demonstration examples for the three real-world manipulation tasks. Top : Put-Three-Obj (open drawer $\to$ pick&place three fruits $\to$ close drawer). Middle : Wipe-Board (grasp cloth $\to$ wipe over marked region $\to$ remove visible ink). Bottom : Water-Plant (grasp spray bottle $\to$ reposition toward plant $\to$ activate trigger).

Figure 8: Real-world robot task settings for three multi-step tasks. Put-Three-Obj and Wipe-Board include unseen variants to evaluate robustness to visual shifts.

Figure 9: Successful real-robot rollouts (seen setting). Top: Put-Three-Obj ; Middle: Wipe-Board ; Bottom: Water-Plant .

Figure 10: Successful real-robot rollouts (unseen setting). Put-Three-Obj : the policy corrects an initial misalignment under altered tablecloth texture. Wipe-Board : the policy adaptively revisits residual ink under changed marker color.

Figure 11: Failure cases across real-robot tasks . Put-Three-Obj : spatial misalignment, incomplete contact, and unstable grasp. Wipe-Board : insufficient planar contact and persistent residual ink. Water-Plant : incorrect affordance reasoning leading to wrong bottle orientation.

Model	Put-Three-Obj		Wipe-Board		Water-Plant
	Seen	Unseen	Seen	Unseen	Seen
Being-H0	38.0	16.0	40.0	33.3	36.7
GR00T N1.5	48.0	28.0	56.7	43.3	53.3
JALA w/o align	40.0	32.0	53.3	43.3	56.7
JALA-vjepa	38.0	34.0	66.7	60.0	66.7
JALA-dino	60.0	58.0	83.3	80.0	73.3

Table 5: Real-world robot performance measured by average subtask completion rate (%) on three multi-step manipulation tasks. Each policy is evaluated over 10 rollouts per task.

Explore similar work

LARA: Latent Action Representation Alignment for Vision-Language-Action Models
Jun 5, 2026Mengya Liu, Baoxiong Jia, Jiangyong Huang +2Latent Action Learning Robotic Manipulation
State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence
From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data
May 18, 2026Zhiyuan Feng, Qixiu Li, Huizhi Liang +12Language-Conditioned Robot Manipulation Cross-Embodiment Robot Learning
1Tsinghua University · 5Microsoft Research Asia · 2HKUST +4
Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos
Jun 17, 2026Runze Xu, Yiluo Zhang, Jian Wang +2Latent Action Learning VLM Adaptation
Department of Electronic Engineering, Tsinghua University · Tianfu Jiangxi Laboratory

Jun 5, 2026J/K move · Enter open · S save

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

Mengya Liu, Baoxiong Jia, Jiangyong Huang, Jingze Zhang+1

State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence

Latent Action Learning Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Visual-language action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks.

Abstract

Explore similar work

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data

Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos