LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment

Dujun Nie; Fengjiao Chen; Qi Lv; Jun Kuang; Xiaoyu Li; Xuezhi Cao; Xunliang Cai

cs.CVApr 13, 2026

LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignment

Authors: Dujun Nie, Fengjiao Chen, Qi Lv, Jun Kuang, Xiaoyu Li, Xuezhi Cao, Xunliang Cai

Organizations: Meituan, Beijing, China

Abstract

While the shortage of explicit action data limits Vision-Language-Action (VLA) models, human action videos offer a scalable yet unlabeled data source. A critical challenge in utilizing large-scale human video datasets lies in transforming visual signals into ontology-independent representations, known as latent actions. However, the capacity of latent action representation to derive robust control from visual observations has yet to be rigorously evaluated. We introduce the Latent Action Representation Yielding (LARY) Benchmark, a unified framework for evaluating latent action representations on both high-level semantic actions (what to do) and low-level robotic control (how to do). The comprehensively curated dataset encompasses over one million videos (1,000 hours) spanning 151 action categories, alongside 620K image pairs and 595K motion trajectories across diverse embodiments and environments. Our experiments reveal two crucial insights: (i) General visual foundation models, trained without any action supervision, consistently outperform specialized embodied latent action models. (ii) Latent-based visual space is fundamentally better aligned to physical action space than pixel-based space. These results suggest that general visual representations inherently encode action-relevant knowledge for physical control, and that semantic-level abstraction serves as a fundamentally more effective pathway from vision to action than pixel-level reconstruction.

Figures & tables

Figure 1: LARYBench evaluates vision-to-action transformation on both action generalization and robotic control.

Figure 2: Overall pipeline of our benchmark LARY. First, (a) we construct a comprehensive dataset featuring both atomic and composite actions across varied domains to support classification and regression tasks. Next, (b) we extract the continuous latent action $z$ using various representation paradigms, also highlighting the integration of pre-trained general vision encoders within the VQ-VAE training architecture. Finally, (c) we evaluate these representations utilizing probe-based classification and latent-conditioned regression to quantify their semantic separability (Accuracy) and physical dynamics modeling capabilities (MSE), respectively.

Figure 3: Data curation process for LARY-Bench. To efficiently transform diverse raw videos into standardized evaluation tasks, we integrate a Vision-Language Model (VLM) with robust spatio-temporal video understanding capabilities. The VLM serves as the core reasoning agent for temporal video segmentation and semantic action alignment.

Model	Paradigm	Params(M)	Classification Accuracy $\uparrow$
Model	Paradigm	Params(M)	Avg.	Atomic Robot	Composite Human	Composite Robot
V-JEPA2 ( Assran et al., 2025 )	Semantic Encoder	303.89	76.62	79.09	80.35	70.43
DINOv3 ( Siméoni et al., 2025 )	Semantic Encoder	303.13	68.68	60.79	76.19	69.06
Wan2.2 ( Wan et al., 2025 )	Pixel Encoder	704.69	49.36	14.91	67.77	65.39
FLUX.2-dev ( Labs, 2024 )	Pixel Encoder	84.05	47.48	51.95	46.12	44.36
LAPA ( Ye et al., 2024 )	Embodied LAM	343.80	20.17	22.27	14.61	23.64

Table 1: Action Classification results of latent action representations. Avg. denotes the mean accuracy across Atomic Robot, Composite Human, and Composite Robot. Best results are in bold and second best are underlined .

Model	Paradigm	Params(M)	Regression MSE $\downarrow$
Model	Paradigm	Params(M)	Avg.	CALVIN	VLABench	RoboCOIN	AgiBotWorld-Beta
V-JEPA 2 ( Assran et al., 2025 )	Semantic Encoder	303.89	0.25	0.27	0.07	0.32	0.33
DINOv3 ( Siméoni et al., 2025 )	Semantic Encoder	303.13	0.19	0.22	0.06	0.22	0.24
Wan2.2 ( Wan et al., 2025 )	Pixel Encoder	704.69	0.30	0.39	0.09	0.34	0.39
FLUX.2-dev ( Labs, 2024 )	Pixel Encoder	84.05	0.35	0.25	0.04	0.47	0.62
LAPA ( Ye et al., 2024 )	Embodied LAM	343.80	0.97	0.96	0.95	0.96	1.00

Table 2: Action Regression results of latent action representations. Avg. is the mean MSE across all datasets. Best results are in bold and second best are underlined .

Figure 4: Performance Evolution of Latent Action Models. The (*) denotes the default quantization settings for LAPA-DINOv3 ( $cs=8,sl=16,dim=32$ ). $cs$ , $sl$ , and $dim$ represent Codebook Size, Sequence Length and Latent Dimension respectively. Composite Human classification on the left and AgiBotWorld-Beta AgiBot-World-Contributors et al. (2025) regression on the right.

Codebook Size	Recon Loss	Codebook Utilization (%)	Accuracy (Composite task) $\uparrow$		MSE (AgiBotWorld-Beta) $\downarrow$
Codebook Size	Recon Loss	Codebook Utilization (%)	Human	Robot	MSE (AgiBotWorld-Beta) $\downarrow$
8	0.00808	100.0	71.31	64.89	0.88
64	0.00751	100.0	70.15	64.04	0.83
256	0.00758	89.5	69.84	63.82	0.85

Table 3: Ablation on Codebook Size. Default settings are Sequence Length = 49, Latent Dimension = 256. Best results are in bold and second best are underlined .

Sequence Length	Recon Loss	Codebook Utilization (%)	Accuracy (Composite task) $\uparrow$		MSE (AgiBotWorld-Beta) $\downarrow$
Sequence Length	Recon Loss	Codebook Utilization (%)	Human	Robot	MSE (AgiBotWorld-Beta) $\downarrow$
16	0.00908	1.6	69.23	63.33	0.79
49	0.00751	100.0	70.15	64.04	0.83
64	0.00773	79.7	71.37	64.70	0.72

Table 4: Ablation on Sequence Length. Default settings are Codebook Size = 64, Latent Dimension = 256. Best results are in bold and second best are underlined .

Latent Dimension	Recon Loss	Codebook Utilization (%)	Accuracy (Composite task) $\uparrow$		MSE (AgiBotWorld-Beta) $\downarrow$
Latent Dimension	Recon Loss	Codebook Utilization (%)	Human	Robot	MSE (AgiBotWorld-Beta) $\downarrow$
32	0.01141	3.1	60.94	57.69	0.87
64	0.00967	100.0	66.94	63.07	0.83
256	0.00751	100.0	70.15	64.04	0.83
512	0.00732	1.6	71.25	64.97	0.80
1024	0.00670	84.4	72.55	65.78	0.81

Table 5: Ablation on Latent Dimension. Default settings are Codebook Size = 64, Sequence Length = 49. Best results are in bold and second best are underlined .

Figure 5: Action classification performance across the long-tail distribution of the Composite Human dataset. The background histogram indicates the number of samples per action class, sorted by descending frequency. Solid lines represent the moving average F1 scores, while transparent scatters denote exact class-wise scores.

Figure 6: Cross-attention heatmaps of the temporal pooler across various models for a 9-frame pour sequence. The spatial sequence length ( $SL$ ) dictates visual granularity: $SL=14*14$ yield fine-grained $14\times 14$ heatmaps, whereas LAMs ( $SL\in\{4,16\}$ ) produce blocky $2\times 2$ or $4\times 4$ attention regions. Temporally, $T=9$ models compute frame-by-frame attention; $T=8$ models extract pairwise latent actions (leaving the 9th frame unmodified); and models with temporal compression ( $T\in\{3,4\}$ ) duplicate attention maps across their corresponding frame groups.

Model	Paradigm	Params(M)	Regression MSE $\downarrow$
Model	Paradigm	Params(M)	stride=5	stride=15	stride=30	Avg.
V-JEPA 2 ( Assran et al., 2025 )	Semantic Encoder	303.89	0.07	0.13	0.16	0.12
DINOv3 ( Siméoni et al., 2025 )	Semantic Encoder	303.13	0.06	0.20	0.25	0.17
Wan2.2 ( Wan et al., 2025 )	Pixel Encoder	704.69	0.09	0.16	0.24	0.16
FLUX.2-dev ( Labs, 2024 )	Pixel Encoder	84.05	0.04	0.57	0.62	0.41
LAPA ( Ye et al., 2024 )	Embodied LAM	343.80	0.95	0.87	0.77	0.86

Table 6: Ablation study on VLABench ( Zhang et al., 2025b ) sampling stride. We report the regression MSE with sampling strides of 5, 15, and 30, and their average. Best results are in bold and second best are underlined .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: Left: Overall composition of the LARYBench dataset. Reg. and Classi. represent regression and classification. Ego and Exo represent Egocentric and Exocentric. AgiBot-B, EPIC, and SSv2 represent AgiBotWorld-Beta [ Gao et al., 2025 ] , EPIC-KITCHENS [ Damen et al., 2020 ] , and Something-Somethingv2 [ Goyal et al., 2017 ] . Right: Duration distribution of all videos (Composite tasks).

Figure 2: Distribution of the action categories of the Atomic Robot , Composite Human , and Composite Robot datasets.

Panel of Figure 3 — Figure 3: Word clouds illustrating the semantic diversity of the LARYBench dataset annotations. (a) Highlights the wide variety of actionable verbs indicating the interaction types, while (b) showcases the extensive range of nouns representing the manipulated objects across both human and robotic domains.

Figure 4: Representative examples of Kinematic Atomic Primitives from the Atomic Robot subset. The visualizations demonstrate precise, discrete end-effector movements and gripper state changes in structured exocentric environments.

Figure 5: Representative examples of Composite Human Behaviors.

Figure 6: Representative examples of Composite Robot Behaviors sourced from AgiBotWorld-Beta [ AgiBot-World-Contributors et al., 2025 ] .

Source Dataset	Domain	Viewpoint	Task Allocation	Data Format	Count	Proportion
Classification Track
LIBERO [ Liu et al., 2023 ]	Robot	Exo	Atomic	Image Pairs	25,940	2.1%
EgoDex [ Hoque et al., 2025 ]	Human	Ego	Composite	Video Clips	502,752	41.3%
SSv2 [ Goyal et al., 2017 ]	Human	Ego	Composite	Video Clips	106,911	8.8%
Ego4D [ Grauman et al., 2022 ]	Human	Ego	Composite	Video Clips	22,744	1.9%
HoloAssist [ Wang et al., 2023 ]	Human	Ego	Composite	Video Clips	10,825	0.9%

Appendix

Table 1: Detailed composition and statistics of the LARY Benchmark dataset. The dataset is divided into two primary tracks: Classification (including both composite videos and atomic image pairs) and Regression. Proportions are calculated independently for each track.

Figure 7: Cross-domain performance gap analysis across shared action categories. The heatmap visualizes the F1 score discrepancy ( $\Delta\text{F1}=\text{Robot}-\text{Human}$ ) between human and robot domains. Red regions highlight superior performance on human actions, whereas blue regions indicate better performance on the corresponding robot actions. Absolute F1 scores for Human (H) and Robot (R) are explicitly reported within each cell.

Figure 9: A case for the action compress .

Figure 10: A case for the action plug .All models failed.

Explore similar work

LARA: Latent Action Representation Alignment for Vision-Language-Action Models
Jun 5, 2026Mengya Liu, Baoxiong Jia, Jiangyong Huang +2Latent Action Learning Robotic Manipulation
State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence
From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models
May 6, 2026Yihan Lin, Haoyang Li, Yang Li +4Latent Action Learning Vision-Language-Action Models
School of Information, Renmin University of China, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China · Engineering Research Center of Database and Business Intelligence, Beijing, China
From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data
May 18, 2026Zhiyuan Feng, Qixiu Li, Huizhi Liang +12Language-Conditioned Robot Manipulation Cross-Embodiment Robot Learning
1Tsinghua University · 5Microsoft Research Asia · 2HKUST +4

Jun 5, 2026J/K move · Enter open · S save

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

Mengya Liu, Baoxiong Jia, Jiangyong Huang, Jingze Zhang+1

State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence

Latent Action Learning Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Visual-language action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks.

Abstract

Explore similar work

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models

From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data