Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.
Figures & tables
Figure 1: Overview of the RynnVLA-Latent pretraining corpus. We aggregate approximately 30.1K hours from 25 datasets, comprising 16.63M episodes and 3.057B frames. Human video and real-robot video each contribute approximately 40% of the duration, with simulation accounting for the remaining 20%. The figure summarizes source composition, dataset contributions by duration, and object and action vocabulary from public tasks.
Figure 2: Data curation pipeline. The video data is collected from diverse sources and partitioned into short clips during data preprocessing. Each clip undergoes a multi-modal annotation process: (1) Video Captioning: Qwen3-VL ( Bai et al., 2025 ) generates detailed natural language descriptions of the video content; (2) Optical Flow Estimation: DPFlow ( Morimitsu et al., 2025 ) computes dense per-frame motion fields; (3) Depth Estimation: Depth Anything 3 ( Lin et al., 2025 ) produces monocular depth predictions.
Figure 3: Architecture of RynnLAM and comparison with existing methods. (a) Conventional latent action models compress a frame transition into a single latent representation and reconstruct the future observation conditioned on the source frame. (b) RynnLAM factorizes transition information into motion tokens ztm , an action latent zta , and a camera latent ztc . The three components are trained through complementary objectives: feature-space reconstruction, 3D dynamic-flow prediction, and relative camera-pose estimation, respectively. This factorization encourages the action representation to capture physical scene dynamics while encouraging camera ego-motion to be represented separately.
Figure 4: Two-stage training pipeline of RynnVLA-Latent. (a) Latent-action pretraining. RynnLAM provides shared transition descriptors for human and heterogeneous robot videos. The policy predicts these descriptors from observations and language. (b) Robot post-training. Both the pretrained RynnBrain backbone and action-expert trunk are reused. The output projection is retargeted to native robot actions, and proprioceptive state is introduced. Post-training adapts the pretrained policy to the target control space without adding a separate latent-to-action decoder.
Method
Layers
VLM
Action Expert
Total Params
Hidden Size
Params
Hidden Size
Params
RynnVLA-Latent-2B
28
2048
2.13 B
768
0.38 B
2.51 B
RynnVLA-Latent-4B
36
2560
4.44 B
768
0.60 B
5.04 B
Table 2: Model configurations for the scaling variants of RynnVLA-Latent . Total parameters denote the total number of model parameters.
Figure 5: Standardized Experimental Platforms. Two dual-arm embodiments used for real-world evaluation: the stationary Marvin-WUJI (dual 7-DOF arms with 20-DOF dexterous hands) and the mobile Astribot-S1 (dual 7-DOF arms with parallel grippers on a wheeled base).
Method
Avg.
CALVIN
VLABench
RoboCOIN
AgiBotWorld-B
General Encoders (Absolute Action)
V-JEPA 2 ( Assran et al., 2025 )
0.25
0.27
0.07
0.32
0.33
DINOv3 ( Siméoni et al., 2025 )
0.19
0.22
0.06
0.22
0.24
Wan2.2 ( Wan et al., 2025 )
0.30
0.39
0.09
0.34
0.39
FLUX.2-dev ( Labs, 2025 )
0.35
0.25
0.04
0.47
0.62
RynnLAM (Raw Features)
0.14
0.08
0.01
0.24
0.22
Table 3: Results and Ablation Study on the LARY benchmark ( Nie et al., 2026 ) . Low-level control regression (MSE, lower is better). ‘RynnLAM (Raw Features)’ probes the latent representation produced directly by the RynnLAM encoder without assembling the 608-dimensional descriptor, following the general-encoder protocol; the ‘RynnLAM’ rows probe the full 608-dimensional latent-action descriptor.
Figure 6: Latent-action similarity visualization. We retrieve the nearest neighbors of several query latent actions across the multi-embodiment corpus and display the corresponding video frames. The displayed high-similarity neighbors exhibit related motion patterns across different embodiments and visual domains, providing a qualitative illustration of cross-domain transition similarity.
Method
Spatial
Object
Goal
Long
Avg.
OpenVLA-OFT ( Kim et al., 2025 )
97.6
98.4
97.9
94.5
97.1
π0 -FAST ( Pertsch et al., 2025 )
96.4
96.8
88.6
60.2
85.5
π0 ( Black et al., 2024 )
98.0
96.8
94.4
88.4
94.4
π0.5 ( Intelligence et al., 2025 )
98.8
98.2
98.0
92.4
96.9
GR00T-N1.7 ( Bjorck et al., 2025 )
97.7
98.5
97.5
94.4
97.0
InternVLA-M1 ( Chen et al., 2025b )
98.0
99.0
93.8
92.6
95.9
Table 4: Results on the LIBERO benchmark.
Method
Cam.
Robot
Lang.
Light
Bkg.
Noise
Layout
Total
OpenVLA ( Kim et al., 2024 )
0.8
3.5
23.0
8.1
34.8
15.2
28.5
15.6
OpenVLA-OFT ( Kim et al., 2025 )
56.4
31.9
79.5
88.7
93.3
75.8
74.2
69.6
π0 ( Black et al., 2024 )
13.8
6.0
58.8
85.0
81.4
79.0
68.9
53.6
π0 -FAST ( Pertsch et al., 2025 )
65.1
21.6
61.0
73.2
73.2
74.4
68.8
61.6
π0.5 ( Intelligence et al., 2025 )
78.4
73.6
80.8
96.2
94.1
89.0
84.5
84.4
StarVLA ( Community, 2026 )
52.5
49.8
88.5
95.7
95.7
73.0
76.9
74.1
Table 5: Results on LIBERO-Plus , evaluated zero-shot with the LIBERO-finetuned checkpoint.
Method
Clean
Randomized
Avg.
π0 ( Black et al., 2024 )
65.9
58.4
62.2
π0.5 ( Intelligence et al., 2025 )
82.7
76.8
79.8
X-VLA ( Zheng et al., 2026 )
72.9
72.8
72.9
StarVLA-FAST ( Community, 2026 )
72.5
83.2
77.9
ABot-M0 ( Yang et al., 2026 )
86.1
85.1
85.6
LingBot-VLA ( Wu et al., 2026 )
88.6
86.7
87.7
Table 6: RoboTwin 2.0 Native protocol. Values are success rates in percent. Baseline values are taken from the respective reports. Avg. in our rows is the arithmetic mean of Clean and Randomized.
In-Distribution
Cross-Category
Common Sense
Method
SR
PS
IS
SR
PS
IS
SR
PS
IS
π0 ( Black et al., 2024 )
47.0
62.7
67.8
21.2
33.6
44.0
29.1
43.0
54.9
ACoT-VLA ( Zhong et al., 2026 )
-
66.1
79.8
-
38.9
54.1
-
37.8
52.3
π0 -FAST ( Pertsch et al., 2025 )
56.2
66.8
72.4
31.0
38.2
47.8
38.0
48.6
56.8
X-VLA ( Zheng et al., 2026 )
-
67.8
-
-
25.1
-
-
48.2
-
π0.5 ( Intelligence et al., 2025 )
65.4
77.8
80.4
38.2
49.7
52.0
43.9
57.3
60.0
Table 7: Results on VLABench. SR, PS, and IS denote success rate, progress score, and intention score. The Average columns report arithmetic means across the five tracks when all five values are available. Dashes denote unreported measurements.
Model
Marvin-WUJI
Astribot-S1
Plate Placement
Steak Cooking
Plate Wiping
Table Cleaning
Steak Serving
Plate Retrieval
π0 ( Black et al., 2024 )
80.00
34.29
71.43
22.86
37.14
45.71
π0.5 ( Intelligence et al., 2025 )
82.86
28.57
77.14
25.71
40.00
45.71
GR00T-N1.7 ( Bjorck et al., 2025 )
74.29
28.57
74.29
17.14
45.71
37.14
RynnVLA-Latent-2B
80.00
34.29
68.57
34.29
42.86
48.57
RynnVLA-Latent-4B
85.71
42.86
82.86
40.00
57.14
54.29
Table 8: Success rates on real-world robotic manipulation tasks . Values are closed-loop success rates in percent on six tasks after robot-specific post-training. Each setting is evaluated with 35 trials per task. Bold indicates the highest observed rate per task among the listed comparisons.
Method
LIBERO
LIBERO-Plus
Avg.
RynnVLA-Latent-2B-QwenVL ( Bai et al., 2025 )
98.1
78.8
88.5
RynnVLA-Latent-4B-QwenVL ( Bai et al., 2025 )
98.6
79.7
89.2
RynnVLA-Latent-2B w/o camera slots
98.5
81.9
90.2
RynnVLA-Latent-4B w/o camera slots
98.3
82.9
90.6
RynnVLA-Latent-2B Shared vector representation
98.7
80.5
89.6
RynnVLA-Latent-4B Shared vector representation
99.2
83.5
91.4
Table 9: Policy variants on LIBERO and LIBERO-Plus. Values are success rates in percent. Avg. is the arithmetic mean of the two benchmark scores. All rows, including the latent-action references, are trained on the same approximately 1,000-hour subset and are therefore directly comparable; they differ from the full-corpus results in Tabs. 4 and 5 .
Figure 7: Post-training evaluation on real-world tasks. Each row shows the language instruction and representative execution frames for one task on the corresponding platform.
Figure 8: Scaling of latent-action pretraining. We pretrain RynnVLA-Latent-2B and RynnVLA-Latent-4B with 10%, 30%, 50%, 70% and 100% of the latent-action pretraining corpus, and evaluate the resulting policies on LIBERO-Plus ( Fei et al., 2025 ) . Both models achieve their highest success rates with the full corpus, although improvements are not strictly monotonic across all data fractions.
Figure 9: Qualitative visualization of 3D scene flow decoded from latent actions predicted by RynnVLA-Latent. Given the current observation and language instruction, RynnVLA-Latent predicts a latent-action chunk, which is fed into the RynnLAM flow decoder to visualize the implied 3D motion.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Marvin-WUJI demonstration collection. Motion trackers control the dual robot arms, and data gloves control the dexterous hands.
Method
Avg.
CALVIN
VLABench
RoboCOIN
AgiBotWorld-B
Training & Objective Variants
RynnLAM w/ 2D flow
0.56
0.44
0.44
0.69
0.67
RynnLAM w/o contrastive feat.
0.57
0.44
0.45
0.70
0.68
RynnLAM w/o feat. prediction
0.63
0.49
0.50
0.78
0.74
RynnLAM w/ zm from latent tokens
0.79
0.64
0.66
0.94
0.92
RynnLAM
0.38
0.40
0.05
0.42
0.64
Appendix
Table 10: Extended ablations on the LARY benchmark ( Nie et al., 2026 ) (Relative Action, MSE, lower is better). All retrained variants use the same data, budget, and probe protocol as the main table. Avg. is the arithmetic mean of the displayed per-dataset MSE values, rounded to two decimal places.
Task Name
Description
Episodes
Plate-Placement
Take the plate from the shelf and place it to the left of the oven.
500
Steak-Cooking
Flip the steak in the pan to fry it.
800
Plate-Wiping
Take the dirty plate from the shelf, wipe it with a cloth, and return it to the same shelf.
500
Table-Cleaning
Collect all scattered items on the table into the box, then place the box onto the shelf.
850
Steak-Serving
Carry the steak plate horizontally to the designated dining spot and place it gently on the placemat.
600
Plate-Retrieval
Collect the dirty plate and the knife and fork from the dining table, and return them to their original positions.
600
Appendix
Table 11: Statistics of the real-world dataset used for RynnVLA-Latent training.
Latent Action Models (LAMs) have emerged as an effective paradigm for handling heterogeneous datasets during Vision-Language-Action (VLA) model pretraining, offering a unified action space across embodiments. However, existing LAMs often rely on discrete quantization encode and decode pipelines, which can lead to trivial frame reconstruction behavior, limited representational capacity, and a lack of physically meaningful structure. We introduce RotVLA, a VLA framework built on a continuous rotational latent action representation. Latent actions are modeled as elements of SO(n), providing continuity, compositionality, and structured geometry aligned with real-world action dynamics. A triplet frame learning framework further enforces meaningful temporal dynamics while avoiding degeneration. RotVLA consists of a VLM backbone and a flow-matching action head, pretrained on large-scale cross-embodiment robotic datasets and human videos with latent-action supervision. For downstream robot control, the flow-matching head is extended into a unified action expert that jointly denoises latent and robot actions. Here, latent actions serve as a latent planner, providing high-level guidance that conditions action generation. With only 1.7B parameters and 1700+ hours of pretraining data, RotVLA achieves 98.2% on LIBERO and 89.6% / 88.5% on RoboTwin2.0 under clean and randomized settings, respectively. It also demonstrates strong real-world performance on manipulation tasks, consistently outperforming existing VLA models.
Qiwei Li, Xicheng Gong, Xinghang Li +5
Wangxuan Institute of Computer Technology, Peking University · Xiaomi Robotics · CASIA
Visual-language action (VLA) models enable robots to predict actions directly from observations and language instructions, but their performance depends on large-scale, high-quality data and is limited by the scarcity of real-world robot action datasets. To facilitate VLA model learning with abundant unlabeled human videos, Latent Action Models (LAM) learn latent action representations from visual dynamics to provide additional supervision for VLA learning. However, LAM and VLA are typically trained separately, leaving LAM ungrounded during VLA training and VLA models constrained by frozen LAM representations. To address these issues, we propose Latent Action Representation Alignment (LARA), a plug-and-play framework that jointly optimizes LAM and VLA via representation alignment. This enables reciprocal benefits where LAMs learn with action trajectories to avoid spurious visual changes, while VLAs are regularized by forward dynamics learned within LAMs to reduce hallucinations of functionally ineffective trajectories. We demonstrate LARA versatility and effectiveness for pre-training, post-training enhancement of pre-trained VLA models, and LAM refinement, achieving an average of ~10%, ~5%, and ~15% improvement over 3 simulation and 1 meticulously designed real-world robotic manipulation benchmarks.
Mengya Liu, Baoxiong Jia, Jiangyong Huang +2
State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence
Human videos provide rich information about object manipulation at a scale difficult to reproduce with robots, but often lack action annotations that can directly supervise robot policies. Realizing this potential requires distinguishing interaction-relevant changes from static appearance and connecting the learned interaction knowledge to executable robot actions. We present WALA, a framework named for World- and Action-supervised Latent Actions. Its semantic-geometric latent action model (LAM) learns latent actions by encoding and predicting semantic and geometric changes, emphasizing interactions over static appearance while retaining spatial detail. During robot policy training, the LAM encoder and decoder provide guidance through distillation and visual prediction, respectively. Together with robot action supervision, these signals shape policy-generated latent actions to support world prediction and capture the information needed to directly generate executable robot actions. Because LAM supervision requires no action labels, WALA enables co-training on action-labeled robot demonstrations and task-relevant action-free videos. In simulation experiments, our method achieves 92.33% average success on RoboTwin and 75.2% on RoboCasa-GR1-Tabletop, exceeding the previous state of the art on RoboCasa by 5.0 percentage points. WALA also shows strong real-robot performance, further enhanced by task-relevant action-free human videos that improve data efficiency and support target-task zero-shot and few-shot transfer.