RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

Haoyu Zhao; Xingyue Zhao; Zhian Su; Bohan Hou; Yinzhou Tang; Fengyi Wang; Kehan Li; Yuming Jiang

cs.ROOct 25, 2026

RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

Authors: Haoyu Zhao, Xingyue Zhao, Zhian Su, Bohan Hou, Yinzhou Tang, Fengyi Wang, Kehan Li, Yuming Jiang,

Organizations: DAMO Academy, Alibaba Group · Hong Kong Embodied AI Lab · CUHK · PUMC ·

Abstract

Scaling vision-language-action (VLA) models is constrained by scarce action-labeled demonstrations and incompatible control spaces across robot embodiments. We present RynnVLA-Latent, a framework that turns heterogeneous interaction videos into scalable supervision for robot policy pretraining. At its core, RynnLAM learns continuous, 3D-grounded latent actions from visual transitions without requiring native robot controls. By combining residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion decoupling, RynnLAM encourages representations that capture control-relevant motion rather than appearance changes alone. We apply RynnLAM to 30.1K hours of human, robot, and simulation videos, producing 745.52M latent-action descriptors. RynnVLA-Latent then learns to predict latent-action chunks from visual observations and language through flow matching, before adapting its pretrained backbone and action expert to embodiment-specific robot actions. The resulting 4B policy achieves 98.8% success on LIBERO, 84.7% on LIBERO-Plus without benchmark-specific adaptation, 51.9% average success across five VLABench tracks, and 88.6% average success on the RoboTwin 2.0 bimanual benchmark. It also achieves the highest observed success rates among the compared methods on six real-world manipulation tasks across dexterous-hand and mobile-gripper platforms. Together with representation ablations and data-scaling results, these findings demonstrate the potential of latent-action supervision for scaling robot policy learning beyond native action-labeled data.

Figures & tables

Figure 1: Overview of the RynnVLA-Latent pretraining corpus. We aggregate approximately 30.1K hours from 25 datasets, comprising 16.63M episodes and 3.057B frames. Human video and real-robot video each contribute approximately 40% of the duration, with simulation accounting for the remaining 20%. The figure summarizes source composition, dataset contributions by duration, and object and action vocabulary from public tasks.

Figure 2: Data curation pipeline. The video data is collected from diverse sources and partitioned into short clips during data preprocessing. Each clip undergoes a multi-modal annotation process: (1) Video Captioning: Qwen3-VL ( Bai et al., 2025 ) generates detailed natural language descriptions of the video content; (2) Optical Flow Estimation: DPFlow ( Morimitsu et al., 2025 ) computes dense per-frame motion fields; (3) Depth Estimation: Depth Anything 3 ( Lin et al., 2025 ) produces monocular depth predictions.

Figure 3: Architecture of RynnLAM and comparison with existing methods. (a) Conventional latent action models compress a frame transition into a single latent representation and reconstruct the future observation conditioned on the source frame. (b) RynnLAM factorizes transition information into motion tokens $z_{t}^{m}$ , an action latent $z_{t}^{a}$ , and a camera latent $z_{t}^{c}$ . The three components are trained through complementary objectives: feature-space reconstruction, 3D dynamic-flow prediction, and relative camera-pose estimation, respectively. This factorization encourages the action representation to capture physical scene dynamics while encouraging camera ego-motion to be represented separately.

Figure 4: Two-stage training pipeline of RynnVLA-Latent. (a) Latent-action pretraining. RynnLAM provides shared transition descriptors for human and heterogeneous robot videos. The policy predicts these descriptors from observations and language. (b) Robot post-training. Both the pretrained RynnBrain backbone and action-expert trunk are reused. The output projection is retargeted to native robot actions, and proprioceptive state is introduced. Post-training adapts the pretrained policy to the target control space without adding a separate latent-to-action decoder.

Method	Layers	VLM		Action Expert		Total Params
Method	Layers	Hidden Size	Params	Hidden Size	Params	Total Params
RynnVLA-Latent-2B	28	2048	2.13 B	768	0.38 B	2.51 B
RynnVLA-Latent-4B	36	2560	4.44 B	768	0.60 B	5.04 B

Table 2: Model configurations for the scaling variants of RynnVLA-Latent . Total parameters denote the total number of model parameters.

Figure 5: Standardized Experimental Platforms. Two dual-arm embodiments used for real-world evaluation: the stationary Marvin-WUJI (dual 7-DOF arms with 20-DOF dexterous hands) and the mobile Astribot-S1 (dual 7-DOF arms with parallel grippers on a wheeled base).

Method	Avg.	CALVIN	VLABench	RoboCOIN	AgiBotWorld-B
General Encoders (Absolute Action)
V-JEPA 2 ( Assran et al., 2025 )	0.25	0.27	0.07	0.32	0.33
DINOv3 ( Siméoni et al., 2025 )	0.19	0.22	0.06	0.22	0.24
Wan2.2 ( Wan et al., 2025 )	0.30	0.39	0.09	0.34	0.39
FLUX.2-dev ( Labs, 2025 )	0.35	0.25	0.04	0.47	0.62
RynnLAM (Raw Features)	0.14	0.08	0.01	0.24	0.22

Table 3: Results and Ablation Study on the LARY benchmark ( Nie et al., 2026 ) . Low-level control regression (MSE, lower is better). ‘RynnLAM (Raw Features)’ probes the latent representation produced directly by the RynnLAM encoder without assembling the 608-dimensional descriptor, following the general-encoder protocol; the ‘RynnLAM’ rows probe the full 608-dimensional latent-action descriptor.

Figure 6: Latent-action similarity visualization. We retrieve the nearest neighbors of several query latent actions across the multi-embodiment corpus and display the corresponding video frames. The displayed high-similarity neighbors exhibit related motion patterns across different embodiments and visual domains, providing a qualitative illustration of cross-domain transition similarity.

Method	Spatial	Object	Goal	Long	Avg.
OpenVLA-OFT ( Kim et al., 2025 )	97.6	98.4	97.9	94.5	97.1
$\pi_{0}$ -FAST ( Pertsch et al., 2025 )	96.4	96.8	88.6	60.2	85.5
$\pi_{0}$ ( Black et al., 2024 )	98.0	96.8	94.4	88.4	94.4
$\pi_{0.5}$ ( Intelligence et al., 2025 )	98.8	98.2	98.0	92.4	96.9
GR00T-N1.7 ( Bjorck et al., 2025 )	97.7	98.5	97.5	94.4	97.0
InternVLA-M1 ( Chen et al., 2025b )	98.0	99.0	93.8	92.6	95.9

Table 4: Results on the LIBERO benchmark.

Method	Cam.	Robot	Lang.	Light	Bkg.	Noise	Layout	Total
OpenVLA ( Kim et al., 2024 )	0.8	3.5	23.0	8.1	34.8	15.2	28.5	15.6
OpenVLA-OFT ( Kim et al., 2025 )	56.4	31.9	79.5	88.7	93.3	75.8	74.2	69.6
$\pi_{0}$ ( Black et al., 2024 )	13.8	6.0	58.8	85.0	81.4	79.0	68.9	53.6
$\pi_{0}$ -FAST ( Pertsch et al., 2025 )	65.1	21.6	61.0	73.2	73.2	74.4	68.8	61.6
$\pi_{0.5}$ ( Intelligence et al., 2025 )	78.4	73.6	80.8	96.2	94.1	89.0	84.5	84.4
StarVLA ( Community, 2026 )	52.5	49.8	88.5	95.7	95.7	73.0	76.9	74.1

Table 5: Results on LIBERO-Plus , evaluated zero-shot with the LIBERO-finetuned checkpoint.

Method	Clean	Randomized	Avg.
$\pi_{0}$ ( Black et al., 2024 )	65.9	58.4	62.2
$\pi_{0.5}$ ( Intelligence et al., 2025 )	82.7	76.8	79.8
X-VLA ( Zheng et al., 2026 )	72.9	72.8	72.9
StarVLA-FAST ( Community, 2026 )	72.5	83.2	77.9
ABot-M0 ( Yang et al., 2026 )	86.1	85.1	85.6
LingBot-VLA ( Wu et al., 2026 )	88.6	86.7	87.7

Table 6: RoboTwin 2.0 Native protocol. Values are success rates in percent. Baseline values are taken from the respective reports. Avg. in our rows is the arithmetic mean of Clean and Randomized.

	In-Distribution			Cross-Category			Common Sense
Method	SR	PS	IS	SR	PS	IS	SR	PS	IS
$\pi_{0}$ ( Black et al., 2024 )	47.0	62.7	67.8	21.2	33.6	44.0	29.1	43.0	54.9
ACoT-VLA ( Zhong et al., 2026 )	-	66.1	79.8	-	38.9	54.1	-	37.8	52.3
$\pi_{0}$ -FAST ( Pertsch et al., 2025 )	56.2	66.8	72.4	31.0	38.2	47.8	38.0	48.6	56.8
X-VLA ( Zheng et al., 2026 )	-	67.8	-	-	25.1	-	-	48.2	-
$\pi_{0.5}$ ( Intelligence et al., 2025 )	65.4	77.8	80.4	38.2	49.7	52.0	43.9	57.3	60.0

Table 7: Results on VLABench. SR, PS, and IS denote success rate, progress score, and intention score. The Average columns report arithmetic means across the five tracks when all five values are available. Dashes denote unreported measurements.

Model	Marvin-WUJI			Astribot-S1
Model	Plate Placement	Steak Cooking	Plate Wiping	Table Cleaning	Steak Serving	Plate Retrieval
$\pi_{0}$ ( Black et al., 2024 )	80.00	34.29	71.43	22.86	37.14	45.71
$\pi_{0.5}$ ( Intelligence et al., 2025 )	82.86	28.57	77.14	25.71	40.00	45.71
GR00T-N1.7 ( Bjorck et al., 2025 )	74.29	28.57	74.29	17.14	45.71	37.14
RynnVLA-Latent-2B	80.00	34.29	68.57	34.29	42.86	48.57
RynnVLA-Latent-4B	85.71	42.86	82.86	40.00	57.14	54.29

Table 8: Success rates on real-world robotic manipulation tasks . Values are closed-loop success rates in percent on six tasks after robot-specific post-training. Each setting is evaluated with 35 trials per task. Bold indicates the highest observed rate per task among the listed comparisons.

Method	LIBERO	LIBERO-Plus	Avg.
RynnVLA-Latent-2B-QwenVL ( Bai et al., 2025 )	98.1	78.8	88.5
RynnVLA-Latent-4B-QwenVL ( Bai et al., 2025 )	98.6	79.7	89.2
RynnVLA-Latent-2B w/o camera slots	98.5	81.9	90.2
RynnVLA-Latent-4B w/o camera slots	98.3	82.9	90.6
RynnVLA-Latent-2B Shared vector representation	98.7	80.5	89.6
RynnVLA-Latent-4B Shared vector representation	99.2	83.5	91.4

Table 9: Policy variants on LIBERO and LIBERO-Plus. Values are success rates in percent. Avg. is the arithmetic mean of the two benchmark scores. All rows, including the latent-action references, are trained on the same approximately 1,000-hour subset and are therefore directly comparable; they differ from the full-corpus results in Tabs. 4 and 5 .

Figure 7: Post-training evaluation on real-world tasks. Each row shows the language instruction and representative execution frames for one task on the corresponding platform.

Figure 8: Scaling of latent-action pretraining. We pretrain RynnVLA-Latent-2B and RynnVLA-Latent-4B with 10%, 30%, 50%, 70% and 100% of the latent-action pretraining corpus, and evaluate the resulting policies on LIBERO-Plus ( Fei et al., 2025 ) . Both models achieve their highest success rates with the full corpus, although improvements are not strictly monotonic across all data fractions.

Figure 9: Qualitative visualization of 3D scene flow decoded from latent actions predicted by RynnVLA-Latent. Given the current observation and language instruction, RynnVLA-Latent predicts a latent-action chunk, which is fed into the RynnLAM flow decoder to visualize the implied 3D motion.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Marvin-WUJI demonstration collection. Motion trackers control the dual robot arms, and data gloves control the dexterous hands.

Method	Avg.	CALVIN	VLABench	RoboCOIN	AgiBotWorld-B
Training & Objective Variants
RynnLAM w/ 2D flow	0.56	0.44	0.44	0.69	0.67
RynnLAM w/o contrastive feat.	0.57	0.44	0.45	0.70	0.68
RynnLAM w/o feat. prediction	0.63	0.49	0.50	0.78	0.74
RynnLAM w/ $z^{m}$ from latent tokens	0.79	0.64	0.66	0.94	0.92
RynnLAM	0.38	0.40	0.05	0.42	0.64

Appendix

Table 10: Extended ablations on the LARY benchmark ( Nie et al., 2026 ) (Relative Action, MSE, lower is better). All retrained variants use the same data, budget, and probe protocol as the main table. Avg. is the arithmetic mean of the displayed per-dataset MSE values, rounded to two decimal places.

Task Name	Description	Episodes
Plate-Placement	Take the plate from the shelf and place it to the left of the oven.	500
Steak-Cooking	Flip the steak in the pan to fry it.	800
Plate-Wiping	Take the dirty plate from the shelf, wipe it with a cloth, and return it to the same shelf.	500
Table-Cleaning	Collect all scattered items on the table into the box, then place the box onto the shelf.	850
Steak-Serving	Carry the steak plate horizontally to the designated dining spot and place it gently on the placemat.	600
Plate-Retrieval	Collect the dirty plate and the knife and fork from the dining table, and return them to their original positions.	600

Appendix

Table 11: Statistics of the real-world dataset used for RynnVLA-Latent training.

VLA Model Training

See all 48 papers

RotVLA: Rotational Latent Action for Vision-Language-Action Model
May 13, 2026Qiwei Li, Xicheng Gong, Xinghang Li +5Latent Action Learning Vision-Language-Action Models
Wangxuan Institute of Computer Technology, Peking University · Xiaomi Robotics · CASIA
LARA: Latent Action Representation Alignment for Vision-Language-Action Models
Jun 5, 2026Mengya Liu, Baoxiong Jia, Jiangyong Huang +2Latent Action Learning Robotic Manipulation
State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University · Delta Intelligence
WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos
Jul 13, 2026Jiahao Liu, Yupeng Zheng, Zhongpu Xia +10Latent Action Learning Robot Policy Learning
CASIA · Anyverse Dynamics · UCAS +2

May 13, 2026J/K move · Enter open · S save

RotVLA: Rotational Latent Action for Vision-Language-Action Model

Qiwei Li, Xicheng Gong, Xinghang Li, Peiyan Li+4

Wangxuan Institute of Computer Technology, Peking University · Xiaomi Robotics · CASIA

Latent Action Learning Vision-Language-Action Models Vision-Based Robot Control

Open paper AskPDF arXiv

Latent Action Models (LAMs) have emerged as an effective paradigm for handling heterogeneous datasets during Vision-Language-Action (VLA) model pretraining, offering a unified action space across embodiments. However, existing LAMs often rely on discrete quantization encode and decode pipelines, which can lead to trivial frame reconstruction behavior, limited representational capacity, and a lack of physically meaningful structure. We introduce RotVLA, a VLA framework built on a continuous rotational latent action representation. Latent actions are modeled as elements of SO(n), providing continuity, compositionality, and structured geometry aligned with real-world action dynamics. A triplet frame learning framework further enforces meaningful temporal dynamics while avoiding degeneration. RotVLA consists of a VLM backbone and a flow-matching action head, pretrained on large-scale cross-embodiment robotic datasets and human videos with latent-action supervision. For downstream robot control, the flow-matching head is extended into a unified action expert that jointly denoises latent and robot actions. Here, latent actions serve as a latent planner, providing high-level guidance that conditions action generation. With only 1.7B parameters and 1700+ hours of pretraining data, RotVLA achieves 98.2% on LIBERO and 89.6% / 88.5% on RoboTwin2.0 under clean and randomized settings, respectively. It also demonstrates strong real-world performance on manipulation tasks, consistently outperforming existing VLA models.

Abstract

VLA Model Training

Explore similar work

RotVLA: Rotational Latent Action for Vision-Language-Action Model

LARA: Latent Action Representation Alignment for Vision-Language-Action Models

WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos