TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control

Menghao Yin; Jiaju Yin; Xiyue Dong; Lixin Xu; Zhenhui Zhang; Ke Zhang; Yating Feng; Renjing Xu

cs.ROOct 15, 2026

TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control

Authors: Menghao Yin, Jiaju Yin, Xiyue Dong, Lixin Xu, Zhenhui Zhang, Ke Zhang, Yating Feng, Renjing Xu

Organizations: The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Many vision-language-action policies rely on action-labeled robot demonstrations and lack an explicit mechanism for predicting future visual states. These limitations hinder learning from action-free videos and can encourage overfitting to demonstration conditions, weakening generalization and robustness. We present TS-WAM, a lightweight Teacher-Student World-Action Model that addresses these challenges through self-supervised temporal learning and explicit future prediction. Its vlm-jepa framework learns from heterogeneous data, including robot demonstrations and action-free human videos, using an EMA target branch without requiring a separately trained external teacher. By coupling vlm-jepa with delta-lwm, TS-WAM learns to predict latent actions from the current observation and instruction and uses them to predict future visual features that guide action generation at inference. A unified end-to-end training architecture jointly optimizes temporal representation learning, visual dynamics, and robot control without a separate latent-action or world-model pretraining stage. The EMA target branch is removed at inference, keeping the deployed policy lightweight. Experiments on LIBERO, LIBERO-Plus, and SimplerEnv, together with real-world manipulation experiments on Franka Research 3 and RevoTron, demonstrate that TS-WAM achieves strong manipulation performance, generalization, and robustness.

Figures & tables

Fig. 1: Comparison of conventional IDM-based world-action models and our JEPA-based TS-WAM. Pre-training followed by fine-tuning often requires substantial computational resources and time, whereas TS-WAM achieves strong performance with limited resources through end-to-end training without a separate pre-training stage.

Fig. 2: Overview of TS-WAM. Vlm-jepa predicts temporal latent actions, delta-lwm predicts visual feature changes, and the flow-matching action expert generates robot actions conditioned on predicted future features and VLM context. The EMA teacher provides supervision during training.

Fig. 3: Vlm-jepa architecture. The online $X$ -VLM and predictor infer latent actions from the current observation and instruction; the EMA $Y$ -VLM uses the future frame to provide a stop-gradient alignment target.

Table 4

Method	Params	Camera	Robot	Language	Light	Background	Noise	Layout	Overall
WorldVLA [ 27 ]	7B	0.1	27.9	41.6	43.7	17.1	10.9	38.0	25.0
Mantis [ 28 ]	5.8B	15.7	41.8	45.9	45.1	28.9	39.2	62.5	39.8
UniVLA [ 17 ]	7B	4.3	50.3	71.8	59.1	80.0	25.3	34.3	41.5
$\pi_{0}$ [ 24 ]	3.3B	13.8	6.0	58.8	85.0	81.4	79.0	68.9	53.6
VLA-JEPA [ 29 ]	2B	40.3	55.7	72.9	88.2	70.5	38.2	74.6	62.9
Spatial Forcing [ 35 ]	7B	20.1	13.4	40.9	29.1	33.4	25.7	39.3	28.8

TABLE III: Zero-shot transfer results on LIBERO-Plus. Success rates (%) are reported across seven perturbation categories and overall.

Fig. 4: Latent-action representation diversity. Each point is a temporal window, projected onto two principal components; colors identify tasks. The upper and lower rows show student and teacher latent actions, respectively.

Fig. 5: Effect of latent-action conditioning on LWM predictions. Rows A–D show four task examples. The first two columns contain the observed temporal pair; the remaining columns visualize the ground-truth feature change and predictions using the normal student LA (TS-WAM), a fixed mean LA, and a zero LA. All feature maps share the same GT-fitted PCA basis and RGB scale.

Method	Franka Research 3		RevoTron		Overall
	Stack Bowls	Drawer- Long	Item Sorting	Drawer- Long
$\pi_{0.5}$
IND	75	40	60	45	55.00
OOD-LC	60	20	50	35	41.25
OOD-OL	45	20	40	25	32.50
OOD-BT	40	25	30	30	31.25

TABLE IV: Real-world performance comparison under IND and OOD conditions. Success rates (%) are reported over 20 trials per task and condition; Overall is the mean across four tasks. OOD-LC, OOD-OL, and OOD-BT denote OOD-Light Conditions, OOD-Objects Layout, and OOD-Background Textures, respectively.

Panel of Figure 9 — Fig. 6: LIBERO ablation results. “w/o human video” denotes training without ego data; “w/o Pred delta F” denotes an action expert that receives only semantic features from the VLM, without future features from delta-lwm; “w/o vlm-jepa” denotes training with a single VLM without the JEPA architecture.

Fig. 8: TS-WAM executing Drawer-Long on RevoTron under IND and three OOD conditions. Rows from top to bottom show the original IND scene, OOD-Light conditions (an additional light switched on), OOD-Objects Layout (bread placed on a plate instead of directly on the table), and OOD-Background Textures (the white tabletop covered with a black cloth). Time progresses from left to right. Each sequence shows successful completion of opening the drawer, placing the bread inside, and closing the drawer despite the visual changes.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,190 papers

JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling
Aug 10, 2026Yihan Lin, Jiawei He, Shifeng Bao +6Joint-Embedding Predictive Architecture Robotic Manipulation
School of Information, Renmin University of China, Beijing, China · XYZ Embodied AI, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China +3
LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies
Jun 14, 2026Jialei Chen, Kai Wang, Kang Chen +9Latent Action Learning Action-Conditioned World Models
Jilin University · Zhongguancun Academy · Nankai University +4
WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction
Oct 13, 2026Haomin Wang, Anqiong Ji, Ke Wang +4World Models for Robotics World Action Models
AutoLab, School of Artificial Intelligence, Shanghai Jiao Tong University · Baidu AI Cloud · Awaken Robotics +1

Aug 10, 2026J/K move · Enter open · S save

JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling

Yihan Lin, Jiawei He, Shifeng Bao, Chen Zhao+5

School of Information, Renmin University of China, Beijing, China · XYZ Embodied AI, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China+3

Joint-Embedding Predictive Architecture Robotic Manipulation Vision-Language-Action Models Latent World Models Embodied World Models

Open paper AskPDF arXiv

Robust robot control benefits from explicitly modeling state transitions, but video-generation world action models (WAMs) introduce substantial deployment cost. Existing latent WAMs avoid explicit future generation, but often compress predictive representations or separate predictive modeling from the representations used for action generation. We introduce JEPA-WAM, a latent WAM built in a pretrained V-JEPA space, which couples latent transition prediction with continuous action generation through a shared predictor. JEPA-WAM predicts a spatially structured joint current-future target that captures task-shared visual temporal structure between current and future observations, while preserving dense patch-level correspondence. Through the shared predictor, transition supervision directly shapes the backbone, from which dedicated representations are extracted for action prediction. The same design can also be instantiated in pretrained VLA policies while preserving their original perception and action pathways. On LIBERO-Plus, JEPA-WAM achieves 79.2%, the best result without large-scale robot-policy pretraining, while its pretrained

π_{0.5}

instantiation reaches 86.3%, achieving the best overall performance. Experiments on RoboTwin 2.0 and real-world bimanual manipulation further demonstrate strong generalization under visual and spatial shifts.

Abstract

Vision-Language-Action Models

Explore similar work

JEPA-WAM: Learning Vision-Language-Action Policies with Joint-Embedding World Modeling

LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

WAVE-0: Bridging VLA Policies and WAMs with Lightweight Latent Future Prediction