CoT-WAM: Collaborative Training of Vision-Language Models and World-Action Models via Plan Latents
Authors: Jinhao Zhang, Yushen Zuo, Ke Rui, Weitao Zhou, Haibao Yu, Honglong Tian, Xiaoheng Ma, Jie Mei, Jiawei Wang, Minglei Li
Organizations: Harbin Institute of Technology, Shenzhen · Simple AI · Tsinghua University · Tuojing Intelligence · The University of Hong Kong · The Hong Kong University of Science and Technology
General robot manipulation requires both semantic competence and physical foresight. World action models (WAMs) acquire an understanding of and predictive capability over the physical world by jointly modeling visual foresight and action distributions. However, existing WAMs still lack semantic competence: they have limited ability to perform open-world scene understanding, semantic reasoning, and long-horizon planning. Although vision-language models (VLMs) provide these capabilities, effectively integrating their semantic priors into WAM prediction and control remains challenging. To address this challenge, we propose CoT-WAM, a jointly trained WAM-VLM framework. CoT-WAM uses a learnable embedding to exchange information between the VLM and the WAM, which is supervised by textual and goal-image signals during training to encode rich intent information. A tailored two-stage training strategy enables joint optimization and more effective collaboration between the VLM and WAM. We evaluate CoT-WAM on six benchmarks covering control performance, zero-shot generalization, and long-horizon planning. CoT-WAM achieves SOTA performance on LIBERO, CALVIN, and RoboCasa-Kitchen, and demonstrates stronger zero-shot generalization on LIBERO-Plus and LIBERO-Pro without fine-tuning. On RoboCasa365, it further improves long-horizon planning over baselines with and without external planners. Real-world experiments further demonstrate the effectiveness of our method in practical deployments.
Figures & tables
Figure 1: Overview of CoT-WAM. The VLM encodes the current observation and instruction into continuous intent tokens Ct . The world action model uses these tokens to predict a visual goal, which conditions the video and action experts. Text and visual supervision train the interface in Stage 1; Stage 2 learns control with the VLM frozen. Full future-video generation is omitted during action inference.
Figure 2: Training and inference attention masks. Rows are queries and columns are keys. Colored cells allow attention; triangles denote causal attention. Inference omits the VLM task-text tokens and WAM future-video tokens, while retaining the latent interface and goal-conditioned action path.
Method
Emb. PT.
LIBERO
CALVIN
RoboCasa
Spatial
Object
Goal
Long
Avg.
VLA-based methods
OpenVLA [ 11 ]
✓
84.70
88.40
79.20
53.70
76.50
3.27
–
OpenVLA-OFT [ 10 ]
✓
97.60
98.40
97.90
94.50
97.10
4.28
33.1
π0.5 [ 9 ]
✓
98.80
98.20
98.00
92.40
96.90
3.97
52.9
CoT-VLA [ 31 ]
✓
87.50
91.60
87.60
69.00
83.93
–
–
Table 1: Control performance. Success rate (%) on LIBERO/RoboCasa-Kitchen; average successful chain length (ACL) on CALVIN. Emb. PT.: additional embodied pretraining. Bold/underline: group best/second. Δ : gains over Fast-WAM baseline.
Table 3: Long-horizon planning. † indicates the use of privileged simulator state.
Variant
LIBERO
LIBERO-Plus
CoT-WAM (ours)
99.00
78.60
Architecture and training schedule
w/o VLM
98.00
68.65
w/o goal branch
97.80
73.65
single-stage training
98.50
76.10
Supervision on latent interface
Table 4: Ablation studies. Success rates (%) on LIBERO and LIBERO-Plus.
Figure 3: Attention maps under scene and language shifts. Comparison on LIBERO-Plus under scene OOD (top) and language OOD (bottom).
Figure 4: Visual goals and execution. Successive predicted visual goals (top) and corresponding execution outcomes (bottom), from the initial state to successful task completion.
Figure 5: Subtask planning in long-horizon tasks. Subtask predictions across three RoboCasa365 tasks, with the overall instruction fixed within each task. “Next” denotes annotated valid subtasks; “Latent decode” shows predictions decoded from the VLM’s latent queries.
Task
Fast-WAM
CoT-WAM
Produce sorting
52.5
67.5
Table wiping
42.5
55.0
Average
47.5
61.3
Table 5: Real-world control performance.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Stage 1
Stage 2
Training steps
10k
60k
Learning rate
10−5
5×10−5
Global batch size
64
64
Optimizer
AdamW
AdamW
Weight decay
0.01
0.01
Learning-rate schedule
Cosine
Cosine
Appendix
Table 6: Training configuration. Settings for LIBERO, CALVIN, and RoboCasa-Kitchen.
Figure 6: Real-world task demonstrations. Produce sorting (top) and table wiping (bottom). Each sequence progresses from left to right.