RM-WAM: Reliable Memory-Augmented World Action Models for Long-Horizon Robot Manipulation

Cui Miao; Junhao Yu; Ping Liu; Ning Yang; Xuhua Chen; He Zheng; Xiaodong Wang; Tao Zhang

cs.ROOct 11, 2026

RM-WAM: Reliable Memory-Augmented World Action Models for Long-Horizon Robot Manipulation

Authors: Cui Miao, Junhao Yu, Ping Liu, Ning Yang, Xuhua Chen, He Zheng, Xiaodong Wang, Tao Zhang

Organizations: National University of Defense Technology · Magic-Lab Team, Magiclab Robotics Inc. · Beijing Institute of Technology · Nanjing University

Abstract

World Action Models (WAMs) improve robotic manipulation by jointly predicting future visual states and actions, yet most existing methods make decisions primarily from the current observation. This limits their reliability in long-horizon tasks, where visually similar states may correspond to different interaction histories and task stages. Moreover, directly incorporating historical observations can introduce stale, stage-mismatched, or error-contaminated information, leading to inaccurate world prediction and accumulated closed-loop errors. We propose RM-WAM, a reliable memory-augmented World Action Model that treats memory as a context-dependent predictive resource rather than universally useful historical context. RM-WAM represents history using complementary short-term visual state memory and latent event memory, which respectively capture recent interaction dynamics and longer-term state transitions. A counterfactual memory learning scheme varies the accessible history while keeping the current inputs and prediction targets fixed, enabling the model to learn a context-dependent utility estimate for historical memory. Based on this estimate, a utility-aware gating mechanism adaptively controls memory injection into the world-action backbone. Experiments demonstrate that RM-WAM highlights the importance of reliable and selective memory utilization in long-horizon tasks.

Figures & tables

Fig. 1: Motivation of RM-WAM. (a) Existing memory-augmented VLAs use memory for action prediction, but retrieved memory is not always useful and they do not model future world prediction. (b) RM-WAM evaluates memory utility and incorporates reliable memory into a WAM for both world prediction and action generation.

Fig. 2: Overview of RM-WAM. The left panel shows the overall world-action framework with selective memory injection, while the right panel details the construction of the dual-scale memory.

Method	Spatial	Object	Goal	Long	Avg.
DP	78.3%	92.5%	68.3%	50.5%	72.4%
$\pi_{0.5}$	98.8%	98.2%	98.0%	92.4%	96.9%
HiMem-WAM	98.2%	99.8%	98.4%	94.5%	97.7%
FastWAM	98.2%	100.0%	97.0%	95.2%	97.6%
RM-WAM (Ours)	98.8%	100.0%	98.2%	97.0%	98.5%

TABLE I: Comparisons with other baselines on Libero benchmark. Success rates(%) are reported, and bold denotes the best performance among all methods.

Task	DP	$\boldsymbol{\pi_{0.5}}$	FastWAM	HiMem-WAM	RM-WAM
Observe Pick Up	1%	9%	0%	28%	27%
Rearrange Blocks	0%	13%	0%	33%	65%
Put Back Block	0%	11%	0%	32%	70%
Swap Blocks	11%	24%	0%	38%	62%
Swap T	2%	15%	7%	27%	58%
Battery Try	10%	16%	20%	28%	30%

TABLE II: Comparisons with baselines on RMBench. Success rates (%) are reported, and bold denotes the best performance.

Fig. 3: Real-world long-horizon tabletop organization task. The task consists of three sequential manipulation subtasks, denoted by ①–③, requiring the robot to complete multiple object organization steps over an extended temporal horizon.

Method	Standard	Disturbance	Unseen
FastWAM	80.0	70.0	76.7
RM-WAM	90.0	83.3	86.7

TABLE III: Real-world evaluation results. Success rates (%) are reported, and bold denotes the best performance.

Fig. 4: Visualization of short-term visual memory in RM-WAM. The top row shows spatial attention over recent historical observations, and the bottom row shows the corresponding raw observations together with the current frame. The right panel shows the short-memory utility coefficient learned through counterfactual memory supervision.

Fig. 5: Visualization of latent event memory in RM-WAM. The model attends to earlier event $e_{\tau}$ when returning the block to its original position.

Fig. 6: Visualization of future evolution on LIBERO-Long.

Method	Goal	Object	Spatial	Long	Avg.
w/o short-mem.	97.0	100.0	97.8	96.0	97.7
w/o event-mem.	96.8	100.0	98.2	95.0	97.5
RM-WAM	98.2	100.0	98.8	97.0	98.5

TABLE IV: Ablation study on LIBERO. Success rates (%) are reported, and bold denotes the best performance.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 386 papers

MemoryWAM: Efficient World Action Modeling with Persistent Memory
Jun 18, 2026Sizhe Yang, Juncheng Mu, Tianming Wei +8Long-Horizon Robotic Manipulation World Models for Robotics
The Chinese University of Hong Kong · Tsinghua University · Zhejiang University
DIM-WAM: World-Action Modeling with Diverse Historical Event Memory
Jun 26, 2026Kai Wang, Zhaopeng Gu, Yixiang Chen +7Long-Horizon Robotic Manipulation World Models for Robotics
Institute of Automation, Chinese Academy of Sciences (CASIA), Beijing, China · Shenzhen Yinwang Intelligent Technology Co., Ltd., Shenzhen, China · FiveAges, Beijing, China
HiMem-WAM: Hierarchical Memory-Gated World Action Models for Robotic Manipulation
Jun 9, 2026Xiaoquan Sun, Ruijian Zhang, Chen Cao +12Latent Action Learning Long-Horizon Robotic Manipulation
3Huazhong University of Science and Technology · 1The University of Hong Kong · 2INFIFORCE +3

Jun 18, 2026J/K move · Enter open · S save

MemoryWAM: Efficient World Action Modeling with Persistent Memory

Sizhe Yang, Juncheng Mu, Tianming Wei, Chenhao Lu+7

The Chinese University of Hong Kong · Tsinghua University · Zhejiang University

Long-Horizon Robotic Manipulation World Models for Robotics Action-Conditioned World Models Efficient Inference for World Action Models World Action Models Memory-Augmented Agents

Open paper AskPDF arXiv

Robust robotic manipulation in the real world requires not only an understanding of the current observation, but also memory and dynamics modeling. World action models (WAMs) possess these capabilities by jointly modeling visual foresight and actions conditioned on both current and historical observations, making them a promising paradigm for robotic manipulation. However, existing WAMs face a fundamental trade-off: methods with efficient inference typically condition only on a bounded window of recent observations and therefore struggle in non-Markovian environments, whereas methods that preserve long histories incur time and space costs that grow substantially with sequence length. To address this challenge, we introduce MemoryWAM, a world action model with efficient persistent memory. MemoryWAM uses a hybrid memory design that combines recent frames, event-boundary anchor frames, and compact gist tokens that summarize long-range history. A tailored attention mechanism enables retrieval of both detailed short-term context and compressed long-term context, supporting memory-dependent decision-making with reduced inference latency and GPU memory usage. Across long-horizon, memory-dependent manipulation tasks in both simulation and the real world, MemoryWAM outperforms strong vision-language-action (VLA) and WAM baselines while maintaining favorable computational efficiency.

Abstract

World Models for Robotics

Explore similar work

MemoryWAM: Efficient World Action Modeling with Persistent Memory

DIM-WAM: World-Action Modeling with Diverse Historical Event Memory

HiMem-WAM: Hierarchical Memory-Gated World Action Models for Robotic Manipulation