Organizations: University of Electronic Science and Technology of China · Shanghai AI Lab · Fudan University · Shanghai Jiao Tong University · Tongji University · Shanghai Innovation Institute
World action models (WAMs) built on pretrained video generation models adapt quickly to new robotic tasks, but they forget previously learned tasks under sequential fine-tuning. Experience replay (ER) mitigates such forgetting by rehearsing data of previous tasks, yet this data is often unavailable in robotics, where a policy is deployed as weights only and its training data is not shared along with it. In this work, we ask whether a WAM can achieve data-free continual learning by regenerating its own past. Our key observation is that a WAM can indeed autoregressively regenerate complete rollouts of previously learned tasks and replay them to mitigate forgetting. However, replaying such raw rollouts still lags far behind replaying real data, which we attribute to two issues. First, the WAM is unable to recognize task completion and keeps generating purposeless motion after the task is done. Second, under open-loop generation, the generated actions gradually drift away from the generated observations. To address both issues, we present rectified generative replay (R2-WAM), which truncates generated rollouts at task completion with an agentic evaluator and filters out rollouts whose actions are physically inconsistent with the observations. On the four LIBERO suites and five real-world tasks, R2-WAM reduces forgetting by over 90% relative to sequential fine-tuning and nearly matches ER without accessing any data from previous tasks. Our analysis further reveals that forgetting in WAMs concentrates in inverse dynamics modeling: a forgotten WAM still produces visually plausible observations of previous tasks while its generated actions fail. Project page: https://kaipengfang.github.io/r2-wam/
Figures & tables
Figure 1: (a) In continual learning, a WAM is sequentially fine-tuned on new tasks and evaluated on all tasks learned so far. Experience replay (ER) mitigates forgetting of earlier tasks but requires access to their data. (b) R 2 -WAM exploits the autoregressive generation of WAMs to achieve data-free continual learning, and its rectified generative replay attains performance comparable to ER.
Figure 2: Overview of R 2 -WAM. After learning task k−1 , WAMk−1 ① generates M candidate rollouts of that task with the instruction ℓk−1 , which are then ② rectified by truncating each rollout at task completion and filtering out physically inconsistent ones. The top- N retained rollouts Rk−1 join those of earlier tasks to form the buffer R<k , which is ③ replayed with the new-task demonstrations Dk when learning task k . The same procedure then repeats for the next stage k+1 .
Figure 3: Real-robot setup.
Method
LIBERO-Spatial
LIBERO-Object
LIBERO-Goal
LIBERO-10
Average
SR ↑
NBT ↓
Plast. ↑
SR ↑
NBT ↓
Plast. ↑
SR ↑
NBT ↓
Plast. ↑
SR ↑
NBT ↓
Plast. ↑
SR ↑
NBT ↓
Plast. ↑
Seq-FT
11.2
86.07
95.8
21.2
67.84
97.4
9.8
84.87
94.8
7.6
82.00
89.6
12.5
80.19
94.4
Seq-LoRA
9.6
75.42
86.2
17.6
52.47
87.0
9.8
68.72
84.2
7.4
71.24
80.4
11.1
66.96
84.4
EWC
12.0
85.59
96.2
23.6
62.19
98.0
10.0
84.55
95.2
7.8
82.24
90.6
13.3
78.64
95.0
LwF
13.0
85.14
96.0
27.6
61.76
96.8
10.4
83.70
95.0
8.6
78.96
88.8
14.9
77.39
94.2
ER
94.0
0.54
94.3
97.3
− 1.83
95.5
92.8
2.11
95.7
94.1
− 0.46
92.7
94.6
0.09
94.6
Table 1: Continual learning results on the four LIBERO suites. We compare methods without replay (top) and with replay (bottom); all replay methods use 20 episodes per previous task. ER (gray, italic) replays real demonstrations and serves as an upper bound. R-WAM † replays generated rollouts without rectification. ± denotes the standard deviation over three training seeds. Plast. denotes Plasticity; all metrics follow Section 4.1 .
Table 5
Setting
Acc. (%) ↑
VLM
GPT-5.6 Sol ( OpenAI, 2026 )
93.5
GPT-5.6 Terra
91.0
GPT-5.6 Luna
88.0
Qwen3-VL ( Bai et al., 2025 )
73.5
Prompt
Full prompt
91.0
w/o in-context example
78.0
Table 4: Ablation of completion-aware adaptive truncation. Accuracy is the fraction of predicted truncations that fall inside the human-annotated range. We ablate the VLM (top) and the prompt design (bottom); Terra with the full prompt is our default.
Figure 4: What the physical-consistency score measures. (a) We use the WAM to generate 400 rollouts on LIBERO-Spatial, group them by sflow from most to least consistent, and execute their actions in simulation. Each bar is the fraction of rollouts in that group whose actions complete the task. (b) Optical flows behind sflow for a low-score and a high-score rollout: blue is simulated from the generated actions, orange is estimated from the generated observations.
Figure 8
Figure 7: Which capabilities of a WAM are forgotten. (a) Relative retention r(Δ) (Eq. 6 ) of two probed capabilities, Δ tasks after a task was learned. Circles with solid lines: inverse dynamics modeling (actions from real observations); squares with dashed lines: video prediction (observations from real actions). Yellow: Seq-FT; green: R 2 -WAM. (b) Rollouts generated by the final checkpoints from the same real first frame of T0, with the R 2 -WAM rollout shown after truncation. The rightmost frame is rendered by executing the generated actions in simulation, which reveals whether they accomplish the task.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The five real-world sandwich tasks and their scored steps. Each row shows one demonstration of a task from the head camera, with the wrist camera inset at the lower left; columns show the initial state and the four scored steps, each worth one point of the four-point progress score. Tasks are learned in the order T1 to T5 and share the same scene, so a policy must attend to the instruction to pick the right ingredient.
Spatial
Object
Goal
LIBERO-10
Mean demonstration length
101–150
133–161
89–204
189–416
Generation horizon
128–176
160–192
112–224
208–432
Appendix
Table 5: Demonstration lengths and generation horizons per suite (range over tasks, in steps).
Figure 9: A truncated rollout for the task “put the bowl on the plate”. Top: frames every 8 steps; shaded frames are removed. Bottom: normalized end-effector speed and gripper command, with the cut point of the evaluator. The placement, the release, and the upward retreat are retained, while the subsequent descent and re-grasping loop are removed.
Robot-only simulation, scoring, and top- N selection
2 min
Rectified generative replay in total
18 min
Fine-tuning on the new task (1,000 steps)
1.8 h
Appendix
Table 6: Per-task cost of R 2 -WAM on LIBERO-Spatial (4 × H200). Rectification stages are measured on M=40 candidate rollouts of 128–176 steps.
Variant
SR ↑
NBT ↓
Plast. ↑
ReGen-style replay
68.0
19.15
92.2
R 2 -WAM (ours)
94.0
2.90
95.7
Appendix
Table 7: Comparison with ReGen-style generative replay on LIBERO-Spatial. The ReGen-style variant initializes each rollout from a real observation of the task and applies no consistency filtering; both variants are truncated at task completion by the same VLM evaluator.
Figure 10: Task matrices when scaling the replay buffer on LIBERO-Spatial. Rows are replay buffer sizes of 1, 5, 20, and 50 episodes and columns are the ten tasks. Each panel shows how the success rate of one task evolves as the model continues to learn the following tasks, and the last column lists the final SR and NBT of each row. ER (yellow) replays real demonstrations, and R 2 -WAM (green) replays generated rollouts without real data. Shading is the standard error.
Figure 11: Qualitative real-robot rollouts. The robot makes a sandwich with the instructed ingredient in four steps: grasp the ingredient, place it on the bread, grasp the top bread, and cover the sandwich. Each row is one trial from the head camera with the wrist view inset. Its label gives the method, the instruction, and how many further tasks were learned before testing. The four boxes mark which steps the robot completes, and red ellipses mark the error.
Going beyond predicting robot actions, World Action Models (WAMs) can also generate future visual observations. We build on this generative capability to propose Recurrent Generative Replay (REGEN), a continual imitation learning framework that synthesizes pseudo-replay trajectories, enabling a robot policy to rehearse previously learned tasks without storing their original human demonstrations. During continual adaptation, REGEN recursively queries the WAM to synthesize pseudo-replay trajectories conditioned only on prior task instructions and current-task observations. Experiments in both simulation and real-world manipulation settings show that REGEN reduces catastrophic forgetting by up to 50% relative to sequential fine-tuning, while approaching the performance of privileged experience replay methods that require access to real replay data. Finally, we analyze the factors limiting generated replay, identifying long-horizon visual degradation and action-observation inconsistency as the primary bottlenecks. Our results establish WAMs as a promising foundation for continual robot learning without stored demonstrations.
World Action Models (WAMs) emerge as a promising foundation for policy refinement by predicting the consequences of sampled actions. However, visually plausible predictions can mislead policy refinement if they fail to reflect the input actions. To address this mismatch, we introduce R2-WAM, a two-stage repair-and-reject post-training framework that first improves the consistency of predicted futures with input actions, then uses these futures to select inferior action samples for negative fine-tuning. The repair stage grounds imagination in observed robot behavior through a kinematic alignment score that measures agreement between predicted and demonstrated motion, enabling the predicted video to faithfully reflect its input actions. Using the repaired video model, the rejection stage compares imagined outcomes of sampled and demonstrated actions, selectively applying negative fine-tuning to samples whose predicted task progress falls below the demonstrated reference by a prescribed margin. Together, the two stages extend video prediction from representation learning to consequence-based policy refinement without additional environment interaction or changes to the inference procedure. R2-WAM achieves 93.8% average success on RoboTwin 2.0 across clean and randomized settings. On the long-horizon real-world Fold Shirt task, it achieves 87.5% average success, compared with 0% for Fast-WAM. Our project page is available at https://r2-wam.github.io/.
Ruiyan Xu, Haisheng Su, Sixu Lin +4
The Chinese University of Hong Kong, Shenzhen · Manifold AI · Southeast University +1
Cross-task generalization remains difficult for world action models (WAMs), even when their predicted futures depict plausible manipulation. This gap can be attributed directly: we make this gap directly testable by decoupling a WAM into a world model and an inverse-dynamics model (IDM), then holding the IDM fixed while replacing the source of future images. We introduce Align Once, Evaluate Everywhere (AoE), a camera-space inverse-dynamics model pretrained on large-scale robot data, which provides this recovery interface for different future sources across tasks and environments. Through it, we find that actions recovered from recorded futures of unseen tasks execute almost as well as replayed demonstrations, whereas futures predicted by world models collapse across benchmarks. The failures are not generic: they concentrate on the gripper and interaction geometry while generation capacity is spent on task-irrelevant background. This diagnosis suggests that supervision should target exactly these regions. We therefore introduce the vision-to-action (VITA) loss, which passes generated futures through the frozen AoE and penalizes discrepancies in the recovered actions, redirecting generation toward action-relevant detail. Across two world models and three manipulation benchmarks, this simple change raises the execution success of generated futures from near zero to 47.0%. The same loss also transfers to WAMs and yields consistent gains in native control.
Chen Zhao, Zipeng Ye, Yihan Lin +2
Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and KnowledgeEngineering, MOE, China · Meituan, Beijing 100102, China