Organizations: University of Electronic Science and Technology of China · Shanghai AI Lab · Fudan University · Shanghai Jiao Tong University · Tongji University · Shanghai Innovation Institute
World action models (WAMs) built on pretrained video generation models adapt quickly to new robotic tasks, but they forget previously learned tasks under sequential fine-tuning. Experience replay (ER) mitigates such forgetting by rehearsing data of previous tasks, yet this data is often unavailable in robotics, where a policy is deployed as weights only and its training data is not shared along with it. In this work, we ask whether a WAM can achieve data-free continual learning by regenerating its own past. Our key observation is that a WAM can indeed autoregressively regenerate complete rollouts of previously learned tasks and replay them to mitigate forgetting. However, replaying such raw rollouts still lags far behind replaying real data, which we attribute to two issues. First, the WAM is unable to recognize task completion and keeps generating purposeless motion after the task is done. Second, under open-loop generation, the generated actions gradually drift away from the generated observations. To address both issues, we present rectified generative replay (R2-WAM), which truncates generated rollouts at task completion with an agentic evaluator and filters out rollouts whose actions are physically inconsistent with the observations. On the four LIBERO suites and five real-world tasks, R2-WAM reduces forgetting by over 90% relative to sequential fine-tuning and nearly matches ER without accessing any data from previous tasks. Our analysis further reveals that forgetting in WAMs concentrates in inverse dynamics modeling: a forgotten WAM still produces visually plausible observations of previous tasks while its generated actions fail. Project page: https://kaipengfang.github.io/r2-wam/
Figures & tables
Figure 1: (a) In continual learning, a WAM is sequentially fine-tuned on new tasks and evaluated on all tasks learned so far. Experience replay (ER) mitigates forgetting of earlier tasks but requires access to their data. (b) R 2 -WAM exploits the autoregressive generation of WAMs to achieve data-free continual learning, and its rectified generative replay attains performance comparable to ER.
Figure 2: Overview of R 2 -WAM. After learning task k−1 , WAMk−1 ① generates M candidate rollouts of that task with the instruction ℓk−1 , which are then ② rectified by truncating each rollout at task completion and filtering out physically inconsistent ones. The top- N retained rollouts Rk−1 join those of earlier tasks to form the buffer R<k , which is ③ replayed with the new-task demonstrations Dk when learning task k . The same procedure then repeats for the next stage k+1 .
Figure 3: Real-robot setup.
Method
LIBERO-Spatial
LIBERO-Object
LIBERO-Goal
LIBERO-10
Average
SR ↑
NBT ↓
Plast. ↑
SR ↑
NBT ↓
Plast. ↑
SR ↑
NBT ↓
Plast. ↑
SR ↑
NBT ↓
Plast. ↑
SR ↑
NBT ↓
Plast. ↑
Seq-FT
11.2
86.07
95.8
21.2
67.84
97.4
9.8
84.87
94.8
7.6
82.00
89.6
12.5
80.19
94.4
Seq-LoRA
9.6
75.42
86.2
17.6
52.47
87.0
9.8
68.72
84.2
7.4
71.24
80.4
11.1
66.96
84.4
EWC
12.0
85.59
96.2
23.6
62.19
98.0
10.0
84.55
95.2
7.8
82.24
90.6
13.3
78.64
95.0
LwF
13.0
85.14
96.0
27.6
61.76
96.8
10.4
83.70
95.0
8.6
78.96
88.8
14.9
77.39
94.2
ER
94.0
0.54
94.3
97.3
− 1.83
95.5
92.8
2.11
95.7
94.1
− 0.46
92.7
94.6
0.09
94.6
Table 1: Continual learning results on the four LIBERO suites. We compare methods without replay (top) and with replay (bottom); all replay methods use 20 episodes per previous task. ER (gray, italic) replays real demonstrations and serves as an upper bound. R-WAM † replays generated rollouts without rectification. ± denotes the standard deviation over three training seeds. Plast. denotes Plasticity; all metrics follow Section 4.1 .
Table 5
Setting
Acc. (%) ↑
VLM
GPT-5.6 Sol ( OpenAI, 2026 )
93.5
GPT-5.6 Terra
91.0
GPT-5.6 Luna
88.0
Qwen3-VL ( Bai et al., 2025 )
73.5
Prompt
Full prompt
91.0
w/o in-context example
78.0
Table 4: Ablation of completion-aware adaptive truncation. Accuracy is the fraction of predicted truncations that fall inside the human-annotated range. We ablate the VLM (top) and the prompt design (bottom); Terra with the full prompt is our default.
Figure 4: What the physical-consistency score measures. (a) We use the WAM to generate 400 rollouts on LIBERO-Spatial, group them by sflow from most to least consistent, and execute their actions in simulation. Each bar is the fraction of rollouts in that group whose actions complete the task. (b) Optical flows behind sflow for a low-score and a high-score rollout: blue is simulated from the generated actions, orange is estimated from the generated observations.
Figure 8
Figure 7: Which capabilities of a WAM are forgotten. (a) Relative retention r(Δ) (Eq. 6 ) of two probed capabilities, Δ tasks after a task was learned. Circles with solid lines: inverse dynamics modeling (actions from real observations); squares with dashed lines: video prediction (observations from real actions). Yellow: Seq-FT; green: R 2 -WAM. (b) Rollouts generated by the final checkpoints from the same real first frame of T0, with the R 2 -WAM rollout shown after truncation. The rightmost frame is rendered by executing the generated actions in simulation, which reveals whether they accomplish the task.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The five real-world sandwich tasks and their scored steps. Each row shows one demonstration of a task from the head camera, with the wrist camera inset at the lower left; columns show the initial state and the four scored steps, each worth one point of the four-point progress score. Tasks are learned in the order T1 to T5 and share the same scene, so a policy must attend to the instruction to pick the right ingredient.
Spatial
Object
Goal
LIBERO-10
Mean demonstration length
101–150
133–161
89–204
189–416
Generation horizon
128–176
160–192
112–224
208–432
Appendix
Table 5: Demonstration lengths and generation horizons per suite (range over tasks, in steps).
Figure 9: A truncated rollout for the task “put the bowl on the plate”. Top: frames every 8 steps; shaded frames are removed. Bottom: normalized end-effector speed and gripper command, with the cut point of the evaluator. The placement, the release, and the upward retreat are retained, while the subsequent descent and re-grasping loop are removed.
Robot-only simulation, scoring, and top- N selection
2 min
Rectified generative replay in total
18 min
Fine-tuning on the new task (1,000 steps)
1.8 h
Appendix
Table 6: Per-task cost of R 2 -WAM on LIBERO-Spatial (4 × H200). Rectification stages are measured on M=40 candidate rollouts of 128–176 steps.
Variant
SR ↑
NBT ↓
Plast. ↑
ReGen-style replay
68.0
19.15
92.2
R 2 -WAM (ours)
94.0
2.90
95.7
Appendix
Table 7: Comparison with ReGen-style generative replay on LIBERO-Spatial. The ReGen-style variant initializes each rollout from a real observation of the task and applies no consistency filtering; both variants are truncated at task completion by the same VLM evaluator.
Figure 10: Task matrices when scaling the replay buffer on LIBERO-Spatial. Rows are replay buffer sizes of 1, 5, 20, and 50 episodes and columns are the ten tasks. Each panel shows how the success rate of one task evolves as the model continues to learn the following tasks, and the last column lists the final SR and NBT of each row. ER (yellow) replays real demonstrations, and R 2 -WAM (green) replays generated rollouts without real data. Shading is the standard error.
Figure 11: Qualitative real-robot rollouts. The robot makes a sandwich with the instructed ingredient in four steps: grasp the ingredient, place it on the bread, grasp the top bread, and cover the sandwich. Each row is one trial from the head camera with the wrist view inset. Its label gives the method, the instruction, and how many further tasks were learned before testing. The four boxes mark which steps the robot completes, and red ellipses mark the error.
Renmin University of China, Beijing 100872, China · Key Laboratory of Data Engineering and KnowledgeEngineering, MOE, China · Meituan, Beijing 100102, China