cs.AIOct 12, 2026
SaveRAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations
Organizations: Peking University · PKU-PsiBot Joint Lab · Beijing Jiaotong University · University of Macau
Abstract
Recent world action models draw on video generation priors to jointly predict future observations and robot actions. Adapting these priors to control requires determining how future predictions should guide policy learning. RAP-WAM combines predictive Rehearsal and Action Perturbations (RAP) with future-supervised queries and Coupled Action Projection for policy learning. Predictive rehearsal trains on model-generated inputs while retaining recorded targets, and accumulates recurring errors into a spatial prior. We perturb actions and measure changes in video predictions. Action-response-based spatiotemporal credit assignment combines these responses, prediction error, and the rehearsal prior to allocate visual supervision; spatially aggregating the responses determines supervision intensity across future action steps. No separate scoring network or deployment-time probe is required. The base learns from a video-pretrained backbone without intermediate pretraining on external embodied data. We evaluate the fixed base and its agent-refined, task-conditioned system. On 42 RoboDojo tasks, their average success rates are 12.85% and 15.62%, with Scores of 17.86 and 20.59. Across 15 real-robot tasks, the complete system achieves 89.00% task-macro-average success, exceeding the strongest of five baselines by 37.67 percentage points. Across 315 fixed-output simulation windows, 45.21% of response-top20 positions fall outside error-top20, revealing information beyond prediction error.
Figures & tables
Figure 1: Predictive learning and agent-guided policy refinement. a , future latents supervise query alignment during training. CAP predicts action flow-matching velocity ; denoising and unnormalization yield action chunk . b , rehearsal updates spatial prior . The matched probe produces video-denoising velocity predictions and under original and perturbed action inputs, sharing model, condition, video noise, and diffusion level. denotes channel-mean squared discrepancy: compares with , and compares with flow-matching target . Spatial weights combine separately normalized sources, . Temporal weights use , repeated over action steps. Weight construction is detached. c , the agent proposes post-training interventions. Evaluation retains improvements by task; other tasks keep their policies. Accepted and rejected outcomes inform subsequent rounds. Each task selects one complete policy. Scenes and fields are schematic.
| Model | Pretr. | Gen-Std | Gen-Rand | Precision | Long-Hor. | Memory | Open | Average |
|---|---|---|---|---|---|---|---|---|
| VLA and other action policies | ||||||||
| G0.5 ( Liu et al., 2026 ) | Yes | 27.72/21.44 | 9.20 / 4.22 | 28.25 / 20.42 | 44.12/32.25 | 8.61/7.33 | 1.73/1.58 | 20.23 / 14.88 |
| Xiaomi-Robotics-1 ( Xiaomi Robotics Team et al., 2026 ) | Yes | 35.65 / 28.00 | 11.44 / 6.00 | 26.69/18.83 | 38.39/23.67 | 7.81/6.56 | 3.94/3.58 | 20.07/13.93 |
| Hy-Embodied-0.5-VLA ( Zhang et al., 2026 ) | Yes | 21.98/17.00 | 1.57/0.00 | 13.81/8.00 | 25.74/14.92 | 13.37 / 12.11 | 0.65/0.58 | 13.07/8.80 |
| KinRT ( Yang et al., 2026c ) | Yes | 20.94/14.78 | 7.09/2.44 | 15.65/9.92 | 26.40/18.08 | 4.82/3.56 | 4.23 / 3.83 | 13.02/8.80 |
| SimpleMemVLA ( Yin et al., 2026 ) | No | 10.38/7.11 | 2.33/0.78 | 7.42/2.92 | 14.58/5.50 | 33.71 / 33.22 | 0.85/0.75 | 12.58/9.27 |
Table 1: RoboDojo comparison: Score / SR (%). Pretr.: external embodied pretraining reported for the method. Video/VLM initialization and target-task demonstrations are excluded. : AHA-WAM uses robot-data pretraining for real-world experiments, but not in its default simulation setup (Appendix B ). Score and SR are ranked separately per column; highest and second-highest values are bold and underlined , including ties. : VLA supervised by a frozen video generation model during training. : matched 42-task coverage for our RAP-WAM base and post-training system; each task is averaged over its valid evaluation episodes.
Figure 2: Complementary response signals and spatiotemporal supervision. a , signed spatial credit redistribution. b , sensitivity to removing prior, response, or half-error. c , temporal action credit in three phone-packing windows (W1–W3), with a unit reference. d , response-top20 membership within or outside the error-top20 set. e , rank correlation across 35 condition medians. Panels a, b, d, and e use the same 315 RoboDojo windows. Squares and segments indicate medians and IQRs; these fixed-output diagnostics involve no retraining.
Figure 3: Credit across three timescales. a , prior spatial patterns at 3.3%, 60%, and 100% of the same can-placement run (top, independently normalized), and the raw spatial mean at six checkpoints (bottom, logarithmic axis). b , response-top40 locations across four future groups for the newly trained three-can tray-placement policy. c , stacked component shares (top) and fixed-output removal sensitivity (bottom) for , , and across all eleven exported windows of the same demonstration. Shares exclude the unit floor of ; sensitivity is total variation (TV) between normalized full and component-removed weights, retaining the unit floor. Panel c uses all spatial cells and future groups. All panels use the same training run; b–c use its final checkpoint. Definitions and display protocols appear in Appendix F .
Figure 4: Real-robot evaluation and four recorded task sequences. (a) Success rates across fifteen tasks and six methods; bar labels show percentages. (b–e) Tasks 04, 06, 08, and 11 show height-based bottle ordering, numbered-block rearrangement, covering numerical extrema, and fan insertion. Each task occupies four consecutive frames in a two-row, eight-column grid. Timestamps refer to the source video. Frames are taken directly from the original videos with torsos and incidental text masked; task numerals are retained. The phone-box assembly task is provided in Appendix G .
Appendix figures & tables63 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | RoboDojo | Physical-robot data |
|---|---|---|
| Video initialization | Wan2.2-TI2V-5B | Wan2.2-TI2V-5B |
| Input | Head/two-wrist mosaic | Head |
| Action/state dimension | 14/14 | Phone 28/28; numbers 32/32 |
| Language | Precomputed instruction embeddings | None |
| Observation/future/action length | 1/16/16 | 1/16/16 |
| Queries/alignment width | 16/128 | 16/128 |
Table 2: Training settings. Physical training uses the same core with task-specific interfaces; physical success rates appear in Figure 4 .
| Task | Capability | SR | Score |
|---|---|---|---|
| Align blocks | Open | 0.00 | 0.00 |
| Arrange largest number | Generalization | 4.00 | 5.80 |
| Build tower | Precision | 72.00 | 79.60 |
| Classify objects | Long-Horizon | 2.00 | 6.50 |
| Classify objects by language | Open | 0.00 | 1.60 |
| Cover blocks | Memory | 22.00 | 26.10 |
Table 3: Post-training system results across 42 tasks. SR is in percent; Score measures partial completion. Each task is averaged over its valid evaluation episodes.
| Task | Capability | SR | Score |
|---|---|---|---|
| Align blocks | Open | 0.00 | 0.00 |
| Arrange largest number | Generalization | 2.00 | 3.80 |
| Build tower | Precision | 60.00 | 69.80 |
| Classify objects | Long-Horizon | 0.00 | 4.40 |
| Classify objects by language | Open | 0.00 | 1.60 |
| Cover blocks | Memory | 16.00 | 20.60 |
Table 4: RAP-WAM base results across 42 tasks, using the single step-742500 checkpoint before post-training. Task order and metric definitions match Table 3 . SR (%) and Score are computed over valid evaluation episodes.
Figure 5: Why error, response, and rehearsal prior provide different supervision. An exact constructed example, not measured robot results. At four locations, let , , , and , with , target zero, and prior . For , use the expected response . (a) A has error without action sensitivity; B has sensitivity without error. (b) The four locations separate residual, response, combined, and historical signals. (c) Applying the spatial rule to gives positive weights . (d) After normalization, error-only allocation gives , whereas the full rule (Full) gives . Thus current error alone cannot reproduce the allocation. This illustration uses expected response; training uses a single sampled perturbation, as discussed in Appendix C.2 .
Figure 6: Task-level changes under the selected assignment. Aligned rows show all 42 tasks, grouped by capability: (a) SR changes and (b) partial-completion Score changes relative to the base. Red indicates 24 tasks with higher SR, blue four tasks with higher Score only, and gray 14 unchanged tasks. These are observed results under the selected task-policy assignment, not an independent test of regression-free improvement.
| Index | Task |
|---|---|
| 00 | Arrange largest number |
| 01 | Build tower |
| 02 | Classify objects |
| 03 | Cover blocks |
| 04 | Deposit coin |
| 05 | Spell RoboDojo |
Table 5: Task identities for the 35 diagnostic conditions.
Figure 7: Condition-resolved supervision allocation. Both panels retain all 35 conditions, each with nine windows; IDs follow Table 5 . (a) Median total-variation distance after removing each source from the normalized spatial weights, in percent. (b) Median normalized weight mass assigned to response-top20 positions under error-only reweighting (blue) and the full credit rule (red, labeled Full); each connector pairs the same condition. The dashed line is uniform allocation, 20%. The fields and predictions are fixed throughout.
Figure 8: Additional recorded task sequences (I). Tasks 01, 02, 03, and 05 show circle-size selection, cake unpacking and placement, fruit placement, and bowl nesting. Frames are event-aligned within each source video; torsos and incidental text are masked before display downsampling and compression.
Figure 9: Additional recorded task sequences (II). Tasks 07, 09, and 10 show can placement, food placement, and coaster placement. Task 12 aligns disposable towels and places a thick towel on top; the torso and packaging text are masked; the recorded scene is otherwise retained. Tasks 13 and 14 show shoe organization and mahjong kong formation. Task 15 places a hamburger in a box and closes its lid, ending with both arms returned. Timestamps and frame-selection conventions follow Figure 8 .
Figure 10: Additional recorded task sequence (III). Task 16 packs a phone into its box and fits the lid. The four original-video frames show the initial configuration, phone positioning, lid placement, and the final configuration with both arms returned. Torso and incidental packaging and equipment text are masked; timestamps refer to the source video.
Figure 11: Task assignments in the retained repertoire. Each cell counts tasks assigned to the base or a retained specialist within one capability group; blank cells denote zero assignments. P01–P16 identify existing checkpoints, with identities recorded in the accompanying manifest. This is the final repertoire snapshot, not an iteration history or a policy-removal experiment.
Figure 12: Why retain task specialists? (a) All 30 candidate checkpoint identities, evaluated using the same 42-task aggregation. Filled blue circles are retained specialists, open circles other candidates, the square is the base, and the diamond is the task-conditioned system. Axis origins are truncated to display the observed range. (b) For each of the 29 non-base candidates, tasks with lower SR (left) or higher SR (right) relative to the base; unchanged tasks are omitted from the bars. Candidates are ordered by average SR, with Score breaking ties. P identifiers match Figure 11 ; U identifiers denote unused candidates.
Figure 13: Prediction error over the full future horizon. Solid blue curves show the world model. Dashed orange curves show the LPIPS between the unchanged current image and each recorded future frame: the current image is reused as the future prediction, without model inference. Lines are task means, and shaded bands show the 25th–75th percentiles across the selected task windows, not confidence intervals. Both panels use the same vertical scale. Relative reductions use all 16 frames; the condition is excluded.
Figure 14: Task 01: Cover the largest and smallest circles.
Figure 15: Task 02: Unpack the cake and place it in a bowl.
Figure 16: Task 03: Place fruits on a tray.
Figure 17: Task 04: Reorder three different types of bottles by height.
Figure 18: Task 05: Nest three bowls.
Figure 19: Task 06: Rearrange numbered blocks.
Figure 20: Task 07: Place three cans on a tray.
Figure 21: Task 08: Cover the largest and smallest numbers.
Figure 22: Task 09: Place food in separate dishes.
Figure 23: Task 10: Place a can on a coaster.
Figure 24: Task 11: Insert a folding fan into an open-top vessel.
Figure 25: Task 12: Align disposable towels and place a thick towel on top.
Figure 26: Task 13: Arrange a pair of shoes.
Figure 27: Task 14: Mahjong kong formation.
Figure 28: Task 16: Pack a phone into its box.
Figure 29: RoboDojo task 00: Arrange numbers.
Figure 30: RoboDojo task 01: Build a tower.
Figure 31: RoboDojo task 02: Sort objects by category.
Figure 32: RoboDojo task 03: Remember covered block colors.
Figure 33: RoboDojo task 04: Insert a coin.
Figure 34: RoboDojo task 05: Spell RoboDojo.
Figure 35: RoboDojo task 06: Tighten color-matched screws.
Figure 36: RoboDojo task 07: Store eggs and close the holder.
Figure 37: RoboDojo task 08: Put pens into a holder.
Figure 38: RoboDojo task 09: Fold clothes.
Figure 39: RoboDojo task 10: Hang mugs.
Figure 40: RoboDojo task 11: Recall object placement order.
Figure 41: RoboDojo task 12: Insert a thick card.
Figure 42: RoboDojo task 13: Place tubes in a rack.
Figure 43: RoboDojo task 14: Form a mahjong kong.
Figure 44: RoboDojo task 15: Toast bread.
Figure 45: RoboDojo task 16: Match a remembered conveyor object.
Figure 46: RoboDojo task 17: Organize a desk.
Figure 47: RoboDojo task 18: Pack objects facing left.
Figure 48: RoboDojo task 19: Play the xylophone.
Figure 49: RoboDojo task 20: Assemble a stacking toy.
Figure 50: RoboDojo task 21: Play tic-tac-toe.
Figure 51: RoboDojo task 22: Plug in a charger.
Figure 52: RoboDojo task 23: Pour balls into a vase.
Figure 53: RoboDojo task 24: Pour liquid into a cup.
Figure 54: RoboDojo task 25: Count button presses.
Figure 55: RoboDojo task 26: Align a T-shaped block.
Figure 56: RoboDojo task 27: Discard bottles.
Figure 57: RoboDojo task 28: Order nesting dolls.
Figure 58: RoboDojo task 29: Stack textured blocks.
Figure 59: RoboDojo task 30: Stack bowls.
Figure 60: RoboDojo task 31: Store headphones and a laptop.
Figure 61: RoboDojo task 32: Swap T-shaped blocks.
Figure 62: RoboDojo task 33: Swap blocks via an empty mat.
Figure 63: RoboDojo task 34: Sweep blocks.