cs.AIOct 12, 2026

RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations

Authors: Yuanpeng He, Ceyao Zhang, Fangjing Li, Kefei Zhu, Lijian Li, Yueheng Li, Tianjia He, Wenpin Jiao, +3 more

Organizations: Peking University · PKU-PsiBot Joint Lab · Beijing Jiaotong University · University of Macau

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent world action models draw on video generation priors to jointly predict future observations and robot actions. Adapting these priors to control requires determining how future predictions should guide policy learning. RAP-WAM combines predictive Rehearsal and Action Perturbations (RAP) with future-supervised queries and Coupled Action Projection for policy learning. Predictive rehearsal trains on model-generated inputs while retaining recorded targets, and accumulates recurring errors into a spatial prior. We perturb actions and measure changes in video predictions. Action-response-based spatiotemporal credit assignment combines these responses, prediction error, and the rehearsal prior to allocate visual supervision; spatially aggregating the responses determines supervision intensity across future action steps. No separate scoring network or deployment-time probe is required. The base learns from a video-pretrained backbone without intermediate pretraining on external embodied data. We evaluate the fixed base and its agent-refined, task-conditioned system. On 42 RoboDojo tasks, their average success rates are 12.85% and 15.62%, with Scores of 17.86 and 20.59. Across 15 real-robot tasks, the complete system achieves 89.00% task-macro-average success, exceeding the strongest of five baselines by 37.67 percentage points. Across 315 fixed-output simulation windows, 45.21% of response-top20 positions fall outside error-top20, revealing information beyond prediction error.

Figures & tables

Figure 1: Predictive learning and agent-guided policy refinement. a , future latents supervise query alignment during training. CAP predicts action flow-matching velocity vθav_{\theta}^{a} ; denoising and unnormalization yield action chunk A^\hat{A} . b , rehearsal updates spatial prior PP . The matched probe produces video-denoising velocity predictions vv and v+v^{+} under original and perturbed action inputs, sharing model, condition, video noise, and diffusion level. Δ\Delta denotes channel-mean squared discrepancy: SS compares v+v^{+} with vv , and EE compares vv with flow-matching target vz∗v_{z}^{*} . Spatial weights combine separately normalized sources, W=1+N(P)+N(S)+0.5N(E)W=1+\mathcal{N}(P)+\mathcal{N}(S)+0.5\mathcal{N}(E) . Temporal weights use 1+Nt(mean⁡hwS)1+\mathcal{N}_{t}(\operatorname{mean}_{hw}S) , repeated over action steps. Weight construction is detached. c , the agent proposes post-training interventions. Evaluation retains improvements by task; other tasks keep their policies. Accepted and rejected outcomes inform subsequent rounds. Each task selects one complete policy. Scenes and fields are schematic.
ModelPretr.Gen-StdGen-RandPrecisionLong-Hor.MemoryOpenAverage
VLA and other action policies
G0.5 ( Liu et al., 2026 )Yes27.72/21.449.20 / 4.2228.25 / 20.4244.12/32.258.61/7.331.73/1.5820.23 / 14.88
Xiaomi-Robotics-1 ( Xiaomi Robotics Team et al., 2026 )Yes35.65 / 28.0011.44 / 6.0026.69/18.8338.39/23.677.81/6.563.94/3.5820.07/13.93
Hy-Embodied-0.5-VLA ( Zhang et al., 2026 )Yes21.98/17.001.57/0.0013.81/8.0025.74/14.9213.37 / 12.110.65/0.5813.07/8.80
KinRT ( Yang et al., 2026c )Yes20.94/14.787.09/2.4415.65/9.9226.40/18.084.82/3.564.23 / 3.8313.02/8.80
SimpleMemVLA ( Yin et al., 2026 )No10.38/7.112.33/0.787.42/2.9214.58/5.5033.71 / 33.220.85/0.7512.58/9.27
Table 1: RoboDojo comparison: Score / SR (%). Pretr.: external embodied pretraining reported for the method. Video/VLM initialization and target-task demonstrations are excluded. ∗* : AHA-WAM uses robot-data pretraining for real-world experiments, but not in its default simulation setup (Appendix B ). Score and SR are ranked separately per column; highest and second-highest values are bold and underlined , including ties. ‡\ddagger : VLA supervised by a frozen video generation model during training. \lx@sectionsign\lx@sectionsign : matched 42-task coverage for our RAP-WAM base and post-training system; each task is averaged over its valid evaluation episodes.
Figure 2: Complementary response signals and spatiotemporal supervision. a , signed spatial credit redistribution. b , sensitivity to removing prior, response, or half-error. c , temporal action credit in three phone-packing windows (W1–W3), with a unit reference. d , response-top20 membership within or outside the error-top20 set. e , rank correlation across 35 condition medians. Panels a, b, d, and e use the same 315 RoboDojo windows. Squares and segments indicate medians and IQRs; these fixed-output diagnostics involve no retraining.
Figure 3: Credit across three timescales. a , prior spatial patterns at 3.3%, 60%, and 100% of the same can-placement run (top, independently normalized), and the raw spatial mean at six checkpoints (bottom, logarithmic axis). b , response-top40 locations across four future groups for the newly trained three-can tray-placement policy. c , stacked component shares (top) and fixed-output removal sensitivity (bottom) for P\mathcal{P} , S\mathcal{S} , and 0.5E0.5\mathcal{E} across all eleven exported windows of the same demonstration. Shares exclude the unit floor of W\mathcal{W} ; sensitivity is total variation (TV) between normalized full and component-removed weights, retaining the unit floor. Panel c uses all spatial cells and future groups. All panels use the same training run; b–c use its final checkpoint. Definitions and display protocols appear in Appendix F .
Figure 4: Real-robot evaluation and four recorded task sequences. (a) Success rates across fifteen tasks and six methods; bar labels show percentages. (b–e) Tasks 04, 06, 08, and 11 show height-based bottle ordering, numbered-block rearrangement, covering numerical extrema, and fan insertion. Each task occupies four consecutive frames in a two-row, eight-column grid. Timestamps refer to the source video. Frames are taken directly from the original videos with torsos and incidental text masked; task numerals are retained. The phone-box assembly task is provided in Appendix G .
Appendix figures & tables63 assets

Supplementary material from the paper’s appendix.

Appendix

SettingRoboDojoPhysical-robot data
Video initializationWan2.2-TI2V-5BWan2.2-TI2V-5B
InputHead/two-wrist 384×320384\times 320 mosaicHead 224×224224\times 224
Action/state dimension14/14Phone 28/28; numbers 32/32
LanguagePrecomputed instruction embeddingsNone
Observation/future/action length1/16/161/16/16
Queries/alignment width16/12816/128
Appendix
Table 2: Training settings. Physical training uses the same core with task-specific interfaces; physical success rates appear in Figure 4 .
TaskCapabilitySRScore
Align blocksOpen0.000.00
Arrange largest numberGeneralization4.005.80
Build towerPrecision72.0079.60
Classify objectsLong-Horizon2.006.50
Classify objects by languageOpen0.001.60
Cover blocksMemory22.0026.10
Appendix
Table 3: Post-training system results across 42 tasks. SR is in percent; Score measures partial completion. Each task is averaged over its valid evaluation episodes.
TaskCapabilitySRScore
Align blocksOpen0.000.00
Arrange largest numberGeneralization2.003.80
Build towerPrecision60.0069.80
Classify objectsLong-Horizon0.004.40
Classify objects by languageOpen0.001.60
Cover blocksMemory16.0020.60
Appendix
Table 4: RAP-WAM base results across 42 tasks, using the single step-742500 checkpoint before post-training. Task order and metric definitions match Table 3 . SR (%) and Score are computed over valid evaluation episodes.
Figure 5: Why error, response, and rehearsal prior provide different supervision. An exact constructed example, not measured robot results. At four locations, let fA(a)=1f_{A}(a)=1 , fB(a)=2af_{B}(a)=2a , fC(a)=1+2af_{C}(a)=1+2a , and fD(a)=0f_{D}(a)=0 , with a=0a=0 , target zero, and prior P=(0,0,0,1)P=(0,0,0,1) . For δ∼N(0,0.12)\delta\sim\mathcal{N}(0,0.1^{2}) , use the expected response Sˉ=EδS\bar{S}=\mathbb{E}_{\delta}S . (a) A has error without action sensitivity; B has sensitivity without error. (b) The four locations separate residual, response, combined, and historical signals. (c) Applying the spatial rule to P,Sˉ,EP,\bar{S},E gives positive weights (1.5,2,2.5,2)(1.5,2,2.5,2) . (d) After normalization, error-only allocation gives (30,20,30,20)%(30,20,30,20)\% , whereas the full rule (Full) gives (18.75,25,31.25,25)%(18.75,25,31.25,25)\% . Thus current error alone cannot reproduce the allocation. This illustration uses expected response; training uses a single sampled perturbation, as discussed in Appendix C.2 .
Figure 6: Task-level changes under the selected assignment. Aligned rows show all 42 tasks, grouped by capability: (a) SR changes and (b) partial-completion Score changes relative to the base. Red indicates 24 tasks with higher SR, blue four tasks with higher Score only, and gray 14 unchanged tasks. These are observed results under the selected task-policy assignment, not an independent test of regression-free improvement.
IndexTask
00Arrange largest number
01Build tower
02Classify objects
03Cover blocks
04Deposit coin
05Spell RoboDojo
Appendix
Table 5: Task identities for the 35 diagnostic conditions.
Figure 7: Condition-resolved supervision allocation. Both panels retain all 35 conditions, each with nine windows; IDs follow Table 5 . (a) Median total-variation distance after removing each source from the normalized spatial weights, in percent. (b) Median normalized weight mass assigned to response-top20 positions under error-only reweighting (blue) and the full credit rule (red, labeled Full); each connector pairs the same condition. The dashed line is uniform allocation, 20%. The fields and predictions are fixed throughout.
Figure 8: Additional recorded task sequences (I). Tasks 01, 02, 03, and 05 show circle-size selection, cake unpacking and placement, fruit placement, and bowl nesting. Frames are event-aligned within each source video; torsos and incidental text are masked before display downsampling and compression.
Figure 9: Additional recorded task sequences (II). Tasks 07, 09, and 10 show can placement, food placement, and coaster placement. Task 12 aligns disposable towels and places a thick towel on top; the torso and packaging text are masked; the recorded scene is otherwise retained. Tasks 13 and 14 show shoe organization and mahjong kong formation. Task 15 places a hamburger in a box and closes its lid, ending with both arms returned. Timestamps and frame-selection conventions follow Figure 8 .
Figure 10: Additional recorded task sequence (III). Task 16 packs a phone into its box and fits the lid. The four original-video frames show the initial configuration, phone positioning, lid placement, and the final configuration with both arms returned. Torso and incidental packaging and equipment text are masked; timestamps refer to the source video.
Figure 11: Task assignments in the retained repertoire. Each cell counts tasks assigned to the base or a retained specialist within one capability group; blank cells denote zero assignments. P01–P16 identify existing checkpoints, with identities recorded in the accompanying manifest. This is the final repertoire snapshot, not an iteration history or a policy-removal experiment.
Figure 12: Why retain task specialists? (a) All 30 candidate checkpoint identities, evaluated using the same 42-task aggregation. Filled blue circles are retained specialists, open circles other candidates, the square is the base, and the diamond is the task-conditioned system. Axis origins are truncated to display the observed range. (b) For each of the 29 non-base candidates, tasks with lower SR (left) or higher SR (right) relative to the base; unchanged tasks are omitted from the bars. Candidates are ordered by average SR, with Score breaking ties. P identifiers match Figure 11 ; U identifiers denote unused candidates.
Figure 13: Prediction error over the full future horizon. Solid blue curves show the world model. Dashed orange curves show the LPIPS between the unchanged current image and each recorded future frame: the current image is reused as the future prediction, without model inference. Lines are task means, and shaded bands show the 25th–75th percentiles across the selected task windows, not confidence intervals. Both panels use the same vertical scale. Relative reductions use all 16 frames; the condition is excluded.
Figure 14: Task 01: Cover the largest and smallest circles.
Figure 15: Task 02: Unpack the cake and place it in a bowl.
Figure 16: Task 03: Place fruits on a tray.
Figure 17: Task 04: Reorder three different types of bottles by height.
Figure 18: Task 05: Nest three bowls.
Figure 19: Task 06: Rearrange numbered blocks.
Figure 20: Task 07: Place three cans on a tray.
Figure 21: Task 08: Cover the largest and smallest numbers.
Figure 22: Task 09: Place food in separate dishes.
Figure 23: Task 10: Place a can on a coaster.
Figure 24: Task 11: Insert a folding fan into an open-top vessel.
Figure 25: Task 12: Align disposable towels and place a thick towel on top.
Figure 26: Task 13: Arrange a pair of shoes.
Figure 27: Task 14: Mahjong kong formation.
Figure 28: Task 16: Pack a phone into its box.
Figure 29: RoboDojo task 00: Arrange numbers.
Figure 30: RoboDojo task 01: Build a tower.
Figure 31: RoboDojo task 02: Sort objects by category.
Figure 32: RoboDojo task 03: Remember covered block colors.
Figure 33: RoboDojo task 04: Insert a coin.
Figure 34: RoboDojo task 05: Spell RoboDojo.
Figure 35: RoboDojo task 06: Tighten color-matched screws.
Figure 36: RoboDojo task 07: Store eggs and close the holder.
Figure 37: RoboDojo task 08: Put pens into a holder.
Figure 38: RoboDojo task 09: Fold clothes.
Figure 39: RoboDojo task 10: Hang mugs.
Figure 40: RoboDojo task 11: Recall object placement order.
Figure 41: RoboDojo task 12: Insert a thick card.
Figure 42: RoboDojo task 13: Place tubes in a rack.
Figure 43: RoboDojo task 14: Form a mahjong kong.
Figure 44: RoboDojo task 15: Toast bread.
Figure 45: RoboDojo task 16: Match a remembered conveyor object.
Figure 46: RoboDojo task 17: Organize a desk.
Figure 47: RoboDojo task 18: Pack objects facing left.
Figure 48: RoboDojo task 19: Play the xylophone.
Figure 49: RoboDojo task 20: Assemble a stacking toy.
Figure 50: RoboDojo task 21: Play tic-tac-toe.
Figure 51: RoboDojo task 22: Plug in a charger.
Figure 52: RoboDojo task 23: Pour balls into a vase.
Figure 53: RoboDojo task 24: Pour liquid into a cup.
Figure 54: RoboDojo task 25: Count button presses.
Figure 55: RoboDojo task 26: Align a T-shaped block.
Figure 56: RoboDojo task 27: Discard bottles.
Figure 57: RoboDojo task 28: Order nesting dolls.
Figure 58: RoboDojo task 29: Stack textured blocks.
Figure 59: RoboDojo task 30: Stack bowls.
Figure 60: RoboDojo task 31: Store headphones and a laptop.
Figure 61: RoboDojo task 32: Swap T-shaped blocks.
Figure 62: RoboDojo task 33: Swap blocks via an empty mat.
Figure 63: RoboDojo task 34: Sweep blocks.

Explore similar work

CardsList
Oct 4, 2026

R2R^2-WAM: Repair-and-Reject Post-Training for World Action Models

World Action Models (WAMs) emerge as a promising foundation for policy refinement by predicting the consequences of sampled actions. However, visually plausible predictions can mislead policy refinement if they fail to reflect the input actions. To address this mismatch, we introduce R2R^2-WAM, a two-stage repair-and-reject post-training framework that first improves the consistency of predicted futures with input actions, then uses these futures to select inferior action samples for negative fine-tuning. The repair stage grounds imagination in observed robot behavior through a kinematic alignment score that measures agreement between predicted and demonstrated motion, enabling the predicted video to faithfully reflect its input actions. Using the repaired video model, the rejection stage compares imagined outcomes of sampled and demonstrated actions, selectively applying negative fine-tuning to samples whose predicted task progress falls below the demonstrated reference by a prescribed margin. Together, the two stages extend video prediction from representation learning to consequence-based policy refinement without additional environment interaction or changes to the inference procedure. R2R^2-WAM achieves 93.8% average success on RoboTwin 2.0 across clean and randomized settings. On the long-horizon real-world Fold Shirt task, it achieves 87.5% average success, compared with 0% for Fast-WAM. Our project page is available at https://r2-wam.github.io/.

Ruiyan Xu, Haisheng Su, Sixu Lin +4

The Chinese University of Hong Kong, Shenzhen · Manifold AI · Southeast University +1

World Action ModelsAction-Conditioned World ModelsModel-Based RLRL Post-Training
AskPDF
Aug 23, 2026

WAM-OPD: Joint Video-Action Supervision for World Action Model Post-Training with On-Policy Distillation

World Action Models (WAMs) generate both future video and robot actions, offering two connected outputs for post-training supervision. How can a pretrained WAM learn from a stronger Teacher on the histories it encounters during execution? We present WAM-OPD, which collects Student rollout histories and queries a Teacher for paired video and action targets. The Student learns from both targets while retaining its one-step video and action generation at deployment. Across 12 RoboTwin 2.0 tasks, WAM-OPD improves average success from 33.8% to 65.7%; across four real-robot tasks, it improves average success from 51.4% to 64.6%. With the collected Student histories held fixed, joint video-action supervision achieves the highest observed success on all three ablation tasks, while either modality alone also improves performance. A separate comparison with Teacher-generated histories finds task-dependent differences between the two history sources. These results demonstrate the value of paired video-action supervision for improving WAM policies without increasing their deployed sampling budget.

Liuhaichen Yang, Zhengyang Zhong, Hanshang Zhu +13

University College London · Shanghai Jiao Tong University · The University of Manchester +3

World Action ModelsAction-Conditioned World ModelsRobotic ManipulationOn-Policy Distillation
AskPDF
Sep 13, 2026

World-Action Models for Robot Learning and Control: A Survey

Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs), which couple future world prediction with executable action generation. This survey provides a robotics-oriented review of WAMs. We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. We further review applications of WAMs in manipulation, navigation, and autonomous driving, and we summarize the datasets, benchmarks, metrics, and protocols used to evaluate WAM systems. Finally, we discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference. Taken together, this survey aims to provide a concise technical foundation for integrating predictive world modeling with action generation, toward more reliable embodied robot intelligence. Project page: https://rcl-robotics.github.io/Awesome-World-Action-Models.

Zuxing Lu, Hongjia Zhai, Guanzhi Wang +13

MBZUAI · Caltech · Amazon FAR +4

World Action ModelsAction-Conditioned World ModelsWorld Models for RoboticsWorld Model LearningWorld Model-Based Planning
AskPDF