Imitation policies can perform complex manipulation, yet small execution errors can accumulate and cause failure. Refinement can correct these errors, but existing approaches often require additional on-policy learning or costly online planning. Our key insight is that a pretrained policy already supplies a useful nominal behavior: refinement can focus on predicting how small action corrections change its outcome. We therefore learn this local response in a form that is cheap to optimize. Our action-affine world model (A\textsuperscript{2}WM) learns local action sensitivities offline from action-perturbed policy rollouts. Given interaction history and a policy proposal, it predicts task-relevant geometry as a nominal trajectory plus a linear response to corrections. It predicts the full planning horizon directly, avoiding the propagation of intermediate errors through recursive model evaluations. Quadratic penalties on predicted task errors and linear correction constraints then yield a cheap convex quadratic program (QP). Online refinement requires one model evaluation and one QP solve, while both policy and model weights remain frozen. Across four simulated manipulation tasks, our framework achieves the highest success rate among evaluated baselines, adding only 3.8 ms of mean refinement overhead per control step, excluding base-policy sampling.
Figures & tables
Fig. 1: (a) Success rate comparison across four simulated manipulation tasks. Our method achieves the highest observed success rate on each task among the evaluated refinement methods, tying GPC-OPT on PushT. (b) Success–latency trade-off. Our method achieves the highest task-averaged success rate with only a modest increase in mean per-control-step inference time over the frozen diffusion policy (DP). Inference time includes both policy sampling and online refinement. GPC-QP is our baseline that linearizes the GPC world model around the action proposal via its Jacobian and solves a quadratic program for refinement.
Fig. 2: Overview of fast inference-time policy refinement. A frozen base policy generates action proposals every K control steps. At each control step, the action-affine world model uses interaction history and the current proposal to predict a nominal trajectory Zˉ and a causal action-response matrix F , yielding the direct multi-step prediction Z=FΔA+Zˉ . With F and Zˉ fixed, quadratic task costs and linear correction bounds define a convex QP over ΔA . The first optimized correction is applied to the base action, and refinement repeats after the next observation. Both the base policy and world model remain frozen during deployment.
Fig. 3: Offline learning of the action-affine world model. A frozen imitation policy collects rollouts with action perturbations. Each training window contains interaction history H , a recorded proposal Abase , executed action corrections ΔA , and future observables Z . Conditioned on history and proposal, the model outputs F and Zˉ , forming the direct multi-step prediction Z=FΔA+Zˉ . Prediction errors over the full horizon jointly supervise the nominal forecast and action response.
Fig. 4: Task visualizations. Snapshots of the FurnitureBench assembly tasks and the planar PushT task. The furniture tasks require grasping, transporting, and precisely aligning parts for assembly; PushT requires pushing a T-shaped block into alignment with the green target pose.
OneLeg
RoundTable
Lamp
PushT
Mean
Method
Success Rate ↑
Latency ↓
Success Rate ↑
Latency ↓
Success Rate ↑
Latency ↓
Success Rate ↑
Latency ↓
Success Rate ↑
Latency ↓
DP
51.95
–
6.12
–
5.08
–
45.57
–
27.18
–
LPB
50.00
1.05
6.64
3.04
6.45
1.19
43.75
0.31
26.71
1.40
GPC-RANK
54.56
7.94
3.91
15.69
9.24
8.39
75.65
70.97
35.84
25.75
GPC-OPT
70.70
438.64
7.42
416.27
19.79
402.83
78.78
346.47
44.17
401.05
GPC-QP
71.88
39.22
15.23
40.43
25.91
42.94
75.26
35.30
47.07
39.47
TABLE I: Success rate (%) and mean refinement latency per control step (ms).
Fig. 5: Breakdown of online refinement latency. Times are amortized per control step and averaged equally across the four tasks, excluding simulation and each method’s base-policy sampling time. For GPC-RANK, we instead subtract our method’s single-proposal sampling time, retaining excess candidate-sampling cost as Extra DP. Model Computation includes Jacobian evaluation for GPC-QP; Optimization includes scoring, gradient-based refinement, and QP processing; Other covers remaining overhead.
Method
On-Policy Interaction
Data for Refinement
ResiP [ 6 ]
Yes
≈5×105 episodes
KoRR [ 8 ]
Yes
≈1×105 episodes
A 2 WM (ours)
No
256–512 episodes
TABLE II: Learning paradigm and additional interaction data per task.
Fig. 6: Multi-step prediction accuracy across four tasks. (a) A 2 WM versus recursive nonlinear MLPs trained with one- or eight-step losses. (b) Direct versus recursive action-affine predictors, all compatible with QP-based refinement. Errors are normalized per task by the eighth-step RMSE of A 2 WM in (a) and Direct in (b), then averaged equally across tasks. In (b), thick curves average two training seeds and faint curves show individual-seed results.
Fig. 7: Local action-response prediction in two PushT configurations. For each configuration, we fix the initial state, interaction history, and base-policy proposal, and vary the perturbation amplitude α along a fixed action direction. We compare the block’s x -position after eight steps from independent simulator branches (Ground Truth), A 2 WM, and the baseline MLP world model trained with eight-step supervision. Shaded regions indicate the QP correction bounds.
Task
Direct
Rec-1
Rec-8
OneLeg
1.14(±0.02)
2.20(±0.03)
2.23(±0.04)
RoundTable
1.19(±0.03)
2.32(±0.04)
2.28(±0.04)
Lamp
1.19(±0.03)
2.31(±0.03)
2.29(±0.03)
PushT
1.04(±0.02)
2.14(±0.05)
2.07(±0.04)
TABLE III: Model-computation latency for direct and recursive prediction [ms per control step], reported as mean ± std.
Computationally lightweight manipulation policies predict executable actions with low inference latency, but prediction errors can limit task success. Iterative generative policies can achieve higher task success rates, but repeated network evaluations increase inference latency. Refining action proposals from computationally lightweight policies can combine these strengths, but the amount of refinement needed varies across states. We introduce RAIN (Refine Actions If Needed), a framework that predicts action errors at different refinement levels to decide whether and how much to refine each executable proposal, before performing any refinement. We train an ensemble to predict these errors by comparing action proposals and their refined versions with expert actions from offline demonstrations, without requiring explicit labels of refinement needs. Calibration balances uncertainty, computation, and temporal consistency. We evaluate RAIN on eight ManiSkill tasks and four real-robot tasks. In simulation, our policy achieves 66.6% success, approaching the best evaluated diffusion-policy success rate with 5.71 times lower policy latency. On the real robot, success increases from 55.0% with Action Chunking with Transformers (ACT) to 72.5%, with comparable average successful completion time. Matched controls show that dynamic routing maintains the strongest observed fixed-refinement success level with 26.6% fewer refiner evaluations. Project page: https://rain-policy.github.io/.
Hanlin Liu, Zichen Zhou, Jiacheng Zhang +6
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Institute of Technology · Shanghai Jiao Tong University +2
World Action Models (WAMs) emerge as a promising foundation for policy refinement by predicting the consequences of sampled actions. However, visually plausible predictions can mislead policy refinement if they fail to reflect the input actions. To address this mismatch, we introduce R2-WAM, a two-stage repair-and-reject post-training framework that first improves the consistency of predicted futures with input actions, then uses these futures to select inferior action samples for negative fine-tuning. The repair stage grounds imagination in observed robot behavior through a kinematic alignment score that measures agreement between predicted and demonstrated motion, enabling the predicted video to faithfully reflect its input actions. Using the repaired video model, the rejection stage compares imagined outcomes of sampled and demonstrated actions, selectively applying negative fine-tuning to samples whose predicted task progress falls below the demonstrated reference by a prescribed margin. Together, the two stages extend video prediction from representation learning to consequence-based policy refinement without additional environment interaction or changes to the inference procedure. R2-WAM achieves 93.8% average success on RoboTwin 2.0 across clean and randomized settings. On the long-horizon real-world Fold Shirt task, it achieves 87.5% average success, compared with 0% for Fast-WAM. Our project page is available at https://r2-wam.github.io/.
Ruiyan Xu, Haisheng Su, Sixu Lin +4
The Chinese University of Hong Kong, Shenzhen · Manifold AI · Southeast University +1
This paper presents the World-Action Model (WAM), an action-regularized world model that jointly reasons over future visual observations and the actions that drive state transitions. Unlike conventional world models trained solely via image prediction, WAM incorporates an inverse dynamics objective into DreamerV2 that predicts actions from latent state transitions, encouraging the learned representations to capture action-relevant structure critical for downstream control. We evaluate WAM on enhancing policy learning across eight manipulation tasks from the CALVIN benchmark. We first pretrain a diffusion policy via behavioral cloning on world model latents, then refine it with model-based PPO inside the frozen world model. Without modifying the policy architecture or training procedure, WAM improves average behavioral cloning success from 59.4% to 71.2% over DreamerV2 and DiWA baselines. After PPO fine-tuning, WAM achieves 92.8% average success versus 79.8% for the baseline, with two tasks reaching 100%, using 8.7x fewer training steps.
Yuci Han, Alper Yilmaz
Photogrammetry and Computer Vision Lab, The Ohio State University, Columbus, OH 43210, USA.