Fast Inference-Time Policy Refinement via Action-Affine World Models
Organizations: Department of Automation, Tsinghua University, Beijing, China · Johns Hopkins University, Baltimore, MD 21218, USA
Abstract
Imitation policies can perform complex manipulation, yet small execution errors can accumulate and cause failure. Refinement can correct these errors, but existing approaches often require additional on-policy learning or costly online planning. Our key insight is that a pretrained policy already supplies a useful nominal behavior: refinement can focus on predicting how small action corrections change its outcome. We therefore learn this local response in a form that is cheap to optimize. Our action-affine world model (A\textsuperscript{2}WM) learns local action sensitivities offline from action-perturbed policy rollouts. Given interaction history and a policy proposal, it predicts task-relevant geometry as a nominal trajectory plus a linear response to corrections. It predicts the full planning horizon directly, avoiding the propagation of intermediate errors through recursive model evaluations. Quadratic penalties on predicted task errors and linear correction constraints then yield a cheap convex quadratic program (QP). Online refinement requires one model evaluation and one QP solve, while both policy and model weights remain frozen. Across four simulated manipulation tasks, our framework achieves the highest success rate among evaluated baselines, adding only 3.8 ms of mean refinement overhead per control step, excluding base-policy sampling.
Figures & tables
| OneLeg | RoundTable | Lamp | PushT | Mean | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | Success Rate | Latency | Success Rate | Latency | Success Rate | Latency | Success Rate | Latency | Success Rate | Latency |
| DP | 51.95 | – | 6.12 | – | 5.08 | – | 45.57 | – | 27.18 | – |
| LPB | 50.00 | 1.05 | 6.64 | 3.04 | 6.45 | 1.19 | 43.75 | 0.31 | 26.71 | 1.40 |
| GPC-RANK | 54.56 | 7.94 | 3.91 | 15.69 | 9.24 | 8.39 | 75.65 | 70.97 | 35.84 | 25.75 |
| GPC-OPT | 70.70 | 438.64 | 7.42 | 416.27 | 19.79 | 402.83 | 78.78 | 346.47 | 44.17 | 401.05 |
| GPC-QP | 71.88 | 39.22 | 15.23 | 40.43 | 25.91 | 42.94 | 75.26 | 35.30 | 47.07 | 39.47 |
| Method | On-Policy Interaction | Data for Refinement |
|---|---|---|
| ResiP [ 6 ] | Yes | episodes |
| KoRR [ 8 ] | Yes | episodes |
| A 2 WM (ours) | No | 256–512 episodes |
| Task | Direct | Rec-1 | Rec-8 |
|---|---|---|---|
| OneLeg | |||
| RoundTable | |||
| Lamp | |||
| PushT |
Inference-Time Optimization
Inference-time optimization adapts model computation, outputs, or runtime settings to improve quality under deployment constraints.
See all 102 papers