cs.ROMar 11, 2026

ResWM: Residual-Action World Model for Visual RL

Authors: Jseen Zhang, Gabriel Adineera, Jinzhou Tan, Jinoh Kim

Organizations: University of California, San Diego USA · Texas A&M University-Commerce USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Learning predictive world models from raw visual observations is a central challenge in reinforcement learning (RL), especially for robotics and continuous control. Conventional model-based RL frameworks directly condition future predictions on absolute actions, which makes optimization unstable: the optimal action distributions are task-dependent, unknown a priori, and often lead to oscillatory or inefficient control. To address this, we introduce the Residual-Action World Model (ResWM), a new framework that reformulates the control variable from absolute actions to residual actions -- incremental adjustments relative to the previous step. This design aligns with the inherent smoothness of real-world control, reduces the effective search space, and stabilizes long-horizon planning. To further strengthen the representation, we propose an Observation Difference Encoder that explicitly models the changes between adjacent frames, yielding compact latent dynamics that are naturally coupled with residual actions. ResWM is integrated into a Dreamer-style latent dynamics model with minimal modifications and no extra hyperparameters. Both imagination rollouts and policy optimization are conducted in the residual-action space, enabling smoother exploration, lower control variance, and more reliable planning. Empirical results on the DeepMind Control Suite demonstrate that ResWM achieves consistent improvements in sample efficiency, asymptotic returns, and control smoothness, significantly surpassing strong baselines such as Dreamer and TD-MPC. Beyond performance, ResWM produces more stable and energy-efficient action trajectories, a property critical for robotic systems deployed in real-world environments. These findings suggest that residual action modeling provides a simple yet powerful principle for bridging algorithmic advances in RL with the practical requirements of robotics.

Figures & tables

Figure 1. Architecture of the proposed Residual-Action World Model (ResWM). (1) Observation Difference Encoder (ODL) : Consecutive frames ot−1o_{t-1} and oto_{t} are processed to extract dynamic deltas, producing a dynamics-aware latent vector ztz_{t} . (2) Residual Policy : The actor network predicts a residual update δat\delta a_{t} conditioned on ztz_{t} and the previous action at−1a_{t-1} , enforcing temporal smoothness. (3) Latent Dynamics : A Recurrent State-Space Model (RSSM) rolls out future latent states st+1s_{t+1} driven by these residual actions, enabling stable, long-horizon imagination for actor-critic optimization.
Taskpixel SAC+ResWMDeepMDP+ResWMRAD+ResWMDeepRAD+ResWM
100K Step Scores
Cartpole, Swingup237 ±\pm 49476 ±\pm 42389 ±\pm 44526 ±\pm 59694 ±\pm 28813 ±\pm 36703 ±\pm 32845 ±\pm 67
Reacher, Easy239 ±\pm 183489 ±\pm 152471 ±\pm 173596 ±\pm 132734 ±\pm 87894 ±\pm 32792 ±\pm 77942 ±\pm 43
Cheetah, Run118 ±\pm 13314 ±\pm 17306 ±\pm 25413 ±\pm 29364 ±\pm 38498 ±\pm 49453 ±\pm 39542 ±\pm 64
Walker, Walk95 ±\pm 19386 ±\pm 26384 ±\pm 197572 ±\pm 68552 ±\pm 87714 ±\pm 57582 ±\pm 91694 ±\pm 87
Finger, Spin230 ±\pm 194493 ±\pm 112509 ±\pm 72706 ±\pm 113813 ±\pm 65928 ±\pm 86832 ±\pm 101986 ±\pm 86
Table 1. Performance comparison on DMControl 6 common tasks at 100K and 500K environment steps.
100K STEP SCORESCURL ( Srinivas et al., 2020 ) (ICML’20)SVEA ( Hansen et al., 2021 ) (NeurIPS’21)PlayVirtual ( Yu et al., 2021 ) (NeurIPS’21)MLR ( Yu et al., 2022 ) (NeurIPS’22)PSRL ( Choi et al., 2023 ) (CVPR’23)TACO ( Zheng et al., 2024 ) (NeurIPS’23)MaDi ( Grooten et al., 2023 ) (AAMAS’24)ResAct ( Anonymous, 2025 ) (AAAI’25)ResWM (Ours)
Cartpole, Swingup582 ±\pm 146727 ±\pm 86816 ±\pm 36806 ±\pm 48849 ±\pm 63782 ±\pm 51704 ±\pm 54819 ±\pm 44845 ±\pm 67
Reacher, Easy538 ±\pm 233811 ±\pm 115785 ±\pm 142866 ±\pm 103621 ±\pm 202821 ±\pm 97766 ±\pm 101917 ±\pm 59942 ±\pm 42
Cheetah, Run299 ±\pm 48375 ±\pm 54474 ±\pm 50482 ±\pm 38398 ±\pm 71402 ±\pm 62432 ±\pm 44503 ±\pm 42542 ±\pm 64
Walker, Walk403 ±\pm 24747 ±\pm 65460 ±\pm 173643 ±\pm 114595 ±\pm 104601 ±\pm 103574 ±\pm 94772 ±\pm 65694 ±\pm 87
Finger, Spin767 ±\pm 56859 ±\pm 77915 ±\pm 49907 ±\pm 58882 ±\pm 132876 ±\pm 67810 ±\pm 95974 ±\pm 42986 ±\pm 8
Ball in cup, Catch769 ±\pm 43915 ±\pm 71929 ±\pm 31933 ±\pm 16922 ±\pm 60902 ±\pm 54884 ±\pm 36948 ±\pm 44963 ±\pm 26
Table 2. Performance comparison on DMControl 6 common tasks at 100K and 500K environment steps (mean ±\pm std over 5 seeds).
TaskFlare ( Shang et al., 2021 )TACOMaDiDeepRADResActResWM (500K)FlareTACOMaDiDeepRADResActResWM (1M)
Quadruped, Walk296 ±\pm 139345 ±\pm 89277 ±\pm 92307 ±\pm 142385 ±\pm 81408 ±\pm 136488 ±\pm 221665 ±\pm 144621 ±\pm 172586 ±\pm 193690 ±\pm 128715 ±\pm 108
Pendulum, Swingup242 ±\pm 152485 ±\pm 167372 ±\pm 101308 ±\pm 137618 ±\pm 380645 ±\pm 112809 ±\pm 31784 ±\pm 42751 ±\pm 41626 ±\pm 220817 ±\pm 6836 ±\pm 36
Hopper, Hop90 ±\pm 55112 ±\pm 4280 ±\pm 2451 ±\pm 1999 ±\pm 49116 ±\pm 32217 ±\pm 59221 ±\pm 45201 ±\pm 43212 ±\pm 13233 ±\pm 32246 ±\pm 48
Finger, Turn hard282 ±\pm 67372 ±\pm 174311 ±\pm 143173 ±\pm 195465 ±\pm 153513 ±\pm 124661 ±\pm 315672 ±\pm 167695 ±\pm 133310 ±\pm 278857 ±\pm 80843 ±\pm 134
Walker, Run426 ±\pm 33355 ±\pm 89382 ±\pm 87375 ±\pm 177467 ±\pm 27496 ±\pm 62556 ±\pm 93582 ±\pm 63562 ±\pm 68508 ±\pm 125554 ±\pm 21584 ±\pm 57
Average267.2333.8284.4242.8406.8435.6546.2584.8566.0448.4630.2644.8
Table 3. Performance on DMControl 5 hard tasks at 500K and 1M environment steps (mean ±\pm std over 5 seeds).
GameRandomHumanFlareCURLDeepMDPMaDiDeepRADTACOResActResWM
Assault222.4742846967126414271824196321642408
BeamRider363.916926.514621824199621722936254235243928
Berzerk123.72630.4452397542567642723653674
Carnival0380018762043246725372896345742034752
Centipede2090.91201732463721352643864673472650825431
ChopperCommand8117387.84631579162821363436296233623528
Table 4. Performance comparison on Atari 10 benchmark. Scores are raw environment returns;
Figure 2. The figure compares the full model ( V4 ) with three variants ( V1–V3 ) across four DMControl tasks. Results show a performance hierarchy: Residual Policy >> ODL >> Regularization . V4 achieves the best results, confirming the synergistic benefits of all components.
Figure 3. This figure highlights a key difference in attention strategies: while DeepRAD exhibits diffuse attention across limb contours, our model focuses sharply on key joints and effectors, enabling more efficient and task-relevant decision-making.
More in this topic

Model-Based RL

Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.

See all 245 papers

Explore similar work

CardsList
Oct 1, 2026

World Action Modeling with Progressive Visual Planning

World action models (WAMs) have emerged as a promising paradigm for robotic control by jointly predicting future visual dynamics and actions from an initial observation and instruction. However, existing WAMs struggle with long-horizon prediction, as generating dense video rollouts is highly inefficient. Some recent WAMs address this by predicting a single future frame without generating the full video, but this approach neglects how to progress toward the goal. We present ProWAM, a progressive world action model that jointly predicts actions and an ordered sequence of sparse visual sub-goals, providing explicit visual guidance to anchor action generation throughout task execution. This design scales naturally, as sub-goal prediction can be learned from large-scale action-free videos, allowing the video backbone to offload complex visual planning from the action policy. For efficient action generation, ProWAM executes a single video-backbone forward pass to cache sparse sub-goal features, eliminating iterative full-video generation and requiring only lightweight action denoising during replanning. Across extensive evaluations, ProWAM achieves superior out-of-distribution robustness. On simulation benchmarks, it sets new state-of-the-art results on LIBERO-Plus (85.8%) and randomized RoboTwin (75.7%), outperforming the strongest baseline with relative gains of up to +35.9%. On RoboCasa365, ProWAM achieves a 48.1% success rate and 18.2% on the challenging Composite-Unseen split, ranking 4th overall. Crucially, in zero-shot real-world experiments, ProWAM achieves 70.0% success, outperforming the strongest baseline by +15.0 (from 55.0% to 70.0%, a +27.3% relative gain) in novel scenes. These results demonstrate the value of progress-indexed visual foresight for closed-loop control. Our program is in https://sii-ferenas.github.io/ProWAM-page.

Fei Zhang, Zhaochong An, Duncan Frost +5

Shanghai Jiao Tong University · SII · Meta +1

World Model-Based PlanningRobot Task PlanningVision-Language-Action ModelsEfficient Inference for World Action ModelsRobustness of VLA Models
AskPDF
May 8, 2026

Learning Visual Feature-Based World Models via Residual Latent Action

World models predict future transitions from observations and actions. Existing works predominantly focus on image generation only. Visual feature-based world models, on the other hand, predict future visual features instead of raw video pixels, offering a promising alternative that is more efficient and less prone to hallucination. However, current feature-based approaches rely on direct regression, which leads to blurry or collapsed predictions in complex interactions, while generative modeling in high-dimensional feature spaces still remains challenging. In this work, we discover that a new type of latent action representation, which we refer to as Residual Latent Action (RLA), can be easily learned from DINO residuals. We also show that RLA is predictive, generalizable, and encodes temporal progression. Building on RLA, we propose RLA World Model (RLA-WM), which predicts RLA values via flow matching. RLA-WM outperforms both state-of-the-art feature-based and video-diffusion world models on simulation and real-world datasets, while being orders of magnitude faster than video diffusion. Furthermore, we develop two robot learning techniques that use RLA-WM to improve policy learning. The first one is a minimalist world action model with RLA that learns from actionless videos, and improves VLA on LIBERO and real robot. The second one is a visual RL framework trained entirely inside a world model learned from offline videos only, using a video-aligned reward and no online interactions. Project page: https://mlzxy.github.io/rla-wm

Xinyu Zhang, Zhengtong Xu, Yutian Tao +3

Rutgers University · Purdue University · University of Wisconsin-Madison

Latent Action LearningWorld Model LearningAction-Conditioned World ModelsVisual RLVideo World Models
AskPDF
Mar 30, 2026

Enhancing Policy Learning with World-Action Model

This paper presents the World-Action Model (WAM), an action-regularized world model that jointly reasons over future visual observations and the actions that drive state transitions. Unlike conventional world models trained solely via image prediction, WAM incorporates an inverse dynamics objective into DreamerV2 that predicts actions from latent state transitions, encouraging the learned representations to capture action-relevant structure critical for downstream control. We evaluate WAM on enhancing policy learning across eight manipulation tasks from the CALVIN benchmark. We first pretrain a diffusion policy via behavioral cloning on world model latents, then refine it with model-based PPO inside the frozen world model. Without modifying the policy architecture or training procedure, WAM improves average behavioral cloning success from 59.4% to 71.2% over DreamerV2 and DiWA baselines. After PPO fine-tuning, WAM achieves 92.8% average success versus 79.8% for the baseline, with two tasks reaching 100%, using 8.7x fewer training steps.

Yuci Han, Alper Yilmaz

Photogrammetry and Computer Vision Lab, The Ohio State University, Columbus, OH 43210, USA.

Robot Policy LearningWorld Action ModelsAction-Conditioned World ModelsModel-Based RLRobot Imitation Learning
AskPDF