cs.LGMar 18, 2026

R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation

Authors: Naoki Morihira, Amal Nahar, Kartik Bharadwaj, Yasuhiro Kato, Akinobu Hayashi, Tatsuya Harada

Organizations: Honda R&D Co., Ltd. · The University of Tokyo · RIKEN AIP

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

A central challenge in image-based Model-Based Reinforcement Learning (MBRL) is to learn representations that distill essential information from irrelevant visual details. While promising, reconstruction-based methods often waste capacity on large task-irrelevant regions. Decoder-free methods instead learn robust representations by leveraging Data Augmentation (DA), but reliance on such external regularizers limits versatility. We propose R2-Dreamer, a decoder-free MBRL framework with a self-supervised objective that serves as an internal regularizer, preventing representation collapse without resorting to DA. The core of our method is a redundancy-reduction objective inspired by Barlow Twins, which can be easily integrated into existing frameworks. On DeepMind Control Suite and Meta-World, R2-Dreamer is competitive with strong baselines such as DreamerV3 and TD-MPC2 while training 1.59x faster than DreamerV3, and yields substantial gains on DMC-Subtle with tiny task-relevant objects. These results suggest that an effective internal regularizer can enable versatile, high-performance decoder-free MBRL. Code is available at https://github.com/NM512/r2dreamer.

Figures & tables

Figure 1: Comparison of representation learning mechanisms in world models. (a) R2-Dreamer learns representations without a decoder or DA. It uses an internal redundancy reduction objective LBT\mathcal{L}_{\mathrm{BT}} that aligns the latent state sts_{t} (via a projector) with the embedding of the observation oto_{t} . (b) Dreamer relies on a decoder to learn representations by reconstructing the observation o^t\hat{o}_{t} from the latent state sts_{t} , guided by a reconstruction loss Lrecon\mathcal{L}_{\mathrm{recon}} . (c) DreamerPro removes the decoder but depends on DA. It enforces consistency between augmented views of the observation aug(ot)\mathrm{aug}(o_{t}) using a spatial loss LSwAV\mathcal{L}_{\mathrm{SwAV}} and a temporal loss LTemp\mathcal{L}_{\mathrm{Temp}} that leverages an Exponential Moving Average (EMA) of the encoder weights.
Figure 2: Examples drawn from benchmark tasks. Left: Meta-World Assemble. Center: DMC Reacher (hard). Right: DMC-Subtle Reacher with a significantly smaller target.
Figure 3: Mean and median performance over 20 DMC tasks, with standard deviation across seeds. R2-Dreamer is competitive with the baselines on average without requiring a decoder or DA.
Figure 4: Aggregated performance on Meta-World 50 tasks using the mean and the median across tasks. R2-Dreamer achieves strong results even on contact-rich manipulation tasks, remaining competitive with the baselines on average.
Figure 5: Performance on five challenging DMC-Subtle tasks. R2-Dreamer substantially outperforms the baselines, demonstrating its robustness to subtle but critical visual information.
Figure 6: Policy saliency maps on the DMC-Subtle Reacher task. For clarity, the target location is marked with a yellow dot. The two rows show results from two different environment seeds, which are identical across all methods.
Figure 7: Ablation results on 20 DMC tasks using the mean and the median across tasks. Our internal redundancy reduction objective proves more effective and robust than reliance on heuristic DA.
Figure 8: Comparison of R2-Dreamer with and without DA on the DMC-Subtle benchmark. The results highlight that DA can be detrimental in tasks requiring high precision, as it may distort subtle but critical information, underscoring the importance of a DA-free approach for such environments.
MethodTraining Time (hours)
R2-Dreamer4.4
Dreamer7.0
DreamerPro10.4
Dreamer (Author’s JAX impl.)6.6
Table 1: Computational efficiency comparison on the DMC Walker Walk task. Wall-clock time is measured for 1 million environment steps on a single NVIDIA GeForce RTX 3080 Ti GPU.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: DMC-Subtle benchmark. Top: original versions of the five tasks (left to right: Ball in Cup Catch, Cartpole Swingup, Finger Turn, Point Mass, Reacher). Bottom: modified versions with downscaled task-critical objects in the same order.
Figure 10: Per-task learning curves for all 20 DMC tasks.
Figure 11: Per-task learning curves for all 50 Meta-World tasks.
Figure 12: Per-task learning curves for the ablation study across all 20 DMC tasks.
NameSymbolValue
General
Replay capacity—5×1065\times 10^{6}
Batch sizeBB16
Batch lengthTT64
Activation—RMSNorm⁡+SiLU⁡\operatorname{RMSNorm}+\operatorname{SiLU}
Learning rate—4×10−54\times 10^{-5}
Appendix
Table 2: Main hyperparameters. Our settings are identical to DreamerV3, with key changes to the representation learning loss.

Explore similar work

CardsList

  1. Dreamer-CDP: Improving Reconstruction-free World Models Via Continuous Deterministic Representation Prediction

    Mar 7, 2026Michael Hauri, Friedemann ZenkeWorld Model LearningWorld Models

    Friedrich Miescher Instiute for Biomedical Research, Fabrikstrasse 24, 4056 Basel, Switzerland

  2. Next Embedding Prediction Makes World Models Stronger

    Mar 3, 2026George Bredis, Nikita Balagansky, Daniil Gavrilov +1Model-Based RLPartially Observable RL

    T-Tech.

  3. VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning

    Oct 2, 2026Mingyu Park, Samyeul Noh, Hyun Myung +1Generalization in RLModel-Based RL

    Robotics Program KAIST, ETRI · Electrical Engineering KAIST