cs.RONov 10, 2026

Few-Shot Residual Adaptation of Pretrained Latent World Models

Authors: Jakub Suliga, Pankhuri Vanjani, Nisarga Nilavadi, Wolfram Burgard, Rudolf Lioutikov

Organizations: Karlsruhe Institute of Technology · University of Technology Nuremberg

Abstract

Pretrained latent world models can support visual planning and control, but their predictions may fail when physical dynamics change at deployment. We study whether such models can adapt from only a few interactions without modifying their pretrained weights or identifying the changed physical parameters. We introduce a few-shot residual adaptation approach that learns self-supervised corrections from latent prediction errors. The corrections are applied both inside the latent predictor and directly to its predicted latent state, while the pretrained world model remains frozen. We evaluate the approach in simulation and on a real robot under center-of-mass and friction shifts. These changes reduce the success of the frozen model to near zero. Using three interaction episodes for adaptation, our dual-site approach recovers 90% success under the center-of-mass shift and 100% under the modified friction on the real robot. Ablations show that correcting both locations outperforms single-site adaptation. Our diagnostics further suggest that much of the learned correction reduces a general prediction mismatch, while successful control also depends on shift-specific information captured during adaptation.

Figures & tables

Explore similar work

CardsList
  1. Sandwich-Residuals: Parameter-Efficient Test-time Adaptation of World Models

    Sep 18, 2026Krishnam Soni, Aditya Sehgal, Vedant Dave +1Test-Time AdaptationLatent World Models

  2. AdaJEPA: An Adaptive Latent World Model

    Jun 30, 2026Ying Wang, Oumayma Bounou, Yann LeCun +1Continual Test-Time AdaptationWorld Model-Based Planning

  3. Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation

    Mar 16, 2026Jacob Levy, Tyler Westenbroek, Kevin Huang +6World Models for RoboticsRobot Policy Adaptation