cs.RONov 9, 2026

Library of Hands: Fix the Hands, Not the World Model

Authors: Hasaan Ahmad, Qizhen Ying, Yuxuan Sun, Tianjin Huang, Guangming Wang

Organizations: University of the West of England, Bristol, UK · University of Oxford, Oxford, UK · Imperial College London, London, UK · University of Exeter, Exeter, UK · 

Abstract

A world model predicts how a scene changes after an action, and a planner uses it to search for actions that reach a goal. If the camera moves or the controller changes, the same command moves the robot differently: when its commands are turned by 90 degrees, a frozen planner with the LeWM world model that succeeds in 77.5% of episodes succeeds in only 7.8%. Current fixes retrain the model, need training data, or forget a set-up when it returns. We propose Library of Hands (LoH), a training-free method that keeps the world model frozen and corrects only the map from commands to applied actions, which we call the hand. LoH reads the hand through the frozen model, with no extra sensor: it finds the action that best explains the change between two camera images (a reading). It stores every hand it identifies, so a returning set-up is recognised rather than re-learnt. In closed-loop tests, LoH more than doubles success on three simulated tasks, from 28.6% to 62.3%, and loses only 0.6 percentage points when nothing changes. It also works, unchanged, with two other world models and on two of three models pretrained on real robot data, and its gain grows with the quality of its readings.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Training

    Apr 23, 2026Yaxuan Li, Zhongyi Zhou, Yefei Chen +5Robot Policy LearningRobot Policy Adaptation

  2. The Planning Limits of Latent World Models

    Sep 30, 2026Ali Alrasheed, Basim Azam, Naveed AkhtarWorld Model-Based PlanningAction-Conditioned World Models