Library of Hands: Fix the Hands, Not the World Model
Organizations: University of the West of England, Bristol, UK · University of Oxford, Oxford, UK · Imperial College London, London, UK · University of Exeter, Exeter, UK ·
Abstract
A world model predicts how a scene changes after an action, and a planner uses it to search for actions that reach a goal. If the camera moves or the controller changes, the same command moves the robot differently: when its commands are turned by 90 degrees, a frozen planner with the LeWM world model that succeeds in 77.5% of episodes succeeds in only 7.8%. Current fixes retrain the model, need training data, or forget a set-up when it returns. We propose Library of Hands (LoH), a training-free method that keeps the world model frozen and corrects only the map from commands to applied actions, which we call the hand. LoH reads the hand through the frozen model, with no extra sensor: it finds the action that best explains the change between two camera images (a reading). It stores every hand it identifies, so a returning set-up is recognised rather than re-learnt. In closed-loop tests, LoH more than doubles success on three simulated tasks, from 28.6% to 62.3%, and loses only 0.6 percentage points when nothing changes. It also works, unchanged, with two other world models and on two of three models pretrained on real robot data, and its gain grows with the quality of its readings.
Figures & tables
| Approach | (1) | (2) | (3) | (4) | (5) | (6) |
|---|---|---|---|---|---|---|
| Test-time training JEPA-TTT, AdaJEPA | ✗ | ✗ | ✗ | ✓ | ||
| Trained in-context reader SyncWorld | ✓ | ✗ | ✗ | ✗ | ✗ | |
| Calibration per episode V-JEPA 2 | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
| Coarse shift classifier ShiftWM-Bench | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ |
| One kept correction one memory | ✓ | ✓ | ✓ | ✗ | ✓ | |
| LoH (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Hand | Change | Part | Real cause |
|---|---|---|---|
| N | none | none | the training set-up |
| A, D | turn 90 ∘ , 37 ∘ | direction | a moved camera |
| T | random turn | direction | a manually set camera |
| E | mirror of | handedness | an inverted axis |
| C | half gain, delay 2 | scale, timing | a slower controller |
| O | offset on | translation | an actuator bias |
| Method | PushT | Reach. | 2-Room | All | Returns | S4 |
|---|---|---|---|---|---|---|
| Frozen planner | 20.6 | 28.6 | 36.6 | 28.6 | 26.6 | 78.1 |
| ShiftWM-style | 36.2 | 41.0 | 42.0 | 39.7 | 35.8 | 77.3 |
| No memory † | 48.4 | 44.3 | 47.0 | 46.6 | 43.8 | 76.9 |
| One memory † | 54.4 | 48.7 | 48.6 | 50.6 | 45.0 | 76.3 |
| MBCD-style † | 61.5 | 53.7 | 48.1 | 54.4 | 52.0 | 74.2 |
| Fixed Share † | 63.4 | 58.7 | 47.1 | 56.4 | 54.9 | 75.0 |
| Model | Task | S1 | 1 mem. | S4 | S2 | 1 mem. | |
|---|---|---|---|---|---|---|---|
| LeWM | PushT | 0.98 | |||||
| Reacher | 0.70 | ||||||
| 2-Room | 0.90 | ||||||
| DINO-WM | PushT | 0.998 | |||||
| Reacher ∗ | 0.981 | ||||||
| 2-Room ∗ | 0.950 |
| Model | Frozen | True hand | LoH | Gap | 1 mem. | No change | |
|---|---|---|---|---|---|---|---|
| DINO-WM | 0.80 | 7.8 | 35.9 | 67% | 29.7 | ||
| JEPA-WM | 0.70 | 14.1 | 35.9 | 21.9 | 36% | 25.0 | |
| V-JEPA 2-AC | 0.68 | 21.9 | 35.9 | 100% | 31.2 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Constants |
|---|---|
| Sense | Adam, 100 steps, learning rate 0.1, nine starts: the command turned by , , and , the mirrored command, the mirrored command turned by , and zero. A block counts if , or if and ( : the error of the command). Jacobian: central differences, step 0.01. , eigenvalues clipped to ; is the median of over the last 400 blocks. : least squares on 1,200 nominal blocks. |
| Explain | Class weights 1.0, 1.0, 0.3, 0.5 and 0.25 (identity, similarity, reflection, linear, affine); delay weight 1 for and 0.1 for ; prior 0.5 on the identity with . Noise prior with , where is the expected squared norm of a command block sum. Map prior with and (offset: ). Clip model: at most 12 refits, tolerance 0.002. |
| Remember | and : Beta(1,1) posterior means over the last 20 episodes and the last 20 switches. Change point: the last 4 blocks, posterior above 0.6. New entry: and at least 4 blocks. Learning: entries with . Merge: log Bayes factor above 3. Use counts decay by 0.98 per episode. Removal: and no win for 50 episodes, or no win within 20 episodes of creation. At most 64 blocks per entry and 16 entries. Floor: smallest singular value at least 0.2. |
| Act | CEM: 300 samples, 30 iterations, 30 elites, initial standard deviation 1.0 (normalised), horizon 5 blocks, warm start. : the 95th percentile of the frozen one-block error on 20 unshifted episodes (9.02 on PushT, 39.7 on Reacher, 59.0 on Two-Room, 272 on Cube, 0.100 on DINO-WM PushT). |
| Task | Raw action | Clip | Success | ||
|---|---|---|---|---|---|
| PushT | 2-D pusher action | none | agent and block within 20 px, block angle within 20 ∘ | 0.940 | 0.98 |
| Reacher | 2-D joint torques | every joint within 0.05 rad of the target | 0.957 | 0.70 | |
| Two-Room | 2-D velocity | agent within 16 px of the target | 0.967 | 0.90 | |
| Cube | 5-D end effector | none | cube within 4 cm of the target | 1.054 | 0.80 |
| Code | Definition | Real change with the same effect |
|---|---|---|
| B, H | turn by ; turn by | a moved camera; a camera on the other side |
| M | mirror of , then a turn by | swapped and turned axes |
| OA | turn by , then the offset of O | a moved camera and a bias |
| Z, X | dead band; saturation (non-linear) | a worn or limited actuator |
| ZA, XD | turn by , then dead band; turn by , then saturation | both of the above |
| F, G | PushT friction; centre of gravity moved by 20 px | a new surface or object |
| Schedule | Content | Episodes |
|---|---|---|
| S1 | A, C, A, C, D, A, D in blocks of 5 episodes, 4 times | 140 |
| S2 | keep the last hand with ; else a new random turn with , else uniform over A, C, D, E, N | 140 |
| S3 | a new random turn in every episode | 140 |
| S4 | N in every episode | 60 |
| S5, S6 | as S1 and S2 with the non-linear hands | 140 |
| S7, S8 | PushT: hands and world changes in blocks of 5 | 140, 160 |
| ID | Prediction | Result | Held | Failed |
|---|---|---|---|---|
| P1a | The size of predicts the residual: slope in | 1.19 / 0.92 / 0.97 | 3 | 0 |
| P1b | The direction of is informative | Reacher: ratio 1.93 | 1 | 0 |
| P1c | The calibrated gain is through the origin | 0.740 / 1.018 / 1.042 | 1 | 2 |
| P1d | does not flag clipped blocks | trace ratio 1.01 (Reacher), 10.0 (Two-Room) | 0 | 2 |
| P2a | Angle error falls as : slope in | / / | 2 | 1 |
| P2b | Posterior scale calibrated: median in | 0.96 / 1.32 / 0.65 | 2 | 1 |
| LoH minus | Measure | PushT | Reacher | Two-Room | Pooled | SD |
|---|---|---|---|---|---|---|
| Frozen planner | after returns | [31.1, 36.6] | 15.5 | |||
| Frozen planner | S4 | [ , ] | 1.4 | |||
| ShiftWM-style | after returns | [22.6, 26.6] | 11.3 | |||
| No memory † | after returns | [14.6, 18.7] | 11.6 | |||
| One memory † | after returns | [13.6, 17.4] | 10.7 | |||
| MBCD-style † | after returns | [6.7, 10.0] | 9.2 |
| Arm | A ( ) | C (gain, delay) | D ( ) | E (mirror) | N (no change) | T (new turn) |
|---|---|---|---|---|---|---|
| Frozen planner | 7.8 | 34.5 | 31.4 | 9.1 | 77.5 | 28.6 |
| One memory † | 41.4 | 51.6 | 54.2 | 36.6 | 74.0 | 49.4 |
| Fixed Share † | 60.5 | 56.6 | 62.9 | 54.6 | 63.1 | 51.8 |
| LoH | 66.1 | 64.6 | 69.1 | 59.7 | 68.2 | 57.0 |
| True hand | 77.9 | 72.6 | 80.0 | 78.3 | 76.7 | 76.1 |
| Stage | Comparison | PushT | Reacher | Two-Room |
|---|---|---|---|---|
| Sense | Full minus earlier sensor (sticky prior) | [ , 5.4] | [5.3, 11.9] | [3.5, 9.5] |
| Sense | Full minus earlier sensor (Fixed Share) | [ , 4.1] | [3.5, 12.5] | [6.0, 13.1] |
| Explain | LoH minus one model (linear, no delay) | [8.8, 16.4] | [7.7, 14.3] | [0.5, 6.6] |
| Explain | LoH minus five classes with no delay | [7.8, 15.0] | [0.3, 9.7] | [ , 4.5] |
| Remember | LoH minus no memory | [15.3, 23.6] | [18.7, 27.2] | [3.3, 12.1] |
| Remember | LoH minus one memory | [8.9, 17.1] | [12.2, 18.9] | [5.8, 10.8] |
| Test | Result |
|---|---|
| Offset hand (S9) | The offset costs the frozen planner 41.8 pp [30.1, 52.7] on hand O. LoH minus frozen on O: [14.0, 32.6] (56% of the gap); after returns: [32.0, 42.4]. Fresh seeds: , , . On Two-Room, the offset costs only 3.1 pp, and LoH is 6.2 [3.1, 9.4] below the frozen planner, mostly in the first episode after hand A (post hoc). |
| Offset hand on DINO-WM and JEPA-WM | Partial: the grid reached its time limit with 15 and 13 of 18 complete LoH runs, so we give no verdict. On these runs, after returns: [52.5, 69.2] and [49.0, 68.3]; on O: [4.7, 40.0] and [4.6, 38.5]. On Two-Room, LoH again loses on O ( [ , ] and [ , ]). |
| One hand (L1) | LoH minus the true hand, episodes 5–139: [ , ]; Two-Room [ , ]. The library ends with one entry. |
| Long stay (L2) | First five episodes on N after 120 episodes of B: [ , ]; Two-Room [ , ]. On PushT and Reacher, the whole loss is the first episode on N (37.5% against 81.2%). |
| One switch (L3) | After a switch from N to B: [73.3, 80.0]; Two-Room [50.7, 60.0]. Before the switch, the cost is 0.7 pp. |
| 2, 4, 8 hands | All episodes (PushT and Reacher, 32 runs per cell): LoH 76.0, 71.3, 67.6; Fixed Share 75.2, 71.5, 67.2; one memory 71.7, 66.5, 60.7; frozen 45.3, 27.1, 20.4. The library ends with 2.4, 4.3 and 8.6 entries. |
| PushT | Reacher | Two-Room | DINO-WM | ||||
|---|---|---|---|---|---|---|---|
| gain | gain | gain | gain | ||||
| 0 | |||||||
| 0.3 | – | ||||||
| 0.6 | |||||||
| 1.0 | |||||||
| 1.5 | |||||||
| Locked test | Streams (AUC) | ||||
|---|---|---|---|---|---|
| Arm | S1–S3 | After returns | S4 | No change | Rotation |
| LoH | 62.3 | 60.4 | 77.5 | 82.5 | 78.1 |
| LoH without the floor | 61.9 | 60.3 | 77.4 | 52.9 | 64.5 |
| Frozen planner | 28.6 | 26.6 | 78.1 | 83.3 | 2.0 |
| Task | Shift | Frozen | AdaJEPA | JEPA-TTT | LoH |
|---|---|---|---|---|---|
| PushT | action rotation | 2.0 | 4.3 | 68.0 / 50.9 | / |
| contact rotation | 19.3 | 19.3 | 19.3 / 14.6 | 19.3 / 18.3 | |
| Two-Room | wave | 42.3 | 40.3 | / | 49.7 / 45.9 |
| grid | 46.0 | 48.3 | / | 49.0 / 45.2 | |
| Reacher | phase | 23.3 | 18.0 | 22.0 / 17.4 | / |
| harmonic | 20.7 | 20.3 | 22.3 / 17.7 | / |
| Arm | After returns | S4 | LoH minus arm, after returns | Arm minus frozen, S4 |
|---|---|---|---|---|
| Continual test-time training | 26.1 | 74.3 | [31.3, 36.9] | [ , ] |
| AdaJEPA (reset each episode) | 26.6 | 76.5 | [30.2, 37.2] | [ , 0.0] |
| JEPA-TTT (persistent) | 26.4 | 71.4 | [30.0, 37.6] | [ , ] |
| Calibration in every episode | 41.8 | 64.7 | [15.7, 21.2] | [ , ] |
| LoH | 60.3 | 77.7 | – | |
| Frozen planner | 26.1 | 78.1 | – |
| DINO-WM | JEPA-WM | V-JEPA 2-AC | |
| Frozen encoder | DINOv2 ViT-S/14 | DINOv3 ViT-L | V-JEPA 2 ViT-g |
| Calibration: / gain / turn of the map | 0.80 / 1.08 / 39.3 ∘ | 0.70 / 0.73 / 22.3 ∘ | 0.68 / 0.95 / 35.2 ∘ |
| S1 schedule | A, E, A, E, D, A, D | A, B, A, B, D, A, D | A, E, A, E, D, A, D |
| Frozen / true hand / LoH / one memory | 7.8 / 35.9 / 26.6 / 29.7 | 14.1 / 35.9 / 21.9 / 25.0 | 21.9 / 35.9 / 35.9 / 31.2 |
| P0: sensor (bar 0.5) | pass | pass | pass |
| P1: LoH frozen (bar ) | [ , ] (8), pass | [ , ] (8), fail | [ , ] (8), pass |
| Rule | Measure | Bar | Value | Verdict | ||
|---|---|---|---|---|---|---|
| Locked test, 20 seeds | ||||||
| L1 | ours minus frozen, after returns | [31.1, 36.6] | 120 | pass | ||
| L2 | ours minus four baselines † , after returns | , CI | to | lower | 120 | pass |
| L3 | ours minus frozen, S4 | [ , ] | 60 | pass | ||
| L4 | ours minus ShiftWM-style, S1–S3 | [20.6, 24.7] | 180 | pass | ||
| L5 | ours minus no memory † , S1–S3 | [14.1, 17.5] | 180 | pass | ||
Test-Time Adaptation
Test-time adaptation updates or recalibrates pretrained models at inference to handle distribution shifts, often without target labels.
See all 342 papers