Right Ranking, Wrong Scale: Reproducing the Robotic World Model's Dynamics Model and Testing the Uncertainty of Its Follow-up
Organizations: Independent Researcher, Chandigarh, India
Abstract
We reimplement the dynamics model of the Robotic World Model (RWM) and test the uncertainty estimate that its follow-up, RWM-U, uses to penalize reward in offline policy optimization. Our implementation matches the released model's gradients. On held-out data, training on the model's own rollouts beats one-step training at the preregistered long horizon for all ten episodes, but loses on short one-step windows. RWM-U's ensemble disagreement ranks rollout errors better than forecast step (Spearman rho = 0.90 at 100 steps), yet its spread substantially understates error: one-sigma coverage on the released checkpoint is 4.6% at 100 steps, versus 68% for a calibrated Gaussian. Rolling ensemble members on their own predictions raises coverage to 33%. The implemented squared-error loss on sampled predictions drives each member's variance toward zero. A horizon-specific rescaling restores one-sigma coverage under leave-one-episode-out evaluation, but not two-sigma coverage. We also identify two defects in the released evaluation pipeline. The study covers one robot, one gait and one dataset; it trains no policy.
Figures & tables
| Claim | Evidence | Where it stands | Section |
|---|---|---|---|
| RWM: training on rollouts beats teacher forcing at long horizons | 20 trajectories from all ten episodes, each held out once | holds at h = 368 on every episode; reverses at one step on short windows | §4.1 |
| RWM: moderate history and forecast lengths trade accuracy against training time | held-out pair, 4 trajectories | centre not shown best at our budget; the ranking depends on training length; the time trade-off is untested | §4.2 |
| RWM: it beats MLP, RSSM and transformer baselines | held-out pair, 4 trajectories | consistent with the claim, a weak test | §4.3 |
| RWM-U: the learned variance captures aleatoric uncertainty | our own runs, a synthetic test with known noise, and the released checkpoint’s (20 trajectories) | not supported: the objective’s optimum is zero | §5.5 |
| RWM-U: disagreement is a trust metric | released checkpoint, 20 trajectories; our arms, 4 and 20 | supported as a ranking; wrong as a scale at every horizon, including one step | §5.2, §5.6 |
| RWM-U: uncertainty is effectively propagated over long horizons | released checkpoint and our arms, 20 and 20 trajectories | fails under the release’s rollout; per-member rollout partly repairs it | §5.4 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| claim, and where | tested | what the original reports | verdict |
|---|---|---|---|
| RWM-AR consistently outperforms RWM-TF (2501.10100 §IV-D) | yes | no quantitative figure. "significantly outperforms"; the gap is plotted in Fig. 7 and stated nowhere in text, caption or table | reproduces at long horizon (§4.1); retrained on fixed folds with each episode held out once it replicates, +2.2275 [+1.3683, +3.5053] at h = 368 over all ten episodes’ 20 trajectories at 2,500 iterations, not 10,000 (rule X3: REPLICATES ACROSS FOLDS , §4.1, Appendix H), and scored with the released evaluation’s timing rule M-23’s verdict stands (rule X8: SAME VERDICT , Appendix E) |
| Teacher forcing gives "poor autoregressive performance" (§IV-C) | yes | quantitative : Fig. 6 prints e for the teacher-forced N=1 row, 3.99 at (32, 1) against 0.47 at the centre, on unstated data and horizon; the passage itself gives no number | reproduces, and more strongly: on the 400-step unit Arm B is worse than the hold-last floor at every horizon (on M-64’s shorter units only at the longest they reach), and the configuration sweep’s (32, 1) is 6.35 the centre’s error at h = 368 (Appendix H) |
| M=32, N=8 gives the optimal trade-off between accuracy and training time (§IV-C) | yes | quantitative : Fig. 6 prints the error and the training hours of every cell of its grid, and the centre’s error is the tied-lowest ( docs/presubmission/ORIGINAL_SPECS.md a.5) | NOT OPTIMAL AT OUR BUDGET on accuracy alone (§4.2): (32, 32), (32, 16), (8, 8) and (2, 8) beat it at our budget (no reading its rule reports puts the centre ahead of any of them), but the ranking depends on training length: trained longer, the centre passes each of them on at least one reading (post hoc, §4.2, Appendix H). The trade-off with training time is not tested; the original’s printed hours are consistent with it. Under the released evaluation’s timing the rule keeps its verdict (rule X8: SAME VERDICT , Appendix E) |
| Beats MLP, RSSM and transformer baselines (§IV-D) | yes | no quantitative figure. "consistently achieves the lowest prediction errors across all environments"; plotted in Fig. 7, with no number in text, caption or table | REPRODUCES with the baselines teacher-forced, as the original trains them, and RWM AHEAD OF ALL THREE with them trained autoregressively, which compares architectures at one training regime rather than testing this claim (§4.3). Both are the rules’ verdicts at h = 368; from h = 32 and h = 128 respectively, every reading the rules report gives them, and at shorter horizons they split. The baselines are built to our reading of Table S7 and trained with RWM’s settings; at h = 368 every one is worse than predicting no change, and rule X1’s other read-outs and retraining variants did not rescue our RSSM ( NOT RESCUED BY THE SETTINGS TRIED ), so the RSSM comparison stays uninformative and the claim rests on the MLP and the transformer. Matched to RWM’s parameter count, they return PARTIAL (rule X4): the MLP is behind (RWM BETTER) and the transformer cannot be settled (CANNOT BE SETTLED), matching size being one axis only (§4.3). Under the released evaluation’s timing both rules keep their verdicts (rule X8: SAME VERDICT , Appendix E). One robot where the original has several |
| Zero-shot hardware transfer (§IV-E) | no | — | [hardware: zero-shot transfer] no hardware; this is a dynamics-model reproduction |
| Policies transfer to hardware from ~6M state transitions against ~250M for the model-free baseline (its Limitations section and table, from v2 on; not in v1) — the paper’s headline sample-efficiency result | no | 6M against 250M state transitions at equal real tracking reward (0.90 +- 0.04 against 0.90 +- 0.03), Table 1 from v2 on — the only table of numbers in either paper | [policy, hardware: the sample-efficiency result] not tested. It is a claim about policy learning and hardware deployment, and requires the RL loop, a simulator and an ANYmal. We reproduce the dynamics model only; no policy is trained anywhere in this work, so no transition count of ours is comparable |
| claim (§) | arena (n_independent) | checkpoint | in-sample for the model measured? | verdict | survives multiplicity correction? |
|---|---|---|---|---|---|
| Autoregressive training beats teacher forcing at h = 368 (§4.1) | out-of-sample (4) | 10,000 iterations | no | gap excludes zero, favouring autoregressive training (under the released evaluation’s timing, rule X8: SAME VERDICT) | not applicable at 10,000; the 500- and 2,500-iteration cells survive Holm (Appendix H) |
| The same comparison reverses at h = 1, at the short unit M-64 built (§4.1) | out-of-sample (60) | 10,000 iterations | no | gap excludes zero, favouring teacher forcing | not applicable |
| The same comparison at h = 368, retrained on fixed folds with each episode held out once, at 2,500 iterations (rule X3) (§4.1) | all ten episodes (20) | 2,500 iterations | no | REPLICATES ACROSS FOLDS: gap +2.2275 [+1.3683, +3.5053]; 10 of 10 episodes positive | not applicable |
| (M, N) = (32, 8) is the optimal configuration on accuracy alone (the accuracy half of the trade-off) (§4.2) | out-of-sample (4) | 2,500 iterations | no | NOT OPTIMAL AT OUR BUDGET; 4 of 8 neighbours beat the centre, and no reading the rule reports puts the centre ahead of any of them; the ranking depends on training length (Appendix H) (under the released evaluation’s timing, rule X8: SAME VERDICT) | yes |
| RWM beats MLP, RSSM and transformer baselines trained with RWM’s settings, at h = 368 (teacher-forced, as the original trains them; the RSSM comparison uninformative, rule X1) (§4.3) | out-of-sample (4) | 2,500 iterations | no | REPRODUCES; every reading the rule reports returns it from h = 32, and below that the readings split; at h = 368 every baseline is worse than predicting no change (under the released evaluation’s timing, rule X8: SAME VERDICT) | yes |
| RWM beats MLP, RSSM and transformer baselines trained with RWM’s settings, at h = 368 (trained autoregressively: architecture at one training regime, not the original’s claim; the RSSM comparison uninformative, rule X1) (§4.3) | out-of-sample (4) | 2,500 iterations | no | RWM AHEAD OF ALL THREE; every reading the rule reports returns it from h = 128, and below that the readings split; at h = 368 every baseline is worse than predicting no change (under the released evaluation’s timing, rule X8: SAME VERDICT) | yes |
| rule | what it governs | commit | lead time | tested by | verdict |
|---|---|---|---|---|---|
| M-16 | The Arm A / Arm B comparison | C004 Step 5: pre-register the decision rule before launching any main run | +1.3 h | first main-run data | CANNOT BE SETTLED AT THIS BUDGET |
| M-22 | Whether episode difficulty biases the A/B comparison | C013 Pre-register the Task 4b difficulty-bias rule, and the two-arena convention | +5 min | M-16 re-evaluated | RESOLVED — branch 1, 4c not run |
| M-23 | The 10,000-iteration comparison | C016 5.1: pre-register M-23, the long-horizon decision rule | +2 min | 10k runs launched | RESOLVED — reproduces at long horizon |
| M-43 | The ensemble-5 replication | C081 PRE-REGISTER the ensemble-5 replication rule, before the runs exist | +13.3 h | ens5 result committed | DOES NOT GENERALISE |
| M-44 | The trunk-sharing mechanism | C090 PRE-REGISTER M-44 and M-45, with the power check M-43 was committed without | +6.4 h | R2 result committed | MECHANISM SUPPORTED |
| M-45 | The within-trajectory control | C090 PRE-REGISTER M-44 and M-45, with the power check M-43 was committed without | +4.4 h | A2 result committed | DISAGREEMENT CARRIES WITHIN-ROLLOUT INFORMATION |
| configuration (cost per iteration ÷ the centre’s) | centre trained to (iterations) | compute ratio (centre’s total ÷ configuration’s) | h = 100, held-out pair (4) | h = 100, in-sample (16) | h = 368, held-out pair (4) | h = 368, in-sample (16) |
|---|---|---|---|---|---|---|
| (2, 8) (0.26) | 2,500 | 3.78 | -0.1274 [-0.2175, -0.0373] | -0.0219 [-0.0341, -0.0107] | -0.1139 [-0.2988, -0.0088] | -0.0219 [-0.0364, -0.0081] |
| (2, 8) (0.26) | 5,000 | 7.55 | -0.0742 [-0.1586, +0.0101] | +0.0619 [+0.0448, +0.0813] | -0.0301 [-0.1976, +0.0711] | +0.0927 [+0.0681, +0.1230] |
| (2, 8) (0.26) | 10,000 | 15.11 | -0.0176 [-0.0921, +0.0475] | +0.1085 [+0.0846, +0.1367] | +0.1135 [+0.0983, +0.1357] | +0.1403 [+0.1075, +0.1802] |
| (8, 8) (0.40) | 2,500 | 2.50 | -0.0755 [-0.1410, -0.0269] | -0.0142 [-0.0239, -0.0053] | -0.1004 [-0.2449, -0.0167] | -0.0208 [-0.0462, -0.0009] |
| (8, 8) (0.40) | 5,000 | 4.99 | -0.0223 [-0.0782, +0.0244] | +0.0696 [+0.0538, +0.0870] | -0.0166 [-0.1434, +0.0719] | +0.0937 [+0.0762, +0.1123] |
| (8, 8) (0.40) | 10,000 | 9.99 | +0.0343 [-0.0198, +0.0719] | +0.1162 [+0.0924, +0.1436] | +0.1270 [+0.0987, +0.1589] | +0.1414 [+0.1163, +0.1710] |
| quantity | released checkpoint: cells unseen by the multiplier within 10 points of 68.27%, per-horizon c (unpowered) | same, constant c | per-horizon c fitted on the other model, both models scored (unpowered) | range of fitted c | Arm A, its own per-horizon c: cells within the band, unseen by the model too |
|---|---|---|---|---|---|
| aleatoric | 12 / 12 | 2 / 12 | 0 / 72 | 592.6 – 7782 (13.1 ) | 10 / 36 |
| epistemic | 12 / 12 | 2 / 12 | 12 / 72 | 5.082 – 47.33 (9.31 ) | 17 / 36 |
| h | epistemic, per-horizon multiplier (the reading) | aleatoric, per-horizon multiplier | epistemic, one constant multiplier | epistemic, no multiplier | epistemic multiplier across the ten folds (smallest – largest) |
|---|---|---|---|---|---|
| 1 | 68.6 [63.9, 72.9] , WITHIN BAND | 68.2 [57.4, 77.6], UNRESOLVED | 68.6 [63.9, 72.9], WITHIN BAND | 16.2 [13.4, 18.9] | 6.29 – 6.81 |
| 8 | 68.4 [61.2, 74.9] , WITHIN BAND | 68.5 [57.6, 78.3], UNRESOLVED | 50.6 [44.7, 55.8], OUTSIDE BAND | 10.0 [8.6, 11.3] | 10.44 – 12.35 |
| 32 | 68.3 [61.3, 74.6] , WITHIN BAND | 68.4 [57.2, 78.3], UNRESOLVED | 37.1 [32.4, 41.4], OUTSIDE BAND | 7.0 [6.0, 7.8] | 16.44 – 18.84 |
| 100 | 68.2 [64.5, 71.7] , WITHIN BAND | 67.9 [59.4, 75.7], WITHIN BAND | 26.5 [24.0, 28.9], OUTSIDE BAND | 4.6 [4.1, 5.1] | 28.87 – 31.90 |
| 128 | 68.2 [65.3, 71.0] , WITHIN BAND | 68.0 [60.2, 75.2], WITHIN BAND | 25.0 [23.1, 26.9], OUTSIDE BAND | 4.4 [4.0, 4.8] | 31.86 – 34.20 |
| 368 | 68.2 [65.3, 71.1] , WITHIN BAND | 68.1 [60.9, 75.3], WITHIN BAND | 21.2 [19.7, 22.7], OUTSIDE BAND | 3.6 [3.3, 3.9] | 40.31 – 43.02 |
| finding | evidence (ledger ID, artifact) | bearing on the paper |
|---|---|---|
| The reset guard tests index values for truthiness, so a reset at the first row would be missed | B-02; train.py:143 | Latent twice over: this dataset has no reset at its first row and marks none at all (§6) |
| The train/test split draws windows at random, and adjacent windows share all but one of their 40 rows, so the released test loss is not held out | B-03; model_training.py:33 | The held-out arena (§3) is built from whole episodes and does not use this split |
| finding | evidence (ledger ID, artifact) | bearing on the paper |
|---|---|---|
| The papers name a forecast decay factor and set it to one in every version (RWM Table S9; RWM-U Table S8); the training loss averages every forecast step with equal weight, which is that setting | C-17 (which corrects C-09); system_dynamics.py:186-231 and both config files | A difference of naming, not of behaviour; our training does the same |
| The auxiliary heads (contact and termination) are trained on true next states, the state head on its own samples | M-13; system_dynamics.py:217, 261 | In a rollout the auxiliary heads read predicted states they never trained on; this paper compares states only (§4.3) and does not measure them |
| Training feeds back a reparameterised sample, inference the mean | C-05; system_dynamics.py:115, 217 | Our arms train and roll out the same way; for the released checkpoint, whose per-member has collapsed (§5.5), the sample is numerically the mean |
| Actions enter the network raw (the config’s action mean and standard deviation are zeros and ones), and no action scale is recorded in either repository; measured against joint motion, they are not joint targets in radians | C-07, D-07; anymal_d_flat_cfg.py | A perturbation of one size means different things for actions and for states; no reported result perturbs either |
| The release’s configuration, the follow-up’s paper (its Table S8 in the first version, 2,500) and the checkpoint’s own tag state three different training lengths; the first author believes the configuration’s is a typo | C-13; base_cfg.py:97 , the checkpoint’s iter field | §6: none of them reaches the released variance state at a constant rate |
World Models for Robotics
World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.
See all 419 papers