Perception into Imagination: Executable Memory for World Models
Abstract
Physical agents must use what they previously saw or touched after visibility or contact is lost. Their actions change the implications of these remembered facts. We introduce executable perceptual memory (EPM), a world-model framework that advances explicit perceptual state alongside neural latent dynamics during imagination. Its evolving read-out informs reward prediction, value estimation and policy learning. We align reward inputs with the transitions that produced them and supervise the reward head on open-loop rollouts of recorded actions. On Memory Maze , EPM reaches 30.37 return at 1M interactions, compared with 23.37 when the same evolving read-out informs only policy and value on a common perceptual front end: a gain of 7.00 [3.80, 10.21]. The complete method also improves control on and in MiniWorld. Probes of remembered, out-of-view arrivals show stronger reward anticipation in both visual implementations. Concentration, tactile recovery and real tactile frames demonstrate complementary roles of remembered facts in imagined-policy training, motion-dependent decisions and sensory-content retention. EPM connects persistent perception to the predicted consequences and learning signals of a world model for Physical AI.
Figures & tables
| Arm | Policy/value | Reward | OL |
|---|---|---|---|
| Observation+OL | – | – | Yes |
| Held+OL | Held | – | Yes |
| Held+R+OL | Held | Held | Yes |
| Evolved | Evolved | – | No |
| Evolved+OL | Evolved | – | Yes |
| EPM | Evolved | Evolved | Yes |
| Memory Maze | Memory Maze | MiniWorld | |||||||
| 1M, 12 labels | 1M, 6 labels | 2M, 12 labels | |||||||
| Observation+OL | 17.32 | 14.18 | 20.74 | ||||||
| Held+OL | 17.18 | – | 18.52 | ||||||
| Held+R+OL | 12.08 | – | 3.37 | ||||||
| Evolved | 19.12 | 23.37 | 18.09 | ||||||
| Evolved+OL | 19.69 | 18.98 | 20.83 | ||||||
| Method | Interactions | Return |
|---|---|---|
| Native DreamerV3 | 1M | 2.90 |
| Native R2I | 3M | 6.74 |
| R2I + front end | 3M | 6.08 |
| R2I + front end + state | 3M | 9.36 |
| Policy/value-only interface | 1M | 11.17 |
| With heading correction | ||
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| 0.25M | 0.5M | 1M | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Contrast | 95% CI | 95% CI | 95% CI | |||||||||
| EPM Evolved+OL | [ , ] | 12/12 | 0.0005 | [ , ] | 11/12 | 0.0010 | [ , ] | 12/12 | 0.0005 | |||
| EPM Obs+OL | [ , ] | 12/12 | 0.0005 | [ , ] | 12/12 | 0.0005 | [ , ] | 12/12 | 0.0005 | |||
| EPM Held+R+OL | [ , ] | 12/12 | 0.0005 | [ , ] | 12/12 | 0.0005 | [ , ] | 12/12 | 0.0005 | |||
| Evolved+OL Held+OL | [ , ] | 11/12 | 0.0044 | [ , ] | 12/12 | 0.0005 | [ , ] | 11/12 | 0.0015 | |||
| Held+R+OL Held+OL | [ , ] | 4/12 | 0.2153 | [ , ] | 2/12 | 0.0635 | [ , ] | 0/12 | 0.0005 | |||
| 0.25M | 0.5M | 1M | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Contrast | 95% CI | 95% CI | 95% CI | |||||||||
| EPM Evolved+OL | [ , ] | 4/6 | 0.3125 | [ , ] | 6/6 | 0.0312 | [ , ] | 6/6 | 0.0312 | |||
| EPM Evolved | [ , ] | 5/6 | 0.2188 | [ , ] | 3/6 | 0.7500 | [ , ] | 6/6 | 0.0312 | |||
| EPM Obs+OL | [ , ] | 6/6 | 0.0312 | [ , ] | 6/6 | 0.0312 | [ , ] | 6/6 | 0.0312 | |||
| Evolved+OL Evolved | [ , ] | 3/6 | 0.8125 | [ , ] | 1/6 | 0.0625 | [ , ] | 0/6 | 0.0312 | |||
| Evolved+OL Obs+OL | [ , ] | 4/6 | 0.1250 | [ , ] | 6/6 | 0.0312 | [ , ] | 6/6 | 0.0312 | |||
| 1M | 2M | |||||||
|---|---|---|---|---|---|---|---|---|
| Contrast | 95% CI | 95% CI | ||||||
| EPM Evolved+OL | [ , ] | 10/12 | 0.0098 | [ , ] | 10/12 | 0.0620 | ||
| EPM Obs+OL | [ , ] | 12/12 | 0.0005 | [ , ] | 9/12 | 0.0493 | ||
| EPM Held+R+OL | [ , ] | 12/12 | 0.0005 | [ , ] | 12/12 | 0.0005 | ||
| Evolved+OL Held+OL | [ , ] | 12/12 | 0.0005 | [ , ] | 11/12 | 0.0034 | ||
| Held+R+OL Held+OL | [ , ] | 0/12 | 0.0005 | [ , ] | 0/12 | 0.0005 | ||
| Memory Maze , 1M | ||||||
|---|---|---|---|---|---|---|
| Label | Obs+OL | Held+OL | Held+R+OL | Evolved | Evolved+OL | EPM |
| 1 | 17.50 | 16.17 | 11.54 | 20.35 | 16.31 | 24.88 |
| 2 | 16.56 | 17.38 | 9.75 | 18.56 | 21.56 | 25.73 |
| 3 | 20.81 | 17.54 | 14.23 | 12.77 | 19.60 | 25.25 |
| 4 | 19.67 | 16.23 | 11.25 | 21.17 | 20.75 | 24.06 |
| 5 | 15.12 | 19.29 | 11.38 | 18.96 | 21.33 | 25.38 |
| Arm | |||
|---|---|---|---|
| Memory Maze , 1M | |||
| Observation+OL | [ , ] | [ , ] | [ , ] |
| Held+OL | [ , ] | [ , ] | [ , ] |
| Held+R+OL | [ , ] | [ , ] | [ , ] |
| Evolved | [ , ] | [ , ] | [ , ] |
| Evolved+OL | [ , ] | [ , ] | [ , ] |
| Comparison | Steps | Return difference [95% CI] | Positive | Role | |
|---|---|---|---|---|---|
| Evolved observation-only | 250k | +0.60 [ 0.74, +1.94] | 5/6 | 0.34375 | secondary |
| Evolved observation-only | 500k | +2.72 [+0.61, +4.83] | 6/6 | 0.03125 | primary |
| Evolved observation-only | 1M | +5.24 [+1.05, +9.43] | 6/6 | 0.03125 | primary |
| Evolved held | 250k | +0.83 [ 0.69, +2.36] | 4/6 | 0.1875 | secondary |
| Evolved held | 500k | +2.17 [ 0.44, +4.77] | 6/6 | 0.03125 † | secondary |
| Evolved held | 1M | +5.35 [+0.31, +10.40] | 5/6 | 0.0625 † | secondary |
| Comparison | Steps | Dependence difference [95% CI] | Positive | |
|---|---|---|---|---|
| Evolved observation-only | 250k | +0.76 [ 0.05, +1.56] | 6/6 | 0.03125 |
| Evolved observation-only | 500k | +1.57 [ 0.24, +3.38] | 5/6 | 0.0625 |
| Evolved observation-only | 1M | +2.59 [+0.52, +4.67] | 6/6 | 0.03125 |
| Evolved held | 250k | +0.82 [ 0.05, +1.69] | 6/6 | 0.03125 |
| Evolved held | 500k | +1.46 [ 0.63, +3.54] | 5/6 | 0.09375 |
| Evolved held | 1M | +1.82 [ 0.65, +4.29] | 4/6 | 0.125 |
| Label | Evolved | F | O | Evolved O | Evolved F | F O | Dep. Evolved | Dep. F | Dep. O |
|---|---|---|---|---|---|---|---|---|---|
| 250k | |||||||||
| 1 | 1.229 | 0.250 | 0.625 | +0.604 | +0.979 | 0.375 | 0.708 | 0.000 | 0.083 |
| 2 | 0.729 | 0.458 | 0.542 | +0.188 | +0.271 | 0.083 | 0.229 | 0.021 | 0.021 |
| 3 | 3.875 | 0.250 | 0.854 | +3.021 | +3.625 | 0.604 | 2.417 | 0.021 | 0.167 |
| 4 | 0.583 | 0.896 | 0.562 | +0.021 | 0.312 | +0.333 | 0.354 | 0.083 | 0.021 |
| 5 | 0.562 | 0.750 | 1.292 | 0.729 | 0.188 | 0.542 | 0.312 | 0.083 | 0.042 |
| Label | Return (E / H / C) | E C [95% CI] | C H [95% CI] | Dependence (E / H / C) |
|---|---|---|---|---|
| 250k | ||||
| 1 | 0.8 / 0.8 / 0.9 | 0.08 [ 0.58, +0.48] | +0.12 [ 0.21, +0.46] | +0.15 / +0.10 / 0.04 |
| 2 | 2.1 / 1.5 / 0.6 | +1.52 [+0.81, +2.35] | 0.90 [ 1.35, 0.46] | +0.90 / 0.10 / 0.00 |
| 3 | 2.1 / 0.8 / 0.4 | +1.73 [+1.19, +2.33] | 0.48 [ 0.71, 0.25] | +0.83 / 0.00 / +0.06 |
| 4 | 1.1 / 1.1 / 0.5 | +0.65 [+0.33, +0.96] | 0.60 [ 0.94, 0.27] | +0.42 / +0.21 / +0.10 |
| 5 | 1.7 / 1.0 / 1.6 | +0.12 [ 0.46, +0.71] | +0.54 [+0.02, +1.02] | +1.25 / +0.04 / +0.27 |
| State update | Error before (mm) | Error after (mm) | Top-1 | Regret |
|---|---|---|---|---|
| Evolving, object held after contact | 2.44 | 4.36 | 0.64 | 0.0070 |
| Evolving, object follows tip | 2.39 | 2.48 | 0.42 | 0.0201 |
| Frozen state | 2.95 | 3.90 | 0.02 | 0.0658 |
| Q1 accuracy | Q2 accuracy | |||||
| Arm | =8 | =16 | =32 | =8 | =16 | =32 |
| byte-state memory (B=48) | 0.290 0.010 | 0.354 0.008 | 0.417 0.020 | 0.656 0.007 | 0.694 0.004 | 0.700 0.011 |
| 128-dim. GRU (reference) | 0.356 0.005 | 0.456 0.016 | 0.485 0.025 | 0.700 0.003 | 0.719 0.007 | 0.710 0.008 |
| no history | 0.194 0.009 | 0.192 0.010 | 0.187 0.010 | 0.667 0.002 | 0.671 0.002 | 0.659 0.004 |
| proprioception only | 0.142 0.005 | 0.152 0.003 | 0.152 0.006 | 0.636 0.003 | 0.673 0.001 | 0.683 0.001 |
| byte-state, B=16 | 0.271 0.030 | 0.292 0.025 | 0.305 0.024 | |||
| Budget | Contrast | 95% CI | Role | ||
|---|---|---|---|---|---|
| 25 | Synthetic pretraining none | [ , ] | 17/20 | Primary | |
| 25 | Synthetic pretraining Taxim | [ , ] | 11/20 | Secondary | |
| 25 | Taxim none | [ , ] | 13/20 | Secondary | |
| 450 | Synthetic pretraining none | [ , ] | 12/20 | Secondary |