When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?
Organizations: Technology School Electronic Systems (TUES) · Baba Tonka High School of Mathematics
Abstract
Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty. We present an evaluation framework that separates what frozen representations contain, how they organize appearance and physical differences, and what changes a model's own prediction error. On IntPhys2, a benchmark of physically possible and impossible events, we compare representations from latent prediction, masked pixel reconstruction, and simple visual statistics. Appearance remains easy to decode, while V-JEPA2 exposes modest information about whether events are possible or impossible. Looking at the latent geometry, appearance changes become smaller compared with differences between videos, their directions line up more consistently across scenes, and their overlap with physical changes decreases in deeper layers. Across action-conditioned models, incorrect actions raise prediction error, while appearance changes also produce substantial responses. In Physion++ video clips, repeating a single frame can produce higher error than the original clip, while changing color throughout the clip can lower it. Using Kubric, a simulation and rendering framework, we find that impossible events can produce lower error than possible events. A fixed error threshold also incorrectly flags many possible events as impossible. A model can therefore represent physical information and respond to incorrect actions without its native prediction error reliably measuring physical surprise.
Figures & tables
| Check | Controlled comparison and measured property |
|---|---|
| Readout | Train a linear classifier on frozen representations and test it on scenes excluded from training. Measure how accurately it distinguishes physically possible from impossible events. |
| Geometry | Measure appearance and physical shifts across layers. Relative size shows how strongly appearance changes affect representations compared with differences between unrelated videos. Stronger alignment than in shuffled pairs indicates a consistent response across scenes. Greater overlap than with random subspaces of the same dimension indicates shared directions between appearance and physical changes. |
| Action | Keep the observed outcome fixed and compare predictions made with correct and mismatched actions. Check whether incorrect actions increase prediction error. |
| Appearance/static | Change appearance without changing the simulated physical state, or repeat a single frame throughout a clip. Compare prediction error with the original clip. |
| Physical validity | Compare possible and impossible events while varying motion or appearance. Check whether impossible events consistently receive higher error within each condition. |
| Calibration | Choose an error threshold on separate calibration data and apply it unchanged to new scenes and appearances. Measure the rates of false alarms and missed violations. |
| Contrast | Hue | Pixel darkening |
|---|---|---|
| Persistent original | [ , ] | [ , ] |
| Target-only original | [ , ] | [ , ] |
| Persistent target-only | [ , ] | [ , ] |
| Feature family or contrast | Appearance classification | Physics classification | Clean-only physics |
|---|---|---|---|
| V-JEPA2-G | 97.7% [95.7, 99.3] | 57.9% [54.2, 61.8] | 54.8% [51.3, 58.8] |
| VideoMAE2-G | 95.3% [92.8, 97.6] | 49.2% [47.3, 50.9] | 49.6% [46.5, 52.2] |
| PixelSummary | 79.1% [73.7, 84.5] | 50.0% [48.9, 51.0] | 51.8% [48.7, 54.8] |
| V-JEPA2-G VideoMAE2-G | 2.4 pp [ , 5.0] | 8.7 pp [4.4, 13.3] | 5.3 pp [0.4, 10.1] |
| V-JEPA2-G PixelSummary | 18.6 pp [13.5, 24.1] | 7.9 pp [4.1, 11.8] | 3.1 pp [ , 8.3] |
| Model/task | Opposite lawful |
|---|---|
| LeWM TwoRoom | 1.884 [1.821, 1.946] |
| LeWM PushT | 0.3834 [0.3456, 0.4242] |
| DINO-WM PushT | 0.00364 [0.00304, 0.00428] |
| JEPA-WM PushT | 0.00351 [0.00293, 0.00411] |
| Model / population | Readout | Geometry | Action | Appearance/static | Physical validity | Calibration |
|---|---|---|---|---|---|---|
| V-JEPA2-G / IntPhys2 | S | S | – | – | – | – |
| VideoMAE2-G / IntPhys2 | I | S | – | – | – | – |
| V-JEPA2-G / crossed support | – | – | – | D | Contradicted | I |
| V-JEPA2-L / Physion++ Mass | – | – | – | D | – | – |
| LeWM / TwoRoom | – | – | S | D | I | – |
| LeWM / PushT | – | – | S | D | I | – |
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Evidence block | Unit/status | Supports | Does not support |
|---|---|---|---|
| IntPhys2 readout | 57 scenes / 114 pairs; post-hoc five-fold scene cross-fit | Modest out-of-fold physics-label accessibility advantage under the evaluated probe budget | Preregistered confirmation, native-error specificity, calibrated surprise, or general physical reasoning |
| IntPhys2 frozen geometry | 57 scenes / 114 pairs; analysis fixed on development scenes, then run once on the sealed test set | Controlled relative magnitude, null-adjusted direction, and cross-fitted subspace overlap for the evaluated perturbations/models | Semantic disentanglement, causal factorization, residual specificity, calibration, or general physical understanding |
| V-JEPA2-G multi-dose geometry | 38 development scenes / 76 pairs; exploratory, run after the sealed test | How each transformation moves the features, and whether a local appearance basis captures more physical displacement than an endpoint basis of the same size | Confirmation, hue circularity, semantic axes, or generality beyond tested factors |
| LeWM / DINO-WM action controls | 256 episodes per task or scorer; confirmatory | Action-conditioned prediction sensitivity in LeWM TwoRoom, LeWM PushT, and DINO-WM PushT | Specificity of large error to dynamics or physical violation |
| JEPA-WM PushT action control | 256 episodes; post-hoc exploratory | Action sensitivity and qualitative robustness of the DINO-WM ordering under a revised predictor recipe over the same encoder family | Independent encoder-family replication, causal attribution to one model choice, broad cross-model generality, or calibrated physical surprise |
| LeWorldModel appearance controls | 256 fresh PushT episodes; confirmatory plus exact decomposition | Appearance sensitivity, color-direction anisotropy, and partial persistent transport | Pure nuisance invariance or role-aware dynamics |
| Family | Input | Native window | Patch/tubelet | Layers | Temporal scope |
|---|---|---|---|---|---|
| V-JEPA2-G | 64 frames | 0,10,20,30,40 | one joint 64-frame clip | ||
| VideoMAE2-G | 16 frames | 0,10,20,30,40 | four independent chunks | ||
| PixelSummary | 64 frames | n/a | n/a | one | 100 RGB/histogram/difference statistics |
| Model | Layer | Relative magnitude H/B/R/P | [95% CI] | |
|---|---|---|---|---|
| V-JEPA2-G | b00 | .607285 | .571/2.111/.610/.007 | .401 [.320, .483] |
| b10 | .007055 | .392/1.634/.449/.007 | .125 [.078, .178] | |
| b20 | .006974 | .260/1.022/.245/.010 | .063 [.039, .090] | |
| b30 | .015104 | .110/1.497/.142/.012 | .102 [.065, .145] | |
| final | .235825 | .080/.077/.090/.021 | .060 [.050, .071] | |
| VideoMAE2-G | b00 | .010641 | 1.245/1.750/1.310/.005 | .348 [.268, .432] |
| Property | V-JEPA2-G | VideoMAE2-G |
|---|---|---|
| Relative appearance magnitude | strong replication | strong replication |
| Null-adjusted direction | strong replication | strong replication |
| Appearance-physics overlap | qualitative replication | strong replication |
| Model | ||||
|---|---|---|---|---|
| V-JEPA2-G | .005 [.004,.006]/.0007 | .012 [.010,.014]/.0014 | .060 [.050,.071]/.0028 | .153 [.132,.174]/.0057 |
| VideoMAE2-G | .024 [.016,.034]/.0007 | .058 [.046,.073]/.0014 | .077 [.063,.092]/.0028 | .130 [.113,.148]/.0057 |
| Model | Layer | Hue | Brightness | Recolor | Physics |
|---|---|---|---|---|---|
| V-JEPA2-G | b00 | .651/.466/.186 | .835/.785/.050 | .427/.420/.007 | .065/.066/ .002 |
| b10 | .714/.384/.329 | .953/.741/.212 | .499/.330/.168 | .064/.066/ .002 | |
| b20 | .758/.349/.409 | .941/.660/.281 | .615/.282/.333 | .088/.066/.022 | |
| b30 | .784/.250/.534 | .954/.699/.255 | .712/.250/.462 | .119/.066/.052 | |
| final | .693/.212/.481 | .792/.236/.556 | .564/.180/.383 | .134/.067/.067 | |
| VideoMAE2-G | b00 | .666/.499/.167 | .962/.763/.199 | .449/.461/ .012 | .070/.067/.003 |
| Factor | L straightness | G straightness | L effective rank | G effective rank |
|---|---|---|---|---|
| Hue | .739 | .692 | 3.61 | 9.83 |
| Brightness | .887 | .907 | 2.98 | 5.43 |
| Selective recolor | .324 | .553 | 5.94 | 13.72 |
| Factor | Mild dose | Medium dose | Strong dose |
|---|---|---|---|
| Hue ( ) | .640/.064/.576 | .685/.160/.525 | .734/.234/.500 |
| Brightness (.85/.65/.45) | .810/.081/.729 | .830/.164/.667 | .816/.257/.559 |
| Recolor ( ) | .598/.138/.459 | .636/.195/.442 | .625/.213/.412 |
| Model/task | Contrast | Low or matched scale | High or matched scale |
|---|---|---|---|
| LeWM TwoRoom | zero vs. lawful | 0.359 [0.315, 0.403] at 0.25 | 1.463 [1.382, 1.545] at 1.0 |
| LeWM TwoRoom | opposite vs. lawful | 0.909 [0.834, 0.987] at 0.25 | 1.884 [1.821, 1.946] at 1.0 |
| LeWM TwoRoom | hue vs. opposite | 0.222 [0.150, 0.296] at 120 ∘ /0.75 | 0.220 [0.147, 0.295] at 180 ∘ /1.0 |
| LeWM PushT | zero vs. lawful | 0.0085 [0.0070, 0.0102] at 0.25 | 0.1668 [0.1469, 0.1880] at 1.0 |
| LeWM PushT | opposite vs. lawful | 0.0332 [0.0279, 0.0391] at 0.25 | 0.3834 [0.3456, 0.4242] at 1.0 |
| LeWM PushT | hue vs. opposite | 0.146 [0.125, 0.167] at 30 ∘ /0.25 | 0.642 [0.585, 0.696] at 60 ∘ /0.75 |
| Model | ||||||
|---|---|---|---|---|---|---|
| LeWM | 0.008728 | 0.895323 | 0.008358 | 0.253505 | 0.886596 | 0.245146 |
| DINO-WM | 0.061985 | 0.111752 | 0.218553 | 0.221626 | 0.049767 | 0.003073 |
| JEPA-WM | 0.196626 | 0.214008 | 0.335983 | 0.338820 | 0.017381 | 0.002837 |
| Object | Role color | Matched novel | Role/novel RGB L1 | Role/novel changed pixels |
|---|---|---|---|---|
| Block | agent blue | orange | 0.006157/0.006188 | 0.02924/0.02924 |
| Agent | block gray | purple | 0.000563/0.000563 | 0.002844/0.002841 |
| Contrast | Scope | Mean [95% CI] | Positive episodes |
|---|---|---|---|
| Block blue orange | target only | 0.493 [0.450, 0.535] | 233/256 |
| Block blue orange | persistent | 0.111 [0.092, 0.131] | 191/256 |
| Agent gray purple | target only | 0.397 [0.355, 0.440] | 245/256 |
| Agent gray purple | persistent | 0.088 [0.054, 0.123] | 164/256 |
| Pair | Target score gap | Target-shift gap | Persistent score gap | Shift cosines |
|---|---|---|---|---|
| Block blue/orange | 0.493 | 0.495 | 0.111 | 0.916/0.928 |
| Agent gray/purple | 0.397 | 0.400 | 0.088 | 0.815/0.705 |
| Contrast | Collision | Dominoes | Waterpush |
|---|---|---|---|
| Persistent hue original | |||
| Target-only hue original | |||
| Persistent darkening original | |||
| Target-only darkening original | |||
| Persistent target-only hue | |||
| Persistent target-only darkening |
| Condition | Score change | Target contribution | Predictor contribution |
|---|---|---|---|
| persistent hue | |||
| target only hue | |||
| persistent brightness | |||
| target only brightness |
| Video | Segment | Raw score | Mean RGB change | Static residual |
|---|---|---|---|---|
| 0 | 12 | 0.5898 | 0.001867 | |
| 1 | 9 | 0.6021 | 0.001294 | |
| 2 | 11 | 0.6083 | 0.009740 | |
| 3 | 21 | 0.6380 | 0.005892 | |
| 4 | 16 | 0.6215 | 0.001611 | |
| 5 | 20 | 0.6455 | 0.001180 |
| Appearance / world | False alarms | Misses | Balanced accuracy | AUROC |
|---|---|---|---|---|
| Clean / Solid | 100.00 | 45.70 | 27.15 [24.02, 30.27] | 0.083 |
| Clean / Gap | 37.89 | 0.78 | 80.66 [77.54, 83.59] | 0.882 |
| Clean / Balanced | 68.95 | 23.24 | 53.91 [52.05, 55.76] | 0.499 |
| Persistent / Solid | 100.00 | 59.38 | 20.31 [17.38, 23.44] | 0.048 |
| Persistent / Gap | 30.47 | 0.39 | 84.57 [81.64, 87.50] | 0.944 |
| Persistent / Balanced | 65.23 | 29.88 | 52.44 [50.68, 54.20] | 0.492 |
| Location | Geometry-only | Combined | Incremental [interval] | Out of range |
|---|---|---|---|---|
| Block 09 | 0.7580 | 0.7632 | [ , ] | 67 |
| Block 19 | 0.7511 | 0.7381 | [ , ] | 208 |
| Block 29 | 0.7642 | 0.7626 | [ , ] | 137 |
| Final encoder | 0.7671 | 0.7744 | [ , ] | 57 |
| Native target | 0.7653 | 0.7725 | [ , ] | 57 |
| Location | Geometry-only | Combined | Incremental [interval] | Out of range |
|---|---|---|---|---|
| Block 09 | -0.0175 | -0.0447 | [ , ] | 81 |
| Block 19 | -0.0491 | -0.0492 | [ , ] | 56 |
| Block 29 | -0.0407 | -0.0370 | [ , ] | 75 |
| Final encoder | -0.0119 | -0.0062 | [ , ] | 46 |
| Native target | -0.0119 | -0.0062 | [ , ] | 47 |
| Location | Physical margins | Appearance-score changes |
|---|---|---|
| Block 09 | 0.7671 [0.7623, 0.7725] | -0.0751 [-0.0997, -0.0492] |
| Block 19 | 0.7185 [0.7072, 0.7304] | -0.0293 [-0.0452, -0.0128] |
| Block 29 | 0.7231 [0.7098, 0.7364] | -0.0341 [-0.0535, -0.0141] |
| Final encoder | 0.7518 [0.7442, 0.7603] | -0.0316 [-0.0471, -0.0159] |
| Native target | 0.7540 [0.7469, 0.7618] | -0.0315 [-0.0470, -0.0157] |
Video World Models
Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.
See all 282 papers