Counterfactual World Models: JEPAs That Reason About "What If"
Organizations: Brown University · AMI Labs · New York University · Independent Researcher
Abstract
World models operate in a world they cannot fully observe. When a hidden disturbance, such as a current pushing a boat, changes the outcome of an action, a counterfactual question follows: what would have happened under a different action, subject to the same disturbance? Its answer separates bad decisions from bad luck. We prove that world models trained on factual transitions cannot answer it on their own, whatever their data or capacity. For many physical systems, the observed outcome and a simple property of the dynamics suffice: natura non facit saltus, nature makes no leaps. When actions and disturbances act as separate forces, their interaction is second-order in time, so the counterfactual outcome is, to first order, the observed outcome plus the difference between the world model's predictions under the alternative and factual actions. This yields a counterfactual world model with no counterfactual data, extra parameters, or retraining. For transitions of duration , we prove an bound on the counterfactual error relative to the disturbance variance, establish a matching minimax lower bound from unpaired factual data, and characterize failures at discontinuities, coordinate folds, and disturbances that scale the command. We validate these predictions from pixels with JEPA world models on frozen DINOv2 and masked autoencoder (MAE) encoders. On four rendered simulators, our approach improves counterfactual from -- to -- with the same world models; on a fifth, an underwater vehicle, it reaches -- with sixteen times more factual data. Four further frozen encoders, DINOv3, SigLIP~2, CLIP and EVA-02, give -- on the same simulators. In hindsight planning, it finds the action that would have reached the goal in -- of events, against -- for an oracle planner that knows the simulator but not the outcome.
Figures & tables
| Query from factual transitions | Additional requirement | Status | Where |
|---|---|---|---|
| Interventional mean | Exchangeability, overlap | Identified by factual regression | and ; [ 52 , 25 , 55 ] |
| Interventional distribution | Same | Identified | [ 44 , 45 ] |
| Individual counterfactual, unrestricted coupling | None | Not identified ; positive minimax risk | ; [ 24 , 8 ] |
| … with exact factual likelihood | None | Not identified ; action-dependent gauge | ; [ 37 ] |
| … under additive response | Shared additive disturbance | Exact by residual reuse | ; [ 26 , 45 ] |
| … under monotone scalar response | Rank preservation | Identified by rank transport | [ 14 , 5 , 37 , 61 ] |
| Assumption | Stated in | Holds, e.g. | Fails, e.g. |
|---|---|---|---|
| Logged actions depend only on what the world model sees, plus independent randomization | an autopilot steering from the same camera, with exploration noise | a skipper who feels the current and steers against it | |
| The logged actions cover the queried ones | (positivity) | random exploration over the full rudder range | an autopilot that never turns hard, queried about a hard turn |
| The disturbance adds its own force instead of scaling the command | and | a current on a boat, a gust on a drone, an unknown load on a robot arm | a worn motor that delivers a fraction of the command; braking on ice |
| No contact, impact or saturation within the transition | a vehicle or an arm moving freely through water or air | a pusher that may or may not touch the block; a ball hitting a wall | |
| The transition is short compared with the interaction | control at tens of hertz; servos that settle to their setpoint | a weak damping fault, whose window narrows with its size; long horizons | |
| The encoded context determines the response to the action | and | a frame showing the boat’s pose, with past frames for its velocity | an occluded object; a single frame of a fast-moving one |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Environment | Coordinates | Mechanism | Theory | Oracle | Copy | Ideal resid. | |
|---|---|---|---|---|---|---|---|
| HandReach | wrist angles † | direct offset | additive | 49.1 | 7.3 | 98.3 | 0.033 |
| HandReach | fingertips ‡ | forward kinematics | smooth | 49.6–57.3 | – | 94.6–98.9 | 0.025–0.116 |
| BlueROV2 | body velocities | hydrodynamics | smooth | 50.4 | 0.5 | 96.9 | 0.062 |
| BlueROV2 | direct thrust | additive | 51.8 | 1.4 | 100.0 | 0.000 | |
| BlueROV2 | folded coordinate | fold | 16.9 | 1.666 | |||
| Push-T | pusher | kinematic | additive | 51.0 | 3.5 | 100.0 | 0.000 |
| Environment | Oracle | Copy | Ideal residual | measured | predicted | |
|---|---|---|---|---|---|---|
| HandReach | 0 | 49.1 | 7.3 | 98.3 | 0.033 | 0 |
| 0.5 | 53.5 | 76.9 | 0.497 | 0.5 | ||
| 1 | 53.5 | 6.7 | 2.006 | 2 | ||
| Ball-in-Cup | 0 | 51.1 | 6.7 | 99.8 | 0.003 | 0 |
| 0.5 | 52.1 | 77.0 | 0.479 | 0.5 | ||
| 1 | 52.2 | 5.9 | 1.969 | 2 |
| Oracle | Counterfactual world model | ||||||
|---|---|---|---|---|---|---|---|
| Encoder | Environment | Copy | Predictor | (no outcome) | Feature | Readout | Readout, ens. |
| MAE-B | HandReach | 7.5 | 38.1 | 49.1 | 82.4 [80,85] | 85.6 [83,87] | 86.7 |
| MAE-B | Ball-in-Cup | 6.7 | 41.6 | 51.1 | 92.0 [91,93] | 94.2 [93,95] | 94.7 |
| MAE-B | FetchReach | 43.3 | 44.8 | 50.0 | 94.0 [93,95] | 95.7 [94,97] | 96.6 |
| MAE-B | Liquid transport | 7.4 | 42.1 | 50.6 | 82.3 [78,86] | 94.8 [92,96] | 95.9 |
| MAE-B | Push-T † | 5.1 | 18.3 | 51.0 | 57.3 [54,61] | 76.5 [74,79] | 79.2 |
| Counterfactual world model | |||||||
|---|---|---|---|---|---|---|---|
| Encoder | Environment | rec. | Copy | Predictor | Feature | Readout | Readout, ens. |
| DINOv3-B | HandReach | 97.2 | 8.4 | 39.2 | 86.4 [85,88] | 88.8 [87,90] | 90.2 |
| DINOv3-B | Ball-in-Cup | 98.9 | 6.8 | 44.5 | 92.6 [92,94] | 95.0 [94,96] | 95.4 |
| DINOv3-B | FetchReach | 98.4 | 43.7 | 44.6 | 93.7 [93,95] | 94.9 [94,96] | 95.8 |
| DINOv3-B | Liquid transport | 98.2 | 7.6 | 42.8 | 82.3 [76,87] | 92.8 [89,95] | 93.3 |
| DINOv3-B | Push-T † | 96.7 | 3.9 | 38.3 | 75.0 [74,76] | 89.1 [88,90] | 90.9 |
| Predictor | CF WM, feature | CF WM, readout | Factual | ||||
|---|---|---|---|---|---|---|---|
| Environment | orig. | sel. | orig. | sel. | orig. | sel. | MSE gain |
| HandReach | 27.5 | 38.1 | 77.3 | 82.4 | 78.6 | 85.6 | 12.4 |
| Ball-in-Cup | 42.0 | 41.6 | 93.1 | 92.0 | 94.4 | 94.2 | 10.4 |
| FetchReach | 41.0 | 44.8 | 91.9 | 94.0 | 92.6 | 95.7 | 10.4 |
| Liquid transport | 38.0 | 42.1 | 78.7 | 82.3 | 91.2 | 94.8 | 8.4 |
| Push-T † | 30.4 | 18.3 | 70.9 | 57.3 | 83.8 | 76.5 | 8.4 |
| CF WM, readout | Trained predictor | |||||||
|---|---|---|---|---|---|---|---|---|
| Fitting episodes | 819 | 1,638 | 3,276 | 6,553 | 819 | 1,638 | 3,276 | 6,553 |
| HandReach | 69.8 | 71.0 | 80.9 | 85.6 | 22.6 | 29.7 | 31.5 | 38.1 |
| Ball-in-Cup | 80.8 | 80.4 | 92.6 | 94.2 | 23.7 | 26.9 | 40.0 | 41.6 |
| FetchReach | 90.7 | 90.6 | 93.3 | 95.7 | 33.6 | 35.5 | 41.2 | 44.8 |
| Liquid transport | 83.9 | 89.5 | 94.5 | 94.8 | 25.2 | 37.6 | 42.3 | 42.1 |
| Environment | Disturbance | Enters | (s) | Oracle | Copy | Ideal resid. | Gates | |
|---|---|---|---|---|---|---|---|---|
| HandReach | actuator offset (benchmark) | command | 0.16 | 49.1 | 7.3 | 98.3 | 0.033 | pass |
| HandReach | external load on the palm | plant, force | 0.16 | 50.1 | 0.4 | 100.0 | 0.001 | pass |
| HandReach | joint damping | plant, state | 0.04 | 94.4 | 42.3 | 95.9 | 0.736 | fail |
| HandReach | joint damping | plant, state | 0.16 | 80.1 | 69.4 | 82.9 | 0.860 | fail |
| Ball-in-Cup | joint damping | plant, state | 0.08 | 92.7 | 38.3 | 92.1 | 1.081 | fail |
| Ball-in-Cup | joint damping | plant, state | 0.32 | 91.5 | 85.9 | 93.6 | 0.754 | fail |
| Encoder | Disturbance | rec. | Oracle | Copy | Predictor | CF WM, feature | CF WM, readout | Ideal resid. |
|---|---|---|---|---|---|---|---|---|
| MAE-B | actuator offset | 97.1 | 49.1 | 7.5 | 38.1 | 82.4 [80,85] | 85.6 [83,87] | 98.3 |
| MAE-B | external load | 98.1 | 50.0 | 13.8 | 39.0 | 84.4 [80,87] | 88.7 [85,91] | 99.9 |
| DINOv2-S | actuator offset | 96.6 | 49.1 | 8.5 | 39.9 | 82.3 [81,84] | 86.1 [84,88] | 98.3 |
| DINOv2-S | external load | 97.2 | 50.0 | 13.4 | 39.8 | 80.9 [77,84] | 86.9 [83,90] | 99.9 |
| Encoder | rec. | Oracle | Predictor | CF WM, feature | CF WM, readout | Ideal resid. | |
|---|---|---|---|---|---|---|---|
| MAE-B | 0 | 97.1 | 49.1 | 38.1 | 82.4 [80,85] | 85.6 [83,87] | 98.3 |
| MAE-B | 0.5 | 91.0 | 52.9 | 41.0 | 70.5 [67,73] | 67.4 [64,70] | 78.9 |
| MAE-B | 1 | 62.9 | 52.4 | 41.4 | 6.4 [ 1,15] | 11.8 [ 22, 1] | 1.1 |
| DINOv2-S | 0 | 96.6 | 49.1 | 39.9 | 82.3 [81,84] | 86.1 [84,88] | 98.3 |
| DINOv2-S | 0.5 | 91.5 | 52.9 | 43.3 | 70.6 [67,74] | 66.3 [63,69] | 78.9 |
| DINOv2-S | 1 | 66.7 | 52.4 | 41.6 | 11.0 [2,19] | 12.8 [ 23, 2] | 1.1 |
| Environment | Trained predictor | CF WM, feature | CF WM, readout | Copy | Oracle | |
|---|---|---|---|---|---|---|
| Hand wrist | 1 | 57.4 [53.0, 61.1] | 88.3 [86.6, 89.8] | 89.4 [87.5, 90.9] | 18.9 | 67.8 |
| 2 | 56.9 [53.0, 60.3] | 88.3 [86.7, 89.6] | 89.5 [87.7, 91.2] | 18.9 | 67.8 | |
| 3 | 56.8 [52.7, 60.4] | 88.1 [86.2, 89.5] | 89.2 [87.3, 90.9] | 18.9 | 67.8 | |
| Cup position | 1 | 55.0 [52.0, 57.4] | 92.1 [91.1, 93.0] | 93.8 [92.6, 94.8] | 12.5 | 66.4 |
| 2 | 55.3 [51.4, 58.3] | 92.8 [91.3, 94.0] | 94.7 [93.8, 96.0] | 12.5 | 66.4 | |
| 3 | 56.0 [53.5, 58.1] | 94.2 [93.4, 95.0] | 95.4 [94.8, 96.0] | 12.5 | 66.4 |
| Environment | Trained predictor | CF WM, feature | CF WM, readout | Copy | Oracle | |
|---|---|---|---|---|---|---|
| Hand wrist | 1 | 55.9 [51.6, 60.0] | 86.6 [85.0, 88.1] | 89.2 [87.9, 90.4] | 19.2 | 67.8 |
| 2 | 55.9 [51.7, 59.7] | 86.1 [84.6, 87.7] | 88.4 [87.1, 89.7] | 19.2 | 67.8 | |
| 3 | 56.4 [51.8, 60.2] | 86.2 [84.5, 87.7] | 88.6 [87.3, 90.0] | 19.2 | 67.8 | |
| Cup position | 1 | 53.6 [51.3, 55.3] | 90.3 [89.3, 91.1] | 91.6 [90.2, 93.0] | 13.1 | 66.4 |
| 2 | 54.2 [51.8, 56.1] | 91.1 [90.2, 91.7] | 91.7 [90.6, 92.8] | 13.1 | 66.4 | |
| 3 | 54.3 [51.7, 56.3] | 91.2 [90.7, 91.6] | 92.3 [91.5, 92.9] | 13.1 | 66.4 |
| Fresh share | Trained predictor | CF WM, feature | CF WM, readout | Copy | Oracle | |
|---|---|---|---|---|---|---|
| 25% | 1 | 71.7 [68.1, 74.7] | 90.7 [89.0, 92.0] | 91.4 [89.8, 92.7] | 24.3 | 81.1 |
| 2 | 71.7 [67.6, 74.8] | 91.0 [89.3, 92.3] | 91.7 [90.1, 93.1] | 24.3 | 81.1 | |
| 3 | 71.8 [67.7, 74.8] | 90.6 [88.9, 92.0] | 91.1 [89.3, 92.5] | 24.3 | 81.1 | |
| 50% | 1 | 57.4 [53.0, 61.1] | 88.3 [86.6, 89.8] | 89.4 [87.5, 90.9] | 18.9 | 67.8 |
| 2 | 56.9 [53.0, 60.3] | 88.3 [86.7, 89.6] | 89.5 [87.7, 91.2] | 18.9 | 67.8 | |
| 3 | 56.8 [52.7, 60.4] | 88.1 [86.2, 89.5] | 89.2 [87.3, 90.9] | 18.9 | 67.8 |
| Oracle planner | Counterfactual world model minus | ||||||
|---|---|---|---|---|---|---|---|
| Encoder | Environment | Predictor | No outcome | True dist. | CF WM | Predictor | Oracle (no outcome) |
| MAE-B | HandReach | 11 | 15 | 100 | 37 | +27 [+20,+33] | +22 [+16,+28] |
| Ball-in-Cup | 10 | 15 | 100 | 60 | +50 [+45,+56] | +45 [+38,+51] | |
| FetchReach | 11 | 14 | 100 | 69 | +58 [+51,+64] | +55 [+48,+61] | |
| BlueROV2 | 18 | 23 | 100 | 50 | +32 [+25,+38] | +27 [+21,+34] | |
| Liquid transport | 13 | 19 | 100 | 62 | +49 [+39,+59] | +43 [+32,+55] | |
| Disturbance, 2nd attempt | Disturbance, 3rd attempt | No disturbance, 3rd attempt | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Encoder | Environment | Pred. | CF | Pred. | CF | Pred. | CF | |||
| MAE-B | HandReach | 12 | 50 | +38 [+32,+44] | 13 | 60 | +47 [+41,+53] | 34 | 64 | +30 [+23,+37] |
| Ball-in-Cup | 11 | 63 | +52 [+46,+58] | 11 | 86 | +75 [+70,+80] | 33 | 91 | +58 [+52,+63] | |
| FetchReach | 13 | 80 | +68 [+62,+73] | 12 | 97 | +85 [+81,+89] | 62 | 99 | +38 [+32,+43] | |
| BlueROV2 | 23 | 60 | +37 [+31,+43] | 21 | 71 | +50 [+44,+56] | 57 | 75 | +18 [+11,+24] | |
| Liquid transport | 21 | 69 | +48 [+43,+54] | 21 | 87 | +67 [+61,+72] | 63 | 93 | +30 [+25,+36] | |
| Persistent fault | Fresh disturbance | No disturbance | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Encoder | Environment | Pred. | CF | Pred. | CF | Pred. | CF | |||
| MAE-B | HandReach | 4 | 11 | +7 [+4,+11] | 6 | 5 | 2 [ 6,+2] | 24 | 12 | 12 [ 18, 6] |
| Ball-in-Cup | 9 | 22 | +13 [+8,+19] | 7 | 7 | 0 [ 4,+4] | 37 | 24 | 13 [ 20, 6] | |
| FetchReach | 12 | 51 | +39 [+33,+46] | 11 | 7 | 4 [ 9,0] | 65 | 59 | 6 [ 14,+2] | |
| BlueROV2 | 25 | 50 | +25 [+18,+31] | 12 | 13 | +1 [ 4,+6] | 65 | 52 | 14 [ 21, 7] | |
| DINOv2-S | HandReach | 6 | 18 | +12 [+7,+17] | 6 | 7 | +1 [ 3,+5] | 30 | 25 | 4 [ 11,+2] |
| Environment | Coordinate | Oracle (no outcome) | Factual-copy |
|---|---|---|---|
| HandReach | wrist WRJ1 | 52.3 [49.9, 54.5] | 7.2 [-3.3, 17.0] |
| HandReach | wrist WRJ0 | 47.8 [45.1, 50.4] | 7.4 [-7.3, 21.7] |
| Ball-in-Cup | cup | 51.6 [50.4, 52.9] | 5.2 [-1.0, 11.2] |
| Ball-in-Cup | cup | 50.6 [49.2, 52.0] | 8.2 [2.3, 13.6] |
| FetchReach | end-effector | 51.8 [49.3, 54.0] | 48.1 [42.5, 53.2] |
| FetchReach | end-effector | 49.0 [46.5, 51.3] | 40.2 [34.9, 46.2] |
| Environment | Frozen encoder | Train | Test | 99% | Alt. split | |
|---|---|---|---|---|---|---|
| windows | episodes | [95% CI] | upper | range | ||
| AntMaze | DINOv2-S | 985k | 250 | |||
| AntMaze | V-JEPA 2 | 985k | 250 | |||
| HumanoidMaze | DINOv2-S | 1.985M | 250 | |||
| HumanoidMaze | V-JEPA 2 | 1.985M | 250 | |||
| PushT | DINOv2-S | 1.441M | 2,989 | n/a |
| Eddies | Relative risk | Drift (%) | Observability gates, (%) | ||||||||
| Rate | CF | JEPA | Oracle | CF | JEPA | Drift head | Readout | Oracle | Copy | ||
| 1.25 | 0.3 | 0.487 | 0.640 | 1.168 | 1.019 | 20.6 | 9.9 | 92.58 | 99.96 | 97.2 | 7.3 |
| 2.5 | 0.3 | 0.263 | 0.383 | 1.145 | 1.023 | 48.0 | 7.6 | 94.39 | 99.96 | 96.1 | 4.9 |
| 5 | 0.3 | 0.082 | 0.229 | 1.132 | 1.007 | 70.1 | 7.7 | 94.81 | 99.96 | 95.6 | 4.1 |
| 10 | 0.3 | 0.022 | 0.168 | 1.109 | 0.997 | 76.9 | 7.1 | 94.95 | 99.96 | 95.4 | 4.2 |
| 20 | 0.3 | 0.005 | 0.141 | 1.119 | 0.993 | 80.1 | 9.6 | 94.90 | 99.96 | 95.3 | 4.2 |
Counterfactual Prediction
Counterfactual prediction estimates outcomes under hypothetical interventions using observed data and assumptions about how systems generate outcomes.
See all 40 papers