cs.LGNov 18, 2026

Counterfactual World Models: JEPAs That Reason About "What If"

Authors: Randall Balestriero, Yann LeCun, Kerda Varaku

Organizations: Brown University · AMI Labs · New York University · Independent Researcher

Abstract

World models operate in a world they cannot fully observe. When a hidden disturbance, such as a current pushing a boat, changes the outcome of an action, a counterfactual question follows: what would have happened under a different action, subject to the same disturbance? Its answer separates bad decisions from bad luck. We prove that world models trained on factual transitions cannot answer it on their own, whatever their data or capacity. For many physical systems, the observed outcome and a simple property of the dynamics suffice: natura non facit saltus, nature makes no leaps. When actions and disturbances act as separate forces, their interaction is second-order in time, so the counterfactual outcome is, to first order, the observed outcome plus the difference between the world model's predictions under the alternative and factual actions. This yields a counterfactual world model with no counterfactual data, extra parameters, or retraining. For transitions of duration hh, we prove an O(h2)O(h^2) bound on the counterfactual error relative to the disturbance variance, establish a matching minimax lower bound from unpaired factual data, and characterize failures at discontinuities, coordinate folds, and disturbances that scale the command. We validate these predictions from pixels with JEPA world models on frozen DINOv2 and masked autoencoder (MAE) encoders. On four rendered simulators, our approach improves counterfactual R2R^2 from 3838--46%46\% to 8686--96%96\% with the same world models; on a fifth, an underwater vehicle, it reaches 9090--94%94\% with sixteen times more factual data. Four further frozen encoders, DINOv3, SigLIP~2, CLIP and EVA-02, give 8181--95%95\% on the same simulators. In hindsight planning, it finds the action that would have reached the goal in 3535--69%69\% of events, against 1414--23%23\% for an oracle planner that knows the simulator but not the outcome.

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. How Can Driving World Models Do Counterfactual Prediction?

    Aug 12, 2026Jiaru Zhang, Can Cui, Yi Xu +3Counterfactual EvaluationCausal Counterfactual Generation