Chain of Time: In-Context Physical Simulation with Image Generation Models

YingQiao Wang; Eric Bigelow; Boyi Li; Tomer Ullman

cs.CVOct 30, 2025

Chain of Time: In-Context Physical Simulation with Image Generation Models

Authors: YingQiao Wang, Eric Bigelow, Boyi Li, Tomer Ullman

Organizations: Department of Psychology, Harvard University · Sakana AI · NTT Research · Department of Computer Science, UC Berkeley ·

Abstract

We propose a novel cognitively-inspired prompting method to stabilize and interpret the physical simulation produced by autoregressive image generation models, called Chain-of-Time. Chain-of-Time involves generating a series of intermediate images during a simulation, and it is motivated by mental simulation in humans, as well as in-context reasoning in LLMs. Chain-of-Time is used at inference time, and requires no additional fine-tuning. We apply this method to synthetic and real-world domains, including 2-D graphics simulations and natural 3-D videos. These domains cover various physical dynamics, ranging from rigid-body rolling to harmonic oscillation and pulley systems. We found that Chain-of-Time simulation stabilizes the physics simulated by the state-of-the-art image generation models. Beyond examining performance, we also analyzed the specific states of the world simulated by an image model at each time step, which sheds light on the dynamics underlying these simulations. This analysis reveals insights that are hidden from traditional evaluations of physical reasoning, highlighting cases where image generation models can simulate physical properties such as velocity, gravity, oscillations, and pulley systems. Like humans, image generation models are able to accurately predict future world states over short time periods. By providing a mechanism analogous to working memory, Chain-of-Time enables a model to stitch together a series of short-term predictions into an accurate long-term simulation.

Figures & tables

Figure 1: We study physical reasoning in multi-modal image generation models by providing the model with a sequence of input images of a scene at successive time steps, and having the model predict what the scene will look like some time in the future. Accurately predicting future world states requires reasoning about latent physical properties such as gravity. We propose a novel method, Chain-of-Time (Bottom), which generates a sequence of images corresponding to a step-by-step simulation of the scene as it unfolds over time. As a baseline, Direct Prediction (Top) directly generates the final state without intermediate steps. We use object segmentation algorithms to track object dynamics in both input and generated images (yellow dotted lines).

Figure 2: We use nine experimental domains to test models’ ability to reason about different physical properties. Three of these domains ( 2D Motion , 2D Gravity , Fluids ) were synthetically generated and inspired by previous work on the cognitive science of intuitive physics, four of the domains ( Rolling on Slope , Collision , Rolling on Track , Pulley System ) were synthetically generated by prior works that evaluate physical reasoning in VLMs, and two of the domains ( Bouncing , Pendulum ) use real world videos sampled from various sources and demonstrate the generality of our approach.

Figure 3: Chain-of-Time is a composition of three components: De-rendering

\phi

, Simulation

\tau

, and Rendering

\phi^{-1}

. De-rendering operates by converting input images

I_{0}\ldots I_{t}

into world states

X_{t}

, which represent a physical simulation over time. Chain-of-Time begins with an initial prompt

L_{\text{init}}

and iteratively generates a sequence of in-context output images

I_{t+1}\ldots I_{T}

with follow-up prompts

L_{t+1}\ldots L_{T}

Figure 4: Prediction errors for the eight domains included in this analysis, averaged across all data for each domain. Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object, or water level) and the final positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We generally find a monotonic relationship between Chain-of-Time precision and performance when compared against the direct prediction baseline, indicating a more stable physical simulation under Chain-of-Time .

Figure 5: Prediction errors of each predicted frame for the four domains which have the most complicated object motion, averaged across all data for each domain under Chain-of-Time Quarter. Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object) and the positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We found that Chain-of-Time Quarter up to 2 seconds of simulation shows small but consistent error accumulation across these domains with complex motion.

Panel of Figure 6 — Figure 6: Simulation analysis for Rolling on Slope and Pendulum domains (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom Left) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories, first accelerating sharply and then steadily decelerating. (Bottom Right) Extracted X and Y coordinates of the pendulum in each frame. We observe oscillatory motion in the pendulum for both the ground truth and simulated image sequences.

Panel of Figure 7 — Figure 7: (Left) Simulation analysis for Bouncing and Pulley System domains (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. For Bouncing , the images shown correspond to the “during” period of the collision, beginning at frame 5 of 12. (Bottom Left) Extracted Y coordinates of the tennis ball in each frame, for three independent simulation results of four frames each: one simulation before the bouncing collision, one during, and one after collision. We observe parabolic motion in the ball for both the ground truth and simulated image sequences. (Bottom Right) Extracted X and Y coordinates of the purple ball in each frame. Despite the complexity of this domain, image generation models are able to accurately simulate the ball’s future motion.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Prediction errors for the eight domains included in this analysis, averaged across all data for each domain using prompts listed in App. 9 . Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object, or water level) and the final positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We observe similar prediction between the results generated by the prompts in App. 9 and the results generated by the prompts in App. C , which is shown in Fig. 4 .

Figure 9: Prediction errors of each predicted frame for the four domains which have the most complicated object motion, averaged across all data for each domain under Chain-of-Time Quarter, using the data generated by App. 9 . Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object) and the positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We found that the result generated by using prompt listed in App. 9 shows similar quantitative values and patterns when compared with the result generated by using prompt listed in App. C , which is shown in Fig. 5 .

Figure 10: Examples of the SAM3 detecting the red ball for stimuli used in 2D Gravity .

Figure 11: Examples of the SAM3 detecting the tennis ball for stimuli used in Bouncing .

Figure 12: Examples of the SAM3 detecting the silver ball for stimuli used in Pendulum .

Figure 13: Examples of the SAM3 detecting the grey ball for stimuli used in Pulley System .

Figure 14: Examples of the algorithm detecting the water level for stimuli used in Fluids .

Figure 15: In the Fluids domain, we find that the IGMs are able to simulate water levels increasing over time. Here we show water levels in generated images steadily increasing as a function of time. (Left) The model is sensitive to initial water levels, with solid lines representing a low water level at the initial time of simulation

t

and dotted lines representing a high initial water level. (Right) However, the model is insensitive to the flow rate of water, with water level consistently increasing at the same rate during simulations. Solid lines represent a slow flow rate and dotted lines represent a 3x faster flow rate.

Panel of Figure 16 — Figure 16: Simulation analysis for Bouncing domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. The images shown correspond to the “during” period of the collision, beginning at frame 5 of 12. (Bottom) Extracted Y coordinates of the transparent ball in each frame, for three independent simulation results of four frames each: one simulation before the bouncing collision, one during, and one after collision. We observe parabolic motion in the ball for both the ground truth and simulated image sequences.

Panel of Figure 17 — Figure 17: Simulation analysis for Bouncing domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. The images shown correspond to the “during” period of the collision, beginning at frame 5 of 12. (Bottom) Extracted Y coordinates of the smiley face ball in each frame, for three independent simulation results of four frames each: one simulation before the bouncing collision, one during, and one after collision. We observe parabolic motion in the ball for both the ground truth and simulated image sequences.

Panel of Figure 18 — Figure 18: Simulation analysis for Pendulum domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. (Bottom) Extracted X and Y coordinates of the pendulum in each frame. We observe oscillatory motion in the pendulum for both the ground truth and simulated image sequences.

Panel of Figure 19 — Figure 19: Simulation analysis for Pendulum domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. (Bottom) Extracted X and Y coordinates of the pendulum in each frame. We observe oscillatory motion in the pendulum for both the ground truth and simulated image sequences.

Panel of Figure 20 — Figure 20: Simulation analysis for Rolling on Slope domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories, first accelerating sharply and then steadily decelerating.

Panel of Figure 21 — Figure 21: Simulation analysis for Rolling on Slope domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories, first accelerating sharply and then steadily decelerating.

Panel of Figure 22 — Figure 22: Simulation analysis for Collision domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories.

Panel of Figure 23 — Figure 23: Simulation analysis for Collision domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories.

Panel of Figure 24 — Figure 24: Simulation analysis for Rolling on Track domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories.

Panel of Figure 25 — Figure 25: Simulation analysis for Rolling on Track domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories.

Panel of Figure 26 — Figure 26: Simulation analysis for Pulley System domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. (Bottom) Extracted X and Y coordinates of the grey ball in each frame. Despite the complexity of this domain, image generation models are able to accurately simulate the ball’s future motion.

Panel of Figure 27 — Figure 27: Simulation analysis for Pulley System domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. (Bottom) Extracted X and Y coordinates of the white ball in each frame. Despite the complexity of this domain, image generation models are able to accurately simulate the ball’s future motion.

Method	2D Gravity	Bouncing	Rolling on Slope	Fluid	Pendulum
Chain-of-time Quarter	$112\pm 5.2$	$72\pm 3.1$	$103\pm 3.2$	$92\pm 6.7$	$274\pm 4.8$
Chain-of-time Half	$133\pm 4.1$	$87\pm 5.9$	$117\pm 2.8$	$78\pm 4.2$	$312\pm 3.3$
Direct Prediction	$152\pm 6.7$	$103\pm 4.2$	$128\pm 3.6$	$64\pm 5.4$	$357\pm 3.6$

Appendix

Table 1: RMSE results across different methods and domains.

Model Type	RMSE in Pendulum Domain	RMSE in Rolling on Slope Domain
LTX-Video	$143\pm 19.1$	$89.8\pm 11.2$
Nano Banana Pro	$169\pm 14.6$	$46\pm 5.1$
GPT-Image-Gen 1.5	$243\pm 20.2$	$52\pm 6.6$

Appendix

Table 2: RMSE results across pendulum and rolling on slope domains. Three models were tested here: LTX-Video, Nano Banana Pro, and GPT-Image-Gen 1.5.

Domain	2 Frames	3 Frames	4 Frames	5 Frames	6 Frames
Rolling on Track	12.3%	21.7%	27.3%	31.8%	33.2%
Bouncing	18.7%	32.7%	42.0%	49.2%	53.5%
Rolling on Slope	14.3%	35.9%	42.0%	50.5%	56.1%

Appendix

Table 3: Accuracy improvements and gains over DP with Chain-of-Time across different input frame counts in three different domains.

Additional Image Generation	1	2	3
Accuracy Gain	11.07%	13.1%	14.47%

Appendix

Table 4: Accuracy gain across additional image generations.

Additional Image Generation	1	2	3
Tokens needed for 1% Accuracy Gain	309.03	540.98	1043.86

Appendix

Table 5: Tokens needed for each 1% accuracy gain across additional image generations.

Model Type	CoT Quarter RMSE	CoT Half RMSE	Direct Prediction RMSE
Nano Banana Pro	$232\pm 25.3$	$337\pm 24.8$	$420\pm 30.1$
GPT-Image-Gen 1.5	$293\pm 31.2$	$373\pm 29.2$	$436\pm 35.9$

Appendix

Table 6: RMSE results in the Pendulum domain.

Model Type	CoT Quarter RMSE	CoT Half RMSE	Direct Prediction RMSE
Nano Banana Pro	$580\pm 37.1$	$646\pm 25.3$	$730\pm 30.8$
GPT-Image-Gen 1.5	$562\pm 34.2$	$651\pm 28.6$	$710\pm 32.7$

Appendix

Table 7: RMSE results in the Rolling on Track domain.

Domains	Rolling on Track	Bouncing	Rolling on Slope	Pulley System	2D Gravity	Pendulum	Fluid	Collision
Nano-Banana Pro (Original Baseline)	$667\pm 53$	$93\pm 5.7$	$98\pm 6.2$	$462\pm 10.4$	$226\pm 12.8$	$288\pm 47.2$	$34\pm 2.3$	$142\pm 12.4$
Nano-Banana Pro (New Baseline)	$703$	$108$	$94$	$502$	$197$	$253$	$23$	$152$
Nano-Banana Pro (Chain-of-Time Half)	$582\pm 51$	$72\pm 5.2$	$69\pm 6.2$	$411\pm 15.1$	$74\pm 12.5$	$187\pm 16.9$	$43\pm 1.9$	$104\pm 10.6$
Nano-Banana Pro (Chain-of-Time Quarter)	$428\pm 27$	$65\pm 4.3$	$46\pm 5.1$	$339\pm 11.9$	$65\pm 8.4$	$169\pm 14.6$	$72\pm 1.8$	$58\pm 9.9$
GPT-Image-1.5 (Original Baseline)	$693\pm 97$	$74\pm 2.8$	$112\pm 9.7$	$378\pm 8.1$	$246\pm 11.7$	$306\pm 57.1$	$37\pm 1.8$	$152\pm 10.9$
GPT-Image-1.5 (New Baseline)	$682$	$57$	$103$	$355$	$214$	$317$	$34$	$131$

Appendix

Table 8: Prediction errors for the eight domains included in this analysis, averaged across all data for each domain, compared to the new baseline. Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object, or water level) and the final positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We generally find a monotonic relationship between Chain-of-Time precision and performance when compared against the new baseline as well, consistent with the results in Fig. 4 .

Physics-Based Simulation

Physics-based simulation models physical systems with equations and numerical solvers, increasingly augmented by learned models for speed and control.

See all 227 papers

Jun 9, 2026

Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency

Image generation models now produce high-quality static images, yet their ability to represent how a visual world changes over time remains poorly understood. Practical workflows such as storyboarding, step-by-step illustration, reference-guided editing, and video previsualization require models to preserve identities, objects, spatial relations, and causal order across multiple visual states. Existing evaluations largely measure single-image correctness, compositional alignment, or video quality, leaving open whether an image model can coherently imagine a temporally ordered process. We introduce ImageTime, a diagnostic benchmark that uses spatiotemporal consistency as a behavioral probe of visual world modeling in image generation. Given an action instruction, and optionally a reference image specifying the initial state, a model must generate one image containing four ordered key states: initial state, action onset, transition state, and final state. This four-keyframe protocol is more temporally demanding than single-image generation while avoiding the confounds of dense video dynamics. ImageTime organizes tasks with a progressive capability hierarchy and decomposes each scenario into stage-wise state predicates, cross-frame temporal constraints, and forbidden causal violations. GPT-5.5 scores all generated images under a structured VLM-as-judge protocol, producing interpretable capability scores, diagnostic subscores, and failure labels. Through multi-family benchmarking, ImageTime reveals where current image generation systems succeed, fail, and drift when asked to maintain coherent visual world states over time.

Xinrui Wu, Lichen Huang

University of Electronic Science and Technology of China

World Model Learning Conditional Image Generation Image Generation Evaluation

Ask PDF

May 19, 2026

PhyWorld: Physics-Faithful World Model for Video Generation

World simulators can provide safe and scalable environments for training Physical AI systems before real-world deployment. Large video generation models are emerging as a promising basis for such simulators because they can generate diverse and realistic visual futures. However, using them as world simulators requires physically faithful video continuations, namely, generated videos that preserve the physical state implied by the conditioning input, and evolve in ways consistent with basic physical principles. We propose PhyWorld, a video generation world model designed to produce temporally coherent and physically faithful scene continuations through two-stage post-training. In the first stage, we improve video-to-video continuation with flow matching fine-tuning, encouraging stable visual attributes and coherent motion dynamics across frames. In the second stage, we align generated dynamics with physical principles using Direct Preference Optimization (DPO) over physics preference pairs, guiding the model toward outputs with higher physical plausibility. To evaluate PhyWorld, we use both standard video-quality benchmarks and a dedicated physical-faithfulness benchmark with per-law scoring. Experiments show that PhyWorld improves video consistency, achieving an average score of 0.769 on VBench compared with 0.756 or below for state-of-the-art baselines. PhyWorld also improves physical plausibility, reaching an average score of 3.09 on our physical-faithfulness benchmark compared with 2.99 for the strongest baseline. These results suggest that post-training large video generation models with continuation and physics-preference signals can make them more effective world simulators for Physical AI.

Pu Zhao, Juyi Lin, Timothy Rupprecht +10

Northeastern University · University of Georgia · Tulane University +1

Physics-Informed Generative Modeling Physical Consistency in Video Generation Video Generation Temporal Consistency in Video Generation Video World Models

Ask PDF

Aug 5, 2026

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning

Reliable physical reasoning from video requires understanding how objects move, interact, and respond to interventions. Existing vision-language models (VLMs) often struggle to interpret these dynamics and reason reliably about future and counterfactual outcomes. We introduce PhysMind, a training-free agentic framework that constructs one reusable, question-agnostic executable world per video. PhysMind recovers a temporally consistent dynamic scene through object segmentation, mesh reconstruction, and 6D pose tracking, then fits analytic continuous-time dynamics and latent physical parameters without unrolling a time-stepped simulator. Given a question, it inspects, continues, or edits the world and answers from the resulting trajectories and interactions. Relative to direct chain-of-thought (CoT) reasoning with the same VLM, PhysMind improves accuracy by 38.23 points on CLEVRER and 8.08 points on Physion++. On counterfactual questions, it exceeds the strongest evaluated VLM baseline, GPT-5.5, by 19.25 points.

Chen Yang, Shenxiang Zeng, Haoyang Zhao +6

1Tsinghua University · 2The University of Hong Kong

Counterfactual Reasoning Physical Reasoning in Video Embodied World Models Temporal Reasoning in VLMs

Ask PDF

Abstract

Physics-Based Simulation

Explore similar work

Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency

PhyWorld: Physics-Faithful World Model for Video Generation

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning