Chain of Time: In-Context Physical Simulation with Image Generation Models

YingQiao Wang; Eric Bigelow; Boyi Li; Tomer Ullman

cs.CVOct 30, 2025

Chain of Time: In-Context Physical Simulation with Image Generation Models

Authors: YingQiao Wang, Eric Bigelow, Boyi Li, Tomer Ullman

Organizations: Department of Psychology, Harvard University · Sakana AI · NTT Research · Department of Computer Science, UC Berkeley ·

Abstract

We propose a novel cognitively-inspired prompting method to stabilize and interpret the physical simulation produced by autoregressive image generation models, called Chain-of-Time. Chain-of-Time involves generating a series of intermediate images during a simulation, and it is motivated by mental simulation in humans, as well as in-context reasoning in LLMs. Chain-of-Time is used at inference time, and requires no additional fine-tuning. We apply this method to synthetic and real-world domains, including 2-D graphics simulations and natural 3-D videos. These domains cover various physical dynamics, ranging from rigid-body rolling to harmonic oscillation and pulley systems. We found that Chain-of-Time simulation stabilizes the physics simulated by the state-of-the-art image generation models. Beyond examining performance, we also analyzed the specific states of the world simulated by an image model at each time step, which sheds light on the dynamics underlying these simulations. This analysis reveals insights that are hidden from traditional evaluations of physical reasoning, highlighting cases where image generation models can simulate physical properties such as velocity, gravity, oscillations, and pulley systems. Like humans, image generation models are able to accurately predict future world states over short time periods. By providing a mechanism analogous to working memory, Chain-of-Time enables a model to stitch together a series of short-term predictions into an accurate long-term simulation.

Figures & tables

Figure 1: We study physical reasoning in multi-modal image generation models by providing the model with a sequence of input images of a scene at successive time steps, and having the model predict what the scene will look like some time in the future. Accurately predicting future world states requires reasoning about latent physical properties such as gravity. We propose a novel method, Chain-of-Time (Bottom), which generates a sequence of images corresponding to a step-by-step simulation of the scene as it unfolds over time. As a baseline, Direct Prediction (Top) directly generates the final state without intermediate steps. We use object segmentation algorithms to track object dynamics in both input and generated images (yellow dotted lines).

Figure 2: We use nine experimental domains to test models’ ability to reason about different physical properties. Three of these domains ( 2D Motion , 2D Gravity , Fluids ) were synthetically generated and inspired by previous work on the cognitive science of intuitive physics, four of the domains ( Rolling on Slope , Collision , Rolling on Track , Pulley System ) were synthetically generated by prior works that evaluate physical reasoning in VLMs, and two of the domains ( Bouncing , Pendulum ) use real world videos sampled from various sources and demonstrate the generality of our approach.

Figure 3: Chain-of-Time is a composition of three components: De-rendering

\phi

, Simulation

\tau

, and Rendering

\phi^{-1}

. De-rendering operates by converting input images

I_{0}\ldots I_{t}

into world states

X_{t}

, which represent a physical simulation over time. Chain-of-Time begins with an initial prompt

L_{\text{init}}

and iteratively generates a sequence of in-context output images

I_{t+1}\ldots I_{T}

with follow-up prompts

L_{t+1}\ldots L_{T}

Figure 4: Prediction errors for the eight domains included in this analysis, averaged across all data for each domain. Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object, or water level) and the final positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We generally find a monotonic relationship between Chain-of-Time precision and performance when compared against the direct prediction baseline, indicating a more stable physical simulation under Chain-of-Time .

Figure 5: Prediction errors of each predicted frame for the four domains which have the most complicated object motion, averaged across all data for each domain under Chain-of-Time Quarter. Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object) and the positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We found that Chain-of-Time Quarter up to 2 seconds of simulation shows small but consistent error accumulation across these domains with complex motion.

Panel of Figure 6 — Figure 6: Simulation analysis for Rolling on Slope and Pendulum domains (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom Left) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories, first accelerating sharply and then steadily decelerating. (Bottom Right) Extracted X and Y coordinates of the pendulum in each frame. We observe oscillatory motion in the pendulum for both the ground truth and simulated image sequences.

Panel of Figure 7 — Figure 7: (Left) Simulation analysis for Bouncing and Pulley System domains (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. For Bouncing , the images shown correspond to the “during” period of the collision, beginning at frame 5 of 12. (Bottom Left) Extracted Y coordinates of the tennis ball in each frame, for three independent simulation results of four frames each: one simulation before the bouncing collision, one during, and one after collision. We observe parabolic motion in the ball for both the ground truth and simulated image sequences. (Bottom Right) Extracted X and Y coordinates of the purple ball in each frame. Despite the complexity of this domain, image generation models are able to accurately simulate the ball’s future motion.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Prediction errors for the eight domains included in this analysis, averaged across all data for each domain using prompts listed in App. 9 . Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object, or water level) and the final positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We observe similar prediction between the results generated by the prompts in App. 9 and the results generated by the prompts in App. C , which is shown in Fig. 4 .

Figure 9: Prediction errors of each predicted frame for the four domains which have the most complicated object motion, averaged across all data for each domain under Chain-of-Time Quarter, using the data generated by App. 9 . Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object) and the positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We found that the result generated by using prompt listed in App. 9 shows similar quantitative values and patterns when compared with the result generated by using prompt listed in App. C , which is shown in Fig. 5 .

Figure 10: Examples of the SAM3 detecting the red ball for stimuli used in 2D Gravity .

Figure 11: Examples of the SAM3 detecting the tennis ball for stimuli used in Bouncing .

Figure 12: Examples of the SAM3 detecting the silver ball for stimuli used in Pendulum .

Figure 13: Examples of the SAM3 detecting the grey ball for stimuli used in Pulley System .

Figure 14: Examples of the algorithm detecting the water level for stimuli used in Fluids .

Figure 15: In the Fluids domain, we find that the IGMs are able to simulate water levels increasing over time. Here we show water levels in generated images steadily increasing as a function of time. (Left) The model is sensitive to initial water levels, with solid lines representing a low water level at the initial time of simulation

t

and dotted lines representing a high initial water level. (Right) However, the model is insensitive to the flow rate of water, with water level consistently increasing at the same rate during simulations. Solid lines represent a slow flow rate and dotted lines represent a 3x faster flow rate.

Panel of Figure 16 — Figure 16: Simulation analysis for Bouncing domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. The images shown correspond to the “during” period of the collision, beginning at frame 5 of 12. (Bottom) Extracted Y coordinates of the transparent ball in each frame, for three independent simulation results of four frames each: one simulation before the bouncing collision, one during, and one after collision. We observe parabolic motion in the ball for both the ground truth and simulated image sequences.

Panel of Figure 17 — Figure 17: Simulation analysis for Bouncing domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. The images shown correspond to the “during” period of the collision, beginning at frame 5 of 12. (Bottom) Extracted Y coordinates of the smiley face ball in each frame, for three independent simulation results of four frames each: one simulation before the bouncing collision, one during, and one after collision. We observe parabolic motion in the ball for both the ground truth and simulated image sequences.

Panel of Figure 18 — Figure 18: Simulation analysis for Pendulum domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. (Bottom) Extracted X and Y coordinates of the pendulum in each frame. We observe oscillatory motion in the pendulum for both the ground truth and simulated image sequences.

Panel of Figure 19 — Figure 19: Simulation analysis for Pendulum domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. (Bottom) Extracted X and Y coordinates of the pendulum in each frame. We observe oscillatory motion in the pendulum for both the ground truth and simulated image sequences.

Panel of Figure 20 — Figure 20: Simulation analysis for Rolling on Slope domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories, first accelerating sharply and then steadily decelerating.

Panel of Figure 21 — Figure 21: Simulation analysis for Rolling on Slope domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories, first accelerating sharply and then steadily decelerating.

Panel of Figure 22 — Figure 22: Simulation analysis for Collision domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories.

Panel of Figure 23 — Figure 23: Simulation analysis for Collision domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories.

Panel of Figure 24 — Figure 24: Simulation analysis for Rolling on Track domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories.

Panel of Figure 25 — Figure 25: Simulation analysis for Rolling on Track domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for a single trial. Frame indexing begins at 6 because the first 5 frames are provided as input to the model. (Bottom) Extracted X and Y coordinates of the ball in each frame. The ground truth and simulated objects follow very similar trajectories.

Panel of Figure 26 — Figure 26: Simulation analysis for Pulley System domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. (Bottom) Extracted X and Y coordinates of the grey ball in each frame. Despite the complexity of this domain, image generation models are able to accurately simulate the ball’s future motion.

Panel of Figure 27 — Figure 27: Simulation analysis for Pulley System domain (Top) Comparison between ground truth images and images generated by Chain-of-Time (quarter) for an example trial. (Bottom) Extracted X and Y coordinates of the white ball in each frame. Despite the complexity of this domain, image generation models are able to accurately simulate the ball’s future motion.

Method	2D Gravity	Bouncing	Rolling on Slope	Fluid	Pendulum
Chain-of-time Quarter	$112\pm 5.2$	$72\pm 3.1$	$103\pm 3.2$	$92\pm 6.7$	$274\pm 4.8$
Chain-of-time Half	$133\pm 4.1$	$87\pm 5.9$	$117\pm 2.8$	$78\pm 4.2$	$312\pm 3.3$
Direct Prediction	$152\pm 6.7$	$103\pm 4.2$	$128\pm 3.6$	$64\pm 5.4$	$357\pm 3.6$

Appendix

Table 1: RMSE results across different methods and domains.

Model Type	RMSE in Pendulum Domain	RMSE in Rolling on Slope Domain
LTX-Video	$143\pm 19.1$	$89.8\pm 11.2$
Nano Banana Pro	$169\pm 14.6$	$46\pm 5.1$
GPT-Image-Gen 1.5	$243\pm 20.2$	$52\pm 6.6$

Appendix

Table 2: RMSE results across pendulum and rolling on slope domains. Three models were tested here: LTX-Video, Nano Banana Pro, and GPT-Image-Gen 1.5.

Domain	2 Frames	3 Frames	4 Frames	5 Frames	6 Frames
Rolling on Track	12.3%	21.7%	27.3%	31.8%	33.2%
Bouncing	18.7%	32.7%	42.0%	49.2%	53.5%
Rolling on Slope	14.3%	35.9%	42.0%	50.5%	56.1%

Appendix

Table 3: Accuracy improvements and gains over DP with Chain-of-Time across different input frame counts in three different domains.

Additional Image Generation	1	2	3
Accuracy Gain	11.07%	13.1%	14.47%

Appendix

Table 4: Accuracy gain across additional image generations.

Additional Image Generation	1	2	3
Tokens needed for 1% Accuracy Gain	309.03	540.98	1043.86

Appendix

Table 5: Tokens needed for each 1% accuracy gain across additional image generations.

Model Type	CoT Quarter RMSE	CoT Half RMSE	Direct Prediction RMSE
Nano Banana Pro	$232\pm 25.3$	$337\pm 24.8$	$420\pm 30.1$
GPT-Image-Gen 1.5	$293\pm 31.2$	$373\pm 29.2$	$436\pm 35.9$

Appendix

Table 6: RMSE results in the Pendulum domain.

Model Type	CoT Quarter RMSE	CoT Half RMSE	Direct Prediction RMSE
Nano Banana Pro	$580\pm 37.1$	$646\pm 25.3$	$730\pm 30.8$
GPT-Image-Gen 1.5	$562\pm 34.2$	$651\pm 28.6$	$710\pm 32.7$

Appendix

Table 7: RMSE results in the Rolling on Track domain.

Domains	Rolling on Track	Bouncing	Rolling on Slope	Pulley System	2D Gravity	Pendulum	Fluid	Collision
Nano-Banana Pro (Original Baseline)	$667\pm 53$	$93\pm 5.7$	$98\pm 6.2$	$462\pm 10.4$	$226\pm 12.8$	$288\pm 47.2$	$34\pm 2.3$	$142\pm 12.4$
Nano-Banana Pro (New Baseline)	$703$	$108$	$94$	$502$	$197$	$253$	$23$	$152$
Nano-Banana Pro (Chain-of-Time Half)	$582\pm 51$	$72\pm 5.2$	$69\pm 6.2$	$411\pm 15.1$	$74\pm 12.5$	$187\pm 16.9$	$43\pm 1.9$	$104\pm 10.6$
Nano-Banana Pro (Chain-of-Time Quarter)	$428\pm 27$	$65\pm 4.3$	$46\pm 5.1$	$339\pm 11.9$	$65\pm 8.4$	$169\pm 14.6$	$72\pm 1.8$	$58\pm 9.9$
GPT-Image-1.5 (Original Baseline)	$693\pm 97$	$74\pm 2.8$	$112\pm 9.7$	$378\pm 8.1$	$246\pm 11.7$	$306\pm 57.1$	$37\pm 1.8$	$152\pm 10.9$
GPT-Image-1.5 (New Baseline)	$682$	$57$	$103$	$355$	$214$	$317$	$34$	$131$

Appendix

Table 8: Prediction errors for the eight domains included in this analysis, averaged across all data for each domain, compared to the new baseline. Prediction error is measured by taking the average RMSE between the ground-truth positions (location of focal object, or water level) and the final positions predicted by the IGM. Error bars are 95% Bootstrapped CI. We generally find a monotonic relationship between Chain-of-Time precision and performance when compared against the new baseline as well, consistent with the results in Fig. 4 .

Physics-Based Simulation

Physics-based simulation models physical systems with equations and numerical solvers, increasingly augmented by learned models for speed and control.

See all 227 papers

Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency
Jun 9, 2026Xinrui Wu, Lichen HuangWorld Model Learning Conditional Image Generation
University of Electronic Science and Technology of China
PhyWorld: Physics-Faithful World Model for Video Generation
May 19, 2026Pu Zhao, Juyi Lin, Timothy Rupprecht +10Physics-Informed Generative Modeling Physical Consistency in Video Generation
Northeastern University · University of Georgia · Tulane University +1
PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
Aug 5, 2026Chen Yang, Shenxiang Zeng, Haoyang Zhao +6Counterfactual Reasoning Physical Reasoning in Video
1Tsinghua University · 2The University of Hong Kong

Jun 9, 2026J/K move · Enter open · S save

Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency

Xinrui Wu, Lichen Huang

University of Electronic Science and Technology of China

World Model Learning Conditional Image Generation Image Generation Evaluation

Open paper AskPDF arXiv

Image generation models now produce high-quality static images, yet their ability to represent how a visual world changes over time remains poorly understood. Practical workflows such as storyboarding, step-by-step illustration, reference-guided editing, and video previsualization require models to preserve identities, objects, spatial relations, and causal order across multiple visual states. Existing evaluations largely measure single-image correctness, compositional alignment, or video quality, leaving open whether an image model can coherently imagine a temporally ordered process. We introduce ImageTime, a diagnostic benchmark that uses spatiotemporal consistency as a behavioral probe of visual world modeling in image generation. Given an action instruction, and optionally a reference image specifying the initial state, a model must generate one image containing four ordered key states: initial state, action onset, transition state, and final state. This four-keyframe protocol is more temporally demanding than single-image generation while avoiding the confounds of dense video dynamics. ImageTime organizes tasks with a progressive capability hierarchy and decomposes each scenario into stage-wise state predicates, cross-frame temporal constraints, and forbidden causal violations. GPT-5.5 scores all generated images under a structured VLM-as-judge protocol, producing interpretable capability scores, diagnostic subscores, and failure labels. Through multi-family benchmarking, ImageTime reveals where current image generation systems succeed, fail, and drift when asked to maintain coherent visual world states over time.

Abstract

Physics-Based Simulation

Explore similar work

Can Image Models Imagine Time? ImageTime: A Novel Benchmark for Probing Visual World Modeling Through Spatiotemporal Consistency

PhyWorld: Physics-Faithful World Model for Video Generation

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning