Integrating Simulation and Chain-of-thought Reasoning in Multimodal-Language Models For Physical Reasoning

YingQiao Wang; Eric Bigelow; Tomer Ullman; Yujin Tang; Sebastian Risi

cs.AIOct 14, 2026

Integrating Simulation and Chain-of-thought Reasoning in Multimodal-Language Models For Physical Reasoning

Authors: YingQiao Wang, Eric Bigelow, Tomer Ullman, Yujin Tang, Sebastian Risi

Organizations: Department of Psychology, Harvard University · Sakana AI · Goodfire AI

Abstract

Multimodal language models often need to reason about the physical dynamics of the world in order to answer questions appropriately. Various methods have emerged for solving physical reasoning tasks, and in particular, two leading approaches are Chain-of-Thought reasoning and physics-grounded simulation. However, relatively little is known about which particular problems can be solved by each approach. Some tasks may benefit from computationally expensive generative video simulations, whereas others may be solved equally well (or even better) with language-based reasoning. Our work addresses this by systematically comparing simulation and language reasoning across a variety of multimodal tasks involving physical reasoning, such as answering questions about fluid dynamics and elasticity. We draw a parallel with this to resource-rational theories of human cognition, which propose that people optimally trade off simulation and reasoning depending on task demands. We present a novel cognitively-inspired hybrid model which optimally trades off simulation and reasoning to improve model performance and computational efficiency. Our model is able to decide the appropriate method to solve a physical reasoning problem at test time, and finds an optimal balance between accuracy and cost. Further, our model can be trained on a smaller vision-language model that is only 4B parameters and deployed to improve performance on much bigger base models with $>$ 3T parameters like Gemini 3 Pro.

Figures & tables

Figure 1: The overview of HSR model. Our HSR model consists of simulation controller, physics simulator-based simulation, and Chain-of-thought reasoner. The inputs consist of a set of video and a question based on this video, which is passed into simulation controller. Depending on the assignment from the simulation controller, either simulation or Chain-of-thought reasoner will be applied to answer the question.

Figure 2: Evaluation results of simulation only (Simulation) and chain-of-thought only (Chain-of-Thought) when Gemini-3-Pro and Qwen3-VL-4B-Thinking was set as the base model. The evaluation is based on fluid hourglass (FD) and bouncing ball (EC) domains, with questions in property (Property), predictive (Predictive), and counterfactual (Counterfactual) categories. We could observe that HSR performs near the best accuracy across all physical domains and question categories

Figure 3: Evaluation results of simulation only (Simulation) and chain-of-thought only (Chain-of-Thought) when Gemini-3-Pro and Qwen3-VL-4B-Thinking was set as the base model. The evaluation is based on overall performance We could observe that HSR achieved the overall accuracy across all physical domains and question categories

Panel of Figure 4 — Figure 4: Accuracy vs Number of Questions Assigned to Simulation. The first row is the HSR model with Gemini-3-Pro as the base model. The second row is the HSR model with Qwen3-VL-4B-Thinking as the base model. The first column is the property category collasped across two physical domains. The second column is the predictive category collapsed across two physcial domains. The third column is the counterfactual category collapsed across two physical domains. The red dash line is the number of questions assigned to simulation, which is determined by HSR model at threshold $\tau_{t}=0.53$ . This dash line also indicates the local maximum of the function, indicating that HSR found the local optimum routing scheme when considering the tradeoff.

Base Model	Prop	Pred	Counter
Gemini-3-Pro	28%	23%	48%
Qwen3-VL-4B-Thinking	27%	36%	38%

Figure 5: The percentage of questions assigned to simulation. Two models were evaluated as the base model for HSR: Gemini-3-Pro and Qwen3-VL-4B-Thinking. Three question categories were evaluated, including property (Prop), predictive (Pred), and counterfactual (Counter), across all physical domains. Both base model tends to assign less questions to simulation for questions in property category, but more questions to simulation for questions in counterfactual category.

Panel of Figure 6 — Figure 6: First row : Accuracy vs Number of Questions Assigned to Simulation, when VLM-as-a-Judge serves as the simulation controller in the HSR model, with Gemini-3-Pro as the base model. The first column is the property category collasped across two physical domains. The second column is the predictive category collapsed across two physcial domains. The third column is the counterfactual category collapsed across two physical domains. The red dash line is the number of questions assigned to simulation, which is determined by VLM-as-a-Judge at threshold $\tau_{t}=0.53$ . The plot is showing uninterpretable results, showing the ineffectiveness of VLM-as-a-Judge as a simulator controller. Second row: Histogram of the probability of simulation predicted by VLM-as-a-Judge. The probability density concentrates around 0.

Panel of Figure 7 — Figure 7: Accuracy vs Number of Questions Assigned to Simulation. The HSR model with Gemini-3-Pro as the base VLM model and simulation controller trained based on Qwen3-VL-4B-Thinking is evaluated. The first column is the property category collasped across two physical domains. The second column is the predictive category collapsed across two physcial domains. The third column is the counterfactual category collapsed across two physical domains. The red dash line is the number of questions assigned to simulation, which is determined by HSR model at threshold $\tau_{t}=0.53$ . This dash line also indicates the local maximum of the function, indicating that HSR found the local optimum routing scheme when considering the tradeoff.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Base Model	Methods	FD-Pred	EC-Pred	FD-Counter	EC-Counter
Gemini-3-Pro	Sim-High	0.315	0.570	0.435	0.440
	Sim-Low	0.363	0.568	0.541	0.451

Appendix

Table 1: Evaluation results of simulation only (Sim) and chain-of-thought only (CoT) when Gemini-3-Pro set as the base model. We varied the reasoning effort in Gemini-3-Pro, which high reasoning effort (High) and low reasoning effort (Low) were used for evaluation. The evaluation is based on fluid hourglass (FD) and bouncing ball (EC) domains, with questions in predictive (Pred), and counterfactual (Counter) categories. The best accuracy in each question category and physical domains is highlighted in bold.

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

Decoupled Physical Modeling and Execution for Physics Reasoning
Aug 22, 2026Ye Zhang, Xuehang Guo, Rui Pan +4Physical Reasoning VLM Reasoning
University of Pennsylvania · William & Mary · University of Illinois Urbana-Champaign +3
$Ψ$ -0.6: Adaptive Streams of Thought in a Unified Vision-Language World Model
Oct 23, 2026Khai Loong Aw, Klemen Kotar, Baihan Zhang +5VLM Reasoning World Models for VLA Models
Stanford University
ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?
Jun 6, 2026Bin Zhu, Yanhao Jia, Kexin Zhao +12Physical Reasoning Multimodal QA
Peking University, Shenzhen Graduate School · 2Peng Cheng Laboratory · Rabbitpre Intelligence +3

Aug 22, 2026J/K move · Enter open · S save

Decoupled Physical Modeling and Execution for Physics Reasoning

Ye Zhang, Xuehang Guo, Rui Pan, Pengfei Yu+3

University of Pennsylvania · William & Mary · University of Illinois Urbana-Champaign+3

Physical Reasoning VLM Reasoning

Open paper AskPDF arXiv

Physics reasoning requires constructing a consistent model of the underlying physical system rather than relying solely on symbolic or formula-based manipulation. Although large language models have shown strong ability in solving math and coding problems, they still struggle with physics problems, as these problems entangle the physical modeling process with mathematical calculations. Humans approach physics by first building a representation of the system before performing calculations. Inspired by this, we introduce a unified framework that distills intermediate representations that explicitly encode the physical modeling process and adopt a two-stage post-training strategy, where supervised fine-tuning establishes structured modeling, and reinforcement learning with rubric-based feedback improves the quality of the modeling process. Experiments on multiple multimodal physics benchmarks show that our approach generally improves physical reasoning performance across different models and datasets. Across PhysReason, PhyX, and SeePhys, physical modeling outperforms GRPO by ~3% on average. showing that explicit physical modeling is an effective strategy for improving physics reasoning in small VLMs.

Abstract

Multimodal Reasoning

Explore similar work

Decoupled Physical Modeling and Execution for Physics Reasoning

ΨΨΨ-0.6: Adaptive Streams of Thought in a Unified Vision-Language World Model

ChronoPhyBench: Do MLLMs Truly Understand the World or Merely Exploit Language Priors?

$Ψ$ -0.6: Adaptive Streams of Thought in a Unified Vision-Language World Model