PhySimCode: A Benchmark and Evaluation Method for Physics Video to Code Generation

Souraja Kundu; Aditya Gupta; Joel Julin; Yizhou Zhao; Liuyue Xie; Laszlo A. Jeni

cs.CVOct 11, 2026

PhySimCode: A Benchmark and Evaluation Method for Physics Video to Code Generation

Authors: Souraja Kundu, Aditya Gupta, Joel Julin, Yizhou Zhao, Liuyue Xie, Laszlo A. Jeni

Organizations: Carnegie Mellon University · Amazon AGI

Abstract

Multimodal large language models (MLLMs) are good at video understanding and code generation, but it remains unclear whether they can recover the physical laws underlying videos, a capability essential for converting videos into executable simulations and for providing structured physical supervision in robotics. Captioning motion video does not imply that a model has inferred its governing dynamics, estimated quantitative parameters, or can regenerate the observed scene from code. Existing video-physics benchmarks evaluate plausibility, property estimation, or future-state prediction, but offer limited insight into whether models can translate visual dynamics into runnable simulation code. We introduce PhySimCode, a benchmark for physics video-to-code generation: given only a raw video, a model must infer the relevant physics, recover parameter values, and generate runnable Python that reproduces the input video. The dataset contains 160.6K procedurally generated samples spanning 162 physics phenomena across analytic SciPy simulations and PyBullet-rendered rigid-body, articulated, fluid, and contact-rich scenes; each sample bundles the simulation code, physical parameters, rendered video, and a structured chain-of-thought covering observations, laws, ODEs, numerical methods, and validation checks. We evaluate ten state-of-the-art closed- and open-source MLLMs along physical-law correctness, simulation equivalence, code executability, and parameter recovery. Models reliably state physics laws (4.27-4.93/5) yet score only 1.51-2.47/5 on inferring them from video, and the best system (Claude Sonnet 4.6) recovers just 7.7% of ground-truth parameters within $\pm 20\%$ , exposing a substantial video-to-simulation gap. The 2,430-video evaluation set is available at https://huggingface.co/datasets/SourajaKundu123/PhySimCode and the evaluation code at https://github.com/SourajaKundu/PhySimCode

Figures & tables

Figure 1: Categorical distribution of physical domains with some Scipy (2D) and Kubric (3D) examples from our dataset.

Figure 2: The dataset curation pipeline.

Figure 3: Per-sample distributions of (a) video resolution, (b) duration, (c) CoT length, and (d) code length across Kubric and SciPy. Dashed lines mark per-pipeline means.

Stage	Evaluation / Measurement
(i) Video understanding	Appearance equivalence (human), DINOv2/VideoCLIP (Table 4 )
(ii) Physics understanding	Law correctness & equivalence (Table 3 ); parameter recovery (Tables 5 , 9 )
(iii) Programming	Compilation; execution; video generation; CodeBLEU (Table 5 )
End-to-end	Physical plausibility & equivalence (Table 4 )

Table 2: How different stages of capabilities of MLLMs are measured through all evaluation metrics.

Panel of Figure 4 — Figure 4: Cohen’s $\kappa$ for alignment between human-rated physics equivalence of videos and the patch-wise similarity metrics (DINOv2, VideoCLIP), averaged over (a) closed- and (b) open-source models, and inter-judge Cohen’s $\kappa$ averaged over (c) closed- and (d) open-source models.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

New frontier models
	Physical Law Correctness		Physical Law Equivalence
Model	Formula	Overall	Name	Statement	Formula	Overall
Claude Fable 5.1	4.40	4.33	3.47	3.13	2.90	3.00
GPT-6 Astra	4.73	4.50	3.27	2.87	2.63	2.80
Grok 4.6	4.40	4.23	2.87	2.53	2.30	2.40
Kimi K3	4.40	4.37	3.57	2.63	2.40	2.67

Appendix

Table 6: Average Likert scores (on a 1–5 scale) rated by 3 LLM judge models (Gemini-2.5-flash, Grok-4.1-Fast, GPT-5-Nano) for model outputs across physical law correctness and equivalence dimensions, on the 10-sample hard subset.

New frontier models
	Patchwise Video Similarity
Model	DINOv2 sim.	VideoCLIP sim.
Claude Fable 5.1	0.855	0.881
GPT-6 Astra	0.902	0.896
Grok 4.6	0.781	0.869
Kimi K3	0.863	0.903

Appendix

Table 7: Video simulation equivalence table containing verifiable video frame similarity scores using DINOv2 and VideoCLIP, on the 10-sample hard subset. Cosine similarities on

[0,1]

, averaged over successfully rendered videos (scipy and kubric pooled). Human-judgment columns are not available for this subset.

New frontier models
	Param. Estimation	Code Generation and Execution
Model	Acc. ( $\pm$ 20%)	CoT	Code	Compile	No Errors	Video Save	CodeBLEU
Claude Fable 5.1	18.18%	100.0	100.0	100.0	100.0	100.0	0.240
GPT-6 Astra	5.19%	100.0	100.0	100.0	100.0	100.0	0.240
Grok 4.6	9.09%	100.0	100.0	100.0	100.0	100.0	0.237
Kimi K3	12.99%	100.0	100.0	100.0	100.0	100.0	0.244

Appendix

Table 8: Evaluation of parameter estimation accuracy and generated simulation code quality on the 10-sample hard subset (77 ground-truth parameters). Percentages computed as a fraction of all

N=10

inputs that successfully reached each stage (CoT extracted, Code extracted, Compiles, Runs without error, Saves video). CodeBLEU averaged over scored samples.

Model	Parse	Avg pred	Avg	Mean frac	Mean frac	Global	End-to-end
	fails	params	matched	reported	within $20\%$	accuracy	recovery
claude_sonnet_4_6	0	7.7	3.29	0.460	0.171	0.185	0.0769
gemini_2_5_pro	96	5.5	2.83	0.399	0.150	0.170	0.0606
gemma_3_12b_it	330	4.8	1.94	0.281	0.130	0.163	0.0398
glm_4_1v_9b_thinking	420	4.0	1.93	0.273	0.147	0.199	0.0484
gpt_5_mini	465	8.4	3.17	0.426	0.139	0.178	0.0713

Appendix

Table 9: Parameter-estimation evaluation across 10 MLLMs on the 2,430 benchmark samples (162 experiments

\times

15 samples). Parse fails : number of samples in which the model’s response could not be parsed into a valid CoT block (no parameters recovered for those samples). Avg pred params : mean number of physical parameters the model emits per sample. Avg matched : mean number of model parameters that were successfully aligned to a ground-truth parameter, by greedy bipartite matching over alias sets that capture deabbreviation and Greek/Latin synonyms. Mean frac reported : mean over rows of

n_{\text{matched}}/n_{\text{gt}}

, i.e. recall on naming the correct parameters. Mean frac within

\mathbf{20\%}

: mean over rows of

n_{\text{within tol}}/n_{\text{matched}}

, i.e. value -accuracy among matched parameters at

\pm 20\%

relative-error tolerance. Global accuracy :

\sum n_{\text{within tol}}/\sum n_{\text{matched}}

pooled across all 2,430 rows. End-to-end recovery :

\sum n_{\text{within tol}}/\sum n_{\text{gt}}

, the fraction of all ground-truth parameters that the model both named correctly and estimated within

\pm 20\%

. Best per column in bold .

Metric	Claude Sonnet 4.6	GPT-5 Mini	Gap (GPT $-$ Claude)
Law equivalence – Overall	+0.27	+0.30	+0.03
Law validity – Overall	+0.02	$-0.04$	$-0.06$
CodeBLEU	+0.000	$-0.003$	$-0.003$
Runs without error	$-3.2$ pp	$-3.9$ pp	$-0.7$ pp
Video save	$-2.2$ pp	$-1.6$ pp	+0.6 pp
Param. Est. Acc. ( $\pm 20\%$ )	+0.76 pp	$-1.93$ pp	$-2.69$ pp

Appendix

Table 10: Bias scores across evaluation metrics for Claude Sonnet 4.6 and GPT-5 Mini. The gap is computed as GPT-5 Mini minus Claude Sonnet 4.6.

Metric	$\bm{\rho}$	$\mathbf{p}$
CodeBLEU	+1.000	$\sim$ 0
Law validity - Statement	+0.976	$\sim$ 0
Runs without error	+0.976	$\sim$ 0
Law validity - Name	+0.952	$\sim$ 0
Law validity - Overall	+0.905	0.002
Video save	+0.905	0.002

Appendix

Table 13: Model ranking similarity score measured by Spearman’s rank correlation coefficient (

\rho

) and corresponding significance value (

p

) for all the evaluation metrics

Statistical parameters	Value
Median	0.98
Middle 50% (IQR)	[0.93, 1.05]
90th / 95th percentile	1.17 / 1.22
Full range	[0.48, 2.12]
Parameters above 1.5 / below 0.6	3 / 4 (of 108)

Appendix

Table 16: Sensitivity across the 108 parameters

Objects per category	$\leq 2$	$2$ – $5$	$5$ – $10$	$>10$
SciPy	61	9	1	0
Kubric	38	14	15	7

Appendix

Table 13

Figure 5: Distribution of the 162 categories. (a) Nine top-level physical domains. (b) Physical laws invoked across the dataset (multi-label, so counts sum to more than 162); Newton’s 2nd law appears in every category by construction, while the long tail captures laws like the 1-D damped wave equation, gravitation, hydrostatic equilibrium, and the Lorentz force.

Experiment	$n_{\text{rand}}$	Samples	Seeded
complex_collision_plane	2	2500	$\star$
accelerating_cart_sliding_mass	4	2000	$\star$
balls_in_basin	4	2000	$\star$
disk_spinoff	7	1500	$\star$
block_on_sphere_basin	13	1300
molecule_2	12	1200

Appendix

Table 18: Pipeline A (SciPy): per-experiment statistics.

\star

marks seed-based experiments where a single integer seed drives non-factorizable hidden choices, requiring manually-elevated sample counts.

Experiment	$n_{\text{rand}}$	Var	Col	Samples
multi_physics_arena	47	$\bullet$		5000
tumbling_dice	19	$\bullet$	$\bullet$	3705
bouncing_balls_arena	16	$\bullet$	$\bullet$	3120
balloon_tether_wall_pull	15	$\bullet$	$\bullet$	2925
turntable_mirror_blocks	12	$\bullet$	$\bullet$	2340
rope_pendulum_swing	14	$\bullet$		2100

Appendix

Table 19: Pipeline B (Kubric): per-experiment statistics.

\bullet

in Var marks experiments with variable object counts (boost

\times 1.5

);

\bullet

in Col marks experiments with random color/material choice (boost

\times 1.3

). Sample counts are floored at 400 and capped at 5,000.

Model	Frames	Param. Acc.	Naming Recall	Value Acc.	No Errors	Video Save	CodeBLEU	Prompt Tok.
Qwen	8	5.84%	0.384	0.155	78.4%	82.7%	0.203	2,327
Qwen	16	6.23%	0.387	0.165	75.3%	80.9%	0.206	2,863
Qwen	32	6.31%	0.399	0.168	71.0%	72.8%	0.206	3,937
Qwen	60*	6.07%	0.409	0.157	75.3%	80.9%	0.206	5,468
Qwen	80	5.76%	0.408	0.143	77.8%	80.2%	0.208	6,190
InternVL	8	3.12%	0.247	0.126	48.1%	65.4%	0.194	3,905

Appendix

Table 20: How the parameter estimation and coding performance of the two open-source models vary with different number of input frames for Qwen3-VL-30B and InternVL3.5-30B.

Panel of Figure 6 — Figure 6: Cohen’s- $\kappa$ plot of human-rated video appearance similarity score with the verifiable patch-wise similarity metrics (DinoV2 and VideoCLIP) for (a) closed and (b) open-source models.

Figure 7: Videos rendered by code generated by closed-source models. The top row is the input (reference) video.

Figure 8: Videos rendered by code generated by open-source models. The top row is the input (reference) video.

Physical Reasoning in Video

Physical reasoning in video studies how models infer, predict, and generate motion and interactions consistent with physical laws.

See all 74 papers

PhyGround: Benchmarking Physical Reasoning in Generative World Models
May 11, 2026Juyi Lin, Arash Akbari, Yumei He +13Physical Consistency in Video Generation World Model Evaluation
1Northeastern University · 2Tulane University · University of Washington +1
PhyWorld: Physics-Faithful World Model for Video Generation
May 19, 2026Pu Zhao, Juyi Lin, Timothy Rupprecht +10Physics-Informed Generative Modeling Physical Consistency in Video Generation
Northeastern University · University of Georgia · Tulane University +1
Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning
Mar 20, 2026Qin Zhang, Peiyu Jing, Hong-Xing Yu +7Physical Consistency in Video Generation Video Generation Evaluation
Physion Labs · Stanford University · MIT +2

May 11, 2026J/K move · Enter open · S save

PhyGround: Benchmarking Physical Reasoning in Generative World Models

Juyi Lin, Arash Akbari, Yumei He, Lin Zhao+12

1Northeastern University · 2Tulane University · University of Washington+1

Physical Consistency in Video Generation World Model Evaluation Physical Reasoning in Video Video World Models Video Generation Evaluation

Open paper AskPDF arXiv

Generative world models are increasingly used for video generation, where learned simulators are expected to capture the physical rules that govern real-world dynamics. However, evaluating whether generated videos actually follow these rules remains challenging. Existing physics-focused video benchmarks have made important progress, but they still face three key challenges, including the coarse evaluation frameworks that hide law-specific failures, response biases and fatigue that undermine the validity of annotation judgments, and automated evaluators that are insufficiently physics-aware or difficult to audit. To address those challenges, we introduce PhyGround, a criteria-grounded benchmark for evaluating physical reasoning in video generation. The benchmark contains 250 curated prompts, each augmented with an expected physical outcome, and a taxonomy of 13 physical laws across solid-body mechanics, fluid dynamics, and optics. Each law is operationalized through observable sub-questions to enable per-law diagnostics. We evaluate eight modern video generation models through a large-scale, quality-controlled human study, grounded on social science lab experiment design. A total of 459 annotators provided 5,796 complete annotations and over 37.4K fine-grained labels; after quality control, the retained annotations exhibited high split-half model-ranking correlations (Spearman's rho > 0.90). To support reproducible automated evaluation, we release PhyJudge-9B, an open physics-specialized VLM judge. PhyJudge-9B achieves substantially lower aggregate relative bias than Gemini-3.1-Pro (3.3% vs. 16.6%). We release prompts, human annotations, model checkpoints, and evaluation code on the project page https://phyground.github.io/.

Abstract

Physical Reasoning in Video

Explore similar work

PhyGround: Benchmarking Physical Reasoning in Generative World Models

PhyWorld: Physics-Faithful World Model for Video Generation

Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning