InfiniteWorld: Separating Visual Evidence from Familiar Physics

Tong Zhang; Yuxuan Cheng; Yun Peng

cs.AIOct 24, 2026

InfiniteWorld: Separating Visual Evidence from Familiar Physics

Authors: Tong Zhang, Yuxuan Cheng, Yun Peng

Organizations: Fudan University · The Chinese University of Hong Kong, Shenzhen

Abstract

Physical reasoning requires adapting familiar expectations to the dynamics of the observed world. Existing benchmarks establish broad physical capabilities, yet an overall score leaves unclear whether answers follow visual evidence when familiar rules no longer apply. We introduce InfiniteWorld, a benchmark that pairs physical-reasoning questions with simulator videos whose physical parameters are systematically altered. Its generation pipeline connects simulation, scene screening, paired video translation, question construction, and input checks. Five tasks assess law identification, parameter estimation, temporal reasoning, analytic counterfactual comparison, and law induction. The evaluation defines five matched input settings for each question, comprising text-only evaluation and ordered or shuffled observations from simulator and translated videos. We evaluate a diverse set of video-language models under these matched settings. The results expose a persistent preference for familiar physical rules. Frames improve performance for most models, but gains differ across tasks. Strong accuracy on Earth-normal questions coexists with poor recognition of altered dynamics, and Earth-normal answers remain common when the observed physics changes. Task scores and semantic agreement further distinguish matching physical factors from matching their amount, and correct answers from repeated errors. InfiniteWorld makes these behaviors measurable through paired changes in the evidence supplied for the same question, separating visual evidence from familiar physics.

Figures & tables

Figure 1: When familiar expectations conflict with observed motion. Earth-normal expectations (left, schematic) contrast with recorded altered dynamics (center) and two physical judgments (right). Frames are fixed crops from the evaluated simulator videos in chronological order. The numeric reference is $e^{2}$ , where $e$ is the ratio of outgoing to incoming normal collision speed. Questions and answers are shortened for display. Appendix O includes the full questions and paired renderings.

Task	Questions	Earth-normal	Reference answer
Law identification	6,764	2,396	One listed altered factor or no alteration.
Parameter estimation	1,230	599	A physical quantity or bounce-height ratio.
Temporal reasoning	3,024	1,095	Relative height at two times within the video.
Analytic counterfactual comparison	2,715	1,198	Height or distance under an analytic reference.
Law induction	1,496	0	Whether a query and two references share a generator category.
Total	15,229	5,288	3,382 distinct simulator videos.

Table 1: Tasks. Earth-normal counts questions whose designated simulator videos use Earth-normal parameters. Law induction designates altered reference videos, although its query can be Earth-normal. Temporal questions use observations spanning the entire video.

Figure 2: Construction and evaluation. (a) Simulation and scene checks produce accepted scenes, paired videos, questions, and reference answers. The construction drawings are schematic. (b) The question and options stay identical across all input settings. Text only omits frames; the visual settings present simulator or translated frames in ordered or shuffled sequences. Shown observations are fixed crops from the law-identification example in Figure 1 ; letters identify their original positions, and the shuffled settings reorder the same frames. (c) Paired scores measure visual gain, order effect, and rendering gap. Reference answers enter only scoring.

Panel of Table 2 — Table 2: Full-set results. Score (%) on the full question set. Gain is the Simulator ordered score minus the Text only score. Models are grouped by developer. Bold marks column maxima within each output token limit, computed before rounding. Missing evaluations remain unscored; failed responses within completed evaluations receive zero credit.

Figure 3: Task scores across the five input settings. (a) Full-set task scores in Simulator ordered for the models in the radar legend; farther from the center means a higher score. (b,c) Full-set scores (Table 2 ) for the closed and open models with complete five-setting rows. Parentheses in the bar legend give visual gain, the Simulator ordered score minus the Text only score. Dashed lines show each panel’s per-setting mean. Family scaling appears in Figure 10 .

Figure 4: Familiar physical expectations persist. Left shows law-identification accuracy on Earth-normal and altered scenes. Right shows the Earth-normal error rate on altered questions. Open and filled markers denote Text only and Simulator ordered. Table 8 reports these comparisons.

Figure 5: Agreement includes repeated errors. Each bar separates correct agreement, agreement on the same error, and disagreement for the same law-identification question pairs. Simulator ordered increases correct agreement and reduces wrong agreement, but wrong agreement remains the larger agreeing group. Table 4 reports the counts.

Figure 6: Accuracy on four kinds of law-induction questions. Bars report the percentage correct within each group. Match requires the query to share both references’ generator category, so the answer is Yes. Different factors, Earth-normal, and Different amount require No. Each model is compared in Text only, Simulator ordered, and Simulator shuffled. Sol and Luna recognize matching questions well with ordered frames but perform much worse when the amount differs (Table 9 ).

Figure 7: Matching the factor is insufficient. This recorded Law induction question asks whether the query shares the reference videos’ generator category. The reference answer is No. The query keeps air dissipation but changes its amount category, placing it in the Different amount group. Shown frames are excerpts from the evaluated simulator videos.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Paired simulator and translated videos. An inverted-gravity scene in the simulator video (top) and translated video (bottom). Circles and guides show motion centroids in the displayed source frames. Evaluation uses the common frame sample without these overlays.

Model	Setting	LI	TR	AC	PE	IN	Macro
Qwen3-VL-32B	Text only	25.78	33.63	46.45	25.85	50.00	36.34
	Simulator ordered	45.40	51.39	51.31	36.79	51.47	47.27
	Simulator shuffled	44.40	42.79	51.86	36.02	50.67	45.15
	Translated ordered	41.40	34.85	52.56	36.83	51.20	43.37
	Translated shuffled	40.18	37.04	53.59	34.02	50.87	43.14
Qwen3-VL-30B-A3B	Text only	21.58	35.35	40.11	25.85	50.00	34.58

Appendix

Table 3: Full-set task scores (%). LI is law identification, TR temporal reasoning, AC analytic counterfactual comparison, PE parameter estimation, and IN law induction. Settings are Text only, Simulator ordered, Simulator shuffled, Translated ordered, and Translated shuffled. Macro gives equal weight to each task.

Setting	Agree (%)	Correct ( $n$ )	Wrong ( $n$ )	Disagree ( $n$ )
Text only	51.09	425	1,303	1,654
Simulator ordered	67.45	1,072	1,209	1,101
Simulator shuffled	67.92	990	1,307	1,085

Appendix

Table 4: Qwen3VL-8B option-arrangement consistency. Correct agreement, wrong agreement, valid disagreement, and invalid pairs form an exhaustive partition of the full set of question pairs. Agree is the percentage of pairs whose valid responses give identical answer text, computed from Correct plus Wrong before rounding. High agreement need not imply high accuracy.

Panel of Table 5 — Table 5: Balanced-subset results. Task-balanced score (%) on the same questions across input settings. Gain is the Simulator ordered score minus the Text only score. Tokens denotes the output limit. Bold marks column maxima within each output-limit group, using unrounded scores. Failed responses receive zero credit. Previously saved compatible answers are rescored on these exact question identifiers. Model names identify the evaluated systems. Table 2 reports the full-set comparison. SmolVLM2 uses its fixed native video preprocessing while receiving the same external frames.

Figure 9: Paired input-setting comparisons on the balanced subset. Visual gain adds Simulator ordered frames, the order effect restores chronological presentation, and the rendering gap compares Simulator ordered with Translated ordered (Eq. 4 ; Table 5 ). Models are sorted by visual gain within each output-budget group. Both visual gain and the order effect vary in sign, showing that frames and chronological order do not improve every model’s score.

Figure 10: Scaling does not guarantee greater visual benefit. Balanced-subset scores versus model size for Qwen3-VL and Qwen2.5-VL in Text only and Simulator ordered (Table 5 ; Qwen3-VL-30B-A3B is omitted). Qwen2.5-VL-3B loses score with frames, while the family’s largest model remains below its next smaller variant in Simulator ordered.

Figure 11: Public-text rules rival model scores on some tasks. Rules use question text and options and were selected after inspecting labels (Table 6 ). Model maxima use the diagnostic group in Figure 3 (a), comprising Qwen3-VL-8B, Qwen3.8-27B, Gemma4-12B, GPT-5.4 Mini, GPT-5.6 Sol, and GPT-5.6 Luna. The parameter-estimation rule exceeds every visual-setting score in this group; the temporal rule approaches its best Text only score. Matched frame comparisons help distinguish these answer patterns from visual benefit.

Figure 12: Sensitivity to supplied evidence. Differences in full-set score for the same model and questions. Visual gain adds Simulator ordered frames, order effect restores chronological order, and rendering gap compares simulator with translated frames. Positive values favor the first setting in each definition. Only complete pairs are shown.

Figure 13: Visual gains depend on the task. Simulator ordered score minus Text only score for models with both full-set evaluations. Each column retains every question in its task. Differences are computed before rounding; positive values indicate a benefit from frames. The columns separate law identification, parameter estimation, temporal reasoning, analytic counterfactual comparison, and law induction.

Task	Public-text prediction rule	Credit / $N$	Score
Temporal reasoning	Always same height	1425 / 3,024	47.12
Analytic counterfactual comparison	Lower for height; same for total distance	1598 / 2,715	58.86
Parameter estimation	Gravity ratio 1.0; upward fraction 0.25; bounce ratio 0.25	718.5 / 1,230	58.41

Appendix

Table 6: Post hoc analysis of answer distributions (score, %). Rules use public question text and options only, but were selected after inspecting test labels. They are not trained baselines or held-out generalization estimates. Numeric scoring retains half credit.

Model	Exclude AC			Exclude AC and TR
	Simulator ordered	Visual gain	Order effect	Simulator ordered	Visual gain	Order effect
Qwen3-VL-8B	44.43	+13.33	+3.98	44.20	+12.48	+2.80
Qwen3.8-27B	44.64	+4.91	+2.05	44.85	+7.67	+1.73
Gemma4-12B	35.65	+7.83	-0.10	33.64	+7.50	-0.89
GPT-5.4 Mini	38.27	+6.45	+2.23	36.57	+3.05	-0.85
GPT-5.6 Sol	53.12	+19.77	+11.54	55.24	+21.61	+11.33

Appendix

Table 7: Post hoc full-set simulator task-exclusion sensitivity. Simulator ordered scores are percentages and contrasts are points calculated from unrounded credits. Every retained question and failure remains included. Output limits differ across model groups.

Model	Text only	Simulator ordered	Difference (pp)	Accuracy
Qwen3-VL-8B	28.37	34.62	$+6.25$	31.23
Qwen3.8-27B	94.76	63.69	$-31.07$	21.29
Gemma4-12B	39.40	33.15	$-6.25$	26.14
GPT-5.4 Mini	54.37	20.88	$-33.49$	33.79
GPT-5.6 Sol	44.25	10.05	$-34.20$	59.07
GPT-5.6 Luna	2.77	4.65	$+1.88$	52.93

Appendix

Table 8: Earth-normal error rates on altered law-identification questions. Text only and Simulator ordered columns give the percentage choosing the Earth-normal answer. Difference is Simulator ordered minus Text only in percentage points. Altered-scene accuracy is the Simulator ordered accuracy on the same questions. Failures remain in every denominator. This diagnostic uses all altered law-identification questions. Model groups retain their original output budgets. All values are rounded independently from integer counts.

Model	Setting	Match (Yes)	Different factors (No)	Earth-normal (No)	Different amount (No)	All
Mini	Text only	1.60	98.01	97.93	99.35	49.93
Mini	Simulator ordered	14.71	89.49	85.12	89.61	51.40
Mini	Simulator shuffled	14.17	88.92	87.60	83.77	50.80
Sol	Text only	0.67	98.86	98.76	98.70	49.73
Sol	Simulator ordered	86.23	65.34	85.54	31.82	75.60
Sol	Simulator shuffled	88.10	39.20	64.05	20.13	65.71

Appendix

Table 9: Law-induction accuracy by question group (%). Parentheses show the reference answer. Failed responses receive zero credit. The group scores distinguish category matching from a general tendency to answer No.

Figure 14: Law identification. Selected frames from the Simulator and Translated videos.

Figure 15: Parameter estimation. A numeric reference answer for the bounce-height ratio.

Figure 16: Temporal reasoning. The two query times fall between the sampled frames.

Figure 17: Analytic counterfactual comparison. The displayed frames show the recorded scene; the reference answer comes from the analytic rule.

Figure 18: Law induction with Different amount. Simulator frames show a query with a different amount category from the two reference videos.

Figure 19: Law induction with Match. The Simulator query video shares the reference videos’ generator category.

Physical Reasoning

Physical reasoning studies how AI systems infer object properties, forces, and causal dynamics to predict and act in the physical world.

See all 69 papers

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
Aug 5, 2026Chen Yang, Shenxiang Zeng, Haoyang Zhao +6Counterfactual Reasoning Physical Reasoning in Video
1Tsinghua University · 2The University of Hong Kong
PhyGround: Benchmarking Physical Reasoning in Generative World Models
May 11, 2026Juyi Lin, Arash Akbari, Yumei He +13Physical Consistency in Video Generation World Model Evaluation
1Northeastern University · 2Tulane University · University of Washington +1
Apple- $π$ : Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence
Jul 17, 2026Runmao Yao, Kairui Hu, Yukang Cao +11Physical Consistency in Video Generation Physical Reasoning in Video
1S-Lab, Nanyang Technological University · 2The Chinese University of Hong Kong

Aug 5, 2026J/K move · Enter open · S save

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning

Chen Yang, Shenxiang Zeng, Haoyang Zhao, Zhouyuan Xu+5

1Tsinghua University · 2The University of Hong Kong

Counterfactual Reasoning Physical Reasoning in Video Embodied World Models Temporal Reasoning in VLMs

Open paper AskPDF arXiv

Reliable physical reasoning from video requires understanding how objects move, interact, and respond to interventions. Existing vision-language models (VLMs) often struggle to interpret these dynamics and reason reliably about future and counterfactual outcomes. We introduce PhysMind, a training-free agentic framework that constructs one reusable, question-agnostic executable world per video. PhysMind recovers a temporally consistent dynamic scene through object segmentation, mesh reconstruction, and 6D pose tracking, then fits analytic continuous-time dynamics and latent physical parameters without unrolling a time-stepped simulator. Given a question, it inspects, continues, or edits the world and answers from the resulting trajectories and interactions. Relative to direct chain-of-thought (CoT) reasoning with the same VLM, PhysMind improves accuracy by 38.23 points on CLEVRER and 8.08 points on Physion++. On counterfactual questions, it exceeds the strongest evaluated VLM baseline, GPT-5.5, by 19.25 points.

Abstract

Physical Reasoning

Explore similar work

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning

PhyGround: Benchmarking Physical Reasoning in Generative World Models

Apple-πππ: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

Apple- $π$ : Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence