WOVEN: Weaving Visual World Modeling into Multimodal LLMs

Zheyu Fan; Yue Zhang; Mingkai Deng; Kangrui Wang; Qineng Wang; Canyu Chen; Jie Hao; Xing Fan

cs.CVOct 8, 2026

WOVEN: Weaving Visual World Modeling into Multimodal LLMs

Authors: Zheyu Fan, Yue Zhang, Mingkai Deng, Kangrui Wang, Qineng Wang, Canyu Chen, Jie Hao, Xing Fan, +4 more

Organizations: Northwestern University · UNC Chapel Hill · Carnegie Mellon University · Amazon

Abstract

Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning. We hypothesize that these failures reflect a shared deficit in visual transition reasoning, and test whether this capability can serve as a shared training primitive, one that different models can learn from different supervision sources and reuse across different tasks, with a systematic training recipe. Existing benchmarks document these deficits separately but do not support controlled comparisons across scenes, actions, and reasoning operations. We therefore introduce WOVEN, a training source and benchmark for visual transition reasoning that organizes transition supervision by scene, action, and reasoning type, using diverse, realistic rollouts from video-pretrained generative models: 36,076 examples across 20 scene types, 5 action types, and 8 reasoning types. We first evaluate 38 frontier MLLMs (e.g., GPT-5.4 and Qwen3-VL-235B-A22B) and find a substantial and systematic deficit: even the strongest models fall far below humans, and the failures recur across model families and persist with scale. We then train MLLMs at multiple scales on WOVEN and find that they learn a shared capability that transfers broadly: training subsets of only about 2,000 items each collectively improve 22 of 26 external benchmarks by up to 27.3 percentage points, and WOVEN data can replace 30-50% of a task's own training data with comparable accuracy. Controlled comparisons further yield a training recipe for visual world modeling, validated prospectively on held-out benchmarks: select supervision by the reasoning operation it teaches rather than by the actions, scenes, or domains it shows, and prefer larger changes to the visual state for robustness. Our work establishes visual transition reasoning as a reusable foundation for systematic visual world-model training in MLLMs.

Figures & tables

Figure 1: Overview. (1) MLLMs fail across spatial, embodied, physical, and temporal tasks. (2) These failures share a weakness: reasoning about how visual states, actions, and outcomes relate. (3) WOVEN generates $(s,a,s^{\prime})$ transitions with a video generation model, organizes them along controlled axes of reasoning, action, and scene, and turns them into error-typed multiple-choice supervision. (4) Training on WOVEN improves $22$ of $26$ external benchmarks; the gains come from a shared primitive that different sources build and different tasks use; and controlled comparisons yield a training recipe: the reasoning operation decides where transfer goes, larger state changes give robustness, and transfer covers transition tasks rather than static perception.

Family	Reasoning operation	Distribution	Reasoning operation	Distribution
Causal	Forward dynamics	$P(s^{\prime}\mid s,a)$	Inverse dynamics	$P(a\mid s,s^{\prime})$
Counterfactual	Counterfactual removal	$P(s\mid a,s^{\prime})$	Counterfactual substitution	$P(s^{\prime}_{a_{\mathrm{alt}}}\mid a,s^{\prime},a_{\mathrm{alt}})$
Physical	Outcome prediction	$P(s^{\prime}\mid s,a_{\mathrm{exo}})$	Cued prediction	$P(s^{\prime}\mid s,a_{q})$
Temporal	Temporal ordering	$P(\mathbf{s}\mid\widetilde{\mathbf{s}},a)$	Temporal adjacency	$P(s_{-},s_{+}\mid s_{\mathrm{ref}},\widetilde{\mathbf{s}},a)$

Table 1: Eight reasoning operations considered in this work. The same operation can be applied to different actions and scenes (§ 2 ). Notation:

a_{\mathrm{alt}}

, an alternative action from the same initial state;

a_{\mathrm{exo}}

, a passive event;

a_{q}

a_{\mathrm{exo}}

with a cue naming the physical principle;

\mathbf{s}=(s_{0},\ldots,s_{L})

, sampled states in time order with

s_{0}=s

and

s_{L}=s^{\prime}

;

\widetilde{\mathbf{s}}

, the same states shuffled;

s_{-},s_{+}

, the neighbors in time of a reference state

s_{\mathrm{ref}}

. In the physical and temporal families the action description is optional.

Figure 2: The WOVEN taxonomy. Every item is labeled along the three axes shown from left to right (§ 2.2 ). Left: the exogenous action type with its eight principles, and the four endogenous types, where manipulative actions combine subject (human, humanoid) and view (egocentric, allocentric). Middle: the eight reasoning types in four families; in each schematic, the dashed box marks the element of $(s,a,s^{\prime})$ that the question asks for. Right: example scenes; restaurant and beach are held out for the scene-OOD test.

		Action type					Reasoning type								Perturbation
	Overall	Exo	Perc	Insp	Navi	Mani	Fwd	Inv	Rmv	Sub	Otm	Cue	Ord	Adj	Geo	App
Human	92.3	92.8	92.8	92.6	90.6	92.8	92.8	92.8	91.1	92.2	93.3	91.1	92.8	91.7	97.3	98.7
GPT-5.4	65.8	62.4	61.2	58.3	68.8	81.2	83.8	81.7	64.5	76.0	81.9	85.8	57.1	27.0	62.7	95.5
Intern-S1-Pro	63.2	56.2	52.8	53.7	70.1	87.1	77.6	77.5	66.7	73.3	61.5	69.1	63.4	26.5	33.3	79.2
Qwen2.5-VL-72B	58.9	53.7	47.7	47.8	66.4	83.4	74.7	74.7	54.8	72.0	59.4	76.0	49.4	28.0	38.5	83.0
Gemma-4-31B	57.8	50.8	49.4	53.1	61.2	74.3	76.3	76.6	63.6	64.5	64.9	70.5	30.8	25.3	59.0	95.5

Table 2: Visual transition reasoning in frontier MLLMs against the human baseline. Accuracy (%) on the in-distribution test set, overall and by action type and by reasoning type, and on the state-perturbation test set under geometric and appearance changes. Human accuracy is the mean of three annotators. Results for all

38

models are in Appendix E , with the corresponding analyses in Appendix F .

Gain over the base model after training on one subset
	ID test							OOD scenes test	Perturbation test
Training	Overall	Exo	Perc	Insp	Navi	Mani	Adj	Overall	Overall	Geo	App
Base	26.4	23.8	27.0	23.5	30.0	28.8	25.7	27.8	21.1	22.6	19.7
Full training set	89.3	79.4	90.7	90.5	90.0	95.3	69.7	88.2	38.3	35.0	41.6
perc_causal	+22.9	+5.1	+33.0	+13.4	+26.9	+39.8	+0.2	+20.0	+1.7	+0.9	+2.5
perc_temporal	+6.6	+8.4	+10.9	+4.9	+8.0	+1.3	+29.9	+2.9	+4.1	+5.6	+2.7

Table 3: WOVEN test accuracy (%) of Qwen2.5-VL-3B-Instruct after training on the full training set, and gains (percentage points over the base model) after training on each controlled subset. In-distribution results are reported overall, by action type, and on temporal adjacency; bold marks the action type a subset trains on. Results after SFT alone and by every reasoning type are in Appendix G.17 .

	SAT	InPhyRe	MindCube	AEQA	TB-L	WM-AB	WPred	TOMATO	BSwan	ERQA	SpViz	CV-Bench
Base	37.3	22.0	37.2	43.4	16.6	31.0	34.8	31.6	61.8	33.8	27.7	69.3
WOVEN SFT	55.3	24.6	37.8	49.5	11.5	37.1	39.6	31.5	55.9	37.5	28.8	72.0
WOVEN GRPO	61.3	34.9	48.6	54.4	26.8	39.8	41.5	36.1	65.6	34.5	31.1	72.4
$\Delta$	+24.0	+13.0	+11.3	+11.0	+10.3	+8.9	+6.7	+4.5	+3.8	+3.8	+3.4	+3.0

Table 4: Zero-shot transfer after training on the complete WOVEN training set. Accuracy (%) on the

12

improved benchmarks;

\Delta

is the larger of the SFT and GRPO gains. AEQA: ActionEQA; TB-L: TemporalBench-L; WM-AB: WM-ABench; WPred: WorldPrediction; BSwan: BlackSwan; SpViz: SpatialViz. References, results for all

26

benchmarks, and category breakdowns are in Appendix G.13 .

Figure 3: WOVEN supervision substitutes for in-domain training data. Each panel retrains one external benchmark on a fixed $2{,}000$ -item budget in which a share $p$ of the benchmark’s own training data is replaced by items from a selected WOVEN subset (blue dots: $3$ seeds; blue line: seed mean; band: seed range; gray dashed: $p{=}0$ mean; orange dashed: deletion control). Replacing $30$ – $50\%$ of task-specific supervision preserves comparable accuracy; on SAT, training exclusively on WOVEN yields the highest accuracy in the sweep. The protocol, full per-seed tables, and trend statistics for each benchmark are in Appendix G.14 .

Figure 4: Correlation of transfer profiles under the two candidate groupings. Pairwise Pearson correlation of the $11$ subsets’ $26$ -benchmark transfer profiles. Left: raw profiles, all pairs positive (the common component of § 4 ). Middle and right: the same residual matrix after the per-benchmark $z$ -score, ordered by action type and by reasoning family; only the latter produces diagonal blocks.

Subset	World Prediction	Action EQA
perc_causal	$+2.4$	$+12.5$
insp_causal	$+5.1$	$+10.9$
navi_causal	$+4.1$	$+4.4$
mani_causal	$+8.1$	$+11.6$
insp_cf	$-5.1$	$+12.5$
mani_cf	$+7.7$	$+9.8$

Table 6: Gains on the two agent-action benchmarks by non-temporal subset (percentage points over the base model; base accuracy: WorldPrediction

34.8

, ActionEQA

43.4

). Temporal subsets are covered by (a).

Comparison mixture	Item	Questions	Base	Recipe	Comp.	$\Delta$	$95\%$ CI	$p$
Temporal-heavy	(1)	Set A	31.1	37.9	34.4	$+3.5$	$[+2.03,+4.92]$	0.0002
Uniform	(1)	Set A	31.1	37.9	36.4	$+1.5$	$[+0.18,+2.81]$	0.03
Camera-heavy	(1)	Set C	54.1	58.4	57.8	$+0.6$	$[+0.02,+1.14]$	0.04
Exogenous-heavy	(3)	Set A	31.1	37.9	33.2	$+4.7$	$[+3.19,+6.17]$	0.0002
Exogenous-heavy	(3)	Set B	70.3	71.0	71.5	$-0.5$	$[-1.05,-0.02]$	0.04

Table 7: Prospective test of the recipe on held-out benchmarks. Accuracy (%), mean over

8

models;

\Delta

= Recipe

-

comparison mixture; Item = recipe item tested; Base = untrained model on the same questions; Set A: agent-action questions, Set B: passive-physics questions, Set C: camera-motion questions.

Appendix figures & tables76 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Navigative action type: representative QA items showing in-scene navigation targets across all four navigative reasoning types.

Figure 6: Inspective action type: representative QA items showing orbit-L / orbit-R / arc-up actions paired with all six inspective reasoning types.

Figure 7: Perceptive action type: representative QA items across all four perceptive reasoning types (forward dynamics, inverse dynamics, temporal ordering, temporal adjacency).

Figure 8: Construction pipeline: stage-1 textual specification, stage-2 VGM rollout (Wan2.2-I2V-A14B), stage-3 typed-distractor MCQ assembly, with quality-control filters at each stage.

Figure 9: Exogenous physical events (gravity, support, inertia, containment, collision): per-principle illustration showing the action-conditioned violation-of-expectation distractor design.

Figure 10: Exogenous object properties (permanence, cohesion, solidity): per-principle illustration with prompt template, video, and four-way distractor structure.

Figure 11: Manipulative cross-factoring: $\{$ human, humanoid $\}\times\{$ egocentric, allocentric $\}$ derivation showing the four pair-mate cells used by the pair-mate independence analysis (Appendix F.8 ) and the cross-action-type transfer analysis (Appendix G.7 ).

Figure 12: Manipulative action type: representative QA items showing object-interaction A/B/C/D action structure with all four manipulative reasoning types (forward dynamics, inverse dynamics, counterfactual substitution, counterfactual removal).

Figure 13: WOVEN splits: train_pool (used only for training), test_id ( $n{=}6{,}228$ ), test_perturbation_id ( $n{=}1{,}116$ ), and held-out OOD scenes ( beach , restaurant ). The OOD set is reserved for future fine-tuning evaluations.

Measurement	Value
Mean absolute residual, far half of the image, edited vs. source	$3.8$ (median $1.9$ , p90 $8.0$ )
Same measurement for the resize round-trip of the source frame (control)	$0.17$
Ratio edited / control, per frame	median $20.6\times$ ; $0$ of $864$ below $1.5\times$
Lag-1 spatial autocorrelation of the far-region residual	$0.58$
Reduction of the far-region residual by the best global shift ( $\pm 3$ px, $0.5$ px steps, $80$ frames)	$0\%$
Ratio edited / control by violation type ( $23$ types)	$10\times$ – $36\times$ , every type

Appendix

Table 11: The editor re-renders the whole image. Far-region residual between edited distractor frames and their source frames (test set,

864

edited frames; gray levels on a

0

–

255

scale).

Metric	Observed	Permutation null ( $10$ seeds)	$z$
Patch-level balanced accuracy (%)	$51.1$	$50.9\pm 0.4$	$0.5$
Image-level AUC	$0.532$	$0.520\pm 0.008$	$1.4$

Appendix

Table 12: Editing-artifact probe on the released data (test set,

166

sources,

166

correct options,

278

distractors).

Metric	Observed	Permutation null ( $10$ seeds)	$z$
Patch-level balanced accuracy (%)	$55.9$	$50.5\pm 0.5$	$11.3$
Image-level AUC	$0.718$	$0.508\pm 0.015$	$14.2$

Appendix

Table 13: Positive control: the same probe on a constructed subset without the null edit, with the correct option replaced by the original, unedited outcome frame (test set,

166

sources,

166

correct options,

278

distractors).

Model	Family	Params
GPT-5.4	GPT	N/A
GPT-5.2	GPT	N/A
Qwen-VL-Max	Qwen-max	N/A
Qwen2.5-VL-3B	Qwen2.5	3B
Qwen2.5-VL-7B	Qwen2.5	7B
Qwen2.5-VL-32B	Qwen2.5	32B

Appendix

Table 15: Model inventory.

Figure 14: Human study interface. Annotators see the same frames, question, and four options as the models. Each answer is saved when chosen and cannot be changed.

	Per-action type					Per-reasoning type
	Exo	Perc	Insp	Navi	Mani	Cue	Otm	Adj	Ord	Fwd	Inv	Rmv	Sub	Overall
human	92.8	92.8	92.6	90.6	92.8	91.1	93.3	91.7	92.8	92.8	92.8	91.1	92.2	92.3

Appendix

Table 16: Human accuracy (%) on the ID test, averaged over three annotators.

	appearance	geometric	overall
human	98.7	97.3	98.0

Appendix

Table 17: Human accuracy (%) on the perturbation probe, averaged over three annotators.

	Annotator 1	Annotator 2	Pooled $[95\%$ CI $]$
All questions	94.0	93.6	93.8 $[91.3,95.6]$
Main questions (200)	94.0	93.5	93.8
Perturbation questions (50)	94.0	94.0	94.0
Exogenous (43)	95.3	93.0	94.2
Inspective (43)	95.3	93.0	94.2
Manipulative (43)	95.3	93.0	94.2

Appendix

Table 18: Human audit, Part 1: blind answering (accuracy %,

250

questions).

Question	Annotator 1	Annotator 2	Pooled $[95\%$ CI $]$	Agreement
Scripted action carried out (yes)	94	91	92.5 $[88.0,95.4]$	92%
Scripted action carried out (yes or partly)	99	98	98.5 $[95.7,99.5]$	—
Final state plausible	99	98	98.5 $[95.7,99.5]$	97%
Video continuous	98	97	97.5 $[94.3,98.9]$	97%
Follows the stated physical principle (exogenous, 20)	100	100	100 $[91.2,100]$	100%

Appendix

Table 19: Human audit, Part 2: rollout videos (% of

100

videos).

	Annotator 1	Annotator 2	Pooled $[95\%$ CI $]$
Edits carried out as instructed with nothing else changed	149/150	147/150	98.7% $[96.6,99.5]$
Questions with all three edits correct	49/50	47/50	96.0%

Appendix

Table 20: Human audit, Part 3: edited images.

ID	Claim	$n$	Test	Statistic (df)	Raw $p$	Bonf. adj. $p$	Effect size
F.1	Inspective below manipulative, frontier	11 models	sign test	$11/11$ positive	$9.8{\times}10^{-4}$	$6.9{\times}10^{-3}$	gap range $16.2$ – $35.6$ pp
	Mirror confusion, frontier	7 families	1-sided binomial vs. $1/3$	$\hat{p}_{\min}{=}0.55$	$\ll\!10^{-10}$ each	$\ll\!10^{-9}$ each	$\hat{p}{-}1/3\in[0.22,0.56]$
	Orientation concentration, GPT-5.4	$231$ err	$\chi^{2}$ vs. uniform $1/3$	$65.22$ ( $\mathrm{df}{=}2$ )	$6.9{\times}10^{-15}$	$4.8{\times}10^{-14}$	$V{=}0.376$ (med–large)
	Orientation concentration, Qwen2.5-VL-7B	$477$ err	$\chi^{2}$ vs. uniform $1/3$	$31.74$ ( $\mathrm{df}{=}2$ )	$1.3{\times}10^{-7}$	$9.1{\times}10^{-7}$	$V{=}0.182$ (small)
F.2	Appearance vs. geometric scaling slope	4 dense ladders, 11 models	OLS log-linear, per-ladder intercepts, slope $b$	app $b{=}39.8$ , geo $b{=}11.1$ pts/dec ( $t{=}4.43$ , $4.83$ ; $\mathrm{df}{=}6$ )	$4.4{\times}10^{-3}$ , $2.9{\times}10^{-3}$	$3.1{\times}10^{-2}$ , $2.0{\times}10^{-2}$	slope difference $28.7$ pts/dec, $95\%$ CI $[8.1,49.4]$
F.2	Within-model app–geo gap, Qwen2.5-VL-7B	$558{+}558$	two-proportion $z$	$z{=}9.34$	$<10^{-20}$	$<10^{-19}$	$\Delta{=}27$ pp

Appendix

Table 21: Per-finding statistical details for the findings of Appendix F . Family-wise correction applies Bonferroni at

k{=}7

. All seven findings remain significant under correction.

Set	QA count	Sources
ID reasoning type items	6,228	594
ID perturbation items	1,116	320
Scene-OOD reasoning type	3,496	332

Appendix

Table 22: Benchmark scale.

Action type	#Comp	Reasoning type list
exogenous	4	Otm, Cue, Ord, Adj
perceptive	4	Fwd, Inv, Ord, Adj
inspective	6	Fwd, Inv, Rmv, Sub, Ord, Adj
navigative	4	Fwd, Inv, Ord, Adj
manipulative	4	Fwd, Inv, Rmv, Sub

Appendix

Table 23: Action type

\times

reasoning type adjacency.

Model	Family	Params	ID acc
GPT-5.4	GPT	N/A	$65.8\%$
Intern-S1-Pro	Intern	N/A	$63.2\%$
Qwen2.5-VL-72B	Qwen2.5	72B	$58.9\%$
Qwen3-VL-235B-A22B	Qwen3	235B-A22B (MoE)	$58.5\%$
GPT-5.2	GPT	N/A	$58.0\%$
Gemma-4-31B	Gemma4	31B	$57.8\%$

Appendix

Table 24: ID overall accuracy (

n{=}6{,}228

, sorted).

Model	Fwd	Inv	Rmv	Sub	Otm	Cue	Ord	Adj
GPT-5.4	83.8	81.7	64.5	76.0	81.9	85.8	57.1	27.0
GPT-5.2	70.7	78.6	53.2	56.3	78.8	85.4	46.4	27.1
Qwen-VL-Max	69.6	79.4	63.3	74.6	48.6	59.4	34.7	26.3
Qwen2.5-VL-3B	19.7	42.2	21.0	17.6	22.6	23.3	26.9	25.1
Qwen2.5-VL-7B	53.4	53.8	37.1	54.8	36.5	43.8	31.2	24.3
Qwen2.5-VL-32B	49.4	70.5	40.3	46.1	36.1	49.0	49.3	28.6

Appendix

Table 25: ID accuracy by reasoning type (%, all 38 models).

Model	Exo	Perc	Insp	Navi	Mani
GPT-5.4	62.4	61.2	58.3	68.8	81.2
GPT-5.2	58.9	52.3	50.7	63.4	67.4
Qwen-VL-Max	40.8	46.2	52.1	60.8	79.4
Qwen2.5-VL-3B	24.2	24.9	22.8	32.2	28.9
Qwen2.5-VL-7B	33.5	32.2	34.4	46.4	64.3
Qwen2.5-VL-32B	39.6	43.2	39.6	61.0	58.3

Appendix

Table 26: ID accuracy by action type (%, all 38 models).

Model	permanence	cohesion	solidity	gravity	support	inertia	collision	containment
GPT-5.4	56.2	61.8	69.4	75.0	62.5	60.4	43.8	70.1
GPT-5.2	57.6	60.4	62.5	75.0	50.7	55.6	43.8	66.0
Qwen-VL-Max	29.9	44.4	38.9	56.9	41.7	47.2	18.8	48.6
Qwen2.5-VL-7B	37.5	37.5	28.5	47.2	34.0	41.0	21.5	20.8
Qwen2.5-VL-32B	41.0	42.3	36.4	44.2	37.6	41.6	32.4	41.8
Qwen2.5-VL-72B	54.2	51.4	45.8	76.4	58.3	59.0	32.6	52.1

Appendix

Table 27: Exogenous sub-principle accuracy (%, ID; capable models shown, full 38 in repo).

Model	h/ego	h/allo	hh/ego	hh/allo
GPT-5.4	87.2	80.9	82.6	74.3
GPT-5.2	69.8	67.7	64.9	67.4
Qwen-VL-Max	85.1	79.2	81.2	72.2
Qwen2.5-VL-7B	68.8	65.6	66.7	56.2
Qwen2.5-VL-32B	66.7	59.0	60.8	46.9
Qwen2.5-VL-72B	91.3	84.4	85.8	72.2

Appendix

Table 28: T07b Manipulative subject

\times

view accuracy (%, ID; capable models).

Model	overall	geo	app
GPT-5.4	79.1	62.7	95.5
GPT-5.2	60.0	45.7	74.4
Qwen-VL-Max	48.9	33.7	64.2
Qwen2.5-VL-3B	21.1	22.6	19.7
Qwen2.5-VL-7B	37.0	23.5	50.5
Qwen2.5-VL-32B	37.4	30.1	44.6

Appendix

Table 29: Perturbation accuracy (overall / geo / app,

n{=}1{,}116

, all 38).

Figure 15: Diagnostic profiles of eight frontier MLLMs against the human baseline. Performance is grouped by action type (left), reasoning type (middle), and state-axis perturbation (right). Full $38$ -model results are reported in Appendix E .

Model	manipulative	inspective	gap
GPT-5.4	81.2	58.3	${+}23.0$
GPT-5.2	67.4	50.7	${+}16.7$
Qwen-VL-Max	79.4	52.1	${+}27.3$
Qwen2.5-VL-7B	64.3	34.4	${+}29.9$
Qwen2.5-VL-32B	58.3	39.6	${+}18.8$
Qwen2.5-VL-72B	83.4	47.8	${+}35.6$

Appendix

Table 30: Manipulative

-

inspective accuracy gap (frontier).

Model	translation	rotation	gap
GPT-5.4	92.4	74.0	${+}18.4$
GPT-5.2	88.2	56.6	${+}31.6$
Qwen-VL-Max	82.6	42.7	${+}39.9$
Qwen2.5-VL-7B	56.6	17.0	${+}39.6$
Qwen2.5-VL-32B	58.0	41.3	${+}16.7$
Qwen2.5-VL-72B	77.1	42.7	${+}34.4$

Appendix

Table 31: Perceptive translation vs. rotation (frontier).

Figure 16: Appearance and geometric robustness across the Qwen2.5-VL dense model ladder ( $\{3,7,32,72\}$ B). Appearance robustness scales $3.5{\times}$ faster than geometric robustness, with fitted slopes of $31.2$ and $8.9$ percentage points per decade of parameters, respectively.

Model	Params	ID	overall	geo	app
Qwen2.5-VL-3B	3B	26.4	21.1	22.6	19.7
Qwen2.5-VL-7B	7B	41.6	37.0	23.5	50.5
Qwen2.5-VL-32B	32B	47.7	37.4	30.1	44.6
Qwen2.5-VL-72B	72B	58.9	60.8	38.5	83.0

Appendix

Table 32: Qwen2.5-VL dense scaling (%).

Model	Params	ID	overall	geo	app
Qwen3-VL-8B-Instruct	8B	33.1	23.0	25.1	21.0
Qwen3-VL-30B-A3B	30B-A3B (MoE)	42.8	34.1	25.6	42.5
Qwen3-VL-32B-Instruct	32B	55.2	50.4	32.4	68.5
Qwen3-VL-235B-A22B	235B-A22B (MoE)	58.5	55.5	32.8	78.1

Appendix

Table 33: Qwen3-VL family.

Model	Params	ID	overall	geo	app
Gemma-3-4B	4B	28.9	24.4	24.6	24.2
Gemma-3-12B	12B	41.5	40.3	32.6	48.0
Gemma-3-27B	27B	47.0	49.5	36.4	62.5
Gemma-4-26B-A4B	26B-A4B (MoE)	43.9	34.1	25.4	42.8
Gemma-4-31B	31B	57.8	77.2	59.0	95.5

Appendix

Table 34: Gemma-3/Gemma-4.

Model	Params	ID	overall	geo	app
Qwen3.5-9B	9B	30.1	20.6	20.8	20.4
Qwen3.5-27B	27B	33.3	36.8	31.2	42.5
Qwen3.5-35B-A3B	35B-A3B (MoE)	22.5	12.6	10.6	14.7
Qwen3.5-122B-A10B	122B-A10B (MoE)	26.9	25.8	24.4	27.2
Qwen3.5-397B-A17B	397B-A17B (MoE)	55.4	50.0	32.8	67.2

Appendix

Table 35: Qwen3.5 family (reasoning disabled).

Model	ID acc	ID rank	Geo acc	Geo rank	$\Delta$
ERNIE-4.5-VL-424B-A47B	51.7	11	25.1	30	${+}19$
Qwen2.5-VL-7B	41.6	20	23.5	33	${+}13$
Gemma-4-26B-A4B	43.9	17	25.4	28	${+}11$
Llama-4-Scout	32.6	25	17.0	35	${+}10$
Qwen3-VL-30B-A3B	42.8	18	25.6	27	${+}9$
Seed-2.0-Mini	54.3	10	27.1	19	${+}9$

Appendix

Table 36: ID vs. perturbation-geometric rank shift (subset; positive

\Delta

= relatively worse on geometric).

Model	Ord	Adj	Gap
GPT-5.4	57.1	27.0	${+}30.1$
GPT-5.2	46.4	27.1	${+}19.3$
Qwen-VL-Max	34.7	26.3	${+}8.4$
Qwen2.5-VL-32B	49.3	28.6	${+}20.7$
Qwen2.5-VL-72B	49.4	28.0	${+}21.4$
Qwen3-VL-32B	37.1	26.3	${+}10.8$

Appendix

Table 37: Temporal ordering–adjacency gap (frontier).

Comparison	Dense acc	MoE acc	gap
Qwen3-VL-32B vs. 30B-A3B	55.2	42.8	${+}12.4$
Gemma-4-31B vs. 26B-A4B	57.8	43.9	${+}13.9$
Qwen3.5-27B vs. 35B-A3B	33.3	22.5	${+}10.9$

Appendix

Table 38: Dense vs. MoE comparisons (matched total).

Pair	centered cos
Qwen2.5-VL-72B $\leftrightarrow$ Qwen3-VL-235B-A22B	${+}0.951$
GPT-5.4 $\leftrightarrow$ GPT-5.2	${+}0.895$
Qwen3-VL-235B $\leftrightarrow$ Qwen3-VL-32B	${+}0.895$
ERNIE-424B $\leftrightarrow$ GPT-5.4	${+}0.825$
Qwen3-VL-235B $\leftrightarrow$ GPT-5.4	${+}0.761$
Qwen2.5-VL-72B $\leftrightarrow$ GPT-5.4	${+}0.712$

Appendix

Table 39: F-Fingerprint: centered cosine similarity on

8

-dim per-principle accuracy vectors.

			Perturbation probe
Variant	ID	OOD	all	geo	app
Qwen2.5-VL-3B (base)	26.4	—	21.1	22.6	19.7
+full-mix SFT	89.3 ( $\uparrow$ 62.9)	88.2	38.3 ( $\uparrow$ 17.1)	35.0 ( $\uparrow$ 12.4)	41.6 ( $\uparrow$ 21.9)
+inverse-text SFT	95.5 $\dagger$ ( $\uparrow$ 54)	—	—	—	—
Qwen2.5-VL-7B (base)	41.6	—	37.0	23.5	50.5
+mv-balanced SFT	50.1 ( $\uparrow$ 8.5)	57.5	52.4 ( $\uparrow$ 15.4)	38.4 ( $\uparrow$ 14.9)	66.5 ( $\uparrow$ 16.0)

Appendix

Table 40: SFT accuracy on WOVEN’s own test sets (ID, scene-OOD, state-axis perturbation probe) (%). Full-mix SFT is on Qwen2.5-VL-3B; inverse-text is narrow SFT on inverse dynamics with text options (

4

endogenous action types); mv-balanced is balanced mv_fwd + mv_bwd training on Qwen2.5-VL-7B. Downstream transfer is in Tables 5 and 5 . Per-cell own deltas, the remaining narrow-run families, and full recipe details in the appendix.

	inspective	perceptive	navigative	manipulative	mean
base 3B	$16$	$27$	$59$	$62$	$41$
inverse-text SFT	$\mathbf{93}$	$\mathbf{97}$	$\mathbf{98}$	$\mathbf{94}$	$\mathbf{95.5}$
$\Delta$	${+}77$	${+}70$	${+}39$	${+}32$	${+}54$

Appendix

Table 41: Inverse-text SFT per-cell own (

\Delta

vs. base 3B). Columns are inverse-dynamics cells by action type; the last column is the mean over the four own cells.

Source $\downarrow$ / Target $\rightarrow$	Perc Fwd	Insp Fwd	Navi Fwd	Mani Fwd
Base 3B (acc, %)	$28.1$	$28.1$	$30.9$	$21.9$
perceptive ( $n{=}1036$ ) $\Delta$	$\mathbf{{+}68.8}$ own	${+}13.8$	${+}55.6$	${+}50.0$
inspective ( $n{=}1020$ ) $\Delta$	${+}12.9$	$\mathbf{{+}64.9}$ own	${+}57.3$	${+}46.9$
navigative ( $n{=}1020$ ) $\Delta$	${+}11.1$	${+}10.0$	$\mathbf{{+}67.0}$ own	${+}61.4$
manipulative ( $n{=}1020$ ) $\Delta$	${+}7.0$	${+}1.5$	${+}61.5$	$\mathbf{{+}74.6}$ own

Appendix

Table 42: Cross-action-type forward transfer (

4{\times}4

n{\approx}1{,}020

each, shared SFT protocol.

	Fwd	Sub	Rmv	Inv	Adj	Ord
Base 3B (acc, %)	$28.1$	$26.7$	$25.6$	$15.2$	$26.1$	$27.0$
SFT $\Delta$	$\mathbf{{+}65}$ own	${+}48$	${+}25$	${+}19$	$0$	${-}1$

Appendix

Table 43: (a) Within-action-type cross-comp transfer (inspective forward run).

	Fwd	Sub	Rmv	Inv
Base 3B Insp (acc, %)	$28.1$	$26.7$	$25.6$	$15.2$
Sub $\Delta$ Insp	${+}60$	$\mathbf{{+}66}$ own	${-}1$	${+}48$
Base 3B Mani (acc, %)	$21.9$	$20.8$	$24.0$	$62.5$
Sub $\Delta$ Mani	${+}71$	$\mathbf{{+}74}$ own	$0$	${+}20$

Appendix

Table 44: (b) Sub within-action-type cross-comp, per-action-type split.

	orb_l (own)	orb_r (own)	arc_up (orth)
base 3B (%)	$30.0$	$20.0$	$31.1$
3B $\Delta$	${+}40.0$	${+}45.6$	$\mathbf{{-}24.4}$
7B $\Delta$	$\mathbf{{+}50.0}$	$\mathbf{{+}75.6}$	$\mathbf{{+}43.3}$

Appendix

Table 45: Balanced rotation: 3B vs. 7B ( arc_up interference reverses). Columns are inspective

\times

forward cells by camera move.

	rot_l	rot_r
base 3B (%)	$27.8$	$33.3$
rot_l only (7B) $\Delta$	$\mathbf{{+}65.3}$ own	$-15.2$ cross
rot_r only (7B) $\Delta$	$-13.9$ cross	$\mathbf{{+}63.9}$ own

Appendix

Table 46: Perceptive single-direction runs: 3B vs. 7B. Columns are perceptive cells by camera move.

	orb_l	orb_r	arc_up
base 3B (%)	$30.0$	$20.0$	$31.1$
orb_l only (7B) $\Delta$	$\mathbf{{+}58.9}$ own	$\mathbf{{+}26.7}$ cross	${+}36.7$ cross
orb_r only (7B) $\Delta$	${-}4.4$ cross	$\mathbf{{+}72.2}$ own	${+}37.8$ cross

Appendix

Table 47: Inspective single-direction runs: 3B vs. 7B, asymmetric chirality sharing. Columns are inspective cells by camera move.

	mv_fwd (own)	mv_bwd (own)	rot_l (cross)	rot_r (cross)
base 3B (%)	$20.8$	$19.4$	$27.8$	$33.3$
mv-balanced 7B (%)	$\mathbf{100.0}$	$\mathbf{100.0}$	$2.8$	$5.6$
$\Delta$	${+}79.2$	${+}80.6$	$\mathbf{{-}25.0}$	$\mathbf{{-}27.7}$

Appendix

Table 48: mv-balanced (

7

B) move chirality (perceptive forward). Columns are perceptive cells by camera move.

Variant	own_exo subset acc
$V_{1}$ (shuffled)	$33.2\%$
$V_{2}$ (emergence-ordered)	$\mathbf{38.0\%}$
$\Delta$ ( $V_{2}-V_{1}$ )	$\mathbf{{+}4.8}$ pp

Appendix

Table 49: Curriculum vs. shuffled at 1 epoch (own_exo subset).

Table 61

Normalization	same reasoning family	same action type	neither
raw $\Delta$ (points)	$+0.71$	$+0.53$	$+0.42$
relative $\Delta/\text{base}$	$+0.65$	$+0.49$	$+0.42$
headroom $\Delta/(100-\text{base})$	$+0.70$	$+0.46$	$+0.36$
per-benchmark $z$ -score	$+0.32$	$-0.22$	$-0.23$

Appendix

Table 51: Mean pairwise Pearson correlation of the

11

cells’

26

-dimensional transfer profiles, grouped by what the cell pair shares, under four normalizations of the per-benchmark gains.

	SAT	DSI	MVP	WM-AB	TB-L	ViewSp	R2VLM	InPhyRe	ERQA	BSwan	CLEVRER	TVBench
7B SFT	${+}20.00$	${+}6.32$	${+}5.08$	${+}4.39$	${+}4.30$	—	${+}3.24$	—	—	—	—	—
7B GRPO	${+}20.00$	${+}5.96$	${+}4.49$	${+}4.83$	${+}5.34$	${+}2.10$	${+}3.54$	—	${+}2.75$	—	${+}2.19$	${+}2.04$
32B SFT	${+}22.67$	—	${+}6.28$	${+}5.91$	—	${+}4.94$	—	${+}3.54$	—	${+}2.29$	—	—

Appendix

Table 81: Transfer at scale. Zero-shot transfer gains (pp over the same-scale base) after WOVEN full-mix post-training of Qwen2.5-VL-7B and Qwen2.5-VL-32B. The recipe follows the 3B protocol; the 32B run is SFT-only; SAT is under circular evaluation. Abbreviations as in Tables 5 and 5 ; ViewSp: ViewSpatial. ‘—’: not reported.

Model	3DSRBench
Qwen3.5-4B base	$48.1$
Orca-4B	$52.1$ ( ${+}4.0$ )
${+}$ WOVEN (ours)	$\mathbf{54.8}$ ( ${+}6.7$ )

Appendix

Table 82: Same-backbone comparison with video-SSL mid-training. Both models start from Qwen3.5-4B; gains in parentheses are over the shared base. Orca-4B: latent next-state mid-training (

125

K hours of video,

160

M event annotations). Ours: WOVEN post-training (SFT,

{\approx}107

K QA pairs,

{\approx}7

hours of generated video).

Arm	Training data	Relation to the real data
Real	WOVEN perceptive $\times$ causal dynamics (forward and inverse dynamics questions, four option orderings).	—
Per-source permutation	Same items, images, options, and answer-letter balance. Within each source, the four camera actions are mapped to the outcomes by a random permutation with no fixed point: the forward question for action $a_{i}$ has the outcome of $a_{\pi(i)}$ as its answer, and the inverse question showing outcome $s^{\prime}_{k}$ has $a_{\pi^{-1}(k)}$ as its answer. The answer changes in $95.1\%$ of the examples.	Identical format; the action–outcome mapping differs across sources, so no consistent rule can be learned.
Global mirror	Same items, images, options, and answer-letter balance. One mapping for all sources: move forward $\leftrightarrow$ move backward, rotate left $45^{\circ}$ $\leftrightarrow$ rotate right $45^{\circ}$ . The answer changes in $95.0\%$ of the examples.	Identical format; a consistent, learnable rule with the direction of every action reversed.
A-OKVQA	$2{,}072$ four-option questions from the A-OKVQA training set ( Schwenk et al., 2022 ) (COCO images), each in its four option orderings, plus $1{,}462$ additional random orderings.	Multiple-choice training with a balanced answer-letter distribution, without WOVEN content.

Appendix

Table 101: Content control: the four training sets (

9{,}750

training examples each, same number of optimizer steps).

Benchmark	Base	Real	Permutation	Mirror	A-OKVQA
SAT (circular)	39.3	63.3	35.1	37.8	44.0
SPAR-Bench view change (real images)	26.4	54.9	27.5	35.8	21.8
ActionEQA (circular)	25.6	31.8	22.2	26.5	22.6
WorldPrediction (circular)	14.7	16.6	12.4	11.3	10.3
WOVEN OOD	36.2	45.9	36.8	32.9	32.6

Appendix

Table 102: Content control: accuracy (%), mean over

3

seeds.

Benchmark	Real $-$ Permutation	Real $-$ Mirror	Real $-$ A-OKVQA
SAT (circular)	$+28.2$ $[+21.1,+35.8]$	$+25.6$ $[+18.2,+33.3]$	$+19.3$ $[+11.8,+26.9]$
SPAR-Bench view change	$+27.5$ $[+20.7,+34.5]$	$+19.2$ $[+12.4,+25.5]$	$+33.1$ $[+26.4,+39.3]$
ActionEQA (circular)	$+9.6$ $[+7.0,+12.3]$	$+5.3$ $[+3.5,+7.2]$	$+9.2$ $[+6.7,+11.8]$
WorldPrediction (circular)	$+4.2$ $[+1.9,+6.3]$	$+5.3$ $[+3.6,+7.1]$	$+6.3$ $[+4.0,+8.6]$
WOVEN OOD	$+9.1$ $[+7.2,+11.0]$	$+13.0$ $[+11.4,+14.5]$	$+13.3$ $[+11.4,+15.3]$

Appendix

Table 103: Content control: Real minus control, percentage points with bootstrap

95\%

confidence intervals; every difference has

p<0.001

under the paired permutation test.

Benchmark	Arm	s1	s2	s3	Mean
SAT (circular)	Real	65.3	63.3	61.3	63.3
	Per-source permutation	35.3	36.7	33.3	35.1
	Global mirror	37.3	40.0	36.0	37.8
	A-OKVQA	42.7	44.7	44.7	44.0
SPAR-Bench view change	Real	56.3	55.2	53.3	54.9
	Per-source permutation	29.9	25.7	26.8	27.5

Appendix

Table 104: Content control: per-seed accuracy (%).

Arm	Forward: real answer	Forward: mirrored answer	Inverse: real answer	Inverse: mirrored answer
Base	24.7	27.1	25.3	28.8
Real	97.2 / 98.6 / 95.8	1.0 / 0.3 / 1.7	89.2 / 93.1 / 85.8	6.6 / 3.8 / 6.2
Per-source permutation	21.5 / 24.3 / 21.9	18.4 / 20.5 / 21.9	22.9 / 24.3 / 22.2	31.2 / 29.2 / 32.3
Global mirror	37.2 / 42.0 / 20.5	26.7 / 32.3 / 71.2	5.9 / 8.3 / 5.6	87.2 / 84.7 / 92.4
A-OKVQA	24.7 / 25.0 / 22.6	25.0 / 25.7 / 26.7	31.9 / 33.3 / 25.3	29.5 / 30.2 / 31.9

Appendix

Table 105: Content control, manipulation check: answers on WOVEN ID perceptive dynamics questions (%), per seed.

Benchmark	Model	A	B
SAT	Correct answers	50.0	50.0
	Base	62.0	38.0
	Full mixture, SFT	51.0	49.0
	Full mixture, SFT + GRPO	49.0	51.0
	Perceptive $\times$ causal dynamics, SFT	52.3	47.7
	Perceptive $\times$ causal dynamics, SFT + GRPO	55.0	45.0

Appendix

Table 106: Letter-balance control: answer-letter distribution (% of answers), standard scoring.

Benchmark	Scoring	Base	Full mixture, SFT	Full mixture, SFT + GRPO	Perc. $\times$ causal, SFT	Perc. $\times$ causal, SFT + GRPO
SAT	Standard	57.3	68.3	67.7	69.7	66.3
	Circular	39.3	62.7	61.3	64.7	58.7
ActionEQA	Standard	43.1	50.2	49.0	49.5	52.0
	Circular	25.6	35.4	34.2	32.8	35.5
SPAR-Bench view change	Standard	26.4	41.8	39.5	54.8	55.9
	Circular	0.4	20.3	19.5	30.7	36.0

Appendix

Table 107: Letter-balance control: accuracy (%) under standard and circular scoring.

Benchmark	Scoring	Full mixture, SFT	Full mixture, SFT + GRPO	Perc. $\times$ causal, SFT	Perc. $\times$ causal, SFT + GRPO
SAT	Standard	$+11.0$ $[+4.7,+17.7]$	$+10.3$ $[+3.7,+16.7]$	$+12.3$ $[+6.7,+18.0]$	$+9.0$ $[+3.0,+15.0]$
	Circular	$+23.3$ $[+14.0,+32.7]$	$+22.0$ $[+12.7,+31.3]$	$+25.3$ $[+16.7,+34.0]$	$+19.3$ $[+11.3,+27.3]$
ActionEQA	Standard	$+7.1$ $[+3.0,+11.2]$	$+5.9$ $[+1.9,+10.1]$	$+6.4$ $[+3.0,+9.8]$	$+8.9$ $[+5.5,+12.4]$
	Circular	$+9.8$ $[+6.1,+13.4]$	$+8.6$ $[+5.1,+12.2]$	$+7.1$ $[+4.4,+10.0]$	$+9.9$ $[+6.9,+13.0]$
SPAR view change	Standard	$+15.3$ $[+7.3,+23.4]$	$+13.0$ $[+4.6,+21.1]$	$+28.4$ $[+21.1,+35.6]$	$+29.5$ $[+22.2,+36.8]$
	Circular	$+19.9$ $[+15.3,+24.9]$	$+19.2$ $[+14.6,+24.1]$	$+30.3$ $[+24.9,+35.6]$	$+35.6$ $[+29.9,+41.4]$

Appendix

Table 108: Letter-balance control: trained minus base, percentage points with bootstrap

95\%

confidence intervals. Every difference is significant under the paired permutation test (

p\leq 0.006

under standard scoring;

p<0.001

under circular scoring).

Reasoning type	Base	Forward-only	$\Delta$ $[95\%$ CI $]$	$p$
Counterfactual substitution	28.5	93.3	$+64.8$ $[+58.5,+71.1]$	$<0.001$
Inverse dynamics	43.3	60.7	$+17.4$ $[+12.2,+22.6]$	$<0.001$
Forward dynamics (trained)	37.8	87.4	$+49.6$ $[+42.2,+57.0]$	$<0.001$

Appendix

Table 109: Static-free questions (unchanged-scene option removed, three options), accuracy (%).

Reasoning type	Base	Forward-only	$\Delta$ $[95\%$ CI $]$	$p$
Counterfactual substitution	24.8	78.1	$+53.3$ $[+45.9,+60.4]$	$<0.001$
Counterfactual removal (answer: unchanged scene)	27.8	55.2	$+27.4$ $[+19.3,+35.6]$	$<0.001$
Inverse dynamics	17.4	44.1	$+26.7$ $[+21.5,+32.2]$	$<0.001$
Forward dynamics (trained)	28.9	86.3	$+57.4$ $[+50.7,+64.1]$	$<0.001$

Appendix

Table 110: Original questions (four options, unchanged-scene option included), accuracy (%).

Domain	Benchmark	Units	Scoring
Spatial	SAT	150	circular: a question counts only if both answer orders are correct
Spatial	ViewSpatial	5,712	question accuracy
Embodied	ActionEQA	3,782	question accuracy
Embodied	Robo2VLM	6,676	question accuracy
Physical	InPhyRe	19,981	question accuracy
Physical	CLEVRER	12,000	per-option accuracy (official stratified sample)

Appendix

Table 111: Representative benchmarks, evaluation units, and scoring.

Benchmark	Base	Best candidate	Best gain $[95\%$ CI $]$	Null	$p_{\max}$	$p_{\mathrm{Holm}}$
SAT	37.3	Perc. $\times$ causal, SFT	$+26.0$ $[+17.3,+34.7]$	$+10.4$	$<0.001$	0.002
ViewSpatial	35.3	Mani. $\times$ cf, SFT + GRPO	$+3.6$ $[+2.7,+4.5]$	$+1.3$	$<0.001$	0.002
ActionEQA	43.4	Mani. $\times$ causal, SFT	$+11.5$ $[+9.9,+13.1]$	$+2.0$	$<0.001$	0.002
Robo2VLM	33.5	Mani. $\times$ cf, SFT	$+7.6$ $[+6.6,+8.6]$	$+1.2$	$<0.001$	0.002
InPhyRe	22.0	Insp. $\times$ cf, SFT	$+10.3$ $[+9.7,+10.9]$	$+0.8$	$<0.001$	0.002
CLEVRER	58.2	Insp. $\times$ cf, SFT + GRPO	$+4.6$ $[+3.8,+5.4]$	$+1.0$	$<0.001$	0.002

Appendix

Table 112: Best of

22

candidates versus the max-selection null (accuracy %, gains in points). Perc., Insp., Mani.: perceptive, inspective, manipulative; cf: counterfactual. Null: 95th percentile of the best gain under selection alone;

p_{\max}

p

under the max-selection null;

p_{\mathrm{Holm}}

: Holm-corrected over the 22 candidates.

Benchmark	Subset	Seed 1	Seed 2	Seed 3	Mean
SAT	Perc. $\times$ causal	$+20.7$	$+24.0$	$+27.3$	$+24.0$
ViewSpatial	Mani. $\times$ cf	$+3.3$	$+3.1$	$+4.3$	$+3.6$
ActionEQA	Mani. $\times$ causal	$+11.6$	$+11.4$	$+9.5$	$+10.8$
Robo2VLM	Mani. $\times$ cf	$+8.1$	$+7.0$	$+6.8$	$+7.3$
InPhyRe	Insp. $\times$ cf	$+10.2$	$+9.8$	$+10.6$	$+10.2$
CLEVRER	Insp. $\times$ cf	$+5.1$	$+4.6$	$+4.2$	$+4.6$

Appendix

Table 113: Best subset after GRPO: gain over the base model per GRPO seed (points). Every per-seed gain has

p<0.001

under the paired sign-flip test except WorldPrediction seeds 1 and 2 (

p=0.011

and

p=0.001

Benchmark	Base	Seed 1	Seed 2	Seed 3	Mean	$\Delta$ $[95\%$ CI $]$	$p$
WOVEN ID	27.9	44.4	44.5	44.3	44.4	$+16.5$ $[+15.1,+18.0]$	$<0.001$
WOVEN OOD	33.0	45.2	44.3	43.9	44.5	$+11.5$ $[+9.4,+13.5]$	$<0.001$
SPAR-Bench view change (real images)	28.0	46.4	45.6	51.0	47.6	$+19.7$ $[+12.5,+26.6]$	$<0.001$
SAT (circular)	30.0	40.0	46.0	55.3	47.1	$+17.1$ $[+6.9,+27.1]$	$0.002$

Appendix

Table 114: Gemma-3-4B-it: accuracy (%).

\Delta

= trained (mean of

3

seeds)

-

base, percentage points with bootstrap

95\%

confidence intervals.

Benchmark	Base	Mean of $3$ seeds	$\Delta$
WOVEN ID	29.8	47.7	$+17.9$
WOVEN OOD	36.2	45.9	$+9.7$
SPAR-Bench view change (real images)	26.4	54.9	$+28.5$
SAT (circular)	39.3	63.3	$+24.0$

Appendix

Table 115: The same training on Qwen2.5-VL-3B-Instruct, accuracy (%), for reference.

Model	Forward	Inverse	Joint	Indep.	Excess $[95\%$ CI $]$	Cycle	$\Delta$ joint $[95\%$ CI $]$
Base	21.3	40.4	8.0	8.1	$-0.1$ $[-1.2,+0.9]$	19.0	—
Full mixture, SFT	97.1	94.8	93.5	92.1	$+1.4$ $[+0.8,+2.1]$	94.3	$+85.4$ $[+83.2,+87.7]$
Full mixture, SFT + GRPO	96.6	96.9	94.1	93.7	$+0.4$ $[+0.1,+0.9]$	94.4	$+86.1$ $[+83.7,+88.2]$
Perceptive $\times$ causal, SFT	75.3	76.6	65.5	61.8	$+3.7$ $[+2.6,+4.8]$	71.3	$+57.5$ $[+53.6,+61.5]$
Perceptive $\times$ causal, SFT + GRPO	75.7	77.4	66.9	62.8	$+4.1$ $[+3.2,+4.9]$	72.8	$+58.8$ $[+55.1,+62.6]$
Inspective $\times$ causal, SFT	73.7	75.8	61.9	59.3	$+2.7$ $[+1.6,+3.7]$	69.0	$+53.9$ $[+49.6,+58.1]$

Appendix

Table 116: Forward and inverse consistency on WOVEN ID (%). Joint: both questions of a pair correct; Independent: expected joint rate under independence within action type; Excess: joint

-

independent; Cycle: the inverse query recovers the action behind the outcome the model predicted;

\Delta

joint: joint

-

joint of the base model.

Subset	Recipe	Uniform	Temporal-heavy	Camera-heavy	Exogenous-heavy
Perceptive $\times$ causal dynamics	197	137	45	245	45
Inspective $\times$ causal dynamics	197	137	45	245	45
Navigative $\times$ causal dynamics	196	137	45	46	45
Manipulative $\times$ causal dynamics	196	137	45	46	45
Inspective $\times$ counterfactual	226	136	44	245	45
Manipulative $\times$ counterfactual	226	136	44	46	45

Appendix

Table 117: Prospective test: number of training items per subset in each mixture (of

1{,}500

Source	Subset	Questions	Example
MMSI-Bench	Motion (Cam.)	74	“The images are taken continuously from a first-person perspective. In which direction are you moving?”
VLM4D (real videos)	object motion direction	1,111	“Which way does the person turn?”
VLM4D (real videos)	motion direction from a stated viewpoint	260	“From the camera perspective, which direction are the cyclists moving toward?”
PhysBench	relationships $\cdot$ motion	145	“Moving in the direction indicated by the arrow in the picture, which of the following options are you most likely to see?”
PhysBench	dynamics $\cdot$ manipulation	83	“Which of the following options presents the correct sequence to use the spoon to pick the objects in the green bowl and put them in the green pot?”

Appendix

Table 118: Prospective test: Set A.

Source	Subset	Questions	What is asked
MVBench	counterfactual inference	200	which collision happens if an object is removed
MVBench	moving attribute	200	attributes of the objects that move or stay still
MVBench	moving count	200	how many objects of a kind are moving
MVBench	object existence	198	whether moving objects are still present at the end
IntPhys 2	solidity, fixed camera	104	whether the event is physically possible
IntPhys 2	immutability, moving camera	136	whether the event is physically possible

Appendix

Table 119: Prospective test: Set B.

Source	Subset	Questions	What is asked
MMSI-Bench	Motion (Cam.)	74	how the camera moved or rotated across consecutive images
VLM4D	camera motion	6	which way the camera turned or moved in the video
SPAR-Bench	view change, ScanNet / ScanNet++	125 / 136	which motion the camera made between two images
CameraBench	pan	1,200	whether the camera pans left or right
CameraBench	tilt	703	whether the camera tilts up or down
CameraBench	roll	524	whether the camera rolls clockwise or counterclockwise

Appendix

Table 120: Prospective test: Set C.

Questions	Base	Recipe	Uniform	Temporal-heavy	Camera-heavy	Exogenous-heavy
Set D, PhysBench static (327)	55.7	56.1	55.4	56.5	56.5	56.5
Set D, MMSI-Bench static (559)	28.4	29.0	28.6	29.0	29.0	29.6
Set A (1,673)	31.1	37.9	36.4	34.4	36.3	33.2

Appendix

Table 121: Prospective test: static questions versus agent-action questions (accuracy %, mean over

8

models).

Questions	Mixture	s1	s2	s3	s4	s5	s6	s7	s8	Mean
Set A	Recipe	38.7	36.8	38.3	37.0	38.4	38.3	36.3	39.6	37.9
Set A	Uniform	38.4	36.9	35.2	34.5	38.0	36.1	36.2	36.1	36.4
Set A	Temporal-heavy	35.8	35.1	33.0	35.8	35.6	35.3	32.5	32.4	34.4
Set A	Exogenous-heavy	30.6	33.5	34.5	32.2	35.2	33.9	34.4	31.6	33.2
Set B	Recipe	70.3	71.1	71.5	70.7	70.6	71.0	71.3	71.1	71.0
Set B	Exogenous-heavy	71.4	71.2	71.2	70.7	71.8	72.6	71.5	71.6	71.5

Appendix

Table 122: Prospective test: per-seed scores (accuracy %). Base model: Set A

31.1

, Set B

70.3

, Set C

54.1

Explore similar work

May 25, 2026cs.CV

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.

Longteng Guo, Yifan Wang, Pengkang Huo +4

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Visual Reasoning VLM Reasoning Visually Grounded Reasoning Multimodal Model Evaluation Multimodal Understanding

Ask PDF

Jun 4, 2026cs.CV

WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark

In real-world applications, models are expected to perform reliably across diverse settings. Yet, many existing multimodal benchmarks expand task types without capturing the visual diversity needed to handle open-ended visual inputs. We present WorldBench, a challenging and visually diverse reasoning benchmark to evaluate Multimodal Large Language Models (MLLMs). We build a taxonomy of thousands of visual concepts across multiple domains (e.g., living things). Guided by this taxonomy, we curate a broad collection of images from search engines and existing datasets to comprehensively represent the visual world. Through structured trial-and-error, we manually design challenging questions that frontier MLLMs fail to answer. On quantitative and human evaluations, WorldBench achieves higher visual diversity than any existing diverse benchmark. Evaluating 15 MLLMs on WorldBench reveals weaknesses in visual understanding: even the strongest model reaches only 64.0% accuracy, while some models perform marginally above chance-level. We hope our work highlights the importance of visual diversity in building multimodal benchmarks.

Yida Yin, Harish Krishnakumar, Chung Peng Lee +9

Princeton University · NYU · University of Waterloo +1

Multimodal Large Language Models Multimodal Reasoning Visually Grounded Reasoning Multimodal Model Evaluation Multimodal Understanding

Ask PDF

Jul 28, 2026cs.CV

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.

Jiaang Li, Chengzu Li, Zhaochong An +4

University of Copenhagen · University of Cambridge · 3ETH Zürich +1

VLM Evaluation Multimodal Large Language Models Multimodal Model Interpretability Multimodal Understanding

Ask PDF