SCP: Spatial Causal Prediction in Video

Yanguang Zhao; Jie Yang; Shengqiong Wu; Shutong Hu; Hongbo Qiu; Yu Wang; Guijia Zhang; Tan Kai Ze

cs.CVMar 4, 2026

SCP: Spatial Causal Prediction in Video

Authors: Yanguang Zhao, Jie Yang, Shengqiong Wu, Shutong Hu, Hongbo Qiu, Yu Wang, Guijia Zhang, Tan Kai Ze,

Organizations: National University of Singapore · Shenzhen University · Sichuan University · National Tsing Hua University

Abstract

Spatial reasoning, the ability to understand spatial relations, causality, and dynamic evolution, is central to human intelligence and essential for real-world applications such as autonomous driving and robotics. Existing studies, however, primarily assess models on visible spatio-temporal understanding, overlooking their ability to infer unseen past or future spatial states. In this work, we introduce Spatial Causal Prediction (SCP), a new task paradigm that challenges models to reason beyond observation and predict spatial causal outcomes. We further construct SCP-Bench, a benchmark comprising 2,500 QA pairs across 1,181 videos spanning diverse viewpoints, scenes, and causal directions, to support systematic evaluation. Through comprehensive experiments on {23} state-of-the-art models, we reveal substantial gaps between human and model performance, limited temporal extrapolation, and weak causal grounding. We further analyze key factors influencing performance and propose perception-enhancement and reasoning-guided strategies toward advancing spatial causal intelligence. The project page is https://guangstrip.github.io/SCP-Bench.

Figures & tables

Figure 1: Existing benchmarks primarily assess known static or known dynamic reasoning based on fully observable scenes. A more challenging dynamic–unseen setting is to evaluate models’ ability to predict spatial outcomes from partial observations.

Figure 2: Overview of SCP-Bench. Left : Representative examples illustrating the eight task categories. Right : Data distribution across scene categories and task types. The benchmark comprises 2,500 QA pairs over 1,181 video clips.

Figure 3: Overview of the SCP-Bench construction pipeline. The process comprises five stages: (1) collection of diverse video sources, (2) clip selection with spatially dynamic segments, (3) generation of candidate QA pairs, (4) QA filtering and cutpoint identification, and (5) dataset validation and refinement.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
Human Performance	89.61	97.60	81.20	92.26	85.70	86.70	97.62	91.61	84.17
$\bullet$ Closed Models
GPT-5	66.24	79.04	58.12	59.06	64.07	70.48	95.24	77.42	65.11
Gemini-2.5-Pro	55.84	69.28	54.87	52.76	46.20	63.47	88.10	67.10	62.41
Gemini-2.5-Flash	52.10	59.28	52.14	51.74	43.14	57.75	88.10	66.45	55.60
Claude-Sonnet-4.5	56.14	68.86	52.14	57.43	45.65	60.90	80.95	68.39	63.90

Table 1: Evaluation on the SCP-Bench. “Avg.” indicates the overall average accuracy. For each category, the best-performing closed model and open-source model in average score are both indicated in deep blue , and best performance on each task is b o x e d .

Figure 4: Results across perspectives, view directions, and scenes.

Figure 5: Temporal extrapolation horizon analysis. Samples are grouped by the time gap between the cutpoint and future event: short (0–2s), mid (2–5s), and long (

>

5s).

Setting	Qwen3-VL	InternVL3.5	MiniCPM-V-4.5
Base	47.52	50.52	43.80
Gold Video	54.96 ↑7.44	58.64 ↑8.12	43.72 ↓0.08
Caption w/o Video	46.76 ↓0.76	41.24 ↓9.28	45.76 ↑1.96

Table 2: Comparison between perception and reasoning. “Base” is the standard evaluation performance; “Gold Video” evaluates perceptual understanding using unseen part; “Caption w/o Video” tests reasoning based on captions alone. Red and green indicate performance gains and drops, respectively.

Figure 6: Visible Range Comparison. Only cutpoint uses the cutpoint frame as input; Full video provides the entire clip; Ground truth includes only the unseen clips adjacent to the cutpoint.

Figure 7: Performance gap comparison. Accuracy improvements over the chance level for vision-enabled (w/ video) and vision-disabled (w/o video) settings across task categories.

Figure 8: Causal consistency evaluation. Each option shows the model’s reasoning rationality, confidence score, and explanation, assessing whether MLLMs can infer spatial order consistent with physical and causal constraints.

Setting	Qwen3-VL	InternVL3.5	MiniCPM-V-4.5	LLaVA-OV-1.5
Base	47.52	50.52	43.80	45.52
w/ Flip	46.76 ↓0.76	48.56 ↓1.96	42.24 ↓1.56	45.32 ↓0.2
w/ CoT	51.52 ↑4.0	50.36 ↓0.16	44.88 ↑1.08	42.88 ↓2.64

Table 3: Comparison with flipped video input and CoT reasoning. “w/ Flip” reverses the input video temporally; “w/ CoT” applies step-by-step reasoning.

Figure 9: Performance comparison on different model sizes.

Figure 10: Performance with vs. without self-think reasoning.

Figure 11: Comparison of perception enhancement strategies. PureText uses only the question; Caption w/ V combines dense video captions with the video and question; Caption w/o V uses only captions with the question; SpatialGraph introduces spatial-interaction graph; Original is the baseline.

Figure 12: Unseen spatial causal scaffolds for enhanced reasoning. Text provides textual descriptions of future spatial states; Image employs generated future spatial images; Video uses generated future causal videos.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Video Clip	QA Instance
$\bullet$ Scene Category
Sports Related	467	964
Life Records	289	691
Driving-view Related	122	308
Animal Related	85	143
Artistic Performances	64	95

Appendix

Table 4: Distribution of benchmarks over the scene category, perspective, Causal direction, and question type. These attributes are combined to form the final dataset, which consists of 1,181 video clips and 2,500 QA pairs.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Gold Video
Qwen3-VL-8B	54.96 ↑7.44	62.28	58.12	59.47	48.17	54.26	83.33	57.42	57.91
InternVL3.5-8B	58.64 ↑8.12	68.86	57.26	64.97	51.37	60.37	80.95	60.00	58.27
MiniCPM-V-4.5	43.72 ↓0.08	48.50	47.01	44.40	35.47	50.80	78.57	54.19	43.53
$\bullet$ Dense Caption Only
Qwen3-VL-8B	46.76 ↓0.76	51.50	37.61	53.97	40.16	48.94	80.95	60.00	40.29

Appendix

Table 5: Evaluation results for the “Perception vs. Reasoning Decomposition” experiment. “Avg.” indicates the overall average accuracy. Red and green indicate performance gains and drops relative to the Baseline setting, respectively.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Cutpoint Image
Qwen3-VL-8B	49.52 ↑2.00	49.70	52.14	50.92	44.62	50.27	83.33	55.48	51.80
InternVL3.5-8B	51.92 ↑1.40	67.07	52.99	52.14	44.16	57.18	76.19	61.29	50.36
MiniCPM-V-4.5	43.80 0.00	52.69	55.56	44.60	35.35	50.80	76.19	50.97	40.29
Spatial-MLLM	42.04 ↑2.28	53.46	32.73	38.90	37.30	53.55	71.43	55.69	33.33

Appendix

Table 6: Evaluation results for the “Single-Frame vs. Multi-Frame Reasoning” experiment.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Reverse Video
Qwen3-VL-8B	46.76 ↓0.76	52.69	52.14	46.84	43.94	46.54	90.48	48.39	42.45
InternVL3.5-8B	48.56 ↓1.96	56.89	56.41	49.49	42.33	53.19	64.29	59.35	43.53
MiniCPM-V-4.5	42.24 ↓1.56	49.10	45.30	41.96	35.13	48.67	73.81	50.97	41.37
LLaVA-OneVision-1.5-8B	45.32 ↓0.20	52.69	47.01	45.42	39.47	50.00	80.95	49.03	44.60

Appendix

Table 7: Evaluation results for the “Vision Causal Perception” experiment.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Baseline
Qwen3-VL-8B	47.52	54.49	51.28	49.29	42.33	49.47	90.48	46.45	46.40
InternVL3.5-8B	50.52	59.88	54.70	54.79	43.82	54.52	61.90	58.71	44.96
MiniCPM-V-4.5	43.80	53.29	49.57	43.99	36.04	49.20	76.19	52.26	42.81
Spatial-MLLM	39.76	45.51	28.21	33.81	38.33	49.73	66.67	50.97	32.37
$\bullet$ Gold Video

Appendix

Table 8: Evaluation results for the “Visible Range Comparison” experiment.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Pure Text
Qwen3-VL-8B	42.00 ↓5.52	49.10	35.04	47.25	34.67	46.28	69.05	40.65	45.32
InternVL3.5-8B	44.72 ↓5.80	50.90	42.74	48.88	36.61	52.66	73.81	51.61	41.01
MiniCPM-V-4.5	43.32 ↓0.48	53.29	44.44	46.03	35.01	51.60	69.05	52.90	37.77
LLaVA-OneVision-1.5-8B	40.20 ↓5.32	43.11	41.03	42.16	33.41	47.87	73.81	38.71	41.37
$\bullet$ Dense Caption Only

Appendix

Table 9: Evaluation results for the “Is Visual Information Necessary?” experiment.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Qwen3-VL Series
Qwen3-VL-2B	43.04	41.92	42.74	45.01	40.85	44.41	59.52	47.10	40.65
Qwen3-VL-4B	50.24	58.68	47.86	52.55	44.51	55.05	90.48	60.00	42.09
Qwen3-VL-8B	47.52	54.49	51.28	49.29	42.33	49.47	90.48	46.45	46.40
Qwen3-VL-30B-A3B	54.16	65.27	52.14	54.79	46.22	56.65	85.71	66.45	57.19
Qwen3-VL-32B	56.84	59.88	51.28	58.66	52.63	57.98	90.48	67.10	55.04

Appendix

Table 10: Evaluation results for the “Model Scale-up Effect Analysis” experiment.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Vanilla CoT
Qwen3-VL-8B	51.52 ↑4.00	63.47	50.43	51.53	44.39	54.52	90.48	61.29	51.80
InternVL3.5-8B	50.36 ↓0.16	59.28	51.28	49.29	43.02	56.65	61.90	65.16	51.08
MiniCPM-V-4.5	44.88 ↑1.08	55.09	40.17	48.68	35.47	53.19	73.81	59.35	39.93
LLaVA-OneVision-1.5-8B	42.88 ↓2.64	48.50	39.32	46.44	33.30	53.99	69.05	54.19	39.57

Appendix

Table 11: Evaluation results for the “CoT Reasoning” experiment.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Self-Thinking
Qwen3-VL-8B	47.44 ↓0.08	55.69	45.30	50.10	38.79	51.86	83.33	58.71	48.20
InternVL3.5-8B	47.84 ↓2.68	59.28	51.28	51.12	39.36	53.46	66.67	54.19	46.40
MiniCPM-V-4.5	42.72 ↓1.08	49.10	48.72	40.94	35.35	50.53	69.05	50.32	43.88

Appendix

Table 12: Evaluation results for the “Self-Think Reasoning” experiment.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Dense Caption Auxiliary
Qwen3-VL-8B	47.72 ↑0.20	54.49	48.72	52.75	41.76	46.01	88.10	53.55	46.04
InternVL3.5-8B	50.56 ↑0.04	60.48	53.85	54.18	43.02	56.12	64.29	60.65	45.32
MiniCPM-V-4.5	44.28 ↑0.48	56.29	45.30	51.73	33.64	46.28	85.71	54.84	42.09
LLaVA-OneVision-1.5-8B	45.80 ↑0.28	53.89	46.15	49.08	39.13	47.61	83.33	54.84	42.81
$\bullet$ Spatial Interaction Graph

Appendix

Table 13: Evaluation results for the “Perception Enhancement Strategy” experiment.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Text Scaffold
Qwen3-VL-8B	61.23 ↑13.71	64.29	56.67	67.44	54.48	64.08	91.89	64.06	56.25
InternVL3.5-8B	61.72 ↑11.20	64.29	57.78	67.44	53.25	65.49	91.89	67.97	59.13
MiniCPM-V-4.5	60.15 ↑16.35	58.16	52.22	66.28	51.32	64.08	97.30	67.19	59.62
LLaVA-OneVision-1.5-8B	59.77 ↑14.52	55.10	53.33	66.05	50.09	62.68	89.19	71.09	62.02
$\bullet$ Image Scaffold

Appendix

Table 14: Evaluation results for the “Causal Prediction Enhancement Strategy” experiment.

Benchmark	QA pairs	Modality	Dynamic/Static	View Type		Perspective		Causal Reasoning	Seen/Unseen
Benchmark	QA pairs	Modality	Dynamic/Static	Single	Multi	Ego	Exo	Causal Reasoning	Seen/Unseen
3DSRBench [ 33 ]	3,772	Image	Static	✓	✗	✗	✓	✗	Seen
InternSpatial-Bench [ 15 ]	6,008	Image	Static	✓	✗	✗	✓	✗	Seen
OmniSpatial [ 27 ]	$\sim$ 8,400	Image	Static	✓	✗	✓	✓	✗	Seen
Spatial457 [ 46 ]	23,752	Image	Static	✓	✗	Undeclared		✗	Seen
All-Angles Bench [ 57 ]	$\sim$ 2,100	Image	Static	✗	✓	Undeclared		✗	Seen

Appendix

Table 15: Comparison with existing benchmarks. Modality denotes the input type. Dynamic/Static indicates whether the scene content itself undergoes temporal changes, rather than mere camera movement. View Type specifies whether the questions involve single or multiple viewpoints. Perspective denotes whether the benchmark includes explicitly designed ego (first-person) and exo (third-person) perspective settings. Causal Reasoning indicates whether the benchmark requires inferring outcomes or states driven by causal dependencies. Seen/Unseen reflects whether the queried information is directly observable within the given visual content.

Model	Avg. $\downarrow$	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Misclassifying the Correct Answer as Least Plausible
Qwen3-VL-8B	29.76	34.73	4.27	26.27	27.35	42.29	21.43	29.68	35.61
InternVL3.5-8B	30.96	32.34	8.55	26.27	32.49	38.83	33.33	28.39	33.45
MiniCPM-V-4.5	34.36	37.72	15.38	28.11	32.84	45.48	35.71	39.35	38.13
LLaVA-OneVision-1.5-8B	31.64	36.53	2.56	32.38	26.89	46.28	33.33	31.61	34.53

Appendix

Table 16: Results of the “Extended Causal Consistency” experiment, where models are evaluated by how often they misclassify the correct answer as the least plausible option. Lower values indicate stronger causal consistency. “Avg.” denotes the overall average error rate.

Model	Avg.	Appearance Order	Counting	Planning	Relation	Relative Distance	Relative Size	Relative Speed	Spatial State
$\bullet$ Vanilla CoT
Qwen3-VL-8B	51.52 ↑4.00	63.47	50.43	51.53	44.39	54.52	90.48	61.29	51.80
InternVL3.5-8B	50.36 ↓0.16	59.28	51.28	49.29	43.02	56.65	61.90	65.16	51.08
MiniCPM-V-4.5	44.88 ↑1.08	55.09	40.17	48.68	35.47	53.19	73.81	59.35	39.93
LLaVA-OneVision-1.5-8B	42.88 ↓2.64	48.50	39.32	46.44	33.30	53.99	69.05	54.19	39.57
$\bullet$ Physics-aware CoT

Appendix

Table 17: Evaluation results for the “Physical Commonsense Probing” experiment. “Avg.” indicates the overall average accuracy.

Figure 13: Example of causal prediction enhancement. Here we present a comparative analysis of a case study, showing the actual subsequent video and the causal prediction results for text, image, and video modalities.

Figure 14: Analysis sample for extended causal consistency experiment. The models are required to identify the least plausible option, while InternVL3.5-8B does not independently assess the plausibility of each choice but instead derives the least plausible option by reasoning backward from its own incorrect answer, revealing a lack of internal causal consistency.

Figure 15: Visualization of the physical commonsense probing experiment. To answer each question, models must first infer the underlying physical principles depicted in the video. Although current models can often recognize the appropriate physical laws, they still struggle to apply them effectively when generating correct predictions.

Figure 16: Cases of dynamic integration failure. The models fail to perceive dynamic changes and remain confined to static observations.

Figure 17: Cases of dynamic integration failure and prior-driven hallucination. The models struggle to perceive and infer the correct global spatial dynamics and often overlook critical scene details during reasoning due to prior-driven biases.

Figure 18: Cases of causal reasoning failure and cross-modal attribution bias. The models struggle to perform causal reasoning and instead produce unsupported conclusions, leading to erroneous predictions. Moreover, they may overlook information from other modalities and fall into single-modality reasoning.

Figure 19: Manual filtering tool demonstration. Annotators use it to filter appropriate QA candidates and determine their cutpoints.

Figure 20: Validation tool demonstration. Annotators use it to validate each item in strict accordance with the SCP task specification.

Figure 21: Repairing tool demonstration. Annotators use it to repair and optimize any attribute of the item as needed.

Spatiotemporal Reasoning

Spatiotemporal reasoning studies how systems connect objects, events, locations, and changes across space and time.

See all 70 papers

From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models
Oct 2, 2026Feiran Wang, Xiaoqi Wang, Ziwei Li +3Visual Spatial Reasoning VLM Reasoning
University of Illinois at Chicago · Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI)
SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models
Oct 8, 2026Hongxing Li, Jinyue Su, Dingming Li +5Spatial Reasoning Spatial Reasoning Benchmarks
Zhejiang University
CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering
May 22, 2026Mingfang Zhang, Jingjing Pan, Ashutosh Kumar +7VLM Evaluation Temporal Video Grounding
Woven by Toyota · The University of Tokyo

Oct 2, 2026J/K move · Enter open · S save

From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models

Feiran Wang, Xiaoqi Wang, Ziwei Li, Wenbin He+2

University of Illinois at Chicago · Bosch Research North America · Bosch Center for Artificial Intelligence (BCAI)

Visual Spatial Reasoning VLM Reasoning Spatial Reasoning Benchmarks Video Reasoning 3D VLMs

Open paper AskPDF arXiv

Predicting future spatial states supports collision avoidance and timely decision-making in dynamic environments. However, existing vision-language models (VLMs) and benchmarks for spatial reasoning primarily focus on observed scenes, leaving predictive spatial reasoning beyond the observed interval underexplored. To this end, we introduce SpatialMind, a metric-scale VLM for spatial reasoning and future prediction. Its metric depth adapter anchors spatial reasoning to real-world scale, while its progressive state chain establishes current spatial states and observed dynamics as the foundation for future prediction. Given a video prefix, SpatialMind predicts distances, motion directions, and spatial relations in both observed and unseen future frames. For training and evaluation, we build a scalable data engine that grounds entity descriptions in metric geometry to generate question-answer pairs and state supervision. Using this engine, we construct the SpatialMind-30K dataset and the SpatialMind-2K benchmark, both covering driving and everyday egocentric scenes. The benchmark spans eight tasks across three levels: current-state understanding, observed-dynamics understanding, and future prediction. Experiments show that SpatialMind substantially outperforms both general and spatially specialized models on our benchmark while achieving competitive zero-shot performance on VSI-Bench, OSI-Bench, and VLM4D.

Abstract

Spatiotemporal Reasoning

Explore similar work

From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering