SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning

Chaoyue Li; Shoudong Han; Haoyang Wu; Boxue Yang; Sijia Chen; Yufan Chen; Ruping Zhang

cs.CVOct 10, 2026

SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoning

Authors: Chaoyue Li, Shoudong Han, Haoyang Wu, Boxue Yang, Sijia Chen, Yufan Chen, Ruping Zhang

Organizations: Huazhong University of Science and Technology · Shanghai Jiao Tong University

Abstract

Large multimodal models (LMMs) need to understand how entities move and interact as a scene develops. This requires following the same entities over time and relating their observed motion to subsequent changes. We formulate Scene Evolution Understanding (SEU), a task for predicting and interpreting this change. Given observed video, a model estimates entity histories and predicts one scene continuation through future trajectories and Events, accompanied by Visual Evidence from observed motion and a scene description. We construct SceneForesee, a benchmark for evaluating entity continuity and scene evolution reasoning through paired observed and future scenes with shared entity identities. It contains 144.57K temporal pairs, 1.05M entity correspondences, and 272.49K Events paired with observed evidence. We propose a spatiotemporal reasoning framework, EvoSee, that learns future geometry and Event semantics through shared candidate states and joint assignment. It selects one continuation per entity from observation and uses the selected changes for entity refinement, retention, and scene representation. On SceneForesee, EvoSee outperforms Qwen3.5-9B by 10.45 points in Combined Point-F1 and 17.24 points in Event Global F1. On the 4D reasoning benchmark Dyn-Bench, it improves question answering accuracy and grounding J&F by 4.8 and 3.58 points over their respective base systems. These results support scene evolution as both a prediction target and a dynamic representation that brings predicted entity changes into spatiotemporal reasoning.The project is available at https://github.com/mikubaka88/SceneForesee-Spatiotemporal-Reasoning.

Figures & tables

Figure 1: Scene Evolution Understanding. (a,b) Description and entity continuation from observation. (c) Eighteen continuity and reasoning metrics, plotted as percentage scores; labels give EvoSee values. (d) Observed movement, anticipated interaction, and evidence. Future frames show the evaluation reference. Tracking baselines denote task adaptations.

Figure 2: SceneForesee. (a) Corpus statistics. (b) Event and Visual Evidence categories. (c) Temporal pairing, annotation, and sample validation. (d) Entity Continuity and Scene Evolution Reasoning metrics. Ring labels give counts; sector angles are schematic.

Figure 3: EvoSee. Entity state and evidence condition shared geometric and Event continuations. Selected changes inform refinement, retention, and scene representation. Entity in directed attention denotes geometric state.

	Observed			Future			Combined
Method	P-F1	P-R	mIoU	P-F1	P-R	mIoU	P-F1	P-R	mIoU
TrackFormer-style	15.59	13.94	17.38	14.46	13.11	17.08	15.05	13.53	17.24
MeMOTR-style	15.62	13.71	16.55	15.00	13.44	16.43	15.35	13.60	16.50
LTrack-style	14.59	13.19	16.13	12.78	11.60	15.58	13.72	12.42	15.87
SambaMOTR	17.46	14.98	18.94	15.61	14.16	16.61	15.28	14.16	17.84
Qwen3.5-9B	20.55	19.72	21.58	15.49	14.78	17.55	18.16	17.35	19.65

Table 1: Entity Continuity. P-F1 and P-R denote [email protected] and [email protected]. Underlined best baselines define red gains; bold marks best results. Tracking adaptations are trained for SceneForesee.

	Entity			Event			Visual Evidence
Method	GF1	SF1	GPR	GF1	SF1	GPR	GF1	SF1	GPR
Tracking and entity continuity methods
TrackFormer-style	29.59	40.42	37.18	19.56	16.97	28.33	12.17	10.59	17.63
MeMOTR-style	32.45	42.63	40.31	21.03	18.13	28.72	14.18	13.29	19.37
LTrack-style	30.18	41.06	37.60	22.30	20.32	29.87	15.03	14.04	20.12
SambaMOTR	33.91	42.25	43.02	21.87	20.06	28.10	16.46	13.46	21.93

Table 2: Scene Evolution Reasoning. Event and Evidence match category occurrences. Underlined best baselines define red gains; bold marks best results. OV1.5 and OV2 are 8B LLaVA-OneVision models.

Method	BERTScore F1	Micro-F1
Qwen3.5-9B	95.98	19.94
EvoSee (Ours)	96.26	40.76

Table 3: Description similarity and complete Events.

		Trajectory			Reasoning
ID	Variant	Observed	Future	Combined	Entity	Event	Evidence
1	Geometry-only Assignment	27.82	21.89	24.03	44.83	41.27	43.15
2	w/o Evolution Refinement	29.22	18.27	22.11	48.59	42.71	50.12
3	Observation-only Refinement	29.10	22.60	25.93	49.36	44.52	52.41
4	w/o Continuation Selection	25.37	18.89	22.00	45.61	38.70	43.65
5	Single Hypothesis	27.86	21.88	24.16	46.98	40.67	46.47

Table 4: Ablations on SceneForesee. Columns report dimension means.

		Trajectory			Reasoning
Method	Count	Obs.	Fut.	All	Ent.	Evt.	Evd.
Qwen3.5-9B	1–3	28.35	21.90	25.26	58.06	28.30	50.72
	4–6	22.99	17.30	20.31	53.46	28.28	24.95
	$\geq 7$	9.36	7.49	8.47	19.68	31.39	14.81
	1–3	44.06	38.96	41.57	71.50	52.96	65.12
	4–6	29.08	26.90	28.03	69.59	48.30	54.00

Table 5: Results by observed entity count.

		Trajectory			Reasoning
Method	$H$ (s)	Obs.	Fut.	All	Ent.	Evt.	Evd.
Qwen3.5-9B	1–3	21.97	17.29	19.76	41.35	27.20	31.89
	4–6	15.78	11.75	13.82	32.50	37.21	29.41
	$\geq 7$	13.64	7.95	10.78	31.15	38.76	28.15
	1–3	30.46	27.47	29.04	53.45	49.02	57.26
	4–6	31.29	26.37	28.85	46.77	48.12	54.45

Table 6: Results across temporal extents

H

	Trajectory			Reasoning
Source	Obs.	Fut.	All	Ent.	Evt.	Evd.
APIDIS	24.40	5.23	14.53	50.51	65.63	81.25
OVT-B	32.27	29.45	30.88	50.41	50.26	57.18
TAO	23.37	16.25	20.05	59.92	41.48	54.36

Table 7: EvoSee dimension means by source.

Figure 4: Seal interaction at matched timestamps. Identity colors distinguish the seals; solid and dashed paths show observed and future centers. EvoSee retains both through changing overlap.

Table 12

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Count
Train / Val / Test	116,681 / 14,259 / 13,626
Total samples	144,566
Source videos	2,876
Entity correspondences	1,046,165
Observed trajectory records	1,098,141
Future trajectory records	1,046,165

Appendix

Table A1: Scale and Event distribution of SceneForesee. Entity correspondences count identities shared across paired windows within each sample. Trajectory records count observed and future windows separately. Each Event has one associated Visual Evidence annotation. Val denotes validation.

	Observed			Future			Combined
Variant	P-F1	P-R	mIoU	P-F1	P-R	mIoU	P-F1	P-R	mIoU
Qwen3.5-9B	20.55	19.72	21.58	15.49	14.78	17.55	18.16	17.35	19.65
Geometry-only Assignment	26.93	28.48	28.05	22.39	21.24	22.05	24.04	23.09	24.96
w/o Evolution Refinement	29.77	28.67	29.23	18.09	17.03	19.68	21.44	21.63	23.25
Observation-only Refinement	29.31	28.96	29.02	22.71	22.36	22.74	25.74	25.88	26.17
w/o Continuation Selection	25.49	25.46	25.17	18.74	17.24	20.69	22.63	21.22	22.16

Appendix

Table A2: Complete ablation results for Entity Continuity.

	Entity			Event			Visual Evidence
Variant	GF1	SF1	GPR	GF1	SF1	GPR	GF1	SF1	GPR
Qwen3.5-9B	37.45	40.73	40.14	29.85	28.20	29.87	29.82	34.23	29.84
Geometry-only Assignment	41.38	48.42	44.69	40.93	40.26	42.63	41.84	43.84	43.78
w/o Evolution Refinement	45.50	53.61	46.67	40.63	42.10	45.40	48.43	50.31	51.62
Observation-only Refinement	46.82	54.11	47.15	42.86	45.03	45.67	50.31	53.18	53.74
w/o Continuation Selection	43.46	49.00	44.36	38.54	36.96	40.59	41.31	45.43	44.20

Appendix

Table A3: Complete ablation results for Scene Evolution Reasoning.

			Observed			Future			Combined
Entities	Method	$N$	P-F1	P-R	mIoU	P-F1	P-R	mIoU	P-F1	P-R	mIoU
1–3	Qwen3.5-9B	6,400	26.41	27.65	31.00	19.81	20.58	25.31	23.28	24.25	28.26
1–3	EvoSee (Ours)	6,400	43.94	46.65	41.58	38.32	41.18	37.38	41.21	43.95	39.55
4–6	Qwen3.5-9B	2,335	25.08	21.56	22.32	18.35	15.75	17.81	21.92	18.81	20.19
4–6	EvoSee (Ours)	2,335	29.69	28.89	28.65	27.30	27.23	26.16	28.55	28.09	27.45
$\geq 7$	Qwen3.5-9B	4,891	10.71	8.47	8.91	8.48	6.73	7.27	9.65	7.64	8.12

Appendix

Table A4: Entity Continuity by observed entity count.

			Entity			Event			Visual Evidence
Entities	Method	$N$	GF1	SF1	GPR	GF1	SF1	GPR	GF1	SF1	GPR
1–3	Qwen3.5-9B	6,400	59.77	54.54	59.86	28.76	26.50	29.65	48.74	53.16	50.25
1–3	EvoSee (Ours)	6,400	70.82	72.68	71.01	52.44	53.72	52.73	64.28	66.44	64.64
4–6	Qwen3.5-9B	2,335	53.63	50.29	56.46	27.90	28.68	28.26	24.61	25.32	24.93
4–6	EvoSee (Ours)	2,335	69.33	69.68	69.75	46.66	47.57	50.66	51.81	53.93	56.26
$\geq 7$	Qwen3.5-9B	4,891	18.57	18.11	22.35	31.57	30.19	32.42	15.16	13.71	15.57

Appendix

Table A5: Scene Evolution Reasoning by observed entity count.

			Observed			Future			Combined
Entities	Method	$N$	P-F1	P-R	mIoU	P-F1	P-R	mIoU	P-F1	P-R	mIoU
1	Qwen3.5-9B	947	20.68	26.63	33.28	12.79	16.35	24.77	16.88	21.68	29.18
1	EvoSee (Ours)	947	64.73	70.92	56.06	53.57	57.74	47.54	59.29	64.47	51.89
2	Qwen3.5-9B	3,458	30.82	31.39	34.22	23.44	23.68	27.90	27.28	27.63	31.16
2	EvoSee (Ours)	3,458	46.30	46.96	39.88	39.41	40.37	35.48	42.93	43.69	37.74
3	Qwen3.5-9B	1,995	21.50	21.65	24.33	16.85	17.21	21.08	19.38	19.60	22.81

Appendix

Table A6: Entity Continuity at each observed entity count.

			Entity			Event			Visual Evidence
Entities	Method	$N$	GF1	SF1	GPR	GF1	SF1	GPR	GF1	SF1	GPR
1	Qwen3.5-9B	947	58.62	62.84	61.08	36.05	31.49	37.57	71.50	79.13	74.51
1	EvoSee (Ours)	947	68.62	79.86	71.05	81.73	81.73	81.73	84.69	84.69	84.69
2	Qwen3.5-9B	3,458	58.86	51.46	59.12	29.56	26.63	31.04	51.57	54.75	54.15
2	EvoSee (Ours)	3,458	69.55	70.71	69.61	49.83	50.38	49.87	64.44	65.28	64.49
3	Qwen3.5-9B	1,995	61.09	55.94	61.37	25.00	23.89	25.25	36.58	38.07	36.94

Appendix

Table A7: Scene Evolution Reasoning at each observed entity count.

			Observed			Future			Combined
Extent	Method	$N$	P-F1	P-R	mIoU	P-F1	P-R	mIoU	P-F1	P-R	mIoU
1–3 s	Qwen3.5-9B	11,001	22.28	21.30	22.32	17.11	16.31	18.45	19.86	18.93	20.48
1–3 s	EvoSee (Ours)	11,001	30.25	31.02	30.12	26.99	28.21	27.22	28.70	29.67	28.74
4–6 s	Qwen3.5-9B	1,616	14.08	13.95	19.32	10.21	9.90	15.15	12.21	11.96	17.29
4–6 s	EvoSee (Ours)	1,616	31.90	31.69	30.28	26.58	26.51	26.03	29.28	29.10	28.18
$\geq 7$ s	Qwen3.5-9B	1,009	11.97	11.80	17.15	6.34	5.92	11.59	9.14	8.81	14.40

Appendix

Table A8: Entity Continuity across temporal extents.

			Entity			Event			Visual Evidence
Extent	Method	$N$	GF1	SF1	GPR	GF1	SF1	GPR	GF1	SF1	GPR
1–3 s	Qwen3.5-9B	11,001	39.90	41.97	42.17	27.68	26.21	27.71	30.49	34.66	30.51
1–3 s	EvoSee (Ours)	11,001	51.00	57.98	51.38	47.38	49.27	50.40	54.93	58.41	58.44
4–6 s	Qwen3.5-9B	1,616	28.80	35.79	32.90	37.82	35.98	37.82	27.71	32.80	27.71
4–6 s	EvoSee (Ours)	1,616	42.40	54.44	43.46	46.58	48.26	49.51	52.46	55.13	55.77
$\geq 7$ s	Qwen3.5-9B	1,009	26.79	35.20	31.47	39.47	37.35	39.47	26.33	31.78	26.33

Appendix

Table A9: Scene Evolution Reasoning across temporal extents.

			Observed			Future			Combined
Extent	Method	$N$	P-F1	P-R	mIoU	P-F1	P-R	mIoU	P-F1	P-R	mIoU
1 s	Qwen3.5-9B	7,294	27.12	25.91	26.85	20.99	20.00	22.35	24.24	23.10	24.70
1 s	EvoSee (Ours)	7,294	30.59	31.50	30.49	26.83	28.48	27.60	28.79	30.04	29.11
2 s	Qwen3.5-9B	2,498	18.01	17.25	18.93	13.43	12.81	15.23	15.88	15.17	17.19
2 s	EvoSee (Ours)	2,498	29.53	30.16	29.43	27.41	27.79	26.60	28.56	29.03	28.12
3 s	Qwen3.5-9B	1,209	1.92	1.84	2.02	1.27	1.25	1.54	1.61	1.55	1.79

Appendix

Table A10: Entity Continuity at each temporal extent.

			Entity			Event			Visual Evidence
Extent	Method	$N$	GF1	SF1	GPR	GF1	SF1	GPR	GF1	SF1	GPR
1 s	Qwen3.5-9B	7,294	47.43	49.49	49.76	25.40	24.11	25.44	30.49	34.45	30.52
1 s	EvoSee (Ours)	7,294	52.32	58.62	52.62	47.56	49.52	50.64	54.70	58.15	58.25
2 s	Qwen3.5-9B	2,498	35.39	37.71	37.75	30.28	28.50	30.30	30.26	34.85	30.28
2 s	EvoSee (Ours)	2,498	49.53	57.10	49.99	47.25	49.10	50.22	55.62	59.26	59.11
3 s	Qwen3.5-9B	1,209	5.37	5.38	5.87	35.93	34.20	35.93	30.95	35.54	30.95

Appendix

Table A11: Scene Evolution Reasoning at each temporal extent.

		Observed			Future			Combined
Source	$N$	P-F1	P-R	mIoU	P-F1	P-R	mIoU	P-F1	P-R	mIoU
APIDIS	32	13.38	25.79	34.04	2.09	3.44	10.15	7.59	14.31	21.70
OVT-B	11,048	32.97	32.74	31.09	29.55	30.09	28.70	31.29	31.42	29.92
TAO	2,546	20.19	24.04	25.88	13.77	16.39	18.60	17.24	20.42	22.49

Appendix

Table A12: Entity Continuity across video sources for EvoSee.

		Entity			Event			Visual Evidence
Source	$N$	GF1	SF1	GPR	GF1	SF1	GPR	GF1	SF1	GPR
APIDIS	32	48.75	49.90	52.88	65.63	65.63	65.63	81.25	81.25	81.25
OVT-B	11,048	46.99	56.55	47.68	48.47	50.52	51.79	54.82	58.15	58.58
TAO	2,546	59.45	60.81	59.49	40.42	41.72	42.29	52.43	55.81	54.85

Appendix

Table A13: Scene Evolution Reasoning across video sources for EvoSee.

Method	Overall	Present	Future	Evidence
Qwen3.5-9B	95.98	95.59	96.65	96.31
EvoSee (Ours)	96.26	96.06	96.81	96.52

Appendix

Table A14: BERTScore F1 for the complete scene description and its present, future, and evidence parts.

Figure A1: Cube rotation. EvoSee follows the changing spatial extent of the handheld cube, improving Combined Point-F1 by 30.9 points and MeanIoU by 14.8 points.

Figure A2: Device manipulation. Observed and future tracks follow a person and handheld console during reorientation. Combined Point-F1 and MeanIoU gains are 55.0 and 26.2 points.

Figure A3: Hand and insect motion. The comparison shows entity localization at different spatial scales. Combined Point-F1 and MeanIoU improve by 26.4 and 27.3 points.

Figure A4: Underwater motion. EvoSee more closely localizes the animal through low visibility, with Combined Point-F1 and MeanIoU gains of 35.5 and 3.7 points.

Figure A5: Extended body motion. Five seconds of observation precede a forecast of five seconds. Combined Point-F1 and MeanIoU gains are 36.7 and 8.9 points.

Figure A6: Motion near the water edge. Intervals of four seconds show localization around a stationary scene structure. Combined Point-F1 and MeanIoU improve by 19.5 and 5.0 points.

Spatiotemporal Reasoning

Spatiotemporal reasoning studies how systems connect objects, events, locations, and changes across space and time.

See all 70 papers

Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World
Mar 13, 2026Yuzhi Huang, Kairun Wen, Rongxin Gao +14Spatiotemporal Reasoning Multimodal Large Language Models
XMU · SUSTech · THU +4
LMM-Track4D: Eliciting 4D Dynamic Reasoning in LMMs via Trajectory-Grounded Dialogue
May 19, 2026Chaoyue Li, Yongxue Xu, Jie Feng +1Spatiotemporal Reasoning Multimodal Large Language Models
Huazhong University of Science and Technology · Sun Yat-sen University · Beihang University +1
ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?
Jul 23, 2026Han Li, Si Liu, Zehao Huang +6VLM Evaluation Spatial Reasoning Benchmarks
School of Artificial Intelligence, Beihang University, Beijing 100191, China · Zhongguancun Academy, Beijing 100094, China · School of Engineering, Westlake University, Hangzhou 310030, China

Mar 13, 2026J/K move · Enter open · S save

Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World

Yuzhi Huang, Kairun Wen, Rongxin Gao, Dongxuan Liu+13

XMU · SUSTech · THU+4

Spatiotemporal Reasoning Multimodal Large Language Models 4D Scene Understanding Temporal Video Understanding

Open paper AskPDF arXiv

Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time, constituting a dynamic 4D reality (spatial with temporal dimension). While current Multimodal Large Language Models (MLLMs) excel in static visual understanding, can they also be adept at "thinking in dynamics", i.e., perceive, track and reason about spatio-temporal dynamics in evolving scenes? To systematically assess their spatio-temporal reasoning and localized dynamics perception capabilities, we introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets, enabling robust and scalable evaluation of spatio-temporal understanding. Through multi-stage filtering from massive 2D and 4D data sources, Dyn-Bench provides a high-quality collection of dynamic scenes, comprising 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding pairs. We probe general, spatial and region-level MLLMs to express how they think in dynamics both linguistically and visually, and find that existing models cannot simultaneously maintain strong performance in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Notably, conventional prompting strategies (e.g., chain-of-thought or caption-based hints) provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and Spatio-Temporal Textual Cognitive Map (ST-TCM), significantly enhance MLLMs' dynamics perception and spatio-temporal reasoning in the physical 4D world. Code and benchmark are available at https://dyn-bench.github.io/.

Abstract

Spatiotemporal Reasoning

Explore similar work

Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World

LMM-Track4D: Eliciting 4D Dynamic Reasoning in LMMs via Trajectory-Grounded Dialogue

ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?