Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence

Seunghwan Bang; Hwanjun Song

cs.CVMar 13, 2026

Reasoning over Video: Evaluating How MLLMs Extract, Integrate, and Reconstruct Spatiotemporal Evidence

Authors: Seunghwan Bang, Hwanjun Song

Organizations: UNIST · KAIST

Abstract

The growing interest in embodied agents increases the demand for spatiotemporal video understanding, yet existing benchmarks largely emphasize extractive reasoning, where answers can be explicitly presented within spatiotemporal events. It remains unclear whether multimodal large language models can instead perform abstractive spatiotemporal reasoning, which requires integrating observations over time, combining dispersed cues, and inferring implicit spatial and contextual structure. To address this gap, we formalize abstractive spatiotemporal reasoning from videos by introducing a structured evaluation taxonomy that systematically targets its core dimensions and constructs a controllable, scenario-driven synthetic egocentric video dataset tailored to evaluate abstractive spatiotemporal reasoning capabilities, spanning object-, room-, and floor-plan-level scenarios. Based on this framework, we present VAEX-BENCH, a benchmark comprising five abstractive reasoning tasks together with their extractive counterparts. Our extensive experiments compare the performance of state-of-the-art MLLMs under extractive and abstractive settings, exposing their limitations on abstractive tasks and providing a fine-grained analysis of the underlying bottlenecks. The dataset will be released soon.

Figures & tables

Figure 1: Two examples of abstractive spatiotemporal queries from VAEX-Bench : (Left) Global Counting, requiring aggregation of objects observed across the traversal; (Right) Map Direction, requiring inference of relative spatial directions between rooms.

Figure 2: Query-conditioned video construction pipeline: (Step 1) Scenario-based Query Construction, followed by (Step 2) Environment and Trajectory Design and (Step 3) Rendering and Video Recording.

Category	Extractive	Abstractive	Expanded Capability
Chronology	Appearance Order: Order of first appearance: {sofa, TV, table, cabinet}	Memory–Action: What activity is possible in the {third} room?	Frame-local perception $\rightarrow$ Memory-grounded perception
Direction	Relative Direction: Facing {fridge} from {stove} , where is {stool} ?	Map Direction: [room1 is east of room2] What is the direction of {room3} from {room4} ?	Frame-local direction $\rightarrow$ Allocentric orientation
Distance	Relative Distance: Which object is the closest to the {door} ?	Map Scale: [ $D^{2}(room1,room2)=1$ ] What is the $D^{2}$ between {room3} and {room4} ?	Frame-local proximity $\rightarrow$ Global metric estimation
Planning	Route Plan: Facing {TV} from {sofa} , what sequence of turns navigate you to {table} ?	Simulation: Which room is located directly opposite to {kitchen} ?	Local route planning $\rightarrow$ Allocentric simulation
Counting	Object Counting: How many {chairs} are in the {dining room} ?	Global Counting: How many {chairs} are observed across all rooms?	Single-room counting $\rightarrow$ Multi-room aggregation

Table 1: Expansion from extractive to abstractive tasks. Question examples illustrate the shift in cognitive requirements, where items blue denote variable placeholders that can be adapted to different environments.

Resolution	#Query	#Rooms	#Objects	Video Duration(s)
$640\times 480$	30	6.7 $\pm$ 1.25	73.5 $\pm$ 14.27	36.81 $\pm$ 6.05

Table 2: Dataset statistics of VAEX-Bench averaged over 10 scenarios.

	Abstractive						Extractive
Models	Mem.	MDir.	MSca.	Sim.	Glob.	Avg.	Order	Dir.	Dist.	Route	Obj.	Avg.
Reference Baselines
Random	30.7	22.0	18.0	35.3	N/A	26.5	15.3	22.0	42.0	20.0	N/A	24.8
Human-Level	89.3	83.3	60.0	93.3	82.7	81.7	93.3	88.0	77.3	88.7	92.7	88.0
Proprietary Models
GPT-5.2 ( Singh et al., 2025 )	38.0	26.0	34.0	29.3	23.3	30.1	73.3	26.0	56.0	18.7	48.7	44.5

Table 3: Performance on VAEX-Bench over 14 SOTA MLLMs compared with Random and Human baselines.

\dagger

denotes a MoE model with 22B activated parameters.

Panel of Figure 4 — Figure 4: Temporal bottleneck in multi-hop sub-tasks. The outer circle represents the pass/fail ratio for Hop 1 (Blue: Pass, Pink: Fail). The inner ring shows the conditional pass rate for Hop 2 among the cases that passed Hop 1 (Green: Pass, Yellow: Fail).

Figure 5: Visualization of floor-plan prediction. The upper and lower halves, separated by a bold dark line, represent the Ground Truth and Prediction, respectively. Colored icons denote functional areas, while grey regions indicate “None”.

Model Family	Model Name
GPT	gpt-5.2
Gemini	gemini-3-flash-preview
	gemini-3-pro-preview
Claude	claude-haiku-4-5-20251001-v1:0
	claude-sonnet-4-5-20250929-v1:0
Qwen	qwen3-vl-2b-instruct

Table 5: Set of proprietary and open-source multimodal large language models (MLLMs) evaluated on our benchmark.

Panel of Table 6 — Table 6: Question Templates for Abstractive Spatiotemporal Reasoning. Note that the five answer choices shown here are illustrative examples. Blue text indicates placeholders for rooms, objects, and counts/order, which may vary depending on the scenario.

Panel of Table 8 — Table 8: Prompt design for LLM-Judge section.

Model	Stat	Mem.	MDir.	MSca.	Sim.	Glob.	Avg.
GPT-5.2	mean $\pm$ std	38.0 $\pm$ 3.0	26.0 $\pm$ 6.0	34.0 $\pm$ 7.2	29.3 $\pm$ 6.4	23.3 $\pm$ 6.7	30.1 $\pm$ 2.5
	95% CI	[34.3,41.7]	[18.6,33.4]	[25.0,43.0]	[21.4,37.3]	[15.1,31.6]	[27.0,33.3]
	p-value	-	-	-	-	-	0.0085
Gemini-3 Flash	mean $\pm$ std	60.7 $\pm$ 2.8	34.0 $\pm$ 4.9	24.0 $\pm$ 3.7	51.3 $\pm$ 6.9	31.3 $\pm$ 7.3	40.3 $\pm$ 2.8
	95% CI	[57.2,64.1]	[27.9,40.1]	[19.5,28.5]	[42.8,59.9]	[22.3,40.4]	[36.8,43.7]
	p-value	-	-	-	-	-	-

Table 10: Abstractive evaluation statistics. Mean performance and standard deviation (mean

\pm

std), 95% confidence intervals (CI), and p-values across five runs for the abstractive evaluation tasks. Statistical significance is evaluated using paired two-sided t-tests.

Model	Stat	Appr.	Dir.	Dist.	Plan.	Obj.	Avg.
GPT-5.2	mean $\pm$ std	73.3 $\pm$ 6.7	26.0 $\pm$ 4.3	56.0 $\pm$ 4.9	18.7 $\pm$ 7.3	48.7 $\pm$ 6.9	44.5 $\pm$ 3.1
	95% CI	[65.1,81.6]	[20.6,31.4]	[49.9,62.1]	[9.6,27.7]	[40.1,57.2]	[40.6,48.4]
	p-value	-	-	-	-	-	0.0176
Gemini-3 Flash	mean $\pm$ std	93.3 $\pm$ 0.0	40.0 $\pm$ 7.5	62.7 $\pm$ 1.5	16.0 $\pm$ 2.8	38.0 $\pm$ 3.8	50.0 $\pm$ 2.4
	95% CI	[93.3,93.3]	[30.7,49.3]	[60.8,64.5]	[12.5,19.5]	[33.3,42.7]	[47.1,52.9]
	p-value	-	-	-	-	-	0.0341

Table 11: Extractive evaluation statistics. Mean performance and standard deviation (mean

\pm

std), 95% confidence intervals (CI), and p-values across five runs for the extractive evaluation tasks. Statistical significance is evaluated using paired two-sided t-tests.

Category	Contents
Environment typology	Single-story residence
# of rooms	6
Types of rooms	bathroom, bedroom, kitchen, living room, piano room, study
Objects	bathroom : washstand, toilet
	bedroom : wardrobe, lamp, cabinet, bed, books, pillows, windows
	kitchen : fridge, cupboard, stove, sink, window, cups

Table 12: Scenario 1

Category	Contents
Environment typology	Single-story residence
# of rooms	6
Types of rooms	bathroom, bedroom, dining room, kitchen, living room, study
Objects	bathroom : washstand, bath, toilet, window
	bedroom : bed, pillow, wardrobe, window
	dining room : table, chairs, book, fireplace, window, door

Table 14: Scenario 3

Category	Contents
Environment typology	Office
# of rooms	5
Types of rooms	bathroom, file room, locker room, lounge, main office
Objects	bathroom : washstand, toilet
	file room : desk, chair, carpet, bookshelf, window, printer, prints, safe
	locker room : lockers, carpet

Table 16: Scenario 5

Category	Contents
Environment typology	Single-story residence
# of rooms	6
Types of rooms	bathroom, bedroom, dining room, kitchen, living room, study
Objects	bathroom : window, towers, curtain, toilet
	bedroom : bed, pillow, air conditioner, wardrobe, window
	dining room : table, chairs, window

Table 18: Scenario 7

Category	Contents
Environment typology	Multi-story residence
# of rooms	8
Types of rooms	bathroom 1, bathroom 2, bedroom 1, bedroom 2, dining room, dressroom, kitchen, living room
Objects	bathroom 1 : washstand, cup, toilet, mirror, plant
	bathroom 2 : washstand, toilet, mirror
	bedroom 1 : bed, pillows, lamps, picture, vase, chair, window, curtain

Table 20: Scenario 9

Spatiotemporal Reasoning

Spatiotemporal reasoning studies how systems connect objects, events, locations, and changes across space and time.

See all 70 papers

TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs
Jan 30, 2026Baiqi Li, Kangyi Zhao, Ce Zhang +3VLM Evaluation Video Understanding
University of North Carolina at Chapel Hill · University of Pittsburgh · Carnegie Mellon University
VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis
May 21, 2026Jinho Park, Youbin Kim, Hogun Park +1VLM Evaluation Spatiotemporal Reasoning
Department of Artificial Intelligence, Sungkyunkwan University · Department of Artificial Intelligence, Yonsei University
Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning
Jun 26, 2026Hohin Kwan, Hongyu Li, Ray Zhang +5VLM Evaluation Multimodal Large Language Models
1HKUST · 2Colab, Beihang University · 3CUHK

Jan 30, 2026J/K move · Enter open · S save

TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs

Baiqi Li, Kangyi Zhao, Ce Zhang, Chancharik Mitra+2

University of North Carolina at Chapel Hill · University of Pittsburgh · Carnegie Mellon University

VLM Evaluation Video Understanding Multimodal Large Language Models Video Reasoning Temporal Video Understanding Multimodal Understanding

Open paper AskPDF arXiv

Fine-grained spatio-temporal understanding is essential for video reasoning and embodied AI. Yet, while Multimodal Large Language Models (MLLMs) master static semantics, their grasp of temporal dynamics remains brittle. We present TimeBlind, a diagnostic benchmark for compositional spatio-temporal understanding. Inspired by cognitive science, TimeBlind categorizes fine-grained temporal understanding into three levels: recognizing atomic events, characterizing event properties, and reasoning about event interdependencies. Unlike benchmarks that conflate recognition with temporal reasoning, TimeBlind leverages a minimal-pairs paradigm: video pairs share identical static visual content but differ solely in temporal structure, utilizing complementary questions to neutralize language priors. Evaluating over 20 state-of-the-art MLLMs (e.g., GPT-5, Gemini 3 Pro) on 600 curated instances (2400 video-question pairs), reveals that the Instance Accuracy (correctly distinguishing both videos in a pair) of the best performing MLLM is only 48.2%, far below the human performance (98.2%). These results demonstrate that even frontier models rely heavily on static visual shortcuts rather than genuine temporal logic, positioning TimeBlind as a vital diagnostic tool for next-generation video understanding. Dataset and code are available at https://baiqi-li.github.io/timeblind_project/ .

Abstract

Spatiotemporal Reasoning

Explore similar work

TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs

VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis

Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning