STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Models

Thong Nguyen; Tri Cao; Khoi Le; Cong-Duy Nguyen; Quynh Vo; See-Kiong Ng; Bryan Hooi Kuen-Yew

cs.CVAug 28, 2026

STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Models

Authors: Thong Nguyen, Tri Cao, Khoi Le, Cong-Duy Nguyen, Quynh Vo, See-Kiong Ng, Bryan Hooi Kuen-Yew

Organizations: National University of Singapore (NUS), Singapore · Centre for AI Research, VinUniversity

Abstract

While multimodal large language models (MLLMs) have advanced video understanding, they remain highly prone to hallucinations in dynamic scenes. We argue this stems from a failure in spatio-temporal monitoring, the ability to persistently track object identities, states, and relations over time. Existing benchmarks obscure this deficit by relying on single final-answer evaluations for queries that can often be resolved via local visual cues or statistical priors. To rigorously diagnose this, we introduce STRAND, a benchmark of human-verified object-centric facts that evaluates intermediate reasoning by decomposing queries into sub-questions, distinguishing genuine temporal understanding from coincidental correctness. Crucially, we score models with Faithful Accuracy, an unconditional joint metric that credits a prediction only when the target answer and every prerequisite sub-question are correct, so that a model cannot inflate its score by being selectively consistent on the small subset of targets it happens to answer correctly. To address failure modes exposed by STRAND, we further propose an object-centric framework that explicitly constructs and reasons over structured object trajectories via chunk-wise state extraction and temporal aggregation. Extensive experiments, including backbone-, frame-, call-, and token-matched comparisons against both end-to-end MLLMs and modular video harnesses, demonstrate that our object-centric framework significantly reduces hallucinated answers and improves spatio-temporal reasoning consistency over state-of-the-art MLLMs. The code, model, and data have been made available at nguyentthong.github.io/strand.

Figures & tables

Figure 1: Existing benchmarks and MLLM architectures versus ours. Top : a single answer records only that a model is wrong, whereas sub-questions name the facts it rests on. Bottom : we answer from object trajectories rather than raw frames alone.

Figure 2: Overview of the STRAND construction pipeline. Annotators anchor timestamps, register entities, extract typed facts, and link identities across time. Target questions and sub-questions are composed from that record and verified.

Figure 3: Overview of our framework. Stages 1 to 5 run once per video. Stages 6 and 7 run once per question.

Method	STRAND
Method	$A_{\text{faith}}$	$A_{\text{target}}$	$A_{\text{sub}}$	$A_{\text{cons}}$
Ours
Ours (Gemini-3-Flash)	59.4	77.1	76.3	81.3
Ours (Qwen3-VL-235B)	54.2	74.0	72.6	78.1
Proprietary end-to-end MLLMs
Gemini-3.1-Pro	38.9	61.4	64.9	69.2

Table 1: STRAND results (%). Best and second-best per column. Four further open-weight models are in Appendix H .

A_{\text{faith}}

is the primary metric and

A_{\text{cons}}

is reported for reference only (Section 4.3 ).

Method	VideoHallucer	Video-MME	EgoSchema
Ours (Gemini-3-Flash)	76.8	76.8	78.4
Ours (Qwen3-VL-235B)	71.4	73.6	76.4
Gemini-3.1-Pro	63.1	77.3	76.8
GPT-5	67.5	76.2	75.4
Qwen3-VL-235B	59.8	72.6	72.1
LLaVA-OV	48.5	66.3	60.1

Table 2: Condensed external-benchmark results (%). Full sub-scores in Appendix D.1 .

Ablation Focus	Pipeline	Chunk (s)	Frames/chunk	Frames@ $G_{\eta}$	$A_{\text{target}}$ (%)
Reference Base Config
Base Configuration	Ours (Gemini-3-Flash)	15	60	64	77.1
Base Configuration	Ours (Qwen3-VL-235B)	15	60	64	74.0
Query-based retrieval
No filter (full timeline)	Gemini / Qwen	15	60	64	77.0 / 64.6
Random filter (top- $k$ )	Gemini / Qwen	15	60	64	76.5 / 68.0

Table 3: Ablation studies on STRAND Target Accuracy (%).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	State Change	Identity Tracking	Action Sequ.	Multi-Obj Interact.
Target count ( $n_{k}$ )	200	277	300	200
Ours (Gemini-3-Flash)	70.00	49.82	52.00	73.00
Gemini-3.1-Pro	50.00	31.05	32.67	48.00
Qwen3.5-27B	58.00	37.91	38.00	55.00

Appendix

Table 4: Faithful Accuracy (

A_{\text{faith}}

) breakdown across STRAND question categories (%). The four categories partition the 977 target questions.

Model	Obj. Hallucination	State Misattribution	Temporal Order Error
Ours (Gemini-3-Flash)	12.4	34.1	53.5
Gemini-3.1-Pro	38.2	36.5	25.3
Qwen3.5-27B	33.7	38.4	27.9

Appendix

Table 5: Sub-question error types (%), as a share of each model’s incorrect sub-questions. Rows sum to 100.

Method	VideoHallucer			Video-MME (w/o subs)				EgoSchema
Method	Basic	Halluc.	Overall	Short	Med.	Long	Over.	Acc.
Ours
Ours (Gemini-3-Flash)	83.5	79.9	76.8	85.8	78.3	66.2	76.8	78.4
Ours (Qwen3-VL-235B)	80.6	75.1	71.4	82.2	76.0	62.7	73.6	76.4
Frontier end-to-end MLLMs
Gemini-3.1-Pro †	85.2	68.4	63.1	84.1	76.5	71.3	77.3	76.8

Appendix

Table 6: Results on external benchmarks. Basic and Hallucinated sub-columns are unmarked because Video-LLaVA’s yes-bias inflates Basic. VideoHallucer Overall is the paired both-correct score, so

\text{Overall}\leq\min(\text{Basic},\text{Halluc.})

. Video-MME Overall is the mean of the three subset columns.

Ablation Focus	Pipeline	Chunk (s)	Frames/chunk	Frames@ $G_{\eta}$	$A_{\text{target}}$ (%)
Reference Base Config
Base Configuration	Ours (Gemini-3-Flash)	15	60	64	77.1
Base Configuration	Ours (Qwen3-VL-235B)	15	60	64	74.0
Cluster A: Visual Budget at the Answerer
Frames@ $G_{\eta}$ = 32	Gemini / Qwen	15	60	32	76.3 / 66.4
Frames@ $G_{\eta}$ = 16	Gemini / Qwen	15	60	16	76.5 / 65.5

Appendix

Table 7: Full ablation studies on STRAND Target Accuracy (%).

Constraint	End-to-end baseline	Ours	$\Delta$
Equal total frame budget (64 frames)	41.5	54.2	+12.7
Equal number of model calls (single pass)	39.8	49.3	+9.5
Equal token budget	45.1	52.8	+7.7

Appendix

Table 8: Budget-matched comparisons on Faithful Accuracy

A_{\text{faith}}

(%). All configurations use the Qwen3-VL backbone, so only the representation changes within each control.

Temporal constraint $\Delta t_{max}$
Configuration	$A_{\text{target}}$ (%)	$A_{\text{faith}}$ (%)
Base: $\Delta t_{max}=15$ s, $\tau_{conf}=0.75$ , bipartite	74.0	54.2
$5$ s (strict)	71.2	50.4
$30$ s (relaxed)	72.8	51.1
Confidence threshold $\tau_{conf}$
$0.90$ (high precision)	70.5	48.9

Appendix

Table 9: Ablation of identity-linking thresholds and conflict resolution on STRAND. The base row reproduces Ours (Qwen3-VL-235B) in Table 1 .

Single centre frame
Model	Full video		Restricted input
Model	$A_{\text{target}}$	$A_{\text{faith}}$	$A_{\text{target}}$	$A_{\text{faith}}$
Gemini-3.1-Pro	61.4	38.9	31.4	14.1
Gemini-3-Flash	48.7	28.7	29.8	12.5
GPT-5	32.5	20.0	28.2	10.2
Qwen3.5-27B	68.4	45.5	32.7	13.8

Appendix

Table 10: Single-frame solvability and shortcut audit. Full-video columns reproduce Table 1 . The last two rows receive no visual input at all rather than a single frame.

System	Latency (s)		Total cost per video ( $Q=39.7$ )
System	Pre-proc.	Per-query	Calls	Input tokens	Output tokens
Ours (Gemini-3-Flash) ∗	14.2	3.2	91.6	962,400	18,100
Ours (Qwen3-VL-235B)	18.4	2.9	91.6	958,157	17,450
Gemini-3.1-Pro ∗	0.0	3.5	39.7	654,415	1,985
Qwen3.5-27B	0.0	2.8	39.7	654,415	1,985
Qwen3-VL-32B Instruct	0.0	2.9	39.7	654,415	1,985

Appendix

Table 11: Efficiency and cost per video (

Q=39.7

queries). Measurement protocol in Appendix D.7 .

Figure 4: Qualitative analysis of target and sub-question behavior.

Figure 5: Example of the STRAND dataset. To rigorously evaluate model faithfulness, STRAND requires models to correctly answer all underlying supporting sub-questions in addition to the complex target question. This ensures the model accurately tracks temporal action sequences, from passing ( $t_{0}$ ) to scoring ( $t_{2}$ ), and grounds specific visual attributes (e.g., “yellow shirt”, “number 10”) rather than merely guessing the final answer.

Figure 6: STRAND evaluation on multi-step object interactions. Building on our faithfulness criteria, models must resolve all supporting sub-questions to demonstrate true comprehension. In this scenario, the model must accurately follow a complex sequence, from the initial display of all artifacts ( $t_{0}$ ) to cutting machine 1 ( $t_{3}$ ), while distinguishing specific visual attributes (“number 1”, “number 2”, “cake”) to prevent shortcut learning.

Figure 7: STRAND evaluation of temporal ordering. This example further illustrates how mandatory sub-questions enforce reasoning faithfulness. To succeed, the model must track the precise sequence of events, from placing ingredients ( $t_{0}$ ) to smashing both items ( $t_{3}$ ), and ground specific visual objects (“strawberries”, “cookies”). This verifies that the model understands the correct temporal ordering of interactions rather than relying on educated guesses.

Figure 8: STRAND evaluation of actor disambiguation over time. This scenario highlights how STRAND tests complex temporal tracking across multiple subjects. To accurately answer the target question, the model must resolve the chronological sequence of events, from the first student raising his hand ( $t_{0}$ ) and the second student raising his hand ( $t_{1}$ ) to the eventual celebration ( $t_{3}$ ). Mandatory supporting sub-questions force the model to explicitly differentiate between the two actors (e.g., verifying that the second student celebrated, while the first did not) to confirm precise spatiotemporal grounding rather than relying on spurious correlations.

Temporal Video Understanding

Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.

See all 371 papers

May 9, 2026

Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models

While multimodal large language models (MLLMs) have advanced video understanding, they remain highly prone to hallucinations in dynamic scenes. We argue this stems from a failure in spatio-temporal monitoring, the ability to persistently track object identities, states, and relations over time. Existing benchmarks obscure this deficit by relying on single final-answer evaluations for queries that can often be resolved via local visual cues or statistical priors. To rigorously diagnose this, we introduce STEMO-Bench (Spatio-TEmporal MOnitoring), a benchmark of human-verified object-centric facts that evaluates intermediate reasoning by decomposing queries into sub-questions, distinguishing genuine temporal understanding from coincidental correctness. To address failure modes exposed by STEMO, we propose STEMO-Track, a novel object-centric framework that explicitly constructs and reasons over structured object trajectories via chunk-wise state extraction and temporal aggregation. Extensive experiments demonstrate that our object-centric framework significantly reduces hallucinated answers and improves spatio-temporal reasoning consistency over state-of-the-art MLLMs.

Tri Cao, Khoi Le, Thong Nguyen +7

National University of Singapore · VinUniversity · Nanyang Technological University

VLM Evaluation Object Hallucination in VLMs Multimodal Large Language Models Temporal Video Understanding

Ask PDF

May 11, 2026

TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models

Video large language models (Video-LLMs) have made strong progress in general video understanding, but their ability to maintain temporal object consistency remains underexplored. Existing benchmarks often emphasize event recognition, action understanding, or coarse temporal reasoning, while rarely testing whether models can preserve the identity, state, and continuity of the same object across occlusion, disappearance, reappearance, state transitions, and cross-object interactions. We introduce TOC-Bench, a diagnostic benchmark for evaluating temporal object consistency in Video-LLMs. TOC-Bench is object-track grounded: each queried subject is linked to a per-frame trajectory and a structured temporal event timeline. To ensure that questions require temporally ordered visual evidence rather than language priors, single-frame shortcuts, or unordered frame cues, we design a three-layer temporal-necessity filtering protocol, which removes 60.7% of candidate QA pairs and retains 17,900 temporally dependent items across 10 diagnostic dimensions. From this pool, we construct a human-verified benchmark with 2,323 high-quality QA pairs over 1,951 videos. Experiments on representative Video-LLMs show that temporal object consistency remains a major unsolved challenge, with notable weaknesses in event counting, event ordering, identity-sensitive reasoning, and hallucination-aware verification, even when models perform well on general video understanding benchmarks. These results suggest that object-centric temporal coherence is a key bottleneck for current Video-LLMs, and that TOC-Bench provides a focused platform for diagnosing and improving object-aware temporal reasoning. The resource is available at https://github.com/cjzcjz666/toc_bench.git.

Junzhe Chen, Siyuan Meng, Yuxi Chen +3

Tianjin University

Video-Language Model Evaluation Video-Language Models Multimodal Model Evaluation Temporal Video Understanding

Ask PDF

Jun 26, 2026

Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning

Recent interest in multimodal large language models (MLLMs) raises a central question: can they reason over dynamic visual evidence rather than merely recognize objects or events in individual frames? This ability, which we refer to as video temporal-logical reasoning, requires models to maintain, update, and compose evidence as visual states evolve across frames. Existing video benchmarks often conflate this capability with scene complexity, static recognition, or uncontrolled temporal variation. To isolate this capability, we introduce Video-MME-Logical, a controlled benchmark organized around five temporal-logical operations: state tracking, sequential counting, temporal ordering, dynamic spatiality, and structural composition. The benchmark contains 25 fine-grained task categories generated with controlled object states, transitions, temporal dependencies, and logical compositions. It enables difficulty-controlled final-answer evaluation by varying temporal horizon and reasoning complexity, and supports intermediate-state diagnostics by verifying whether models recover the required logical reasoning trace before producing the final answer. Experiments with state-of-the-art MLLMs reveal a substantial human-model gap, especially as temporal-logical complexity increases. Supervised fine-tuning on up to 500K generated samples improves performance but remains insufficient to close the reasoning gap, positioning Video-MME-Logical as a scalable testbed for analyzing and improving temporal-logical reasoning in MLLMs.

Hohin Kwan, Hongyu Li, Ray Zhang +5

1HKUST · 2Colab, Beihang University · 3CUHK

VLM Evaluation Multimodal Large Language Models Video Reasoning Temporal Video Understanding

Ask PDF

Abstract

Temporal Video Understanding

Explore similar work

Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models

TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models

Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning