Existing VLMs have achieved strong performance in video understanding, yet they struggle with long-video spatiotemporal reasoning when target objects become invisible, often mistaking "invisible" for "unknown". We define this challenge as hidden-state spatiotemporal reasoning: inferring object states during prolonged invisible intervals from context interactions. To address this, we propose StateTrace, a novel object-centric framework that endows VideoLLMs with an explicit mechanism for hidden state reasoning in long videos. StateTrace builds a reusable spatiotemporal state memory that organizes object trajectories, inter-object relations, and state-transition events into a structured reasoning substrate. At inference time, it retrieves question-relevant state-evolution trajectories and converts them into compact reasoning cues, enabling the model to explicitly reason about why an object disappears, how its state evolves while invisible, and whether that state should persist at query time. We further build HSR-Bench, a diagnostic benchmark for hidden-state reasoning, containing 1,427 video-QA samples from 1,384 unique videos. Extensive experiments across multiple VideoLLMs show that StateTrace consistently improves performance on both public benchmarks and HSR-Bench (e.g., improving VideoLLaMA3 from 39.6 to 64.2 on HSR-Bench).
Figures & tables
Figure 1. A) Illustration of hidden-state spatiotemporal reasoning; B) StateTrace: Augmented with state summaries. The figure contains two panels. The first panel illustrates a target object becoming invisible in a long video. The second panel shows how StateTrace constructs and retrieves object-state summaries for answer generation.
Figure 2. Overview of StateTrace. The pipeline consists of three stages: (I) offline spatiotemporal state memory construction, (II) query-time retrieval and spatiotemporal evidence summarization, and (III) final answer generation with evidence. The figure illustrates the three-stage StateTrace pipeline. Stage I divides a long video into chunks and constructs an object-centric spatiotemporal memory containing semantic information, object trajectories, spatial relations, visibility changes, and state-transition events. Stage II retrieves question-relevant video chunks and memory records and summarizes them into semantic and spatial evidence. Stage III combines the retrieved visual evidence, the generated summaries, and the question as input to a VideoLLM for final answer generation.
Model
Size
MLVU
VideoMME
LVB
w/o sub.
w/ sub.
InternVL2.5 ( Chen et al., 2024c )
2B
61.4
51.9
54.1
52.0
InternVL2.5 + Video-RAG ( Luo et al., 2024 )
2B
62.4
52.4
54.8
53.1
InternVL2.5 + FlexSelect ( Zhang et al., 2025b )
2B
63.0
52.6
55.3
54.0
InternVL2.5 + StateTrace (Ours)
2B
64.0
52.8
56.0
56.3
VideoLLaMA3 ( Zhang et al., 2025a )
2B
65.4
59.6
63.4
57.1
Table 1. Main results on public video QA benchmarks. StateTrace is compared with base VideoLLMs and other enhancement methods across different benchmarks. The best result in each column is shown in bold.
Model
Size
HSR-Bench
InternVL2.5 ( Chen et al., 2024c )
2B
20.81
InternVL2.5 + Video-RAG ( Luo et al., 2024 )
2B
23.46
InternVL2.5 + FlexSelect ( Zhang et al., 2025b )
2B
26.18
InternVL2.5 + StateTrace (Ours)
2B
49.12
VideoLLaMA3 ( Zhang et al., 2025a )
2B
24.73
VideoLLaMA3 + Video-RAG ( Luo et al., 2024 )
2B
27.95
Table 2. Performance on HSR-Bench. StateTrace is compared with base VideoLLMs and representative enhancement methods. The best result is highlighted in bold.
Figure 3. Performance across video-length buckets on LongVideoBench. ST denotes the proposed StateTrace framework, and Overall is the weighted average accuracy across all duration buckets. The figure compares the accuracy of baseline VideoLLMs and their StateTrace-enhanced variants across multiple video-duration buckets on LongVideoBench. The final group reports the weighted average accuracy over all duration buckets.
Figure 4. Performance across occlusion-severity buckets on HSR-Bench. ST denotes StateTrace, and Overall is the weighted average accuracy over all samples. The figure compares the accuracy of baseline VideoLLMs and their StateTrace-enhanced variants across light, medium, and heavy occlusion severity buckets on HSR-Bench. The final group reports the weighted average accuracy over all samples.
Backbone
Size
Setting
MLVU
VideoMME
LVB
w/o sub.
w/ sub.
InternVL2.5 ( Chen et al., 2024c )
2B
w/o DispC
62.3
52.1
55.0
53.7
InternVL2.5
2B
Full
64.0
52.8
56.0
56.3
InternVL2.5
8B
w/o DispC
71.4
64.6
68.7
61.3
InternVL2.5
8B
Full
73.4
65.6
70.0
64.0
Qwen2.5-VL ( Bai et al., 2025b )
3B
w/o DispC
70.0
61.5
68.1
56.9
Table 3. Ablation on disappearance-cause reasoning under different backbones. DispC denotes disappearance-cause reasoning. The best result in each metric is shown in bold.
Backbone
Size
Setting
MLVU
VideoMME
LVB
w/o sub.
w/ sub.
InternVL2.5 ( Chen et al., 2024c )
2B
w/o SpS
61.8
51.8
54.4
52.9
InternVL2.5
2B
Full
64.0
52.8
56.0
56.3
InternVL2.5
8B
w/o SpS
70.8
64.2
68.2
60.8
InternVL2.5
8B
Full
73.4
65.6
70.0
64.0
Qwen2.5-VL ( Bai et al., 2025b )
3B
w/o SpS
69.5
61.1
67.8
56.1
Table 4. Ablation on spatial summary generation under different backbones. SpS denotes spatial summary. The best result in each metric column is shown in bold.
Figure 5. Two examples illustrating improved hidden-state spatiotemporal reasoning with StateTrace. The figure presents two qualitative examples comparing a baseline VideoLLaMA3-7B model with the StateTrace-enhanced version. The first example involves occlusion, where the model must infer the color of a car after it becomes temporarily hidden behind a black car. The second example involves containment, where the model must determine which cup contains the target nails after repeated exchanges. In both cases, the baseline model gives incorrect answers, while StateTrace preserves hidden object states and answers correctly.
Backbone
Size
Setting
MLVU
VideoMME
LVB
w/o sub.
w/ sub.
InternVL2.5 ( Chen et al., 2024c )
2B
w/o KCR
62.8
52.3
55.3
54.8
InternVL2.5
2B
Full
64.0
52.8
56.0
56.3
InternVL2.5
8B
w/o KCR
72.1
65.0
68.9
62.4
InternVL2.5
8B
Full
73.4
65.6
70.0
64.0
Qwen2.5-VL ( Bai et al., 2025b )
3B
w/o KCR
70.7
61.9
68.5
58.2
Table 5. Ablation on key clip retrieval under different backbones. KCR denotes key clip retrieval. The best result in each metric column is shown in bold.
Temporal video understanding studies how models track actions, events, and changes over time to answer questions, locate moments, and predict what happens next.