cs.CVApr 7, 2026
SaveVideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAG
Organizations: University of Queensland · University of California, Merced · Institute of Automation, CAS · Lancaster University
Abstract
Scaling multimodal large language models (MLLMs) to long videos is constrained by limited context windows. While retrieval-augmented generation (RAG) is a promising remedy by organizing query-relevant visual evidence into a compact context, most existing methods (i) flatten videos into independent segments, breaking their inherent spatio-temporal structure, and (ii) depend on explicit semantic matching, which can miss cues that are implicitly relevant to the query's intent. To overcome these limitations, we propose VideoStir, a structured and intent-aware long-video RAG framework. It firstly structures a video as a spatio-temporal graph at clip level, and then performs multi-hop retrieval to aggregate evidence across distant yet contextually related events. Furthermore, it introduces an MLLM-backed intent-relevance scorer that retrieves frames based on their alignment with the query's reasoning intent. To support this capability, we curate IR-600K, a large-scale dataset tailored for learning frame-query intent alignment. Experiments show that VideoStir is competitive with state-of-the-art baselines without relying on auxiliary information, highlighting the promise of shifting long-video RAG from flattened semantic matching to structured, intent-aware reasoning. Codes and checkpoints are available at https://github.com/RomGai/VideoStir.
Figures & tables
Figure 1: Paradigm shift from flat semantic matching to structured, intent-aware long-video RAG. (Top) Semantic-based retrieval relies on explicit semantic overlap, often missing cues that are only implicitly relevant to the query intent. (Middle) Intent-aware retrieval utilizes MLLM’s reasoning capability to identify cues that may be relevant to the query intent (detailed in Sec. 3.4 ); however, it can be distracted by noisy cues when operating over flattened contexts scattered across long temporal ranges. (Bottom) VideoStir structures the video as a spatio-temporal graph to retrieve cues from coherent contexts, enabling more reliable evidence aggregation. We further discuss the benefits of structured retrieval in Fig. 3 .
Figure 2: Overview of VideoStir. VideoStir achieves long-video understanding via spatio-temporal structuring and two-stage retrieval. (a) Spatio-Temporal Topology Modeling. An event boundary detector segments input video into semantically coherent clips. Their embeddings are used to construct a spatio-temporal graph with temporal edges linking adjacent clips and spatial edges weighted by inter-clip similarity. (b) Graph-based Clip Retrieval . Input query is embedded in the video-language space to retrieve top- matched anchor nodes; multi-hop traversal then expands along spatio-temporal links under hop-distance and edge-weight constraints to aggregate contextual clips. (c) Intent-aware Frame Retrieval. Frames densely sampled from the retrieved clips are assessed by an intent-relevance scorer (LoRA-tuned on IR-600K), which outputs a probability distribution over discrete relevance levels. These distributions are then aggregated via probability-weighted expectation to produce a continuous relevance score; high-scoring, intent-aligned frames are retained and fed to a downstream MLLM to produce the final answer.
| Model | Native Input | LV-Bench (Val) | MLVU | Video-MME-Long (w/o Sub) | |||
| Overall | Gain (%) | Overall | Gain (%) | Overall | Gain (%) | ||
| LLaVA-Video (7B) | ✓ | 56.6 | - | 70.8 | - | - | - |
| + Video-RAG | ✗ | 58.7 | 3.7 | 72.4 | 2.3 | - | - |
| + TV-RAG | ✗ | 58.8 | 2.1 | 72.1 | 1.8 | - | - |
| + VideoStir (Ours) | ✓ | 60.3 | 6.5 | 73.1 | 3.2 | - | - |
| mPLUG-Owl3 (8B) | ✓ | 52.1 | - | 63.7 | - | 50.1 | - |
Table 1: Comparison with long-video RAG baselines. “Native Input” indicates whether the model processes video directly without relying on external auxiliary text or tools. Bold and underlined numbers indicate the best and second-best accuracies, respectively. ‘–’ indicates not applicable.
| (M)LLM Backbone | Method | Overall |
| Methods with Video-to-Text Tools | ||
| GPT-3.5 | LLoVi [ 32 ] | 57.6 |
| DrVideo [ 25 ] | 62.6 | |
| GPT-4 | VideoAgent [ 33 ] | 60.2 |
| LLoVi [ 32 ] | 61.2 | |
| VidAgent [ 34 ] | 62.8 | |
Table 2: System-level comparison on EgoSchema’s test set with SOTA methods built upon fixed (M)LLMs. These approaches can be categorized into two groups: one employs video-to-text tools (e.g., captioners or object detectors) to provide auxiliary textual inputs for downstream (M)LLMs, while the other leverages the native multimodal understanding capability (Native Input) of MLLMs without auxiliary text. Bold and underlined numbers indicate the best and second-best accuracies, respectively. ‘–’ indicates not applicable.
| Method | Overall | Retrieval Acc. |
| w/o Intent-relevance Scorer | ||
| + CLIP L/14 | 52.8 | 69.2 |
| + Video-CLIP | 56.3 | 75.4 |
| + PE | 58.1 | 79.8 |
| w/o Weighted Expectation | 54.2 | 71.6 |
| w/o Event-Boundary Detector | 62.3 | 88.2 |
Table 3: Component-level ablations on the expanded 1k validation set. “Overall” denotes overall accuracy, and “Retrieval Acc.” measures the accuracy of successfully retrieving key cues. “w/o Intent-relevance Scorer” replaces the scorer with similarity-based frame retrieval; “w/o Weighted Expectation” replaces probability-weighted expectation with discrete scores; “w/o Spatio-Temporal Graph” disables structured retrieval, which bypasses the graph and directly selects the top-14 clips by PE-embedding similarity, following the best-performing setting in Table 6 . Bold indicates the best results.
| Embedding Backbone | Overall | Retrieval Acc. |
| Video-CLIP [ 17 ] | 62.1 | 88.3 |
| X-CLIP [ 18 ] | 62.8 | 89.4 |
| PE [ 19 ] | 64.5 | 92.2 |
Table 4: Ablations of VideoStir’s video-language embedding backbones with . “Overall” denotes overall accuracy, and “Retrieval Acc.” refers to the accuracy of successfully retrieving the key evidence for answering the query. The bold numbers represent the best results.
| Base Model | Method | IR-600K CE | LV-Bench CE | Overall | Retrieval Acc. | Trainable Params |
| Teacher Models | ||||||
| InternVL2.5-78B | Zero-Shot | - | - | 64.8 | 92.6 | - |
| Qwen2.5-VL-72B | Zero-Shot | - | - | 65.4 | 95.8 | - |
| Qwen2-VL-72B | Zero-Shot | - | - | 64.9 | 94.7 | - |
| Student Models | ||||||
| InternVL2.5-4B | Zero-Shot | 8.8755 | 8.9423 | 59.3 | 82.1 | - |
Table 5: Ablations on the intent-relevant scorer’s distillation strategies. “IR-600K CE” and “LV-Bench CE” denote the cross-entropy loss computed on the corresponding datasets. To reduce distributional bias, the student model is distilled using teacher models from the same family. “Retrieval Acc.” refers to the accuracy of successfully retrieving the key evidence.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Advantages of structured retrieval over flat clip retrieval. Flattened retrieval mainly returns clips with explicit semantic overlap with the query, while overlooking clips that are contextually relevant but do not contain direct query-matching content. VideoStir structures long videos to reconstruct spatio-temporal context, allowing multi-hop traversal to aggregate contextually related clips around key moments. The resulting evidence is cleaner and more coherent, providing stronger support for subsequent fine-grained retrieval.
| Hyperparameter | Value | Avg. Clips | Retrieval Acc. |
| Number of Anchors ( ) (Fixed ) | 1 | 7.2 | 88.0 |
| 2 | 13.6 | 92.0 | |
| 3 | 21.9 | 90.5 | |
| 4 | 25.4 | 90.0 | |
| Hop Distance ( ) (Fixed ) | 1 | 8.4 | 87.0 |
| 2 | 13.6 | 92.0 |
Table 6: Hyperparameter ablations on Graph-based Clip Retrieval. We investigate the impact of the number of anchors , hop distance , and spatial edge threshold . “Avg. Clips” denotes the average number of retrieved clips per query. The bold numbers represent the best results.
Figure 4: Distribution of Intent Relevance Scores across the IR-600K training and validation sets. The results demonstrate that the validation set maintains a consistent class distribution with the training set across all relevance levels.