GenSpan: Generation-Calibrated Motion Span Priors for Multi-Verb Video Corpus Moment Retrieval

Yunzhuo Sun; Xinyue Liu; Yanyang Li; Nanding Wu; Linlin Zong; Xianchao Zhang; Wenxin Liang

cs.CVMar 23, 2026

GenSpan: Generation-Calibrated Motion Span Priors for Multi-Verb Video Corpus Moment Retrieval

Authors: Yunzhuo Sun, Xinyue Liu, Yanyang Li, Nanding Wu, Linlin Zong, Xianchao Zhang, Wenxin Liang

Organizations: Dalian University of Technology

Abstract

Video Corpus Moment Retrieval (VCMR) aims to retrieve both the correct video and its temporal segment corresponding to a natural-language query, a task that is especially challenging for multi-verb queries where temporal action ordering is critical. Existing approaches often rely solely on text or static images and struggle to capture implicit motion dynamics, leading to retrieval errors and temporal misalignment. We propose GenSpan, a generation-calibrated VCMR framework that constructs short auxiliary videos from LLM-selected subtitle cues and decomposed sub-events, using these as temporal priors rather than direct retrieval targets. A token selector filters candidate-video features aligned with generated motion, and a bidirectional state-space model efficiently predicts video-moment tuples. Experiments on TVR and ActivityNet-Captions demonstrate that GenSpan improves corpus-level retrieval and moment localization, particularly for complex multi-action queries, while reducing computational cost compared to state-of-the-art multimodal baselines.

Figures & tables

Figure 1: Motivation of temporal video-driven VCMR. (a) Text-driven VCMR relies on limited textual cues and may retrieve a wrong video-moment when a query contains ordered actions. (b) Our temporal video-driven VCMR generates an auxiliary motion prior from the query, providing richer temporal evidence for corpus-level video selection and moment localization.

Figure 2: Overview of GenSpan. LLaMA-3.1 selects subtitle cues and decomposes the query; a temporal-prior generator synthesizes a short auxiliary video; GenSpan selects generation-aligned candidate-video tokens and predicts VCMR tuples with a bidirectional SSM.

Figure 3: Example of LLM-guided query decomposition. The original query is segmented into verb-centered sub-events with inferred additional context to enrich temporal details.

Figure 4: Architecture of the proposed GenSpan temporal module. The GenSpan Token Selector filters candidate-video tokens using text and generated-video priors, and the selected tokens are processed by a bidirectional SSM to produce contextualized features for moment prediction.

Method	TVR				ActivityNet-Captions				TVR		ActivityNet-Captions
	VCMR				VCMR				VR		VR
	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10	R@1	R@10
	IoU=0.5	IoU=0.5	IoU=0.7	IoU=0.7	IoU=0.5	IoU=0.5	IoU=0.7	IoU=0.7
HERO [ 22 ]	10.70	26.71	6.95	20.26	1.39	4.67	0.86	2.82	30.70	63.70	8.87	38.64
XML [ 19 ]	6.17	21.66	2.91	10.12	2.20	9.87	1.24	5.62	18.52	53.15	6.14	32.45
ReLoCLNet	8.15	27.95	4.11	14.41	3.09	12.95	1.82	7.34	22.63	57.91	6.66	34.07

Table 1: Quantitative results on VCMR and VR. We report R@1 and R@10 under IoU thresholds 0.5 and 0.7 for VCMR, and R@1/R@10 for VR. The best results are in bold.

Table 6

Table 4: Ablation studies on TVR. Default or best settings are highlighted in light blue.

Figure 5: Generation quality and verb-count analysis on TVR.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: LLM-guided query processing template. Given the input query and matched subtitles, the LLM extracts roles, ordered temporal steps, fine-grained motion details, and uncertain visual priors. This structured representation provides clearer motion instructions than raw query-subtitle concatenation.

Figure 8: Temporal relevance map for different model variants. The full model concentrates high relevance around the ground-truth moment, while the text-only baseline is more diffuse. Removing subtitles weakens character- and dialogue-specific evidence, and removing the GenSpan selector produces broader high-response regions because generated-prior noise is not filtered. The heatmap denotes normalized temporal relevance intensity, not Transformer attention.

Figure 9: Qualitative grounding examples for 1-verb, 2-verb, and multi-verb queries. Timelines compare GT, SgLFT, ICQ, and GenSpan predictions, with subtitles and generated video frames shown for context.

Video Moment Retrieval

See all 16 papers

Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models
Jan 14, 2025Yifang Xu, Yunzhuo Sun, Benxiang Zhai +4Video Moment Retrieval Temporal Video Grounding
Nanjing University · Dalian University of Technology · Nanjing University of Information Science and Technology
Retrieving Any Relevant Moments: Benchmark and Models for Generalized Moment Retrieval
May 4, 2026Yiming Ding, Siyu Cao, Luyuan Jiao +4Temporal Video Grounding Temporal IR
Institute of Automation, Chinese Academy of Sciences · Beijing University of Posts and Telecommunications · Wuhan University +1
Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language
May 28, 2026Xiang Fang, Daizong Liu, Wanlong Fang +5Cross-Modal Learning Temporal Video Grounding
Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science of Technology Wuhan, China · Peking University · Henan University +4

Jan 14, 2025J/K move · Enter open · S save

Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models

Yifang Xu, Yunzhuo Sun, Benxiang Zhai, Ming Li+3

Nanjing University · Dalian University of Technology · Nanjing University of Information Science and Technology

Video Moment Retrieval Temporal Video Grounding Video-Language Models Zero-Shot Learning

Open paper AskPDF arXiv

The target of video moment retrieval (VMR) is predicting temporal spans within a video that semantically match a given linguistic query. Existing VMR methods based on multimodal large language models (MLLMs) overly rely on expensive high-quality datasets and time-consuming fine-tuning. Although some recent studies introduce a zero-shot setting to avoid fine-tuning, they overlook inherent language bias in the query, leading to erroneous localization. To tackle the aforementioned challenges, this paper proposes Moment-GPT, a tuning-free pipeline for zero-shot VMR utilizing frozen MLLMs. Specifically, we first employ LLaMA-3 to correct and rephrase the query to mitigate language bias. Subsequently, we design a span generator combined with MiniGPT-v2 to produce candidate spans adaptively. Finally, to leverage the video comprehension capabilities of MLLMs, we apply VideoChatGPT and span scorer to select the most appropriate spans. Our proposed method substantially outperforms the state-ofthe-art MLLM-based and zero-shot models on several public datasets, including QVHighlights, ActivityNet-Captions, and Charades-STA.

Abstract

Video Moment Retrieval

Explore similar work

Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models

Retrieving Any Relevant Moments: Benchmark and Models for Generalized Moment Retrieval

Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language