Beyond Caption-Based Queries for Video Moment Retrieval

David Pujol-Perich; Albert Clapés; Dima Damen; Sergio Escalera; Michael Wray

cs.CVMar 2, 2026

Beyond Caption-Based Queries for Video Moment Retrieval

Authors: David Pujol-Perich, Albert Clapés, Dima Damen, Sergio Escalera, Michael Wray

Organizations: University of Barcelona · Computer Vision Center · University of Bristol

Abstract

In this work, we investigate the degradation of existing VMR methods, particularly of DETR architectures, when trained on caption-based queries but evaluated on search queries. For this, we introduce three benchmarks by modifying the textual queries in three public VMR datasets -- i.e., HD-EPIC, YouCook2 and ActivityNet-Captions. Our analysis reveals two key generalization challenges: (i) A language gap, arising from the linguistic under-specification of search queries, and (ii) a multi-moment gap, caused by the shift from single-moment to multi-moment queries. We also identify a critical issue in these architectures -- an active decoder-query collapse -- as a primary cause of the poor generalization to multi-moment instances. We mitigate this issue with architectural modifications that effectively increase the number of active decoder queries. Extensive experiments demonstrate that our approach improves performance on search queries by up to 14.82% mAP_m, and up to 21.83% mAP_m on multi-moment search queries. The code, models and data are available in the project webpage: https://davidpujol.github.io/beyond-vmr/

Figures & tables

Figure 1: After watching a video, annotators write detailed, visually-informed captions that map to a single GT moment. However, at inference time, users formulate less detailed, visually-uninformed search queries that often map to multiple GT moments.

Figure 2: Overview of the search-query pipeline. Each of the caption is first processed by an agent that generates per-query under-specifications, which are validated by a second identical agent and manually re-annotated if abnormal. Individual queries mapping to the same under-specified query are then grouped, and a final agent produces a representative search query per group.

Dataset	# videos	# queries	Duration per video(s)	Moments per query	Moments per query (multi)	# multi. queries	% multi. queries	Query length
HD_EPIC [ 33 ]	156	59,454	954	1.00	0.00	0	0.00	$16.47\pm 7.9$
HD_EPIC-S1	156	36,819	954	1.61	3.64	8,560	23.25	$6.09\pm 2.5$
HD_EPIC-S2	156	31,521	954	1.88	3.87	9,717	30.83	$3.73\pm 1.2$
HD_EPIC-S3	156	10,266	954	5.79	11.09	4,873	47.47	$3.03\textpm 1.0$
ANC [ 19 ]	14950	71,957	152.8	1.00	0.00	0	0.00	$13.16\pm 6.1$
ANC-S	14950	59,138	152.8	1.21	2.45	8,818	14.91	$4.83\pm 1.4$

Table 1: Statistics of the search-based VMR benchmarks

Figure 3: Evaluation of the representative models on both the original datasets and their corresponding search query extensions.

Figure 4: Performance degradation for CG-DETR on caption versus search-based evaluation for the “single” and “multi” splits.

Figure 5: Visualization of the active query collapse on HD-EPIC-S2 for the base CG-DETR and our method -SA+QD.

Model	Input	Variant	R m				mAP m
			@0.1	@0.3	@0.5	Avg.	@0.1	@0.3	@0.5	Avg.
CG-DETR	S1	base	28.61	17.95	8.99	18.51	36.21	22.84	11.59	23.54
	S1	-SA+QD	29.87	19.69	10.86	20.14	39.74	26.49	14.87	27.03
	S2	base	24.71	15.52	7.89	16.04	32.15	20.1	10.29	20.84
	S2	-SA+QD	26.17	17.00	9.40	17.52	35.38	23.39	13.04	23.93
	S3	base	9.50	4.61	2.08	5.39	16.20	8.01	3.58	9.26

Table 2: Results of both CG-DETR and LD-DETR on HD-EPIC-S{1,2,3} benchmarks with respect to our proposed modifications.

Model	Variant	R m				mAP m
		@0.1	@0.3	@0.5	Avg.	@0.1	@0.3	@0.5	Avg.
CG-DETR	base	28.92	19.87	11.22	20.00	38.83	26.96	15.21	27.00
CG-DETR	-SA+QD	29.97	20.32	11.38	20.55	41.00	29.40	17.21	29.20
LD-DETR	base	33.13	23.48	11.70	22.70	41.69	30.04	15.58	29.10
LD-DETR	-SA+QD	35.86	24.76	13.17	24.59	45.66	33.09	18.74	32.49

Table 3: Results of both CG-DETR and LD-DETR on YC2-S with respect to our proposed modification.

Model	Variant	R m				mAP m
		@0.1	@0.3	@0.5	Avg.	@0.1	@0.3	@0.5	Avg.
CG-DETR	base	60.44	40.89	24.56	41.96	72.12	54.92	36.42	54.48
CG-DETR	-SA+QD	63.75	43.12	25.50	44.12	74.00	56.42	38.20	56.20
LD-DETR	base	62.58	43.00	26.08	43.88	73.35	56.17	36.79	55.43
LD-DETR	-SA+QD	65.21	43.89	25.77	44.95	74.25	56.31	36.69	55.75

Table 4: Results of both CG-DETR and LD-DETR on ANC-S with respect to our proposed modification.

Figure 6: Dissection of the performance of CG-DETR on HD-EPIC-S2, for single, multi and all the instances, respectively.

Variant	$R_{m}$	$mAP_{m}$	# active	% match P	% match GT
base	16.04	20.84	$3.64\pm 1.18$	$0.06\pm 0.07$	$0.36\pm 0.35$
+ 1-to-5 matching [ 21 ]	14.66	16.30	$9.56\pm 3.20$	$0.11\pm 0.16$	$0.21\pm 0.28$
+ 1-to-k matching [ 21 ]	10.78	11.01	$20.00\pm 0.0$	$0.16\pm 0.33$	$0.07\pm 0.18$
+group_matching [ 8 ]	15.34	17.97	$8.69\pm 3.08$	$0.10\pm 0.13$	$0.27\pm 0.31$
+hybrid [ 16 ]	14.67	17.04	$8.68\pm 2.90$	$0.10\pm 0.13$	$0.28\pm 0.32$
+ms_matcher [ 49 ]	15.75	20.94	$3.58\pm 1.14$	$0.06\pm 0.07$	$0.36\pm 0.34$

Table 5: Ablation of methods to increase number of active queries.

Variant	$R_{m}$	$mAP_{m}$	# active	% match P	% match GT
-SA+QD (ours)	17.52	23.93	$6.43\pm 2.16$	$0.11\pm 0.13$	$0.42\pm 0.37$
+ 1-to-k matcher [ 21 ]	10.38	10.39	$20.00\pm 0.00$	$0.14\pm 0.35$	$0.06\pm 0.17$
+group_matching [ 8 ]	17.30	23.71	$12.70\pm 6.13$	$0.15\pm 0.17$	$0.43\pm 0.36$
+hybrid [ 16 ]	17.91	24.38	$10.10\pm 6.23$	$0.14\pm 0.16$	$0.50\pm 0.36$

Table 6: Effect of 1-to-1 matching in promoting diversity.

Table 13

Figure 7: Evolution of the number of active queries and performance over the total number of decoder queries.

Figure 8: Qualitative results for CG-DETR on HD-EPIC-S2 and YC2-S benchmarks.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A : Intuitive examples that showcase the behavior of both $R_{m}$ and $R1$ . Here the solid lines correspond to prediction, and the dashed ones correspond to GT moments. Moreover, in R1, the checkmark indicates that the entire query is marked as correct, while for $R_{m}$ , since it performs a per-GT evaluation, this indicates whether the corresponding prediction “correctly” retrieves a GT moment or not, based on the criterion defines by the metric. The orange circles indicate the global score of the instance, which is consistent with the single score produced by $R1$ , or by the average of the multiple per-GT scores for $R_{m}$ .

Figure B : Example that showcases the behavior of both $mAP$ and $mAP_{m}$ . $mAP$ computes a global score for the entire ranking, which is depicted in its corresponding orange circles. $mAP_{m}$ , in turn, computes a score for each of the two GT moments separately. The example shows the two rankings that this evaluation leverages, effectively ignoring the influence of predictions that, while match a different GT, should not be considered invalid. The orange circle from $mAP_{m}$ corresponds to the average of the two respective per-GT scores. .

Figure C : Qualitative results of the performance of CG-DETR and our proposed modification (-SA+QD) on HD-EPIC-S1/S2/S3.

Figure D : Qualitative results of the performance of CG-DETR and our proposed modification (-SA+QD) on HD-EPIC-S3, YC2-S and ANC-S.

Model	Input	Variant	$R_{m}$			$mAP_{m}$			$mAP$
			@0.1	@0.3	@0.5	@0.1	@0.3	@0.5	@0.1	@0.3	@0.5
CG-DETR	Original	base	34.44	21.63	11.32	42.69	26.96	14.26	42.69	26.96	14.26
	Original	-SA+QD	34.24	22.68	12.59	45.79	30.52	17.16	45.79	30.52	17.16
	S1	base	28.61	17.95	8.99	36.21	22.84	11.59	38.52	24.33	12.27
	S1	-SA+QD	29.87	19.69	10.86	39.74	26.49	14.87	41.65	27.98	15.85
		base (oracle)	28.85	17.44	9.08	40.42	25.12	13.1	42.58	26.74	14.02

Appendix

Table A : Results of both CG-DETR and LD-DETR on HD-EPIC-S 1,2,3 benchmarks with respect to our proposed modifications.

Model	Input	Variant	$R_{m}$				$mAP_{m}$				$mAP$
			@0.1	@0.3	@0.5	@0.75	@0.1	@0.3	@0.5	@0.75	@0.1	@0.3	@0.5	@0.75
CG-DETR	Orig.	base	63.32	50.71	37.79	20.11	69.47	55.55	41.04	17.18	69.47	55.55	41.04	17.18
	Orig.	-SA+QD	62.46	51.05	37.19	20.27	70.25	57.97	42.79	18.05	70.25	57.97	42.79	18.05
	S	base	28.92	19.87	11.22	4.60	38.83	26.96	15.21	4.07	47.74	33.61	19.40	5.27
	S	-SA+QD	29.97	20.32	11.38	4.36	41.00	29.4	17.21	4.65	49.52	36.41	21.86	6.33
LD-DETR	Orig.	base	68.06	56.34	39.75	19.69	73.15	60.62	42.79	15.88	73.15	60.62	42.79	15.88

Appendix

Table B : Results of both CG-DETR and LD-DETR on YC2-S with respect to our proposed modification.

Model	Input	Variant	$R_{m}$				$mAP_{m}$				$mAP$
			@0.1	@0.3	@0.5	@0.75	@0.1	@0.3	@0.5	@0.75	@0.1	@0.3	@0.5	@0.75
CG-DETR	Orig.	base	75.48	60.00	44.02	26.21	82.31	69.36	53.15	25.7	82.31	69.36	53.15	25.7
	Orig.	-SA+QD	75.19	60.30	44.96	26.47	82.25	69.91	54.5	25.67	82.25	69.91	54.5	25.67
	S	base	60.44	40.89	24.56	12.97	72.18	54.9	36.41	15.07	73.34	55.84	37.49	15.71
	S	-SA+QD	63.75	43.12	25.50	13.36	74.00	56.42	37.2	15.09	75.54	57.52	38.52	15.78
LD-DETR	Orig.	base	75.72	60.63	45.17	27.15	82.73	70.3	54.46	26.62	82.73	70.3	54.46	26.62

Appendix

Table C : Results of both CG-DETR and LD-DETR on ANC-S with respect to our proposed modification.

Benchmark	Input	$R_{m}$			$mAP_{m}$
		@0.1	@0.3	@0.5	@0.1	@0.3	@0.5
HD-EPIC	Original	30.56	23.83	16.46	43.70	36.08	26.89
	S1	21.45	16.50	10.98	31.48	25.71	18.83
	S2	16.97	13.47	8.83	25.93	21.48	15.58
	S3	5.64	4.59	3.07	9.63	8.01	5.91
YC2-S	Original	77.66	66.09	49.31	84.78	75.02	60.22

Appendix

Table D : Results of Flash-VTG on all our proposed benchmarking scenarios.

Figure L : Correlation between the average ratio of matched predictions—i.e., predictions with an IOU of at least 0.1 with one of the GT moments—with respect to their respective confidence score. This highlights the tradeoff between regression quality and confidence score quality. These results correspond to HD-EPIC-S2 for CG-DETR. .

Variant	$R_{m}$			$mAP_{m}$			active
	@0.1	@0.3	@0.5	@0.1	@0.3	@0.5
base	24.71	15.52	7.89	32.15	20.1	10.29	$3.64\pm 1.18$
+actionness [ 27 ]	26.04	16.05	8.11	32.26	19.96	10.23	$3.79\pm 0.98$
+SFL [ 18 ]	21.83	11.22	4.82	27.54	14.48	6.31	$3.05\pm 0.99$
-SA+QD (ours)	26.17	17.00	9.40	35.38	23.39	13.04	$6.43\pm 2.16$

Appendix

Table E : Performance of alternative confidence calibration mechanisms for CG-DETR on HD-EPIC-S2.

Input data	Model	Split	$R_{m}$			$mAP_{m}$
			@0.1	@0.3	@0.5	@0.1	@0.3	@0.5
S1	base	single	31.47	20.09	10.19	39.29	25.02	12.68
	base	multi	22.90	13.73	6.77	29.97	18.46	9.57
	-SA+QD	single	31.61	21.39	12.25	41.72	28.23	16.25
	-SA+QD	multi	26.18	16.16	8.11	35.66	22.89	12.12
S2	base	single	26.62	16.90	8.50	34.55	21.56	10.87

Appendix

Table F : Dissection of the performance between “single” and “multi” for CG-DETR on HD-EPIC-S1/S2/S3

Input data	Model	Split	$R_{m}$			$mAP_{m}$
			@0.1	@0.3	@0.5	@0.1	@0.3	@0.5
S	base	single	39.43	28.70	17.54	50.62	36.31	22.06
	base	multi	24.22	15.89	8.40	33.58	22.77	12.12
	-SA+QD	single	39.26	29.47	17.88	51.83	39.44	24.75
	-SA+QD	multi	25.82	16.08	8.48	36.18	24.81	13.8

Appendix

Table G : Dissection of the performance between “single” and “multi” for CG-DETR on YC2-S.

Input data	Model	Split	$R_{m}$			$mAP_{m}$
			@0.1	@0.3	@0.5	@0.1	@0.3	@0.5
S	base	single	60.63	39.87	24.59	72.39	54.65	37.12
	base	multi	60.26	42.58	24.46	71.72	55.32	35.10
	-SA+QD	single	65.03	42.55	26.00	75.0	56.61	38.45
	-SA+QD	multi	61.72	44.16	24.70	72.42	56.24	35.21

Appendix

Table H : Dissection of the performance between “single” and “multi” for CG-DETR on ANC-S.

Dataset	Split	Specificity	$R_{m}$			$mAP_{m}$
			@0.1	@0.3	@0.5	@0.1	@0.3	@0.5
HD-EPIC-S1	single	Orig.	38.53	24.81	13.04	46.04	29.74	15.74
	single	S	31.47	20.09	10.19	39.29	25.02	12.68
	multi	Orig.	25.97	15.03	7.74	35.75	21.21	11.2
	multi	S	22.90	13.73	6.77	29.97	18.46	9.57
HD-EPIC-S2	single	Orig.	38.79	25.32	13.26	46.26	30.25	16.03

Appendix

Table I : Extended results of the language vs multi-moment gap analysis from Sec. 4 .

Variant	$R_{m}$			$mAP_{m}$
	0.1	0.3	0.5	0.1	0.3	0.5
base	24.71	15.52	7.89	32.15	20.1	10.29
+ 1-to-5 matching [ 21 ]	24.00	15.11	7.97	29.35	18.36	9.6
+ 1-to-k matching [ 21 ]	18.31	8.80	4.12	18.47	8.87	4.16
+group_matching [ 8 ]	24.33	15.85	8.86	31.21	20.12	11.14
+hybrid matching [ 16 ]	23.98	15.19	7.75	30.53	19.07	9.71

Appendix

Table J : Extended ablation of alternative methods to increase the number of active decoder queries, evaluated with CG-DETR on HD-EPIC-S2.

Variant	$R_{m}$			$mAP_{m}$
	0.1	0.3	0.5	0.1	0.3	0.5
-SA+QD (ours)	26.17	17.00	9.40	35.38	23.39	13.04
+ 1-to-k matcher [ 21 ]	17.88	9.18	4.10	17.89	9.19	4.11
+group matcher [ 8 ]	25.72	17.17	9.02	35.36	23.37	12.41
+hybrid matcher [ 16 ]	26.59	17.60	9.55	35.79	24.13	13.24

Appendix

Table K : Extended ablation of the effect of the 1-to-1 matching strategy in promoting diversity across decoder queries, for CG-DETR evaluated on HD-EPIC-S2

-SA	+QD	$R_{m}$			$mAP_{m}$
		0.1	0.3	0.5	0.1	0.3	0.5
		24.71	15.52	7.89	32.15	20.1	10.29
✓		23.97	14.73	7.25	32.17	20.58	10.33
	✓	24.45	16.24	8.83	31.58	21.07	11.66
✓	✓	26.17	17.00	9.40	35.38	23.39	13.04

Appendix

Table L : Extended ablation of the impact of each of the proposed architectural modifications, for CG-DETR evaluated on HD-EPIC-S2.

$k$	$R_{m}$			$mAP_{m}$
	0.1	0.3	0.5	0.1	0.3	0.5
0.00	24.71	15.52	7.89	32.15	20.1	10.29
0.25	26.17	17.00	9.40	35.38	23.39	13.04
0.50	1.82	0.86	0.31	6.72	3.49	1.32

Appendix

Table M : Extended ablation of the effect of the QD dropout rate, for CG-DETR evaluated on HD-EPIC-S2.

Model	# queries	$R_{m}$			$mAP_{m}$			# active
		0.1	0.3	0.5	0.1	0.3	0.5
base	5	22.41	14.12	6.94	27.78	17.46	8.71	$2.87\pm 0.70$
	10	23.60	15.39	7.65	29.4	19.05	9.59	$3.32\pm 1.06$
	20	24.71	15.52	7.89	32.15	20.1	10.29	$3.52\pm 1.05$
	30	21.88	15.00	7.86	28.83	19.65	10.39	$3.98\pm 1.28$
	50	11.66	6.10	2.74	18.12	10.02	4.58	$3.66\pm 1.10$

Appendix

Table N : Extended ablation (corresponding to Fig. 8 of the main text) that shows the impact of the overall number of decoder queries for the CG-DETR and our proposed (-SA + QD), for HD-EPIC-S2.

Figure M : Histogram of the feature similarities of each of the test set of each of the levels of under-specification HD-EPIC-S1/S2/S3 with respect to the original caption-based queries from HD-EPIC

Caption: “Holding the ball of chicken and potato mixture in my left hand while I get some flour in my right hand so that I can sprinkle over the ball”

\downarrow

S1: “Hold the mixture and get some flour to sprinkle over it”

\downarrow

S2: “Sprinkle flour on the mixture”

\downarrow

S3: “Prepare food”

Appendix

Table O : Qualitative results of the under-specified search queries for HD-EPIC-S1/S2/S3.

Representative: “Tear the lettuce and place it on the plate”

\downarrow

Caption1: “Tear the lettuce leaves in half again and place onto the plate that is on top of the weighing scale .”

Caption2: “Tear the lettuce leaves in half again and put the pieces onto the plate .”

Caption3: “Use both hands to tear the lettuce leaves in half again and place the pieces onto the plate .”

Caption4: “Use both hands to tear the lettuce leaves in half .”

Caption5: “Use both hands to tear the lettuce leaf in half”

Appendix

Table P : Example of search queries and the captions that match it for HD-EPIC-S1.

Representative: “Close a cupboard”

\quad\quad\downarrow

Caption1: “With my right hand , close the bottom cupboard .”

Caption2: “With my right hand , close the cupboard .”

Caption3: “Close the cupboard by pushing it with my left hand .”

Caption4: “Having realized this is the wrong cupboard , close it again by pushing it with my left hand .”

Caption5: “I close the cupboard by pushing it with my right hand .”

Appendix

Table Q : Example of search queries and the captions that match it for HD-EPIC-S2.

Representative: “Open a book”

\downarrow

Caption1: “Push down in the center of the recipe book along the spine to try and make sure it stays open onto the countertop.”

Caption2: “Pick up the recipe book using the right hand and flipping it over so that the recipe can be seen.”

Representative: “Clean a bowl.”

\downarrow

Caption1: “Realise there is some old food or dirt on the bottom of the bowl.”

Appendix

Table R : Example of search queries and the captions that match it for HD-EPIC-S3.

Figure N : Histogram of the feature similarities of the test set of YC2-S with respect to the original caption-based queries from YC2

Caption: “Add strained tomato puree to the blond roux and stir the mixture continuously”

\downarrow

S: “Add ingredient.”

Caption: “Add salt to the pan and mix”

\downarrow

S: “Season food.”

Appendix

Table S : Qualitative results of the under-specified search queries for YC2-S.

Representative: “Add ingredients.”

\downarrow

Caption1: “Crack one egg into a bowl”

Caption2: “Add one table spoon of oil salt and cayenne pepper and baking powder and beat”

Caption3: “Add one cup of beer and mix”

Caption4: “Add one quarter cup of corn meal and one cup of flour”

Caption5: “Add onions into batter and drop into hot oil”

Appendix

Table T : Example of search queries and the captions that match it for YC2-S.

Figure O : Histogram of the feature similarities of each of the test set of ANC-S with respect to the original caption-based queries from ANC

Caption: “The man is washing a side of the car.”

\downarrow

S: ‘A person performs a task on a vehicle.”

Caption: “The person then moves back and fourth on the machine while rowing his arms back and fourth.”

\downarrow

S: “A person uses a machine.”

Appendix

Table U : Qualitative results of the under-specified search queries for ANC-S.

Representative: “People play a game.”

\downarrow

Caption1: “A man and woman are shown standing on a tennis court passing a ball back and fourth.”

Caption2: “The players volley and the play is pressed to the back of the court hitting long shots.”

Caption3: “The players volley and the birdie is hit out of view and the player retrieves it then serves.”

Caption4: “The players have a long volley until the play in the foreground misses and the birdie lands at her feet.”

Caption5: “People play games of badminton on indoor courts.”

Appendix

Table V : Example of search queries and the captions that match it for ANC-S.

Video Moment Retrieval

See all 16 papers

May 28, 2026

Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language

Given an untrimmed video and a sentence query, video moment retrieval using language (VMR) aims to locate a target query-relevant moment. Since the untrimmed video is overlong, almost all existing VMR methods first sparsely down-sample each untrimmed video into multiple fixed-length video clips and then conduct multi-modal interactions with the query feature and expensive clip features for reasoning, which is infeasible for long real-world videos that span hours. Since the video is downsampled into fixed-length clips, some query-related frames may be filtered out, which will blur the specific boundary of the target moment, take the adjacent irrelevant frames as new boundaries, easily leading to cross-modal misalignment and introducing both boundary-bias and reasoning-bias. To this end, in this paper, we propose an efficient approach, SpotVMR, to trim the query-relevant clip. Besides, our proposed SpotVMR can serve as plug-and-play module, which achieves efficiency for state-of-the-art VMR methods while maintaining good retrieval performance. Especially, we first design a novel clip search model that learns to identify promising video regions to search conditioned on the language query. Then, we introduce a set of low-cost semantic indexing features to capture the context of objects and interactions that suggest where to search the query-relevant moment. Also, the distillation loss is utilized to address the optimization issues arising from end-to-end joint training of the clip selector and VMR model. Extensive experiments on three challenging datasets demonstrate its effectiveness.

Xiang Fang, Daizong Liu, Wanlong Fang +5

Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science of Technology Wuhan, China · Peking University · Henan University +4

Cross-Modal Learning Temporal Video Grounding Video Moment Retrieval Video Retrieval

Ask PDF

Jan 14, 2025

Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models

The target of video moment retrieval (VMR) is predicting temporal spans within a video that semantically match a given linguistic query. Existing VMR methods based on multimodal large language models (MLLMs) overly rely on expensive high-quality datasets and time-consuming fine-tuning. Although some recent studies introduce a zero-shot setting to avoid fine-tuning, they overlook inherent language bias in the query, leading to erroneous localization. To tackle the aforementioned challenges, this paper proposes Moment-GPT, a tuning-free pipeline for zero-shot VMR utilizing frozen MLLMs. Specifically, we first employ LLaMA-3 to correct and rephrase the query to mitigate language bias. Subsequently, we design a span generator combined with MiniGPT-v2 to produce candidate spans adaptively. Finally, to leverage the video comprehension capabilities of MLLMs, we apply VideoChatGPT and span scorer to select the most appropriate spans. Our proposed method substantially outperforms the state-ofthe-art MLLM-based and zero-shot models on several public datasets, including QVHighlights, ActivityNet-Captions, and Charades-STA.

Yifang Xu, Yunzhuo Sun, Benxiang Zhai +4

Nanjing University · Dalian University of Technology · Nanjing University of Information Science and Technology

Video Moment Retrieval Temporal Video Grounding Video-Language Models Zero-Shot Learning

Ask PDF

May 4, 2026

Retrieving Any Relevant Moments: Benchmark and Models for Generalized Moment Retrieval

Video Moment Retrieval (VMR) aims to localize temporal segments in videos that correspond to a natural language query, but typically assumes only a single matching moment for each query. This assumption does not always hold in real-world scenarios, where queries may correspond to multiple or no moments. Thus, we formulate Generalized Moment Retrieval (GMR), a unified setting that requires retrieving the complete set of relevant moments or predicting an empty set. To enable systematic study of GMR, we introduce Soccer-GMR, a large-scale benchmark built on challenging soccer videos that reflect general GMR scenarios, with realistic negative and positive queries. The benchmark is constructed via a duration-flexible semi-automated pipeline with human verification, enabling scalable data generation while maintaining high annotation quality. We further design a unified evaluation protocol with complementary metrics tailored for null-set rejection, positive-query localization, and end-to-end GMR performance. Finally, we establish strong baselines across two modeling paradigms: a lightweight plug-and-play GMR adapter for discriminative VMR models, and a GMR-tailored GRPO reward for fine-tuning multimodal large language models (MLLMs). Extensive experiments show consistent gains across all metrics and expose key limitations of current methods, positioning GMR as a more realistic and challenging benchmark for video-language understanding.

Yiming Ding, Siyu Cao, Luyuan Jiao +4

Institute of Automation, Chinese Academy of Sciences · Beijing University of Posts and Telecommunications · Wuhan University +1

Temporal Video Grounding Temporal IR Video Retrieval Video Moment Retrieval

Ask PDF

Abstract

Video Moment Retrieval

Explore similar work

Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language

Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models

Retrieving Any Relevant Moments: Benchmark and Models for Generalized Moment Retrieval