SALI: Shot-Aware Late Interaction for Cross-Shot Relation Matching in Text-to-Video Retrieval using Film-Grammar Knowledge

Toya Oyama; Rainer Lienhart; Shin'ichi Satoh

cs.CVSep 24, 2026

SALI: Shot-Aware Late Interaction for Cross-Shot Relation Matching in Text-to-Video Retrieval using Film-Grammar Knowledge

Authors: Toya Oyama, Rainer Lienhart, Shin'ichi Satoh

Organizations: The University of Tokyo · National Institute of Informatics · University of Augsburg

Abstract

Text-to-video retrieval usually represents a video clip by a single embedding. This embedding often loses important relations between people. E.g., an interaction "Anna confronts Mark" is regularly filmed as alternating shot and reverse shot of both (Fig. 1a). No single shot or averaged embedding over clip shots captures this relation. Thus, we propose SALI (Shot-Aware Late Interaction). It extracts the subject and object from a single-sentence query, and matches the query, its subject and object text embeddings against each visual shot embedding of a video clip. The matching operator is greedy max or optimal transport. A film-grammar penalty in fine-tuning adds a small, consistent shift. Built on CLIP4Clip-meanP, SALI keeps overall recall on par on Condensed Movies and ActivityNet while raising R@1 on multi-shot relation queries by 3 and 12 points, the most among all compared methods, and improves such queries on MSR-VTT at a cost of 1.4 R@1 overall.

Figures & tables

Figure 1: Film grammar breaks single-embedding and per-shot matching. (a) An interaction between two people is filmed as alternating shot and reverse shot: no single shot shows the relation. (b) Per-shot max gives the true video and a one-shot look-alike the same score. (c) Mean pooling averages the shots into one vector and dilutes the relation. (d) SALI matches the subject and the object to different shots.

	Shots	rel. (# / %)	Rel-Multi (# / %)	queries (#)
CM [ 6 ] train	33	410 / 4.8	222 / 2.6	8,499
CM [ 6 ] val	40	41 / 4.8	26 / 3.1	846
CM [ 6 ] test	36	82 / 6.4	48 / 3.8	1,276
ANet [ 19 ] test	3	854 / 17.4	419 / 8.5	4,917
MSR-VTT [ 18 ] test	3	61 / 6.1	–	1,000

Table 1: Datasets (official splits; CM: obtainable clips). Shots: median shot count per video; rel.: relational queries; Rel-Multi: rel. queries whose video has at least the median shot count (CM 34, ANet 4); queries: # of queries. ANet test: the val_1 split.

Figure 2: SALI overview. (a) Offline: the same-person matrix $P$ from cross-shot face clustering (training videos only). Training : optionally, the two penalty terms (Eq. 1 ) are computed from the transport plan of the matching operator and shape the encoder (Table 3 ). Inference : the same parameters are used with either operator. (b) The two alternatives for the matching operator, greedy max and OT, and the penalty.

	All ( $n{=}2262$ )			Rel-Multi ( $n{=}113$ )
Method	R@1	R@5	R@10	R@1	R@5	R@10
CLIP4Clip-meanP [ 2 ]	27.23	47.92	56.72	23.89	45.13	57.52
CLIP4Clip-seqTransf [ 2 ] ∗	25.29	46.91	55.13	24.78	42.48	51.33
X-CLIP [ 5 ] ∗	25.99	49.20	57.74	22.12	46.90	52.21
TS2-Net [ 4 ] ∗	20.34	41.73	51.46	18.58	33.63	46.90
SALI+max	26.94	49.19	57.26	26.84	48.08	56.93

Table 2: CM [ 6 ] R@K (%, higher is better) with the same 128 frames per video. SALI+max: greedy max at training and inference; SALI+OT: OT at training and inference. ∗ : position tables extended beyond the released frame cap. Shaded: ours; bold: best per column.

Train			Infer			All		Rel-Multi
max	OT	pen	max	OT	pen	R@1	R@5	R@1	R@5
$\checkmark$	–	–	$\checkmark$	–	–	26.94	49.19	26.84	48.08
$\checkmark$	–	$\checkmark$	$\checkmark$	–	$\checkmark$	27.09	49.01	27.14	44.84
$\checkmark$	–	$\checkmark$	$\checkmark$	–	–	27.09	48.87	26.55	45.13
–	$\checkmark$	–	–	$\checkmark$	–	27.65	48.57	23.30	46.02
–	$\checkmark$	–	$\checkmark$	–	–	26.86	48.73	26.55	47.49

Table 3: Training vs. inference configuration on CM (setting of Table 2 , 3 seeds). Train/Infer: matching operator (max: greedy max; OT) and whether the film-grammar penalty (pen) is applied. Shaded: penalty in training only, penalty-free greedy max at inference; bold: best per column.

	ActivityNet				MSR-VTT
	All		Rel-Multi		All		Rel.
	R@1	R@5	R@1	R@5	R@1	R@5	R@1	R@5
CLIP4Clip	40.25	71.77	40.10	74.22	46.20	70.30	54.10	78.69
SALI+max	41.12	71.23	52.19	78.52	44.83	71.40	55.19	80.87
SALI+OT	40.70	71.50	50.68	77.88	44.97	71.60	53.55	78.14

Table 4: Generality on ActivityNet [ 19 ] and MSR-VTT [ 18 ] (R@K, %). CLIP4Clip [ 2 ] : the meanP variant trained with the official implementation on each dataset. SALI+max and SALI+OT as in Table 2 ; 3 seeds. MSR-VTT relational subset:

n=61

. Shaded: SALI (ours); bold: best per column.

Explore similar work

ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search
Jan 30, 2026Tao Yu, Haopeng Jin, Hao Wang +19
CASIA · UCAS · Peking University +2
IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding
Sep 9, 2026Shiwen Zhao, Qi Zhang, Sezer Karaoglu +2Video Temporal Grounding Vision-Language Model Grounding
School of Computer Science, The University of Sydney, Sydney, NSW 2006, Australia · Informatics Institute at University of Amsterdam, 1098 XH Amsterdam, the Netherlands
Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language
May 28, 2026Xiang Fang, Daizong Liu, Wanlong Fang +5
Hubei Engineering Research Center on Big Data Security, School of Cyber Science and Engineering, Huazhong University of Science of Technology Wuhan, China · Peking University · Henan University +4

Jan 30, 2026, cs.CVJ/K move · Enter open · S save

ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search

Tao Yu, Haopeng Jin, Hao Wang, Shenghua Chai+18

CASIA · UCAS · Peking University+2

Open paper AskPDF arXiv

In recent years, large language models (LLMs) have made rapid progress in information retrieval, yet existing research has mainly focused on text or static multimodal settings. Open-domain video shot retrieval, which involves richer temporal structure and more complex semantics, still lacks systematic benchmarks and analysis. To fill this gap, we introduce ShotFinder, a benchmark that formalizes editing requirements as keyframe-oriented shot descriptions and introduces five types of controllable single-factor constraints: Temporal order, Color, Visual style, Audio, and Resolution. We curate 1,210 high-quality samples from YouTube across 20 thematic categories, using large models for generation with human verification. Based on the benchmark, we propose ShotFinder, a text-driven three-stage retrieval and localization pipeline: (1) query expansion via video imagination, (2) candidate video retrieval with a search engine, and (3) description-guided shot localization. Experiments on multiple closed-source and open-source models reveal a significant gap to human performance, with clear imbalance across constraints: temporal localization is relatively tractable, while color and visual style remain major challenges. These results reveal that open-domain video shot retrieval is still a critical capability that multimodal large models have yet to overcome.

Abstract

Explore similar work

ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search

IAE-VTG: Interaction-Aligned Action-Entity Video Temporal Grounding

Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language