Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos that contain moments relevant to a text query. Since the target moment occupies only a portion of the video, PRVR requires retrieval based on fine-grained understanding beyond coarse video-level matching. However, existing methods often rely on frozen CLIP frame features, which lack temporal understanding. Even with recent progress in parameter-efficient CLIP adaptation, video-level predictions can still be supported by imprecise frame-level evidence. In this paper, we propose an Intrinsic Temporal Adaptation (ITA) framework for PRVR. First, our Backbone-Internal Temporal Adaptation allows the last few visual transformer layers to attend over groups of neighboring frames. This provides temporally aware frame embeddings while keeping CLIP frozen and training only adaptation parameters. Second, we introduce Affinity-Weighted Gradient Propagation to address the weakly supervised nature of PRVR, softly aggregating top-k frames based on text-frame affinities and propagating learning signals to multiple query-relevant frames. Our method achieves state-of-the-art performance on PRVR benchmarks, demonstrates robust cross-dataset transfer, and retrieves substantially more accurate frame-level evidence within ground-truth query-relevant moments. Our code is available at github.com/hynnsk/ITA.
Figures & tables
Figure 1: Comparison between ProPy and Ours. (a) Visualization of cosine similarities between each frame and its preceding frame. Rows and columns are video and frame indices, respectively. ProPy shows vertical stripe patterns across unrelated videos, while ours produces more video-specific transition patterns. (b) Strict frame-level top- k retrieval on ActivityNet Captions. Unlike video-level retrieval, we pool frames from all videos and test whether all top- k frames come from the ground-truth video. This evaluates whether video-level retrieval is supported by coherent frame-level evidence. ProPy rapidly degrades as k increases, whereas ours remains more robust. (c) Comparison of trainable parameters and inference-time GFLOPs of the video encoder.
Figure 2: Illustration of the proposed framework. Given sampled video frames, the CLIP visual encoder produces frame embeddings through Backbone-Internal Temporal Adaptation, where the last few visual transformer layers perform attention over grouped neighboring frames. The resulting frame embeddings are further refined by a lightweight global class-token attention layer. To address the weakly supervised nature of PRVR, we introduce Affinity-Weighted Gradient Propagation, which selects top- k frames according to text-frame affinities and propagates contrastive learning signals to multiple query-relevant frames. The whole framework is trained in a parameter-efficient manner by freezing the CLIP backbone and applying LoRA to the query-key-value projection layers.
Method
TVR
ActivityNet Captions
Charades-STA
R@1
R@5
R@10
R@100
SumR
R@1
R@5
R@10
R@100
SumR
R@1
R@5
R@10
R@100
SumR
MS-SL ( Dong et al., 2022 )
13.5
32.1
43.4
83.4
172.4
7.1
22.5
34.7
75.8
140.1
1.8
7.1
11.8
47.7
68.4
MS-SL++ ( Chen et al., 2025b )
13.6
33.1
44.2
83.5
174.5
7.0
23.1
35.2
75.8
141.1
1.8
7.6
12.0
48.4
69.7
PEAN ( Jiang et al., 2023 )
13.5
32.8
44.1
83.9
174.2
7.4
23.0
35.5
75.9
141.8
2.7
8.1
13.5
50.3
74.7
GMMFormer ( Wang et al., 2024b )
13.9
33.3
44.5
84.9
176.6
8.3
24.9
36.7
76.1
146.0
2.1
7.8
12.5
50.6
72.9
DL-DKD ( Dong et al., 2025 )
14.4
34.9
45.8
84.9
179.9
8.0
25.0
37.5
77.1
147.6
-
-
-
-
-
Table 1: Performance comparison on TVR, ActivityNet Captions, and Charades-STA. Rows highlighted in gray indicate the performance of methods leveraging ResNet152 + I3D + RoBERTa features.
Method
R@1
R@5
R@10
R@100
SumR
GMMFormer
18.2
43.7
56.7
92.5
211.1
MS-SL
20.4
46.7
60.7
94.6
222.5
Proto
22.6
48.8
61.3
93.9
226.6
MS-SL
21.2
50.0
61.0
94.1
226.4
GMMFormer
18.4
42.3
56.2
93.4
210.3
GMMFormer-V2
23.1
50.7
63.9
94.8
232.5
Table 2: Performance on QVHighlights val split. Rows highlighted in gray are results with CLIP-B/16 features.
Backbone
ProPy
ITA
CLIP-B/16
198.0
209.0
CLIP4Clip-B/32
183.6
196.1
InternVideo-MM-B/16
190.7
201.7
Table 3: Generalization across diverse backbones. We report the average SumR over four PRVR benchmarks.
PEA
GTA
TB
AW
TVR
ANet
Charades
QV
Avg.
✓
217.7
195.9
78.5
271.2
190.8
✓
✓
221.1
199.0
82.0
277.6
194.9
✓
✓
✓
223.2
199.6
83.9
277.9
196.2
✓
✓
219.6
199.5
81.7
280.8
195.4
✓
✓
✓
✓
224.7
201.4
87.4
281.9
198.9
Table 4: Ablation study on each component, reporting SumR. PEA, GTA, TB, and AW denote Parameter-Efficient Adaptation, Grouped Temporal Attention, Temporal Bias, and Affinity-Weighting, respectively.
Figure 3: Hyperparameter ablation on the number of selected frames k and the affinity temperature τ . For k , we fix τ=0.05 and vary k from top-1 to all-frame aggregation. For τ , we fix k=4 and vary τ from 0.01 to 1.0. We report average SumR over four datasets.
Figure 4: Ablation study on the group size for grouped temporal attention. GFLOPs denotes the video encoding cost. We report the average SumR over the four datasets.
Figure 5: Comparison between top- k and threshold-based frame selection. The orange dashed lines indicate our final model, while the blue curves show threshold-based selection using different fixed affinity thresholds (x-axis). The y-axis denotes SumR.
Method
QV
TVR
ANet
Charades
MS-SL
226.4
41.2
96.5
30.6
GMMFormer
210.3
38.1
91.1
28.7
AMDNet
221.5
37.2
86.7
30.4
HLFormer
220.5
39.2
92.2
29.6
MSC
220.7
40.5
94.2
27.8
ProPy
275.5
102.5
150.3
46.0
Table 5: Cross-dataset generalization results. All methods are trained only on QVHighlights and evaluated on the other datasets. QV denotes the source-domain evaluation and we report SumR for all results.
Figure 6: Qualitative comparison of frame-level retrieval. Given a text query, we first perform video-level retrieval and consider examples where both ProPy and our method retrieve the correct video. We then rank individual frames and visualize the top-5 retrieved frames for each method.
Figure 7: Visualization of frame-transition score for frames (4th-22nd frames). Orange and blue curves correspond to ProPy and Ours, respectively.
Dataset
Method
R@1
R@5
R@10
TVR
ProPy
6.95
16.26
21.63
ITA
11.90
23.95
31.37
ANet
ProPy
5.61
13.03
18.31
ITA
9.86
18.99
24.64
Charades
ProPy
0.67
2.80
4.14
ITA
2.12
4.70
6.75
Table 6: Frame-level retrieval using GT moments.
Method
Trainable
GFLOPs
GFLOPs
Avg.
Params (M) ↓
(video) ↓
(online) ↓
SumR
MS-SL
2.70
282.8
10.285
166.8
GMM-V2
30.11
284.7
6.475
172.0
MSC
30.24
284.0
6.289
169.8
ProPy
7.98
370.0
6.107
187.7
ITA
1.57
283.7
5.490
198.9
Table 7: Efficiency comparison with existing PRVR methods. GFLOPs (video) denotes the offline cost of encoding a single video. GFLOPs (online) denotes the online retrieval cost when visual embeddings of 10,000 database videos are pre-extracted and stored.
School of Software Engineering, Xi’an Jiaotong University · Faculty of Computer Science, Electrical Engineering and Information Technology, Universität Stuttgart