Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval

Jun Li; Xuhang Lou; Jinpeng Wang; Yuting Wang; Yaowei Wang; Shu-Tao Xia; Bin Chen

cs.CVApr 4, 2026

Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval

Authors: Jun Li, Xuhang Lou, Jinpeng Wang, Yuting Wang, Yaowei Wang, Shu-Tao Xia, Bin Chen

Organizations: Harbin Institute of Technology, Shenzhen · Tsinghua Shenzhen International Graduate School, Tsinghua University · Peng Cheng Laboratory

Abstract

Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos based on text queries that describe only partial events. Existing methods suffer from incomplete global contextual perception, struggling with query ambiguity and local noise induced by spurious responses. To address these issues, we propose DreamPRVR, which adopts a coarse-to-fine representation learning paradigm. The model first generates global contextual semantic registers as coarse-grained highlights spanning the entire video and then concentrates on fine-grained similarity optimization for precise cross-modal matching. Concretely, these registers are generated by initializing from the video-centric distribution produced by a probabilistic variational sampler and then iteratively refined via a text-supervised truncated diffusion model. During this process, textual semantic structure learning constructs a well-formed textual latent space, enhancing the reliability of global perception. The registers are then adaptively fused with video tokens through register-augmented Gaussian attention blocks, enabling context-aware feature learning. Extensive experiments show that DreamPRVR outperforms state-of-the-art methods. Code is released at https://github.com/lijun2005/CVPR26-DreamPRVR.

Figures & tables

Figure 1: (a) Limited contextual awareness causes spurious local spikes on a globally irrelevant “people boating” video, incorrectly outscoring the ground-truth “demonstrating the accordion through the camera” video which exhibits more overall relevance. (b) MIL considers only the closest pair, leading to sparse supervision where others remain undertrained. (c) DreamPRVR first imagines global registers through text-supervised diffusion, then concentrates on fine-grained learning, thereby realizing coarse-to-fine cross-modal alignment and jointly optimizing all video tokens to form a coherent embedding space.

Figure 2: Overview of Dream PRVR . ( a ) The query branch produces embedding $\bm{q}$ and samples $\hat{\bm{q}}$ via TPS to supervise register generation. Video embeddings from a pre-trained model are first processed by a lightweight feature encoder and a probabilistic variational sampler (PVS) to produce the initial register ${\bm{r}}_{T}$ , which is iteratively denoised via a truncated diffusion model to yield optimal registers ${\bm{r}}_{0}$ . ${\bm{r}}_{0}$ subsequently enhance frame- and clip-level representation learning, getting frame embeddings $\bm{V}_{f}$ and clip embeddings $\bm{V}_{c}$ . $\bm{q}$ learns a latent semantic structure through $L_{\text{tssl}}$ and computes similarity scores $S_{f}$ and $S_{c}$ . ( b ) Textual Perturbation Sampler (TPS) models query uncertainty via controllable perturbations and samples $\hat{\bm{q}}$ without trainable parameters. ( c ) Textual Semantic Structure Learning $L_{\text{tssl}}$ employs $L_{\text{div}}$ to diversify queries and $L_{\text{qsp}}$ to align queries from the same video while contrasting across videos. ( d ) The asymmetric attention mask defines two cross-attention patterns, enabling full interactions for video tokens while constraining registers to video-only attention.

Model	ActivityNet Captions					Charades-STA					TVR
Model	R@1	R@5	R@10	R@100	SumR	R@1	R@5	R@10	R@100	SumR	R@1	R@5	R@10	R@100	SumR
Text-to-Video Retrieval (T2VR) Models
RIVRL [ 13 ]	5.2	18.0	28.2	66.4	117.8	1.6	5.6	9.4	37.7	54.3	9.4	23.4	32.2	70.6	135.6
DE++ [ 12 ]	5.3	18.4	29.2	68.0	121.0	1.7	5.6	9.6	37.1	54.1	8.8	21.9	30.2	67.4	128.3
CLIP4Clip [ 42 ]	5.9	19.3	30.4	71.6	127.3	1.8	6.5	10.9	44.2	63.4	9.9	24.3	34.3	72.5	141.0
Cap4Video [ 74 ]	6.3	20.4	30.9	72.6	130.2	1.9	6.7	11.3	45.0	65.0	10.3	26.4	36.8	74.0	147.5

Table 1: Retrieval performance comparison across three standard benchmarks. Models are ranked in ascending order of SumR scores on ActivityNet Captions. State-of-the-art performance is highlighted in bold , while “-” denotes unavailable results.

Model	Train time/epoch (ms)	Model parameters (M)	Inference time (ms)	Retrieval time (ms)	SumR
GMMFormer	26887	12.85	2876	3238	72.9
HLFormer	31463	28.43	3816	3655	78.7
GMMFormerV2	38004	30.79	3843	3688	78.2
Dream PRVR	33609	36.14	4001	3686	80.0

Table 2: Training and Inference Efficiency. Inference time denotes feature extraction for 1334 videos in the evaluation sets, while retrieval time accounts for the total time of query encoding, similarity computation and ranking. Dataset: Charades-STA.

ID	Model	ActivityNet Captions					Charades-STA					TVR
ID	Model	R@1	R@5	R@10	R@100	SumR	R@1	R@5	R@10	R@100	SumR	R@1	R@5	R@10	R@100	SumR
(0)	Dream PRVR (full)	8.7	27.5	40.3	79.5	156.1	2.6	8.7	14.5	54.2	80.0	17.4	39.0	50.4	86.2	193.1
Efficacy of Register Generation Strategy
(1)	$w/o$ registers	8.5	26.6	39.5	78.8	153.4	2.4	7.9	13.6	52.9	76.8	15.7	36.6	48.7	86.0	187.0
(2)	$w/$ AP	8.5	26.2	39.3	77.9	151.9	2.6	8.6	14.4	52.6	78.1	17.0	38.3	49.6	86.4	191.4
(3)	$w/o$ DRE	8.2	25.7	38.7	78.0	150.6	2.4	8.5	14.8	52.6	78.3	16.8	38.2	49.6	86.3	190.8

Table 3: Ablation Study of Dream PRVR . The best scores are marked in bold .

Panel of Figure 3 — Figure 3: The influence of the number of registers and the number of diffusion timesteps, with default settings marked in bold.

Panel of Figure 4 — Figure 4: The t-SNE visualization of the learned textual space. Data points of the same color denote queries from the same video.

Figure 5: A qualitative case study of retrieval results. The same videos in Fig. 1 are selected for a better comparison.

Panel of Figure 6 — Figure 6: The t-SNE visualization of the iterative generation process of global registers. $T$ = 0 indicates the initialization while $T$ = 10 represents the final results. 16 registers are trained for better clarity and data points of the same color correspond to the same video.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7: (a) Illustration of the proposed Diffusion Register Estimator Block (DRE). $\text{Embed}(t)$ , $\bm{q}_{T}$ and $\bm{c}$ denote the temporal embedding, the latent embedding corrupted by $t$ -step noise and the guided condition, respectively. (b) Condition generator for DRE.

Cross-Modal Retrieval

Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.

See all 184 papers

TF-PRVR: Training-Free Partially Relevant Video Retrieval
Oct 6, 2026Giyeol Kim, Chanho EomInformation Retrieval Vision-Language Retrieval
Chung-Ang University
Concentrate After Imagination: Text-Conditioned Evidence Grounding for Partially Relevant Video Retrieval
Sep 8, 2026Shuaiqi Cheng, Siyu You, Yanbi Wu +3Temporal Video Grounding Information Retrieval
1The Hong Kong University of Science and Technology (Guangzhou) · University of Electronic Science and Technology of China · 3The Hong Kong University of Science and Technology
Knowledge-Refined Dual Context-Aware Network for Partially Relevant Video Retrieval
Mar 25, 2026Junkai Yang, Qirui Wang, Yaoqing Jin +3Information Retrieval Temporal Video Grounding
School of Software Engineering, Xi’an Jiaotong University · Faculty of Computer Science, Electrical Engineering and Information Technology, Universität Stuttgart

Oct 6, 2026J/K move · Enter open · S save

TF-PRVR: Training-Free Partially Relevant Video Retrieval

Giyeol Kim, Chanho Eom

Chung-Ang University

Information Retrieval Vision-Language Retrieval Video Retrieval

Open paper AskPDF arXiv

Partially Relevant Video Retrieval (PRVR) aims to retrieve untrimmed videos containing moments relevant to a given text query. Despite recent progress, existing PRVR methods suffer from two key limitations: a fixed video decomposition scheme that causes semantic dilution, and source-domain overfitting induced by task-specific training. In this paper, we propose TF-PRVR, the first training-free framework for PRVR. TF-PRVR leverages frozen vision-language features to construct video-specific hierarchical representations. It derives temporal semantic signals from frame-level features and applies frequency-based multi-scale analysis to identify adaptive temporal boundaries, producing hierarchical segments with coherent event-level semantics. Built on these segments, TF-PRVR constructs a unified multi-scale graph and propagates query relevance across temporally and semantically related nodes. A moment-aware scoring strategy then aggregates temporally aligned relevance across scales, emphasizing consistently supported moments while suppressing isolated false responses. Without task-specific training, TF-PRVR preserves the general-purpose alignment capability of pre-trained vision-language models and avoids dataset-specific overfitting. Extensive experiments demonstrate consistent performance across datasets with diverse visual and temporal characteristics, suggesting a practical direction for training-free PRVR.

Abstract

Cross-Modal Retrieval

Explore similar work

TF-PRVR: Training-Free Partially Relevant Video Retrieval

Concentrate After Imagination: Text-Conditioned Evidence Grounding for Partially Relevant Video Retrieval

Knowledge-Refined Dual Context-Aware Network for Partially Relevant Video Retrieval