VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval

Shahaf Wagner; Gabriele Serussi; Dan Ben Ami; Tomer Galanti; Chaim Baskin

cs.CVOct 8, 2026

VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval

Authors: Shahaf Wagner, Gabriele Serussi, Dan Ben Ami, Tomer Galanti, Chaim Baskin

Organizations: INSIGHT Lab, Ben-Gurion University of the Negev, Israel · Decart AI · Texas A&M University, College Station

Abstract

Finding the right video often requires distinguishing similar scenes in which different events occur. Joint matching improves retrieval, but processing rich video representations for each query is costly. VEDJE compresses features within sampled frames while keeping their representations separate in a reusable cache. Feature-change prediction supplies an auxiliary training signal that improves retrieval from the compressed cache without adding work at query time. On MSR-VTT, MSVD, DiDeMo, and ActivityNet, VEDJE improves R@1 over matched first-stage retrievers in both retrieval directions. On MSR-VTT, it reaches 59.8 text-to-video R@1 with a fine-tuned VideoCLIP-XL first stage. In the VideoPrism configuration, shrinking the per-video cache fourfold to 12 KiB preserves text-to-video recall within 0.2 points. These results show that accurate video search can operate on compact evidence, encoded once and reused as new queries arrive.

Figures & tables

Figure 1: MSR-VTT T2V R@1 (%) versus online parameter count. VEDJE raises fine-tuned VideoCLIP-XL retrieval from 56.2 to 59.8 R@1 with a 33M-parameter cached reranker, bringing the total online footprint to approximately 157M parameters. Published systems retain their reported backbones and training protocols. Counts cover online text encoding and scoring; MLLM points count the base decoder, and Video-ColBERT counts its full text module. Appendix B details accounting and sources; Section 4.7 reports complete-query latency.

Figure 2: VEDJE pipeline. (A) Offline indexing: compression of frame-indexed features yields the ordered cache $Z(v)=[Z_{1};\dots;Z_{T}]$ . (B) Training only: a future-delta head $F_{\omega}$ predicts feature changes from $Z_{t}$ and is discarded after training. (C) Online reranking: a 33M joint encoder processes $(q,Z(v))$ ; the embedded stage-1 score is added to its output before the nonlinear score head. Indexed videos require no query-time visual encoding.

(a) Published systems
Method	T2V R@1	V2T R@1
CLIP4Clip B/32 ( Luo et al., 2022 )	44.5	43.1
X-CLIP B/16 ( Ma et al., 2022 )	49.3	48.9
Video-ColBERT (SigLIP-B/16) ( Reddy et al., 2025 )	51.5	–
CrossTVR-Large ( Dai et al., 2025 )	54.0	51.3
UMT-L (ViT-L/16) ( Li et al., 2023b )	58.8	58.6

Table 1: MSR-VTT retrieval accuracy. R@1 is in percent; gains are percentage points. (a) Source protocols are retained; UMT-L and InternVideo2 report fine-tuned results; UMT-L uses 25M pretraining pairs, and CLIP4Clip reports the best variant per direction. LamRA uses the VideoChat-Flash reproduction of Lee et al. (2025) . (b) Matched reranking uses a 64-token cache from the corresponding backbone; fine-tuned first-stage results are our reproduction. ZS/FT denote zero-shot/fine-tuned first stages; dashes denote unreported results. Full results and uncertainty appear in Appendix A.8 .

Method	Backbone treatment	T2V R@1	V2T R@1
CLIP4Clip ( Luo et al., 2022 )	fine-tuned	46.4	45.4
X-CLIP ( Ma et al., 2022 )	fine-tuned	49.3	48.9
EERCF ( Tian et al., 2024a )	fine-tuned	49.9	47.8
VEDJE-CLIP	frozen	49.8	49.6

Table 2: Shared CLIP ViT-B/16 architecture on MSR-VTT. R@1 (%). Published baselines fine-tune CLIP. VEDJE keeps CLIP fixed and trains its compressor and reranker, using CLIP features and contrastive targets. CLIP4Clip values are the reproduction of Tian et al. (2024a) .

Tokens	KiB/video	Direction	Without $\mathcal{L}_{\mathrm{delta}}$	With $\mathcal{L}_{\mathrm{delta}}$	Gain
64	48	T2V	$54.0\pm 0.1$	$\mathbf{54.6}\pm 0.1$	+0.6
64	48	V2T	$53.6\pm 0.1$	$\mathbf{54.6}\pm 0.1$	+1.0
16	12	T2V	$52.5\pm 0.1$	$\mathbf{54.4}\pm 0.1$	+1.9
16	12	V2T	$52.6\pm 0.1$	$\mathbf{53.1}\pm 0.1$	+0.5

Table 3: Cache size and feature-change supervision. MSR-VTT R@1 (%) is mean

\pm

standard deviation over three seeds. Values are rounded to one decimal, with a minimum displayed SD of 0.1. Storage is BF16 tensor payload per video; gains are percentage points.

Cache construction	$\mathcal{L}_{\mathrm{delta}}$	T2V R@1	V2T R@1
FrameSet-EDJE: mean pooling	No	$51.7\pm 0.2$	$51.6\pm 0.2$
FrameSet-EDJE: attention pooling	No	$52.0\pm 0.2$	$51.6\pm 0.2$
VEDJE: separate frame groups	No	$54.0\pm 0.1$	$53.6\pm 0.1$
VEDJE: separate frame groups	Yes	$\mathbf{54.6}\pm 0.1$	$\mathbf{54.6}\pm 0.1$

Table 4: Cache structure at a fixed storage budget. MSR-VTT R@1 (%) is mean

\pm

standard deviation over three training seeds. The multiframe variants share VideoPrism candidates, the first-stage score input, the 33M reranker, and a 64-token BF16 cache. Pooling combines evidence across frames; VEDJE retains separate frame groups.

Scoring rule	T2V R@1
Stage 1: VideoPrism	50.1
Trained MaxSim with residual prior	51.9
Joint encoder with residual prior (VEDJE)	54.6

Table 5: Scoring the same cache on MSR-VTT. T2V R@1 (%) with VP stage 1 and 20 candidates. Reranking rows share the 64-token cache and candidate pool; trained MaxSim uses the same data and schedule. Appendix A.7 gives the protocols.

	Latency (ms)		T2V R@1 (%)
Candidates $N_{\mathrm{c}}$	64 tokens	16 tokens	64 tokens	16 tokens
20 (default)	5.79	5.82	54.6	54.4
50	6.21	6.12	53.9	54.7
100	7.83	6.33	54.4	54.6
200	10.24	6.88	54.3	54.7

Table 6: Complete-query cost at two cache sizes. MSR-VTT T2V with VP stage 1, host-cache I/O, one L40S, and query batch size 1. Latency is the median of 200 runs after 30 warmups. R@1 at 20 candidates is a three-seed mean; other recalls are single-run evaluations.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

First-stage model	Cache features	VTC video targets
VideoPrism-LvT-B	VideoPrism-Base	VideoPrism-LvT-B
VideoCLIP-XL (ZS)	VideoCLIP-XL	VideoCLIP-XL
VideoCLIP-XL (FT)	VideoCLIP-XL	VideoCLIP-XL
PE-Core-B	PE-Core-B	PE-Core-B
CLIP ViT-B/16 (ZS)	CLIP ViT-B/16	CLIP ViT-B/16

Appendix

Table 7: Backbone sources for the main variants and the CLIP ViT-B/16 control. Each row identifies the first-stage model, the source of cached visual features, and the source of frozen video embeddings for VTC. ZS and FT describe first-stage training.

Cache path	$N_{\mathrm{c}}{=}20$	$N_{\mathrm{c}}{=}100$
Host RAM	5.79	7.83
NVMe, warm	6.3	7.8
NVMe, cold	7.4	11.1

Appendix

Table 8: VP-backed MSR-VTT T2V latency for 64-token caches. Values are milliseconds, measured end to end under host-RAM and batched NVMe reads; warm and cold denote the reported cache-residency conditions.

Representation	Visual payload per video
Uncompressed VideoPrism frame-and-patch representation	6.02 MiB
VEDJE, 64 tokens	48 KiB
VEDJE, 16 tokens	12 KiB

Appendix

Table 9: Per-video visual storage at two bytes per element. The full-feature reference contains

16\times 257\times 768

elements; VEDJE contains

64\times 384

16\times 384

. Values count tensor payload. The first-stage search index and file metadata are accounted for separately.

Method	T2V R@1
Released coarse checkpoint (reproduced)	42.8
EERCF published	47.8
VEDJE on released-checkpoint candidates	49.1

Appendix

Table 10: MSR-VTT T2V using the released EERCF CLIP ViT-B/32 candidate generator (

N_{\mathrm{c}}{=}50

). Stage 1 and VEDJE are evaluated here; the EERCF score is the published reference.

Variant	Joint tokens?	Learned prior?	T2V R@1	V2T R@1
Stage-1 baseline	✗	✗	50.1	49.8
Prior stream only	✗	✓	50.1	49.8
Joint stream only	✓	✗	43.8	41.6
Full VEDJE	✓	✓	54.6	54.6

Appendix

Table 11: Stage-1 prior controls on MSR-VTT 1ka with fixed candidates. Prior-only and joint-only models are trained independently using their respective inputs. The full model combines the prior with joint text-video features.

VideoPrism first stage
	Text-to-video			Video-to-text
Dataset	Stage 1	VEDJE	Gain	Stage 1	VEDJE	Gain
MSR-VTT	50.1	54.6	+4.5	49.8	54.6	+4.8
MSVD	55.6	56.7	+1.1	83.4	83.7	+0.3
DiDeMo	47.1	48.2	+1.1	47.3	47.9	+0.6
ActivityNet	48.8	50.6	+1.8	47.9	48.3	+0.4

Appendix

Table 12: Cross-dataset retrieval before and after VEDJE. R@1 is in percent; gains are in percentage points. Both panels use 64-token caches. Full retrieval results appear in Table 15 .

Dataset	Dir.	Stage-1 R@1	Reranked R@1	Gain	Reranked R@5	Reranked R@10
VideoPrism-LvT-B ( Zhao et al., 2024 ) stage 1 $\rightarrow$ $\text{VEDJE{}}_{\mathrm{VP}}$
MSR-VTT	T2V	50.1	54.6	+4.5	76.9	85.4
MSR-VTT	V2T	49.8	54.6	+4.8	79.3	86.7
MSVD	T2V	55.6	56.7	+1.1	83.7	89.8
MSVD	V2T	83.4	83.7	+0.3	96.0	98.7
DiDeMo	T2V	47.1	48.2	+1.1	72.9	80.3

Appendix

Table 15: Matched first-stage and cached-reranker retrieval. Recall@

K

is in percent; gains are absolute R@1 changes in percentage points. Each panel fixes the candidate generator (named by the VEDJE subscript) and reranker. On MSR-VTT, three-seed T2V R@1 is

54.6\pm 0.1

for VP and

56.5\pm 0.2

for VCLIP (mean

\pm

standard deviation). VideoPrism V2T gains on MSVD, DiDeMo, and ActivityNet were evaluated across independent retrainings. FT denotes a fine-tuned first stage.

Figure 3: T2V R@1 (%) versus candidate-pool size. Each panel uses the same linear recall axis starting at 40%. Hollow markers at 20 candidates repeat main results; filled markers at other candidate counts are single-run evaluations. Dotted lines guide the eye.

Precision	T2V R@1	T2V R@5	T2V R@10	V2T R@1	V2T R@5	V2T R@10
BF16 reference	54.4	78.9	85.6	53.1	78.4	86.5
FP8-E4M3	54.4	78.9	85.5	53.2	78.4	86.5
FP4-E2M1	54.0	78.8	85.5	53.2	78.4	86.5

Appendix

Table 16: MSR-VTT 1ka Recall@K for 16 cached tokens under different cache precisions, in percent. All rows use one cached token per sampled frame.

Variant	T2V R@1	T2V R@5	T2V R@10	V2T R@1	V2T R@5	V2T R@10
16 tokens w/o $\mathcal{L}_{\mathrm{delta}}$	52.5	78.1	84.5	52.6	77.8	85.7
16 tokens with $\mathcal{L}_{\mathrm{delta}}$	54.4	78.9	85.6	53.1	78.4	86.5

Appendix

Table 17: MSR-VTT 1ka Recall@K for 16 cached tokens with and without

\mathcal{L}_{\mathrm{delta}}

, in percent. All rows use one cached token per sampled frame.

Recall	Current-frame reconstruction $X_{t}$	Absolute future feature $X_{t+h}$	Future delta $\Delta_{t,h}$ (default)
T2V R@1	$54.2$	52.8	$\mathbf{54.6}$
T2V R@5	$75.7$	78.2	76.9
T2V R@10	$84.8$	85.3	85.4
V2T R@1	$53.9$	53.8	$\mathbf{54.6}$
V2T R@5	$77.9$	78.6	79.3
V2T R@10	$85.8$	86.5	86.7

Appendix

Table 18: Auxiliary prediction targets with 64 cached tokens on MSR-VTT 1ka. Recall values are percentages. All reconstruction recalls and delta R@1 are mean recall values.

Active training objectives	T2V R@1	T2V R@5	T2V R@10	V2T R@1	V2T R@5	V2T R@10
Stage-1 only ( Zhao et al., 2024 ) No reranker losses	50.1	–	–	49.8	–	–
$\mathcal{L}_{\mathrm{vtm}}$	52.8	76.9	84.0	50.6	77.4	85.2
$\mathcal{L}_{\mathrm{vtm}}+\mathcal{L}_{\mathrm{mlm}}$	52.7	76.2	84.2	51.4	76.0	84.9
$\mathcal{L}_{\mathrm{vtm}}+\mathcal{L}_{\mathrm{mlm}}+\mathcal{L}_{\mathrm{vtc}}$	54.0	77.8	85.3	53.6	77.9	87.0
Full: $\mathcal{L}_{\mathrm{vtm}}+\mathcal{L}_{\mathrm{mlm}}+\mathcal{L}_{\mathrm{vtc}}+\mathcal{L}_{\mathrm{delta}}$	54.6	76.9	85.4	54.6	79.3	86.7

Appendix

Table 19: VideoPrism ( Zhao et al., 2024 ) loss-stack ablation on MSR-VTT 1ka, with Recall@

K

in percent. The stage-1 row reports the VideoPrism retriever before reranking. Reranker rows add the training objectives in stages; the full VEDJE row adds

\mathcal{L}_{\mathrm{delta}}

on top of VTM, MLM, and VTC.

Horizon	T2V R@1	T2V R@5	T2V R@10	V2T R@1	V2T R@5	V2T R@10
$h=2$	53.8	78.2	85.5	53.0	78.7	86.6
$h=3$ (default)	54.6	76.9	85.4	54.6	79.3	86.7
$h=4$	54.4	78.7	85.6	53.0	78.2	86.9

Appendix

Table 20: Future-delta horizon ablation on MSR-VTT 1ka. Recall values are percentages; the no-auxiliary reference in Table 19 has T2V/V2T R@1 of 54.0/53.6.

Variant	Added reranker params	T2V R@1	T2V R@5	T2V R@10	V2T R@1	V2T R@5	V2T R@10
Default VEDJE	33M	54.6	76.9	85.4	54.6	79.3	86.7
BERT-base text encoder ( Devlin et al., 2019 )	110M	52.9	77.5	85.5	54.0	77.9	86.5

Appendix

Table 21: MSR-VTT 1ka reranker-backbone control, with Recall@

K

in percent. The BERT-base ( Devlin et al., 2019 ) control raises the added reranker size from 33M to 110M parameters.

Text encoding and learned scoring
Method	Online parameters	R@1 (%)
CLIP4Clip B/32	63.4M	44.5
CLIP4Clip B/16 (reproduced)	63.4M	46.4
X-CLIP B/16	64.0M	49.3
EERCF B/16	63.7M	49.9
Video-ColBERT SigLIP-B/16	110.3M	51.5

Appendix

Table 22: MSR-VTT T2V R@1 and online parameters. Panels identify the counted modules. Public-architecture counts are rounded to 0.1M; VEDJE totals retain the reported whole-million estimates. Video-ColBERT counts the full SigLIP text module. MLLM rows count the base decoder; their pipelines also use first-stage query encoding and fine-tuning components. A dash denotes an unestablished count or inapplicable entry. ZS/FT denote zero-shot/fine-tuned stage 1; parentheses give matched gains.

Explore similar work

VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement
Jul 1, 2026Seohyun Lee, Seoung Choi, Dohwan Ko +2Temporal Video Grounding Temporal IR
KAIST, Daejeon, Republic of Korea · Korea University, Seoul, Republic of Korea
R^3: Composed Video Retrieval via Reasoning-Guided Recalling and Re-ranking
May 31, 2026Zixu Li, Yupeng Hu, Zhiheng Fu +3Cross-Modal Retrieval Composed Video Retrieval
1Shandong University · 2Harbin Institute of Technology (Shenzhen)
Concentrate After Imagination: Text-Conditioned Evidence Grounding for Partially Relevant Video Retrieval
Sep 8, 2026Shuaiqi Cheng, Siyu You, Yanbi Wu +3Temporal Video Grounding Information Retrieval
1The Hong Kong University of Science and Technology (Guangzhou) · University of Electronic Science and Technology of China · 3The Hong Kong University of Science and Technology

Jul 1, 2026, cs.CVJ/K move · Enter open · S save

VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

Seohyun Lee, Seoung Choi, Dohwan Ko, Jongha Kim+1

KAIST, Daejeon, Republic of Korea · Korea University, Seoul, Republic of Korea

Temporal Video Grounding Temporal IR Query Reformulation Information Retrieval Video Retrieval

Open paper AskPDF arXiv

As video corpora continue to expand in both scale and task complexity, there is increasing demand for approaches that retrieve relevant videos from large-scale corpora (inter-video reasoning) and subsequently perform fine-grained, query-conditioned tasks (intra-video reasoning) within the retrieved content, such as temporal grounding. However, existing approaches typically treat retrieval as a preprocessing step, and consequently, when the initial retrieval fails, there is no mechanism to refine the search, leading to the failure of subsequent fine-grained intra-video reasoning. Moreover, while recent agentic frameworks have advanced video understanding, they typically assume that the query-relevant video is already given, focusing exclusively on intra-video reasoning tasks. To address these limitations, we propose VideoSearch-R1, an agentic framework for iterative video retrieval and reasoning through multi-turn interaction with a video search engine. Specifically, we introduce Soft Query Refinement (SQR) to refine search query tokens in a continuous latent space rather than rewriting queries in the discrete text space, enabling more efficient and fine-grained adjustments. SQR and its reasoning process are trained using Group Relative Policy Optimization (GRPO), guided by task-level reward signals derived from retrieval and downstream tasks. Building upon this, VideoSearch-R1 achieves state-of-the-art performance across three datasets on Video Corpus Moment Retrieval (VCMR), iteratively retrieving videos from large-scale corpora, refining search queries, and performing precise query-conditioned temporal grounding within the retrieved content. Our analyses show that SQR effectively refines the original query, requiring significantly fewer generated tokens than explicit text-level query refinement. Code and model checkpoints are publicly available at mlvlab.github.io/VideoSearch-R1.

Abstract

Explore similar work

VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

R^3: Composed Video Retrieval via Reasoning-Guided Recalling and Re-ranking

Concentrate After Imagination: Text-Conditioned Evidence Grounding for Partially Relevant Video Retrieval