CoVR-R:Reason-Aware Composed Video Retrieval

Omkar Thawakar; Dmitry Demidov; Vaishnav Potlapalli; Sai Prasanna Teja Reddy Bogireddy; Viswanatha Reddy Gajjala; Alaa Mostafa Lasheen; Rao Muhammad Anwer; Fahad Khan

cs.CVMar 20, 2026

CoVR-R:Reason-Aware Composed Video Retrieval

Authors: Omkar Thawakar, Dmitry Demidov, Vaishnav Potlapalli, Sai Prasanna Teja Reddy Bogireddy, Viswanatha Reddy Gajjala, Alaa Mostafa Lasheen, Rao Muhammad Anwer, Fahad Khan

Organizations: Mohamed bin Zayed University of AI · University of Chicago · University of Wisconsin-Madison · Linköping University

Abstract

Composed Video Retrieval (CoVR) aims to find a target video given a reference video and a textual modification. Prior work assumes the modification text fully specifies the visual changes, overlooking after-effects and implicit consequences (e.g., motion, state transitions, viewpoint or duration cues) that emerge from the edit. We argue that successful CoVR requires reasoning about these after-effects. We introduce a reasoning-first, zero-shot approach that leverages large multimodal models to (i) infer causal and temporal consequences implied by the edit, and (ii) align the resulting reasoned queries to candidate videos without task-specific finetuning. To evaluate reasoning in CoVR, we also propose CoVR-Reason, a benchmark that pairs each (reference, edit, target) triplet with structured internal reasoning traces and challenging distractors that require predicting after-effects rather than keyword matching. Experiments show that our zero-shot method outperforms strong retrieval baselines on recall at K and particularly excels on implicit-effect subsets. Our automatic and human analysis confirm higher step consistency and effect factuality in our retrieved results. Our findings show that incorporating reasoning into general-purpose multimodal models enables effective CoVR by explicitly accounting for causal and temporal after-effects. This reduces dependence on task-specific supervision, improves generalization to challenging implicit-effect cases, and enhances interpretability of retrieval outcomes. These results point toward a scalable and principled framework for explainable video search. The model, code, and benchmark are available at https://github.com/mbzuai-oryx/CoVR-R.

Figures & tables

Figure 1: Why reasoning is needed for CoVR . Each row pairs a reference video and edit text with the desired target (highlighted in green), where success depends on after-effects rather than keyword overlap. Examples include: shifting from a brown cow to a solitary brown horse in rolling hills with trees (state and scene changes); switching to a close-up of four hands on a piano (cinematography and object configuration); moving from typing to clenched fists and closing the laptop (temporal phase progression). These cases require inferring consequences such as state transitions, phase order, and shot scale that not merely matching words. Reasoning bridges what the edit says and what the target must show, capturing the visual consequences that define correct retrieval. More examples are in suppl. material.

Figure 2: Data samples from our CoVR-R benchmark . Each row shows an input video, its edit text, the target video, and a concise reasoning trace that makes the implied after-effects explicit (e.g., state/scene shifts, camera/shot changes, temporal phases). Examples illustrate: nighttime aerial → ground-level sunset with silhouetted floodlights and palm trees; lake/fjord → calm sea with sunlit grassy field and sheep activity; couple → solitary woman facing the ocean with wind-blown hair under partly cloudy sky. These annotations ground what the edit implies (not just what it says) and guide evaluation of reasoning-aware retrieval.

Figure 3: Overview of our Reason–then–Retrieve architecture . Given reference video $V_{r}$ and edit text $E$ , Stage-1 uses Qwen3-VL-8B to generate an after-effect reasoning trace $R$ predicting concrete consequences (actions/phases, camera/shot, states, scene, tempo) conditioned on $V_{r}$ and $E$ . Stage–2 converts $(V_{r},E,R)$ into an effect-aware query embedding $\mathbf{q}(V_{r},E,R)$ and performs cosine retrieval against precomputed gallery embeddings ${\mathbf{v}(V)}_{V\in\mathcal{G}}$ via weighted temporal pooling. This reasoning-first pipeline delivers zero-shot CoVR by turning edits into explicit consequences for matching.

		Using Video	Modification			Recall@K				Reasoning
	Model	Captions	Text Fusion	Backbone	Frames	R@1	R@5	R@10	R@50	score
1	CoVR-BLIP [ 29 ]	✘	Avg	BLIP	$15$	$30.30$	$51.07$	$57.05$	$73.82$	$4.85\pm 0.12$
2	Thawakar et al. [ 25 ]	✔	Avg	BLIP	$15$	$31.71$	$52.18$	$59.53$	$74.67$	$5.62\pm 0.11$
3	BSE-CoVR [ 23 ]	✔	Avg	BLIP	$15$	$33.43$	$53.88$	$61.31$	$77.32$	$6.05\pm 0.11$
4	CoVR-BLIP [ 29 ]	✘	CA	BLIP	$15$	$28.06$	$47.80$	$55.93$	$73.76$	$5.10\pm 0.13$
5	Thawakar et al. [ 25 ]	✔	CA	BLIP	$15$	$30.90$	$50.63$	$60.11$	$75.33$	$5.88\pm 0.12$

Table 1: Zero-shot comparison on CoVR-R . Columns show video caption usage, modification text fusion strategy (Avg vs. CA = cross-attention vs. SA = self-attention), backbone model, frames per clip, Recall@K, and Reasoning score (

0–10

; mean

\pm

s.e.m.). MVFT-JI † denotes our video adaptation of [ 26 ] for fair comparison. Among BLIP-based baselines, stronger fusion and video modeling improve both retrieval and reasoning. Our approach with Qwen3-VL-8B achieves the best zero-shot performance, with substantial gains in R@1–R@50 and reasoning scores without requiring video captions. Our Approach+R further improves results through explicit reasoning.

		Using Video		Modification			Recall@K
	Model	Captions	Input Modalities	Text Fusion	Backbone	Frames	R@1	R@5	R@10	R@50
1	Random		-	-	-	-	$0.04$	$0.21$	$0.32$	$1.46$
2	CoVR-BLIP [ 29 ]	✘	Text	-	BLIP	-	$24.12$	$56.02$	$60.16$	$82.34$
3	CoVR-BLIP [ 29 ]	✘	Visual	-	BLIP	$15$	$22.52$	$53.08$	$58.34$	$81.26$
4	CoVR-BLIP [ 29 ]	✘	Visual + Text	Avg	BLIP	$15$	$38.44$	$64.96$	$71.72$	$87.12$
5	Thawakar . et.al. [ 25 ]	✔	Visual + Text	Avg	BLIP	$15$	$40.23$	$66.38$	$74.84$	$88.12$

Table 2: Results on Dense-WebVid-CoVR test set. Our approach with Qwen3-VL-8B (rows

10-11

) outperforms all prior baselines. Row 10 achieves

61.21

% R@1 and

97.61

% R@50, surpassing the strongest baseline BSE-CoVR (row 9:

48.08

% R@1,

93.78

% R@50) by

+13.13

and

+3.83

points. Row 11 (+R) incorporates explicit reasoning traces. ✔: uses video captions; ✘: caption-free. CA: Cross-Attention;SA: Self Attention; Avg: Average pooling. Best results in bold .

Method	R@1	R@5	R@10	R@50	Reasoning Score
Qwen3-VL-8b	$44.32$	$61.91$	$67.33$	$79.90$	–
$+$ Reasoning	$49.88$	$66.99$	$72.97$	$85.14$	$8.31$ ± $0.0980$
$+$ Iterative Refinement	$50.56$	$74.03$	$81.24$	$92.17$	$8.31$ ± $0.1011$

Table 3: Effect of refined reasoning on retrieval .Starting from Qwen3-VL-8B, adding our Reasoning module lifts retrieval (R@1

44.32\%

\rightarrow

49.88\%

) and improves the Reasoning Score. Iterative Refinement further boosts performance (R@1=

50.56\%

) while maintaining reasoning quality (

\sim

8.31), but incurs 5

\times

inference cost. Our final model uses single-pass reasoning.

Strategy	R@1	R@5	R@10	R@50	R ${}_{\text{mean}}$
Last	$1.51$	$3.57$	$4.64$	$10.14$	$3.24$
Mean $\oplus$ Last	$14.41$	$25.11$	$29.20$	$41.77$	$22.91$
Max	$35.95$	$52.02$	$72.05$	$93.98$	$48.78$
Mean $\oplus$ Max	$41.77$	$58.84$	$65.41$	$77.32$	$55.34$
Mean	$44.87$	$63.67$	$69.61$	$82.44$	$59.39$
Weighted	$49.88$	$66.99$	$72.9$	$85.14$	$63.28$

Table 4: Token aggregation strategies ( CoVR-R ). We evaluate pooling methods for aggregating token embeddings in our reasoning model. Importance weighting achieves the best performance.

Method	R@1	R@5	R@10	R@50	Reasoning Score
Qwen3-VL-4B	$43.98$	$61.89$	$68.27$	$80.69$	$7.95\pm 0.11$
Qwen3-VL-8B	$\mathbf{49.88}$	$\mathbf{66.99}$	$\mathbf{72.97}$	$\mathbf{85.14}$	$\mathbf{8.31\pm 0.098}$
Qwen3-VL-72B	$55.48$	$72.69$	$78.57$	$87.99$	$9.05\pm 0.09$
Qwen2.5-VL-7B	$47.56$	$64.08$	$69.84$	$82.33$	$8.06\pm 0.10$
Qwen2.5-VL-3B	$41.78$	$59.59$	$66.17$	$78.59$	$7.72\pm 0.11$

Table 5: Impact of model scaling . Recall@K and Reasoning Score (0–10; checklist satisfaction, mean ± s.e.m.) across Qwen variants. Performance improves with scale: Qwen3-VL-72B is best (R@1=

55.48\%

, R@50=

87.99\%

, Reasoning

9.05\pm 0.09

), while Qwen3-VL-8B is the strongest mid-size model (R@1=

49.88\%

). Overall, Qwen3 series outperforms comparable Qwen2.5 models on both retrieval and reasoning quality.

Figure A : The figure shows the middle frame from each reference and target video pair in the WebVid-CoVR dataset. The arrows indicate the intended modifications described by the text instructions. The targets reveal that the edit instructions are highly generic, making the composed video retrieval task ambiguous and challenging.

Method	R@1	R@5	R@10	R@50
CoVR-BLIP [ 29 ]	45.46	70.46	79.54	93.27
CoVR-BLIP-2 [ 29 ]	45.66	71.71	81.30	94.80
BSE-CoVR [ 23 ]	48.08	73.36	81.06	93.78
Our Approach	49.15	70.72	79.25	93.42

Table A : Comparison of our approach with existing methods on WebVid-CoVR test set . Our approach achieves the highest R@1 (49.15%), demonstrating strong performance on explicit modification instructions.

Strategy	Avg. Trace Length	R@1	R@5	R@10	R@50	Reasoning Score	Inference Time
Minimal (keywords only)	15 tokens	44.10	59.30	65.82	77.45	6.80 $\pm$ 0.13	1.0 $\times$
Compact (atomic assertions)	45 tokens	47.80	64.70	70.15	82.33	7.90 $\pm$ 0.11	1.3 $\times$
Standard (ours)	89 tokens	49.88	66.99	72.97	85.14	8.31 $\pm$ 0.098	1.8 $\times$
Verbose (detailed chains)	186 tokens	48.20	65.10	71.34	83.67	8.42 $\pm$ 0.095	3.2 $\times$

Table B : Impact of reasoning granularity on retrieval performance and efficiency . We evaluate four levels of reasoning detail on CoVR-R. Standard granularity (89 tokens) achieves the best balance between retrieval accuracy (R@K) and reasoning quality, while verbose traces (186 tokens) improve reasoning scores but degrade retrieval by introducing excessive detail that dilutes discriminative signals.

Figure B : Distribution of modification_text in CoVR-R. Mean: 128

\pm

23 words (range: 63-237). These detailed instructions specify not just what changes, but how transformations manifest visually.

Figure C : Distribution of reasoning_detailed in CoVR-R. Mean: 601

\pm

281 words (range: 197-6150). These comprehensive causal explanations are unique to our work and provide temporal dependencies, state transitions, and compositional relationships.

Figure D : Distribution of reasoning (brief summaries) in our dataset. Mean: 192

\pm

32 words (range: 1-372). These concise overviews bridge modification instructions and detailed reasoning.

Figure E : Word cloud visualization of the most frequent content terms in CoVR-R modification texts. Larger words indicate higher frequency and reflect the dataset’s diverse, multi-domain nature, with terms like hand , water , wooden , person , and kitchen highlighting rich contextual variety spanning everyday activities, object interactions, and environmental settings across domains.

Panel of Figure 16 — Figure F : The figure shows two successful retrieval examples where ground truth ranks first: airplane landing (top) and vegetable cooking (bottom) . Each example displays: detailed edit instruction in a text box with key concepts highlighted in red, (Row 1) input reference video, (subsequent rows) top-k retrieved predictions with green border indicating the correctly retrieved ground truth at Top-1. Notably, all top retrievals exhibit highly similar visual contexts, demonstrating the effectiveness of LLM-based reasoning in capturing semantic after-effects for composed video retrieval.

Panel of Figure 17 — Figure G : Example reasoning trace for the first success case (airplane landing in Figure F ). The trace explicitly predicts visual after-effects across multiple dimensions: environmental changes (sky, background), object dynamics (plane motion, smoke), and camera perspective—enabling precise target video matching.

Panel of Figure 18 — Figure H : Reasoning trace for the second failure case (bottom one in Figure I ). The trace captures broad transformations across multiple dimensions but loses critical details like exact sign text (”Crocker Pharmacy Palace”). This incomplete reasoning propagates through the pipeline, causing the generated target description to diverge from ground truth. This illustrates how hyper-specific instructions lead to compounding errors and semantic drift.

Panel of Figure 19 — Figure I : Two failure cases with incorrect Top-1 retrievals: two women smiling (top) and Crocker Pharmacy Palace sign (bottom). Each shows the edit instruction with key concepts highlighted in red, reference video (Row 1), and top-k predictions. Red borders indicate incorrect Top-1 retrievals; green borders indicate ground truth. Despite Top-1 errors, retrieved videos exhibit strong semantic alignment with instructions, demonstrating the model captures conceptual gist but loses the specificity needed for exact matching.

Composed Video Retrieval

Composed video retrieval finds a target video from a reference video and an edit instruction describing how it should change.

See all 13 papers

May 30, 2026

Reason, Retrieve, Re-rank: A Zero-Shot Reasoning-Aware Framework for Composed Video Retrieval

Composed Video Retrieval (CoVR) seeks the target video that results from applying a free-form textual modification to a reference video. We address the Reason-Aware CoVR (CoVR-R) challenge at the CVPR~2026 VidLLMs workshop, where retrieval is strictly zero-shot. We present R3-CoVR (Reason, Retrieve, Re-rank), a training-free pipeline built entirely from frozen foundation models. A multimodal large language model (Qwen3-VL-8B) reasons about the after-effects an edit implies -- state transitions, action phases, scene, camera and tempo -- and verbalises a concise post-edit description; a contrastive video--text encoder (SigLIP-2) embeds this description and the gallery for first-stage retrieval; finally a constraint-aware re-ranking stage uses the same multimodal model as a judge that scores each shortlisted candidate against the intended edited result. On the challenge test set, R3-CoVR attains 91.9% R@1 and 98.2% R@10. Two findings drive these results: (i)~matching the description length to the contrastive encoder's text window lifts \Rk{1} from

67.5

72.7

; and (ii)~the constraint-aware re-ranker, which reorders only the shortlist, lifts \Rk{1} from

72.7

91.9

-- the single largest gain. We analyse the re-ranker's behaviour, the retrieve/re-rank blend, and the shortlist depth, and we release a clean three-layer implementation.

Ali Alavi

The Ohio State University

Multimodal Reranking Cross-Modal Retrieval Composed Video Retrieval Video Retrieval

Ask PDF

May 31, 2026

R^3: Composed Video Retrieval via Reasoning-Guided Recalling and Re-ranking

The CoVR-R challenge evaluates composed video retrieval, where a system must retrieve a target video from a large gallery given a reference video and a textual edit instruction. This setting is not a standard video-text retrieval problem: the query is defined by both the visual evidence in the source video and the transformation implied by the edit. A strong embedding model can provide scalable candidate recall, but it may under-express target-side consequences such as state changes, action replacement, object preservation, or temporal consistency. A pairwise multimodal reranker can verify such details more directly, but exhaustive reranking over the full gallery is computationally infeasible. We present

\mathbb{R}^3

, a zero-shot composed video retrieval pipeline built around Reasoning-guided Recalling and Reranking. The core idea is to turn the source-edit query into a reasoning-grounded retrieval program rather than treating the edit text as a short caption. First, the model generates a reasoning trace that describes the expected target video after applying the edit. Then the trace is encoded together with the source video as a reasoning-augmented query, and its retrieval score is fused with the base composed query through an agreement-gated residual rule. At last, a re-ranker verifies the recalled candidates with direct source-candidate comparison. Experiments have demonstrated the effectiveness of our method in addressing this challenge. Codes are available on https://github.com/Lee-zixu/R-3.

Zixu Li, Yupeng Hu, Zhiheng Fu +3

1Shandong University · 2Harbin Institute of Technology (Shenzhen)

Cross-Modal Retrieval Composed Video Retrieval Video Retrieval

Ask PDF

Jun 1, 2026

Reason-Then-Retrieve for CoVR-R with Structured Edit Prompts and Dense-Sparse Fusion

CoVR-R studies reason-aware composed video retrieval: given a reference video and an edit instruction, the system must retrieve the target video that satisfies the edit. The main difficulty is that the target is not described directly; it must be inferred from fine-grained changes in object identity, action order, final state, hand interaction, and scene transition. We build a zero-shot reason-then-retrieve pipeline around Qwen3.5-27B. For each gallery video, the model generates a retrieval-oriented structured description and a dense embedding by pooling generated-token hidden states with token-dependent weights. For each query, the model first performs edit reasoning over the reference video and instruction, then generates a target-video description whose hidden states serve as the query embedding. We complement dense retrieval with a TF-IDF branch over the generated texts and fuse the two rankings with split-specific weights. On validation, the current best submission reaches 80.81 at R@1, 94.86 at R@5, 97.11 at R@10, and 98.59 at R@50. On the blind test split, it reaches 89.73 at R@1, 95.79 at R@5, 96.63 at R@10, and 97.98 at R@50.

DongQing Liu, MengShi Qi, HongWei Ji

Beijing University of Posts and Telecommunications

Cross-Modal Retrieval Video Reasoning Vision-Language Retrieval Hybrid Retrieval Composed Video Retrieval Video Retrieval

Ask PDF

Abstract

Composed Video Retrieval

Explore similar work

Reason, Retrieve, Re-rank: A Zero-Shot Reasoning-Aware Framework for Composed Video Retrieval

R^3: Composed Video Retrieval via Reasoning-Guided Recalling and Re-ranking

Reason-Then-Retrieve for CoVR-R with Structured Edit Prompts and Dense-Sparse Fusion