Comparing videos to assess their semantic similarity remains a challenging task for current Large Multimodal Models (LMMs), which are primarily trained with single-video objectives. While most prior works have focused on computing global similarity, recent studies have introduced the more challenging problem of concept-based similarity, which measures how similar two videos are with respect to specific concepts (e.g., activity, location). However, no dedicated approaches have yet leveraged LMMs to explicitly address this task. In this paper, we aim to fill this gap introducing Video Caption Entailment for video similarity estimation (VICE): a dual-branch design that leverages LMMs for cross-video reasoning. Each branch jointly processes one video and the caption of the other, measuring their alignment for a specific concept via video-language entailment. The outputs from both branches are fused to produce a final similarity score, along with a textual explanation of the model reasoning. This approach leverages the model's native single-video understanding while enabling fine-grained comparative reasoning between video and text. Experiments on ConViS-Bench and StepDiff show that VICE outperforms models that process both videos simultaneously, highlighting the promise of single-video reasoning for concept-based similarity estimation.
Figures & tables
Figure 1: Order sensitivity of Large Multimodal Models in video-pair understanding. Large Multimodal Models (LMMs) exhibit strong order dependence: swapping the order of videos in a pair leads to different predictions in approximately 25–30% of cases. In contrast, our method is order-invariant (0% change rate).
Figure 2: Overview of ViCE : Given a video pair (V1,V2) and a concept Ci , e.g., Ingredients , ViCE processes the videos through two parallel branches. Each branch generates a concept-conditioned caption (g1,g2) , then performs video–caption entailment by comparing one video with the caption of its counterpart. The branches produce reasoning traces inside <thinking>…</thinking> and similarity scores inside <score>…</score> . The final score s(V1,V2∣Ci) is averaged across both branches. For clarity, we omit visualizations of the reasoning trace summarization (Sec. 3.3 ), as it does not affect the predicted score and is used solely for interpretability.
Model
Main Action
Main Subjects
Main Objects
Location
Order of Actions
Avg.
LLaVA-OV-7B Li et al. (2025)
zero-shot
51.8
48.4
58.6
58.9
41.0
51.7
zero-shot CoT
39.9
28.6
28.3
44.1
28.0
33.8
ViCE
54.4
55.9
50.0
64.8
19.2
48.9
↓2.8
LLaVA-Video-7B Zhang et al. (2025)
zero-shot
44.2
39.8
45.8
56.0
41.2
45.4
Table 1: Correlation performance on ConViS-Bench Liberatori et al. (2025) . We apply ViCE to multiple LMMs, reporting Spearman’s ρ correlations with human judgment ( ×100 ).
Model
Technique
Tools/Equipment
Actions
Visuals
Ingredients
Avg.
LLaVA-OV-7B Li et al. (2025)
zero-shot
31.1
39.4
32.8
32.3
28.5
32.8
zero-shot CoT
29.5
35.3
29.4
25.3
24.1
28.7
ViCE
42.2
36.6
32.8
31.8
26.5
34.0
↑1.2
LLaVA-Video-7B Zhang et al. (2025)
zero-shot
30.6
37.4
28.9
29.2
17.5
28.7
Table 2: Correlation performance on StepDiff Nagarajan and Torresani (2024) . We apply ViCE to multiple LMMs, reporting Spearman’s ρ correlations with human judgment ( ×100 ).
Figure 3: Distributions of predicted scores across datasets and models. We compare zero-shot predictions, predictions with ViCE , and ground truths .
Figure 4: Distributions of predicted scores. We compare ViCE ’s predictions to the ground truth using LLaVA-Video-7B model as the LMM.
Figure 5: Example of a video pair with the captions and reasoning traces generated when conditioning on the target concept. Compared to the zero-shot baseline, ViCE produces scores that more closely align with the ground truth.
Model
Main Action
Main Subjects
Main Objects
Location
Order of Actions
Avg.
LLaVA-OV-7B Li et al. (2025)
Blind
53.4
38.5
42.9
47.0
8.1
38.0
ViCE w/o CoT
48.1
48.9
46.7
57.2
20.2
44.2
ViCE single branch
47.6
48.4
43.0
58.4
25.0
44.5
ViCE ( M=1 )
48.2
49.2
47.1
60.8
40.3
49.1
ViCE
54.4
55.9
50.0
64.8
19.2
48.9
Table 3: Ablation Studies on ConViS-Bench Liberatori et al. (2025) . We report Spearman’s ρ ( ×100 ).
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: User Study Interface: Annotators are asked to rank the descriptions or select the checkbox if none of them apply.
Figure 7: Qualitative results of ViCE : We present three example video pairs for two distinct concepts and datasets Liberatori et al. (2025) ; Nagarajan and Torresani (2024) , along with the captions and reasoning traces generated when conditioning on those concepts. Compared to the zero-shot baseline, ViCE produces scores that more closely align with the ground truth ones.
Model
1 st (%)
2 nd (%)
3 rd (%)
Qwen3-VL-8B Bai et al. (2025a)
38.34
24.87
36.79
LLaVA-Video-7B Zhang et al. (2025)
31.61
42.49
25.90
LLaVA-OV-7B Li et al. (2025)
29.53
33.16
37.31
Appendix
Table 4: User Study Results: We report the percentage of times each model’s explanation generated by ViCE with different LMMs (see Sec. 3.3 ) was ranked 1st, 2nd, or 3rd.
Method
Peak GPU Memory (GiB)
Time / Prediction (s)
Baseline (zero-shot)
17.5
2.5
VICE
22.4
5.8
Appendix
Table 5: Computational Cost: Peak GPU memory and wall-clock time per prediction for the zero-shot baseline and VICE using Qwen3-VL-8B on a single NVIDIA A100 GPU.