cs.CVNov 3, 2025

Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers

Authors: Mohamed Eltahir, Ali Habibullah, Lama Ayash, Tanveer Hussain, Naeemullah Khan

Organizations: King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, King Khalid University (KKU), Abha, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England

Abstract

In the retrieval domain, candidates' fusion from heterogeneous retrievers is a long-standing challenge, particularly for complex, multi-modal data such as videos. While typical fusion techniques are training-free, they rely solely on rank or score signals, disregarding candidates' representations. This work introduces Vote-in-Context (ViC), a generalized, training-free framework that re-thinks list-wise reranking and fusion as a zero-shot reasoning task for a Vision-Language Model (VLM). The core insight is to serialize both content evidence and retriever metadata directly within the VLM's prompt, allowing the model to adaptively weigh retriever consensus against visual-linguistic content. We demonstrate the generality of this framework by applying it to the challenging domain of cross-modal video retrieval. To this end, we introduce the S-Grid, a compact serialization map that represents each video as an image grid, optionally paired with subtitles to enable list-wise reasoning over video candidates. ViC is evaluated both as a single-list reranker, where it dramatically improves the precision of individual retrievers, and as an ensemble fuser, where it consistently outperforms strong baselines like CombSUM. Across video retrieval benchmarks including ActivityNet and VATEX, the framework establishes new state-of-the-art zero-shot retrieval performance, demonstrating its effectiveness in handling complex visual and temporal signals alongside text. In zero-shot settings, ViC achieves Recall@1 scores of 87.1% (t2v) / 89.0% (v2t) on MSR-VTT and 99.6% (v2t) on VATEX, representing massive gains of up to +40 Recall@1 over previous state-of-the-art baselines. We present ViC as a simple, reproducible, and highly effective recipe for turning modern VLMs into powerful zero-shot rerankers and fusers. Code and resources are publicly available at: https://github.com/mohammad2012191/ViC

Figures & tables

Explore similar work

CardsList
  1. UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrieval

    Apr 22, 2026Haokun Wen, Xuemeng Song, Haoyu Zhang +3Cross-Modal LearningLearning to Rank

  2. AiSearch: Interactive Multi-Modal Search with VLMs

    Oct 1, 2026Ali Koksal, Mei Chee Leong, Vicky Sintunata +2Multimodal IRVision-Language Models

  3. Training-Free Composed Video Retrieval via Visual Representation-Guided Video-LLM Reasoning

    Jun 1, 2026Yang Liu, Qianqian Xu, Peisong Wen +2Cross-Modal RetrievalComposed Video Retrieval