Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrieval

Donghoon Han; Eunhwan Park; Seunghyeon Seo

cs.CVMar 6, 2026

Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrieval

Authors: Donghoon Han, Eunhwan Park, Seunghyeon Seo

Organizations: Moreh · Seoul National University

Abstract

Dense image retrieval is accurate but offers limited interpretability and attribution, and it can be compute-intensive at scale. We present BM25-V, which applies Okapi BM25 scoring to sparse visual-word activations from a Sparse Auto-Encoder (SAE) on Vision Transformer patch features. Across a large gallery, visual-word document frequencies are highly imbalanced and follow a Zipfian-like distribution, making BM25's inverse document frequency (IDF) weighting well suited for suppressing ubiquitous, low-information words and emphasizing rare, discriminative ones. BM25-V retrieves high-recall candidates via sparse inverted-index operations and serves as an efficient first-stage retriever for dense reranking. Across seven benchmarks, BM25-V achieves Recall@200 $\geq$ 0.993, enabling a two-stage pipeline that reranks only $K{=}200$ candidates per query and recovers near-dense accuracy within $0.2$ % on average. An SAE trained once on ImageNet-1K transfers zero-shot to seven fine-grained benchmarks without fine-tuning, and BM25-V retrieval decisions are attributable to specific visual words with quantified IDF contributions.

Figures & tables

Text BM25	BM25-V (visual)
Document $d$	Image $I$
Term vocabulary $\{t\}$	SAE dimensions $\{0,\ldots,eD{-}1\}$
Term frequency $f_{t}(d)$	$\mathbf{v}[i]=\sum_{p}\tilde{H}[p,i]$ (sum-pool)
Document length $\|d\|$	$\\|\mathbf{v}\\|_{1}$ (L1 norm)
Document freq $\text{df}_{t}$	$\text{df}_{i}=\|\{j:\mathbf{v}_{j}[i]>0\}\|$
$\text{IDF}_{t}$	$\text{IDF}_{i}=\ln\!\left(1+\dfrac{N-\text{df}_{i}+0.5}{\text{df}_{i}+0.5}\right)$

Table 1: The text–vision BM25 analogy is structural, not metaphorical. Sum-pooling SAE activations across all

P

patch tokens is mathematically equivalent to counting term occurrences across document tokens. Here

\tilde{H}\in\mathbb{R}^{P\times eD}

denotes the SAE activation matrix (

\tilde{H}[p,i]

is the activation of dimension

i

at patch

p

), and

eD

is the SAE feature dimension.

	CUB-200			Cars-196			Aircraft			Pets			Flowers-102			DTD			Food-101
Method	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
Dense $\dagger$	.767	.948	.975	.922	.988	.996	.707	.915	.966	.912	.982	.990	.989	.997	.998	.762	.912	.946	.954	.987	.992
FAISS-HNSW $\dagger$	.768	.948	.975	.922	.988	.996	.708	.915	.966	.912	.982	.990	.989	.997	.998	.762	.912	.946	.954	.987	.992
FAISS-IVF+PQ $\dagger$	.734	.918	.947	.897	.978	.987	.715	.918	.958	.915	.957	.965	.941	.956	.958	.706	.847	.880	.941	.974	.980
BM25-V (ours)	.472	.738	.835	.715	.900	.946	.523	.765	.851	.771	.923	.962	.954	.986	.993	.747	.888	.930	.865	.947	.965
Two-stage $K{=}100$ (ours)	.744	.929	.960	.914	.983	.990	.701	.908	.960	.916	.979	.988	.990	.997	.998	.769	.919	.955	.946	.980	.986

Table 2: R@1/5/10 on cross-domain fine-grained retrieval (7 datasets). SAE trained on ImageNet-1K, zero-shot. Two-stage: BM25-V top-

K

candidates

\to

dense cosine rerank.

\dagger

Dense-only methods (no interpretability).

$k$	R@1	Index size (MB)
16	0.508	35.8
32	0.483	61.7
64	0.396	97.2
128	0.038	140.6

Table 3: Effect of sparsity

k

on CUB-200-2011 (R@1 and sparse index size). Low

k

preserves discriminative structure;

k{=}128

collapses.

Method	Memory/vec	Compression	Query ops
Dense (exact)	$4D$	$1\times$	$2DN$
HNSW [ 17 ]	$4(D{+}2M)$	${\approx}\,1\times$	$2D\cdot\text{ef}$
IVF+PQ [ 10 ]	$m\!\cdot\!b_{\text{PQ}}$	up to $192\times$	$\tfrac{n_{p}}{n_{l}}\,m\,N$
BM25-V (ours)	$6\,L_{0}$	$48\times$	$\tfrac{C\cdot L_{0}^{2}}{V_{\text{active}}}\,N$ $\dagger$
Two-stage (ours)	$4D{+}6\,L_{0}$	${\approx}\,1\times$	$\tfrac{C\cdot L_{0}^{2}}{V_{\text{active}}}\,N{+}KD$

Table 4: Efficiency comparison. Memory is per-vector storage; query cost counts scoring operations for one query against

N

reference images.

D

: dense embedding dimension;

M

, ef: HNSW graph degree and search budget;

m

b_{\text{PQ}}

: PQ sub-quantizers and bytes per code;

n_{p}

n_{l}

: IVF probe and list counts;

L_{0}

: non-zeros per sparse vector;

C

: FLOPs per BM25 posting entry. Our config:

D{=}1152

M{=}32

\text{ef}{=}100

m{=}48

b_{\text{PQ}}{=}1

n_{l}{=}100

n_{p}{=}1

L_{0}{=}16

K{=}200

\dagger

V_{\text{active}}

ranges from 527 to 3,240 across our benchmarks; as

N

grows,

V_{\text{active}}\to D_{s}

and cost

\to CL_{0}^{2}N/D_{s}

( Eq. 11 ).

Figure 1: Normalized rank-frequency log-log plot for all 7 datasets. Each curve is normalized so rank-1 frequency

=1

. Near-parallel lines confirm a common power-law family with

\alpha\in[1.20,2.32]

; slopes steeper than text Zipf (

\alpha\approx 1

) show that visual words are more heavy-tailed than natural language.

Learned Sparse Retrieval

Learned sparse retrieval trains models to expand queries and documents into weighted, indexable terms for efficient semantic search.

See all 18 papers

Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies
May 28, 2026Benjamin Clavié, Sean Lee, Aamir Shakir +1Learned Sparse Retrieval Information Retrieval
Mixedbread AI · National Institute of Informatics (NII) · University of Tsukuba
Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search
May 29, 2026Gyu-Hwung Cho, Youngjune Lee, Kiyoon Jeong +5Multimodal IR Learned Sparse Retrieval
NAVER Corp., Republic of Korea · Seoul National University, Republic of Korea · NAVER Corp. +6
MINER: Mining Multimodal Internal Representation for Efficient Retrieval
May 7, 2026Weien Li, Rui Song, Zeyu Li +8Multimodal Embedding Visual Document Retrieval
McGill University · MIT - Massachusetts Institute of Technology · University of Cambridge +3

May 28, 2026J/K move · Enter open · S save

Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

Benjamin Clavié, Sean Lee, Aamir Shakir, Makoto P. Kato

Mixedbread AI · National Institute of Informatics (NII) · University of Tsukuba

Learned Sparse Retrieval Information Retrieval Dense Retrieval

Open paper AskPDF arXiv

We propose Latent Terms, a method revealing that models trained for dense retrieval, whether single- or multi-vector, learn representations that can trivially be decomposed into retrieval-ready sparse features. When trained on frozen retrievers, Sparse Autoencoders without any retrieval-specific adjustments extract a latent vocabulary with approximately Zipfian collection statistics, directly suitable for classical sparse retrieval scoring via BM25. This approach enables sparse retrieval while requiring no learned expansion objective or sparse retrieval supervision whatsoever, and can be readily applied to any dense retriever. Latent Terms is able to match or outperform single-vector scoring methods from its own base model as well as comparable SPLADE variants. In addition, it substantially outperforms its base model on LIMIT, a task specifically designed to highlight the failures of single-vector retrieval. Overall, our results highlight that neural retrievers contain more expressive and indexable structure than their default scoring functions expose, but that other methods can nonetheless be leveraged.

Abstract

Learned Sparse Retrieval

Explore similar work

Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search

MINER: Mining Multimodal Internal Representation for Efficient Retrieval