Rethinking Fisher Vectors for Second-order Multi-scale Deep Feature Aggregation in Scalable Scene Retrieval

Alexy Bhowmick; Sarat Saharia; Shyamanta M. Hazarika

cs.AIOct 22, 2026

Rethinking Fisher Vectors for Second-order Multi-scale Deep Feature Aggregation in Scalable Scene Retrieval

Authors: Alexy Bhowmick, Sarat Saharia, Shyamanta M. Hazarika

Organizations: Department of Computer Science and Engineering, Assam Don Bosco University, Azara, Guwahati 781017, Assam, India. · Department of Computer Science and Engineering, Tezpur University, Tezpur 784028, Assam, India · Biomimetic Robotics and Artificial Intelligence Lab, Department of Mechanical Engineering, Indian Institute of Technology, Guwahati 781039, Assam, India

Abstract

Aggregating CNN responses into a compact discriminative vector, given their lack of geometric invariance, is a challenge. To address this challenge, we propose a codebook-free method to encode second-order information of multi-scale CNN features within the Fisher Kernel framework. We use a scalable retrieval pipeline to show the efficacy of the proposed vector. The increased discriminative power is confirmed by experimental evaluation of our method on two difficult benchmarks (INRIA Holidays, UKBench). The significant improvement in retrieval accuracy over state-of-the-art approaches comes with only a slight dimension increase.

Figures & tables

Figure 1: Overview of the proposed codebook-free scheme for second-order multi-scale encoding and pooling of CNN-activations.

Figure 2: Accurate retrieval results by baseline Fisher Vector (FV) on INRIA Holidays. Results that do not reflect groundtruth (or are edge cases) are in a red box . A blue box surrounds each query.

Figure 3: Sub-optimal retrieval results by baseline Fisher Vector (FV) on INRIA Holidays. Results that reflect groundtruth are in a green box . A blue box surrounds each query.

Figure 4: The scalable image retrieval framework using the proposed vector representation for retrieval performance evaluation.

Figure 5: Ablation experiments using four different deep learning architectures: Impact of number of Gaussian components k a) on Mean Average Precision(mAP) on Holidays, b) on N-S Score on UKBench. (Higher score is preferable). Parameter $\alpha=0.5$ .

CNN Architecture		Holidays (mAP(%))	UKB (Score/4)
VGG-16	$k=32$	$61.3$	$3.21$
	$k=64$	$66.1$	$3.51$
	$k=128$	$71.6$	$3.52$
	$k=256$	74.5	3.50
VGG-19	$k=32$	$62.0$	$3.33$
	$k=64$	$67.5$	$3.58$

Table 1: Ablation experiments comparing results when using different Deep Learning architectures. k refers to the number of Gaussian components.

Figure 6: Performance comparison with baselines. Impact of number of Gaussian components k a) on Mean Average Precision(mAP) on Holidays, b) on N-S Score on UKBench. (Higher score is preferable). Parameter $\alpha=0.5$ .

Method	Codebook size	D	Holidays (mAP(%))	UKBench (score/4)
Improved FV Perronnin et al. (2010a)	k =64	16384	59.5	3.25
VLAD+SSR Jegou et al. (2012)	k =64	16384	59.8	3.28
VLAD + Intra + Adapt Arandjelovic and Zisserman (2013)	k =64	8192	64.6	-
VLAD* Delhumeau et al. (2013)	k =64	8192	62.2	-
HVLAD Liu et al. (2016a)	k =64	8192	66.8	3.48
Triangulation Embedding Jégou and Zisserman (2014)	k =64	8192	75.6	-

Table 2: Performance comparison of the full-size proposed supervector with state-of-the-art on Holidays and UKBench. Method Codebook size D Holidays (mAP(%)) UKBench (score/4) Improved FV Perronnin et al. (2010a) k =64 16384 59.5 3.25 VLAD+SSR Jegou et al. (2012) k =64 16384 59.8 3.28 VLAD + Intra + Adapt Arandjelovic and Zisserman (2013) k =64 8192 64.6 - VLAD* Delhumeau et al. (2013) k =64 8192 62.2 - HVLAD Liu et al. (2016a) k =64 8192 66.8 3.48 Triangulation Embedding Jégou and Zisserman (2014) k =64 8192 75.6 - CEVLAD Zhou et al. (2016) k =64 8192 67.6 3.43 Fine Residual VLAD Liu et al. (2016b) k =64 8192 61.4 3.38 RVD-W Husain and Bober (2017) - 512 69.0 - SDEVLAD ( Liu et al., 2018 ) k =64 8256 71.4 - FhVLAD ( Bhowmick et al., 2021 ) k =64 8256 67.9 3.51 OxfordNet ( Ng et al., 2015 ) - 512 83.8 - NetVLAD ( Arandjelovic et al., 2018 ) k =64 512 79.9 - MDFA3 ( Wang et al., 2023 ) k =64 512 87.15 - Our proposed method - 16384 83.42 3.76

Method $\downarrow$	k	D’	Holidays	UKBench
Improved FV ( Perronnin et al., 2010a )	64	128	56.5	3.33
VLAD + SSR ( Jegou et al., 2012 )	64	128	55.7	3.35
Multivoc-VLAD ( Jégou and Chum, 2012 )	64	128	61.4	3.36
VLAD+intra+adapt ( Arandjelovic and Zisserman, 2013 )	64	128	62.5	-
HVLAD ( Liu et al., 2016a )	256	128	64.0	3.4
Temb ( Jégou and Zisserman, 2014 )	256	128	65.7	-

Table 3: Performance comparison of the PCA-reduced (D'= 128 ) proposed supervector on Holidays and UKBench with state-of-the-art.

Figure 7: Example retrieval results from the Holidays dataset and the UKB dataset: For query (a) in the blue box, the top row corresponds to our results on Holidays, bottom row corresponds to baseline results from recent FhVLAD Bhowmick et al. (2021) . For query (b) also in the blue box, the top row corresponds to our results on the UKB dataset, and the bottom row corresponds to baseline results from the recent FhVLAD Bhowmick et al. (2021) . In (b), a failure instance of our approach is evident, as the fourth instance in the top row does not match the ground truth. A blue box surrounds each query. A green box corresponds to results that match with the ground-truth. Results that do not reflect groundtruth (or are edge cases) are in a red box .

Image Retrieval

Image retrieval finds and ranks images from visual, textual, or combined queries across large and specialized collections.

See all 85 papers

Deep neural networks with Fisher vector encoding for medical image classification
May 3, 2026Lucas O. Lyra, Antonio E. Fabris, Joao B. FlorindoHybrid CNN-Transformer Architectures Medical Image Classification
Institute of Mathematics and Statistics of the University of Sao Paulo, Rua do Matao, 1010, Sao Paulo, 05508-090, Sao Paulo, Brazil · Institute of Mathematics, Statistics and Scientific Computing of the University of Campinas, Rua Sergio Buarque de Holanda, 651, Campinas, 13083-859, Sao Paulo, Brazil
No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval
May 28, 2026Lixuan Guo, Yifei Wang, Tiansheng Wen +3Learned Sparse Retrieval Multi-Vector Retrieval
Stony Brook University · Xidian University · Amazon AGI SF Lab +4
Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrieval
Mar 6, 2026Donghoon Han, Eunhwan Park, Seunghyeon SeoLearned Sparse Retrieval Fine-Grained Image Retrieval
Moreh · Seoul National University

May 3, 2026J/K move · Enter open · S save

Deep neural networks with Fisher vector encoding for medical image classification

Lucas O. Lyra, Antonio E. Fabris, Joao B. Florindo

Institute of Mathematics and Statistics of the University of Sao Paulo, Rua do Matao, 1010, Sao Paulo, 05508-090, Sao Paulo, Brazil · Institute of Mathematics, Statistics and Scientific Computing of the University of Campinas, Rua Sergio Buarque de Holanda, 651, Campinas, 13083-859, Sao Paulo, Brazil

Hybrid CNN-Transformer Architectures Medical Image Classification

Open paper AskPDF arXiv

Orderless encoding methods have shown to improve Convolutional Neural Networks (CNNs) for image classification in the context of limited availability of data. Additionally, hybrid CNN + Vision Transformers (ViT) models have been recently proposed to address CNN locality bias issues. These models outperformed CNN-only approaches. Despite that, the integration of such hybrid models with more elaborated feature representation can be highly beneficial and remains large unexplored in the literature. In this context, we propose the introduction of an orderless encoding method, Fisher Vectors, to hybrid CNN + ViT architectures, aiming at achieving a model suitable for both small and large datasets. Such enconding method relies on estimating a Gaussian Mixture Model (GMM) on image features. In large datasets, computational costs of the GMM estimation is a limiting factor for the application of Fisher Vectors. Thus, we propose a method to limit the growth of GMM estimation costs as we increase the size of the dataset. We explore the feasibility of our method in the context of medical image classification by appling it to MedMNIST (v2), Clean-CC-CCII and ISIC2018. This collection of datasets contains a wide variety of data scales and modalities. We outperform benchmark results in all MedMNIST (v2) datasets and obtain literature-competitive results in Clean-CC-CCII and ISIC2018.

Abstract

Image Retrieval

Explore similar work

Deep neural networks with Fisher vector encoding for medical image classification

No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval

Visual Words Meet BM25: Sparse Auto-Encoder Visual Word Scoring for Image Retrieval