Learning Retrieval Models with Sparse Autoencoders

Thibault Formal; Maxime Louis; Hervé Dejean; Stéphane Clinchant

cs.LGFeb 27, 2026

Learning Retrieval Models with Sparse Autoencoders

Authors: Thibault Formal, Maxime Louis, Hervé Dejean, Stéphane Clinchant

Organizations: NAVER LABS Europe

Abstract

Sparse autoencoders (SAEs) provide a powerful mechanism for decomposing the dense representations produced by Large Language Models (LLMs) into interpretable latent features. We posit that SAEs constitute a natural foundation for Learned Sparse Retrieval (LSR), whose objective is to encode queries and documents into high-dimensional sparse representations optimized for efficient retrieval. In contrast to existing LSR approaches that project input sequences into the vocabulary space, SAE-based representations offer the potential to produce more semantically structured, expressive, and language-agnostic features. Building on this insight, we introduce SPLARE, a method to train SAE-based LSR models. Our experiments, relying on recently released open-source SAEs, demonstrate that this technique consistently outperforms vocabulary-based LSR in multilingual and out-of-domain settings. SPLARE-7B, a multilingual retrieval model capable of producing generalizable sparse latent embeddings for a wide range of languages and domains, achieves top results on MMTEB's multilingual and English retrieval tasks. We also developed a 2B-parameter variant with a significantly lighter footprint.

Figures & tables

Panel of Figure 1 — Figure 1: (Left) Performance across layers on Llama Scope (Llama-3.1-8B) and Gemma Scope (Gemma-2-2B). (Right) Performance with increasing SAE width on Gemma-2. Evaluation done with $\text{Top-K}=(40,400)$ .

Panel of Figure 2 — Figure 2: (Left) Impact of pruning documents with Top-K (with $k=40$ for queries). (Right) MS MARCO index distribution for SPLARE and SPLADE (8.8 $M$ documents).

	English	Multilingual	Code	Medical	Law	ChemTEB
English Models
SPLADE-v3 [ 49 ]	50.7	38.1	44.5	44.2	40.4	75.6
Lion-SP-8B [ 98 ]	48.5	50.0	53.3	54.4	48.5	71.1
SPLADE-Llama	52.9	54.3	57.3	61.0	49.0	75.9
SPLARE	52.9	56.3	55.1	62.9	51.2	70.0
Multilingual Models

Table 1: Average performance on various MTEB splits. English models are trained on MS MARCO only (§ 5 ). Multilingual models are trained on a large-scale multilingual training set (§ 6 ). Evaluation done with

\text{Top-K}=(40,400)

	indic	sca	deu	fra	kor	XTREME-UP	MIRACL
SPLADE-Llama	91.9	70.4	57.3	65.6	74.8	56.2	69.9
SPLARE	92.3	70.8	57.1	64.8	76.0	58.6	71.7

Table 2: Multilingual comparison of SPLARE and SPLADE (

\text{Top-K}=(40,400)

	English	Multilingual	XTREME-UP
Top Open Source models
e5-mistral-7b-instruct [ 88 ]	57.6	55.8	-
NV-Embed-v2 [ 50 ]	62.8	56.7	-
multilingual-e5-large-instruct [ 89 ]	53.5	57.1	18.7
GritLM-7B [ 67 ]	55.0	58.3	-
SFR-Embedding-Mistral [ 65 ]	59.3	59.4	-

Table 3: Average MTEB retrieval performance of SPLARE (Multilingual) against top models. Multilingual (resp. Eng) refers to MTEB(Multilingual, v2) (resp. MTEB(eng, v2)). At the time of writing (August 24, 2026), SPLARE ranks in the top-10 models on MTEB(Multilingual, v2) retrieval. For XTREME-UP (MRR@10), we report results from [ 51 ] . Unless specified, evaluation for SPLARE is done with

\text{Top-K}=(40,400)

—corresponding to our default model SPLARE or SPLARE-2B.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Value
LoRA rank $r$	64
Max training sequence length (english models)	128
Max training sequence length (multilingual models)	512
Epochs	1
Batch size (w/ gradient accumulation)	128
Learning rate	$5\times 10^{-5}$

Appendix

Table 4: Hyper-parameters.

Dataset	SPLARE	SPLADE-Llama
Arguana	16.0	16.2
Climate-FEVER	18.3	18.0
DBPedia	44.3	44.8
FEVER	76.0	75.8
FiQA-2018	42.4	42.3
HotpotQA	66.8	67.6

Appendix

Table 5: Full results (nDCG@10 unless specified) on BEIR, MS MARCO and TREC DL for English-based SPLARE and SPLADE-Llama models. Evaluation done with

\text{Top-K}=(40,400)

Task Name	SPLARE	SPLADE-Llama
ArguAna	67.4	65.9
CQADupstackGamingRetrieval	59.6	59.3
CQADupstackUnixRetrieval	43.7	44.9
ClimateFEVERHardNegatives	33.4	35.9
FEVERHardNegatives	90.6	91.1
FiQA2018	57.8	57.2

Appendix

Table 6: Full results of SPLARE and SPLADE-Llama on MTEB(Eng, v2). Evaluation done with

\text{Top-K}=(40,400)

Task Name	SPLARE	SPLADE-Llama
AILAStatutes	38.9	36.3
ArguAna	67.4	65.9
BelebeleRetrieval	83.9	83.8
CovidRetrieval	83.3	81.5
HagridRetrieval	98.9	98.8
LEMBPasskeyRetrieval	48.2	48.2

Appendix

Table 7: Full results of SPLARE and SPLADE-Llama on MTEB(Multilingual, v2). Evaluation done with

\text{Top-K}=(40,400)

Task Name	SPLARE	SPLADE-Llama
BelebeleRetrieval	83.9	83.8
MIRACLRetrievalHardNegatives	72.4	70.5
MLQARetrieval	83.8	81.5
StatcanDialogueDatasetRetrieval	30.8	30.5
TwitterHjerneRetrieval	73.2	73.4
WikipediaRetrievalMultilingual	91.7	90.6

Appendix

Table 8: Full results of SPLARE and SPLADE-Llama on the multilingual-only datasets of MTEB(Multilingual, v2). Evaluation done with

\text{Top-K}=(40,400)

Task Name	SPLARE	SPLADE-Llama
Code
AppsRetrieval	28.2	30.7
COIRCodeSearchNetRetrieval	62.1	70.0
CodeEditSearchRetrieval	74.2	75.0
CodeFeedbackMT	56.3	55.5
CodeFeedbackST	78.0	78.8

Appendix

Table 9: Full results of SPLARE and SPLADE-Llama on MTEB domain specific datasets. Evaluation done with

\text{Top-K}=(40,400)

MTEB(Indic, v1)
Task Name	SPLARE	SPLADE-Llama
BelebeleRetrieval	87.1	87.0
XQuADRetrieval	97.5	96.9
Average	92.3	91.9
MTEB(Scandinavian, v1)
DanFeverRetrieval	42.8	41.7

Appendix

Table 10: Full results of SPLARE and SPLADE-Llama on MTEB language-specific benchmarks. Evaluation done with

\text{Top-K}=(40,400)

Baselines (Prior Work)
Model	ar	bn	en	es	fa	fi	fr	hi	id	ja	ko	ru	sw	te	th	zh	de †	yo †	Avg
BM25	39.5	48.2	26.7	7.7	28.7	45.8	11.5	35.0	29.7	31.2	37.1	25.6	35.1	38.3	49.1	17.5	12.0	56.1	31.9
mDPR	49.9	44.3	39.4	47.8	48.0	47.2	43.5	38.3	27.2	43.9	41.9	40.7	29.9	35.6	35.8	51.2	49.0	39.6	41.8
mContriever	52.5	50.1	36.4	41.8	21.5	60.2	31.4	28.6	39.2	42.4	48.3	39.1	56.0	52.8	51.7	41.0	40.8	41.5	43.1
mE5 large	76.0	75.9	52.9	52.9	59.0	77.8	54.5	62.0	52.9	70.6	66.5	67.4	74.9	84.6	80.2	56.0	56.4	78.3	66.6
E5 ${}_{\mathrm{mistral\text{-}7b}}$	73.3	70.3	57.3	52.2	52.1	74.7	55.2	52.1	52.7	66.8	61.8	67.7	68.4	73.9	74.0	54.0	54.1	79.7	63.4

Appendix

Table 11: Multi-lingual retrieval performance on MIRACL dev (nDCG@10). Baseline results are taken from [ 11 , 51 ] . † denotes the two hidden test sets of MIRACL. Evaluation for SPLARE and SPLADE-Llama done with

\text{Top-K}=(40,400)

Model	MRR@10
SPLARE	58.6
SPLARE (Eng Only)	41.6
SPLADE-Llama	56.2
SPLADE-Llama (Eng Only)	30.5
Gemini Embedding	64.3
Gemini Embedding (Eng Only)	49.3

Appendix

Table 12: XTREME-UP dataset (MRR@10) - Average Scores. Baselines taken from [ 51 ] . Evaluation for SPLARE done with

\text{Top-K}=(40,400)

Model	Avg.	as	bho	brx	gbm	gom	gu	hi	hne	kn	mai	ml	mni	mr	mwr	or	pa	ps	sa	ta	ur
Eng Only
SPLARE	42.6	44.4	49.4	10.2	48.2	39.2	40.9	57.0	49.5	46.5	52.6	47.6	20.1	50.8	49.8	26.3	44.4	37.3	43.5	45.6	48.6
SPLADE-Llama	30.5	31.7	46.8	10.7	45.8	30.7	24.2	54.7	44.5	19.0	48.7	21.8	20.0	45.2	48.2	7.6	23.3	2.2	38.5	27.8	19.5
Multilingual
SPLARE	58.6	63.3	63.1	14.2	60.8	58.8	65.1	67.2	63.2	64.7	65.0	68.9	27.4	64.5	63.3	54.5	66.1	52.7	63.8	63.3	61.2
SPLADE-Llama	56.2	61.2	58.8	14.2	59.4	56.5	62.4	64.1	62.5	62.2	64.5	63.2	29.7	60.5	59.6	53.2	62.2	50.7	62.7	59.0	57.6

Appendix

Table 13: XTREME-UP dataset (MRR@10) - Full scores. Evaluation done with

\text{Top-K}=(40,400)

Panel of Figure 4 — Figure 4: Retrieval Latency ( ms ) when pooling documents (Left) or query (Right) representations with Top-K. In low-latency settings, SPLARE enables higher accuracy for a given level of latency.

Parameter	Value
k	1000
query_cut	30
heap_factor	0.5
n_knn	0
sorted	False
num_threads	1

Appendix

Table 14: Seismic retrieval parameters used to measure latency.

SPLADE-Llama at intermediate layers
Layer No.	18	22	26	31
MTEB(Eng, v2)		43.6	44.5	52.9

Appendix

Table 15: Training SPLADE-Llama models at intermediate layers leads to strong deterioration. At layer

<

22, models collapse during training.

Learned Sparse Retrieval

Learned sparse retrieval trains models to expand queries and documents into weighted, indexable terms for efficient semantic search.

See all 18 papers

Rescaling MLM-Head for Neural Sparse Retrieval
Jun 17, 2026Youngjoon Jang, Seongtae Hong, Jonah Turner +1Masked Language Modeling Learned Sparse Retrieval
Korea University Seoul, South Korea · Independent Seoul, South Korea
From Tokens to Concepts: Leveraging SAE for SPLADE
Apr 23, 2026Yuxuan Zong, Mathias Vast, Basile Van Cooten +2Learned Sparse Retrieval Sparse Autoencoders
Sorbonne Université, CNRS, ISIR · Sinequa by ChapsVision · CNRS, Sorbonne Université, ISIR
Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages
Jul 15, 2025Lyzander Marciano Andrylie, Inaya Rahmanisa, Mahardika Krisna Ihsani +3Multilingual Language Models Representation Learning
Faculty of Computer Science, Universitas Indonesia · Department of Natural Language Processing, MBZUAI

Jun 17, 2026J/K move · Enter open · S save

Rescaling MLM-Head for Neural Sparse Retrieval

Youngjoon Jang, Seongtae Hong, Jonah Turner, Heuiseok Lim

Korea University Seoul, South Korea · Independent Seoul, South Korea

Masked Language Modeling Learned Sparse Retrieval Language Model Calibration Information Retrieval

Open paper AskPDF arXiv

Learned sparse retrieval (LSR) models such as SPLADE have traditionally used BERT-style masked language models as backbone encoders. A natural expectation is that replacing BERT with stronger pretrained encoders should improve retrieval effectiveness. However, we find that under standard SPLADE training recipes, backbones with large MLM-head L2 norms can suffer performance degradation and even training collapse under standard SPLADE training recipes. We identify this failure as a scale mismatch in the MLM head: SPLADE directly uses MLM-head outputs to construct sparse lexical representations, and query-document relevance is computed by an unnormalized dot product over these representations. As a result, an inflated MLM-head scale can amplify sparse activations, distort matching scores, and destabilize contrastive training under common training settings. To address this issue, we introduce a simple initialization-time correction that rescales the MLM-head projection by a constant factor before SPLADE training. This zero-cost adjustment improves training stability without modifying the model architecture or training objective. Across both in-domain and out-of-domain retrieval benchmarks, this simple correction substantially improves large-norm backbones such as ModernBERT and Ettin, turning unstable training runs into competitive sparse retrievers. In several settings, the corrected models further match or surpass the classic BERT-SPLADE baseline. These findings suggest that the bottleneck in adapting pretrained encoders to LSR is not encoder capacity alone, but the calibration of the MLM-head scale used to construct sparse lexical representations.

Abstract

Learned Sparse Retrieval

Explore similar work

Rescaling MLM-Head for Neural Sparse Retrieval

From Tokens to Concepts: Leveraging SAE for SPLADE

Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages