On the Challenges and Opportunities of Learned Sparse Retrieval for Code

Simon Lupart; Maxime Louis; Thibault Formal; Hervé Déjean; Stéphane Clinchant

cs.IRMar 23, 2026

On the Challenges and Opportunities of Learned Sparse Retrieval for Code

Authors: Simon Lupart, Maxime Louis, Thibault Formal, Hervé Déjean, Stéphane Clinchant

Organizations: NAVER LABS Europe

Abstract

Retrieval over large codebases is a key component of modern LLM-based software engineering systems. Existing approaches predominantly rely on dense embedding models, while learned sparse retrieval (LSR) remains largely unexplored for code. However, applying sparse retrieval to code is challenging due to subword fragmentation, semantic gaps between natural-language queries and code, diversity of programming languages and sub-tasks, and the length of code documents, which can harm sparsity and latency. We introduce SPLADE-Code, the first large-scale family of learned sparse retrieval models specialized for code retrieval (600M-8B parameters). Despite a lightweight one-stage training pipeline, SPLADE-Code achieves state-of-the-art performance among retrievers under 1B parameters (75.4 on MTEB Code) and competitive results at larger scales (79.0 with 8B). We show that learned expansion tokens are critical to bridge lexical and semantic matching, and provide a latency analysis showing that LSR enables sub-millisecond retrieval on a 1M-passage collection with little effectiveness loss.

Figures & tables

Figure 1: (Top) Performance in nDCG@10 of SPLADE-Code and dense models on code retrieval benchmark. 2 2 2 at the time of the submission, C2-LLM is the top 1 open source model on MTEB Code for 0.5B and 7B. (Bottom) Example from CodeFeedback-ST. Only a few vocabulary dimensions receive non-zero weights, covering both semantic terms (e.g., Fibonacci, Python, recursive) and lexical tokens (e.g., input, f).

Figure 2: SPLADE-Code is a bi-encoder model for retrieval across programming languages (e.g. Text2Code, Code2Code). When queries and documents are in different programming languages, SPLADE-Code learn some shared dimensions in the representation space.

Figure 3: Effectiveness–efficiency trade-off on the CodeSearchNet 1M passage collection. The figure reports retrieval latency and effectiveness of SPLADE-Code-8B compared to the Dense-8B model.

(k_{q},k_{d})

indicates the pruning used on queries and documents for LSR.

Panel of Figure 4 — Figure 4: Examples of sparse vocabulary activations produced by SPLADE-Code-8B for different inputs. Highest-weighted expansion tokens in the representations capturing query intent, or algorithmic concepts highlighted.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	SPLADE-Code 0.6B	SPLADE-Code 1.7B	SPLADE-Code 8B
Base Model	Qwen/Qwen3-0.6B	Qwen/Qwen3-1.7B	Qwen/Qwen3-8B
Batch size	256	256	256
Negatives per query	8	7	7
LoRA Rank	64	64	64
Max Length	512	512	512
Learning Rate	$1\times 10^{-4}$	$1\times 10^{-4}$	$1\times 10^{-4}$

Appendix

Table 7: Main training configurations for the base models. For model merging we merge the same model after 1 and 2 epochs, and with a max length of 1024 (3 ckpts). For 0.6B we also merge a full finetuning model.

Figure 5: Latency of SPLADE-Code-0.6B for code representation with two-step SPLADE on Qwen3-0.6B. On Qwen3-0.6B the dot product for the dense is more efficient since embedding dimensions are lower, hence lower latencies with HNSW, but SPLADE still matches the Pareto frontier.

Dataset	SPLADE-Code-0.6B	SPLADE-Code-8B	SPLADE-EN-0.6B
Average	45.4	49.7	48.1
ArguAna	56.4	59.9	52.3
CQADupstackGamingRetrieval	56.8	59.2	56.0
CQADupstackUnixRetrieval	40.5	46.7	37.6
ClimateFEVERHardNegatives	19.0	21.7	19.6
FEVERHardNegatives	60.8	66.0	62.2

Appendix

Table 11: Results of the SPLADE-Code models on MTEB(eng, v2) with pruning

(40,400)

Dataset	SPLADE-Code-0.6B	SPLADE-Code-8B	SPLADE-EN-0.6B
Average	48.3	54.3	47.0
AILAStatutes	28.1	32.4	27.7
ArguAna	56.4	59.9	52.3
BelebeleRetrieval	54.1	77.1	48.4
CovidRetrieval	73.3	72.7	78.9
HagridRetrieval	98.7	98.7	98.8

Appendix

Table 12: Results of the SPLADE-Code models on MTEB(Multilingual, v2) with pruning

(40,400)

Learned Sparse Retrieval

Learned sparse retrieval trains models to expand queries and documents into weighted, indexable terms for efficient semantic search.

See all 18 papers

SpIDER: Spatially Informed Dense Embedding Retrieval for Software Issue Localization
Dec 18, 2025Shravan Chaudhari, Rahul Thomas Jacob, Jiajun Cao +3Software Engineering Software Engineering Benchmarks
Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Work done while interning at AWS · AWS AI Labs, Seattle, WA, USA
Recall Before Rerank: Benchmarking Deep Learning Models for Large-Scale Code-to-Code Retrieval
Jun 24, 2026Leonardo Venuta, Francesco Tosoni, Paolo FerraginaLLM Evaluation Semantic Search
Sant’Anna School of Advanced Studies, Pisa, Italy
Learning Retrieval Models with Sparse Autoencoders
Feb 27, 2026Thibault Formal, Maxime Louis, Hervé Dejean +1Learned Sparse Retrieval Multilingual IR
NAVER LABS Europe

Dec 18, 2025J/K move · Enter open · S save

SpIDER: Spatially Informed Dense Embedding Retrieval for Software Issue Localization

Shravan Chaudhari, Rahul Thomas Jacob, Jiajun Cao, Shihab Rashid+2

Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA · Work done while interning at AWS · AWS AI Labs, Seattle, WA, USA

Software Engineering Software Engineering Benchmarks Graph-Based Retrieval Dense Retrieval

Open paper AskPDF arXiv

Retrieving code functions, classes or files relevant to a user query, bug report or feature request from large codebases is a fundamental challenge for Large Language Model (LLM)-based coding agents. Agentic approaches typically employ sparse methods like BM25 or dense embedding strategies to identify semantically relevant units. While dense embedding approaches can outperform BM25 by large margins, both ignore the graph-structured characteristics of the codebase. To address this, we propose SpIDER (Spatially Informed Dense Embedding Retrieval), a dense retrieval approach that integrates LLM-based reasoning with graph-based exploration of the codebase. We further introduce SpIDER-Bench, a graph-structured benchmark curated from SWEPolyBench, SWEBench-Verified and Multi-SWE-bench, spanning Python, Java, JavaScript and TypeScript repositories. SpIDER's graph-based candidate expansion attaches a structural reason for inclusion to each surfaced function (its seed and the edge type linking them), making the candidate set auditable while keeping the retrieval budget fixed. The graph is built from per-repository syntax trees, so it can be constructed on-demand at the start of a developer session rather than precomputed offline. Empirical results show that SpIDER consistently improves dense retrieval Recall@20 across every language and benchmark in SpIDER-Bench: by at least 13% relative (+0.05 to +0.12 absolute) along containment edges, rising to at least 27% relative (+0.11 to +0.19 absolute) once call edges are explored.

Abstract

Learned Sparse Retrieval

Explore similar work

SpIDER: Spatially Informed Dense Embedding Retrieval for Software Issue Localization

Recall Before Rerank: Benchmarking Deep Learning Models for Large-Scale Code-to-Code Retrieval

Learning Retrieval Models with Sparse Autoencoders