Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers

Anubhav Shrestha; Safal Shrestha; Minwu Kim; Torsten Suel; Keith Ross

cs.IRSep 11, 2026

Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers

Authors: Anubhav Shrestha, Safal Shrestha, Minwu Kim, Torsten Suel, Keith Ross

Organizations: New York University Abu Dhabi · New York University

Abstract

Recent work has shown that fine-tuning decoder-only large language models (LLMs) for retrieval yields strong first-stage retrievers, with effectiveness improving as backbones grow in size. However, every query and document must pass through the full model, so encoding cost increases with model size. Mixture-of-Experts (MoE) LLMs activate only a subset of parameters per token and are widely used to scale generative models, yet remain underexplored as retrievers. We systematically study MoE backbones for retrieval by training MoE and dense LLMs from several families using the same procedure, evaluating them across diverse datasets, and measuring query encoding time under the same serving configuration. We show that MoE retrievers outperform dense retrievers with comparable active parameter counts by up to 3.0 nDCG@10 points on BEIR. One of our strongest MoE retrievers matches an 8B dense retriever with 59% fewer active parameters and 18% lower query encoding time. We further show that the number of experts used for query encoding can be reduced without retraining or re-indexing, retaining more than 99% of retrieval effectiveness while reducing query encoding time by up to 26%. Recent rerankers provide only modest additional gains over strong MoE first stages, which often match or exceed the reranked configurations we evaluate. Together, these results show that MoE LLMs can be strong and efficient first-stage retrievers.

Figures & tables

Figure 1: Effectiveness–efficiency trade-off for MoE (

\bigstar

) and dense (

\bullet

) LLM retrievers. Each point shows a model’s average BEIR-13 nDCG@10 against its query inference time. Arrows connect each MoE retriever to the larger dense model of its family: MoE retrievers reach comparable or higher effectiveness than dense models that have 1.6 to 6.4 times more active parameters while reducing query inference time by 19% to 78%. MoE retrievers consistently lie near the Pareto frontier (shaded).

Family	Model	Active	Arg	Clm	DBP	FEV	FiQ	Hot	NFC	NQ	Quo	SCI	ScF	TRC	Tou	Average
	OLMo-1B	1.28	57.9	37.5	42.5	88.0	44.3	77.1	37.2	62.1	83.8	24.3	75.5	80.0	26.2	56.6
OLMo	OLMo-MoE-A1B †	1.28	74.9	38.3	46.2	89.7	48.4	79.6	41.1	66.7	82.6	26.1	76.3	80.4	24.7	59.6 ( +5.3% , +1.7% )
	OLMo-7B	6.89	61.7	36.4	46.8	89.0	56.1	81.9	39.4	65.9	87.4	27.8	77.5	68.6	23.8	58.6
	LFM2-1.2B	1.17	67.5	36.5	44.6	87.2	43.9	74.9	38.9	62.6	80.2	24.3	72.9	84.6	31.8	57.7
LFM2	LFM2-MoE-A1B †	1.56	73.9	39.8	46.5	87.5	46.3	77.3	40.6	66.4	78.2	26.7	77.5	82.0	29.1	59.4 ( +2.9% , +1.9% )
	LFM2-2.6B	2.57	69.0	39.1	44.3	86.1	46.7	76.9	39.8	65.8	76.6	26.5	74.6	84.5	27.4	58.3

Table 1: BEIR-13 effectiveness (nDCG@10) of dense and MoE retrievers trained with the same procedure, grouped by model family;

\dagger

marks the MoE retrievers. Active is the active parameter count in billions. Bold marks the best average within each family; for each MoE retriever, the parentheses give its relative improvement over the family’s active-matched dense control and larger dense model, in that order, with – where the family has no such model.

Model	Active	nDCG@10	$\mu$ s/q	$\mu$ s/d
OLMo-1B	1.28	56.6	201	2,455
OLMo-MoE-A1B †	1.28	59.6	316 ( $\downarrow$ 1.6 $\times$ , $\uparrow$ 3.7 $\times$ )	3,451 ( $\downarrow$ 1.4 $\times$ , $\uparrow$ 4.1 $\times$ )
OLMo-7B	6.89	58.6	1,168	14,104
LFM2-1.2B	1.17	57.7	204	2,350
LFM2-MoE-A1B †	1.56	59.4	374 ( $\downarrow$ 1.8 $\times$ , $\uparrow$ 1.2 $\times$ )	3,837 ( $\downarrow$ 1.6 $\times$ , $\uparrow$ 1.4 $\times$ )
LFM2-2.6B	2.57	58.3	460	5,330

Table 2: Query and document encoding time on a single NVIDIA H100 GPU, with the BEIR-13 nDCG@10 average of Table 1 for reference. MoE rows (

\dagger

) use their trained expert count. Active is the active parameter count in billions. Bold denotes the highest nDCG@10 and the lowest time within each family. Parentheses compare each MoE retriever with its active-matched dense control and larger dense reference, respectively (

\uparrow

faster,

\downarrow

slower).

Figure 2: Query-side expert reduction. (a) Average BEIR-13 nDCG@10 against active query parameters as the query expert count

k

is reduced below the trained count (

\blacktriangle

) with the corpus index fixed. (b) The effectiveness–efficiency view of Figure 1 , with each MoE retriever at its trained expert count (

\blacktriangle

) and selected reduced query expert count (

\star

), joined by an arrow; dense retrievers (

\bullet

) and the Pareto frontier (shaded) are as in Figure 1 .

Model	$k$	Active $\downarrow$	nDCG@10 retained	Query time $\downarrow$
OLMo-MoE-A1B	8 $\to$ 6	15.7%	99.5%	17.5%
LFM2-MoE-A1B	4 $\to$ 3	15.5%	98.4%	14.7%
Qwen3-MoE-A3B	8 $\to$ 6	13.5%	99.4%	19.4%
DeepSeek-MoE-A2B	6 $\to$ 3	25.4%	99.4%	21.0%
Gemma-4-MoE-A4B	8 $\to$ 4	18.7%	99.2%	26.2%

Table 3: Query-side expert reduction. We report the smallest query expert count retaining at least 99% of trained-count BEIR-13 nDCG@10, except for LFM2-MoE, for which we use one fewer expert. Reductions in active query parameters and query encoding time are relative to the trained count.

First stage	None	mxbai-v2	Qwen3-0.6B	RankLLaMA-v1-7B
LFM2-MoE-A1B	59.4	61.4 ( +3.4% )	60.4 ( +1.7% )	55.1 ( -7.2% )
OLMo-MoE-A1B	59.6	61.8 ( +3.7% )	60.3 ( +1.2% )	55.5 ( -7.0% )
DeepSeek-MoE-A2B	60.6	61.5 ( +1.5% )	60.4 ( -0.3% )	55.4 ( -8.5% )
Qwen3-MoE-A3B	62.3	61.9 ( -0.6% )	60.6 ( -2.7% )	55.5 ( -10.9% )
Gemma-4-MoE-A4B	62.5	61.4 ( -1.6% )	60.6 ( -3.0% )	55.1 ( -11.8% )

Table 4: Reranking the top 200 documents of each MoE first stage with three released pointwise rerankers (BEIR-13 nDCG@10). None is the unreranked first stage; parentheses give the relative change from it.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Checkpoint	Experts (total / active)	Total (B)	Active (B)	Dim.
OLMo-1B	allenai/OLMo-1B-0724-hf	–	1.28	1.28	2,048
OLMo-MoE-A1B	allenai/OLMoE-1B-7B-0125	64 / 8	6.92	1.28	2,048
OLMo-7B	allenai/OLMo-7B-0724-hf	–	6.89	6.89	4,096
LFM2-1.2B	LiquidAI/LFM2-1.2B	–	1.17	1.17	2,048
LFM2-MoE-A1B	LiquidAI/LFM2-8B-A1B	32 / 4	8.34	1.56	2,048
LFM2-2.6B	LiquidAI/LFM2-2.6B	–	2.57	2.57	2,048

Appendix

Table 5: Backbones used in our experiments. Experts gives the number of routed experts and the number selected per token. Shared or other always-active components are included in the reported active parameter counts. Parameter counts are in billions. Dim. is the hidden size of the backbone, which equals the embedding dimension.

Setting	Value
Optimizer	AdamW
Learning rate	$1\times 10^{-4}$
Warmup	5%
Learning-rate schedule	Linear decay to zero
Maximum query length	192
Maximum document length	192

Appendix

Table 6: Training configuration shared across all retrievers.

Model	LoRA $r$	LoRA $\alpha$	Epochs
OLMo-1B	32	64	2
OLMo-MoE-A1B	16	32	1
OLMo-7B	16	32	1
LFM2-1.2B	32	64	2
LFM2-MoE-A1B	16	32	1
LFM2-2.6B	32	64	2

Appendix

Table 7: Model-specific LoRA settings and number of training epochs.

Figure 3: Effect of router fine-tuning for OLMo-MoE on BEIR-13 nDCG@10. The pretrained routers are either frozen or updated together with the LoRA adapters during retrieval fine-tuning.

Model	$k$	Arg	Clm	DBP	FEV	FiQ	Hot	NFC	NQ	Quo	SCI	ScF	TRC	Tou	Avg.
OLMo-MoE-A1B	8 (trained)	74.8	38.3	46.1	89.7	48.3	79.6	41.0	66.7	82.6	26.2	76.3	80.7	24.4	59.6
	6	73.7	38.6	46.1	89.6	48.3	79.1	40.8	66.6	82.7	25.9	76.3	79.8	23.4	59.3
	4	71.9	38.3	45.2	88.7	47.0	77.5	40.6	65.4	82.7	25.6	77.4	78.1	24.9	58.7
	1	56.6	29.6	34.7	70.1	25.3	50.5	34.4	43.6	77.5	17.1	67.3	65.0	14.8	45.1
LFM2-MoE-A1B	4 (trained)	73.9	39.8	46.5	87.5	46.3	77.3	40.6	66.4	78.2	26.7	77.5	82.0	29.1	59.4
	3	73.9	40.4	45.5	87.0	44.9	75.6	39.8	64.9	78.3	26.3	77.2	80.7	25.2	58.4

Appendix

Table 8: Per-dataset BEIR nDCG@10 for the five MoE retrievers at different query expert counts

k

, with the BEIR-13 average (Avg.). The corpus index remains fixed at the trained expert count, and only query encoding is changed.

Model	$k$	Active (B)	Active $\downarrow$	nDCG@10	Retained	$\mu$ s/q	Query time $\downarrow$
OLMo-MoE-A1B	8 (trained)	1.28	–	59.6	100.0%	316	–
	6	1.08	15.7%	59.3	99.5%	261	17.5%
	4	0.88	31.4%	58.7	98.5%	216	31.6%
	1	0.58	55.0%	45.1	75.7%	141	55.3%
LFM2-MoE-A1B	4 (trained)	1.56	–	59.4	100.0%	374	–
	3	1.32	15.5%	58.4	98.4%	319	14.7%

Appendix

Table 9: Active query parameters, BEIR-13 nDCG@10, and query encoding time for the settings of Table 8 . Active

\downarrow

and Query time

\downarrow

are reductions relative to the trained expert count, and Retained is nDCG@10 relative to the trained count. Query encoding time is measured on one H100 at batch size 128, as in Section 6 .

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 459 papers

Parameter-Efficient Retrievers for Polish and European Languages
Sep 11, 2026Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec +1Multilingual IR Information Retrieval
National Information Processing Institute al. Niepodległości 188b, 00-608 Warsaw, Poland
ExpertLens: Visualizing Embedding Spaces for Post-Hoc Explainability in MoE Enhanced Retrievers
Sep 5, 2026Effrosyni Sokli, Isaac Roberts, Alexander Schulz +2Interpretable ML Neural Network Interpretability
Department of Informatics, Systems and Communication, DISCo University of Milano-Bicocca Milan, Italy · CITEC - Center for Cognitive Interaction Technology Bielefeld University Bielefeld, Germany
REAM: Merging Improves Pruning of Experts in LLMs
Apr 6, 2026Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand +3LLM Compression Mixture-of-Experts Pruning
Mila – Quebec AI Institute · Polytechnique Montréal · Université de Montréal +3

Sep 11, 2026J/K move · Enter open · S save

Parameter-Efficient Retrievers for Polish and European Languages

Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec, Michał Perełkiewicz

National Information Processing Institute al. Niepodległości 188b, 00-608 Warsaw, Poland

Multilingual IR Information Retrieval Dense Retrieval

Open paper AskPDF arXiv

Dense retrieval systems increasingly rely on multi-billion-parameter language models, whose memory and computational requirements make large-scale indexing, frequent corpus updates, and low-latency serving costly. We present a three-stage training pipeline for developing compact and efficient retrievers that remain competitive with substantially larger models. The pipeline combines cross-lingual alignment, relational knowledge distillation, and contrastive fine-tuning. It requires no original ground-truth relevance labels, relying exclusively on supervision generated by strong embedding models and rerankers utilised as teachers. Using this pipeline, we develop PolDense and EuroDense, both supporting contexts of up to 8,192 tokens. PolDense is a family of six Polish retrievers ranging from 17M to 1B parameters. EuroDense is a 435M-parameter retriever supporting nine European languages. We conduct an extensive evaluation covering 41 Polish and 150 multilingual retrieval tasks. The results demonstrate strong quality-efficiency trade-offs. PolDense-1B outperforms the evaluated retrievers with up to 9B parameters, while the PolDense family forms the Pareto frontier across model sizes. Among the evaluated models below 1B parameters, EuroDense ranks first in both task-averaged and language-averaged performance and leads in seven of nine languages. We release all models publicly.

Abstract

Information Retrieval

Explore similar work

Parameter-Efficient Retrievers for Polish and European Languages

ExpertLens: Visualizing Embedding Spaces for Post-Hoc Explainability in MoE Enhanced Retrievers

REAM: Merging Improves Pruning of Experts in LLMs