Recent work has shown that fine-tuning decoder-only large language models (LLMs) for retrieval yields strong first-stage retrievers, with effectiveness improving as backbones grow in size. However, every query and document must pass through the full model, so encoding cost increases with model size. Mixture-of-Experts (MoE) LLMs activate only a subset of parameters per token and are widely used to scale generative models, yet remain underexplored as retrievers. We systematically study MoE backbones for retrieval by training MoE and dense LLMs from several families using the same procedure, evaluating them across diverse datasets, and measuring query encoding time under the same serving configuration. We show that MoE retrievers outperform dense retrievers with comparable active parameter counts by up to 3.0 nDCG@10 points on BEIR. One of our strongest MoE retrievers matches an 8B dense retriever with 59% fewer active parameters and 18% lower query encoding time. We further show that the number of experts used for query encoding can be reduced without retraining or re-indexing, retaining more than 99% of retrieval effectiveness while reducing query encoding time by up to 26%. Recent rerankers provide only modest additional gains over strong MoE first stages, which often match or exceed the reranked configurations we evaluate. Together, these results show that MoE LLMs can be strong and efficient first-stage retrievers.
Figures & tables
Figure 1: Effectiveness–efficiency trade-off for MoE ( ★ ) and dense ( ∙ ) LLM retrievers. Each point shows a model’s average BEIR-13 nDCG@10 against its query inference time. Arrows connect each MoE retriever to the larger dense model of its family: MoE retrievers reach comparable or higher effectiveness than dense models that have 1.6 to 6.4 times more active parameters while reducing query inference time by 19% to 78%. MoE retrievers consistently lie near the Pareto frontier (shaded).
Family
Model
Active
Arg
Clm
DBP
FEV
FiQ
Hot
NFC
NQ
Quo
SCI
ScF
TRC
Tou
Average
OLMo-1B
1.28
57.9
37.5
42.5
88.0
44.3
77.1
37.2
62.1
83.8
24.3
75.5
80.0
26.2
56.6
OLMo
OLMo-MoE-A1B †
1.28
74.9
38.3
46.2
89.7
48.4
79.6
41.1
66.7
82.6
26.1
76.3
80.4
24.7
59.6 ( +5.3% , +1.7% )
OLMo-7B
6.89
61.7
36.4
46.8
89.0
56.1
81.9
39.4
65.9
87.4
27.8
77.5
68.6
23.8
58.6
LFM2-1.2B
1.17
67.5
36.5
44.6
87.2
43.9
74.9
38.9
62.6
80.2
24.3
72.9
84.6
31.8
57.7
LFM2
LFM2-MoE-A1B †
1.56
73.9
39.8
46.5
87.5
46.3
77.3
40.6
66.4
78.2
26.7
77.5
82.0
29.1
59.4 ( +2.9% , +1.9% )
LFM2-2.6B
2.57
69.0
39.1
44.3
86.1
46.7
76.9
39.8
65.8
76.6
26.5
74.6
84.5
27.4
58.3
Table 1: BEIR-13 effectiveness (nDCG@10) of dense and MoE retrievers trained with the same procedure, grouped by model family; † marks the MoE retrievers. Active is the active parameter count in billions. Bold marks the best average within each family; for each MoE retriever, the parentheses give its relative improvement over the family’s active-matched dense control and larger dense model, in that order, with – where the family has no such model.
Model
Active
nDCG@10
μ s/q
μ s/d
OLMo-1B
1.28
56.6
201
2,455
OLMo-MoE-A1B †
1.28
59.6
316 ( ↓ 1.6 × , ↑ 3.7 × )
3,451 ( ↓ 1.4 × , ↑ 4.1 × )
OLMo-7B
6.89
58.6
1,168
14,104
LFM2-1.2B
1.17
57.7
204
2,350
LFM2-MoE-A1B †
1.56
59.4
374 ( ↓ 1.8 × , ↑ 1.2 × )
3,837 ( ↓ 1.6 × , ↑ 1.4 × )
LFM2-2.6B
2.57
58.3
460
5,330
Table 2: Query and document encoding time on a single NVIDIA H100 GPU, with the BEIR-13 nDCG@10 average of Table 1 for reference. MoE rows ( † ) use their trained expert count. Active is the active parameter count in billions. Bold denotes the highest nDCG@10 and the lowest time within each family. Parentheses compare each MoE retriever with its active-matched dense control and larger dense reference, respectively ( ↑ faster, ↓ slower).
Figure 2: Query-side expert reduction. (a) Average BEIR-13 nDCG@10 against active query parameters as the query expert count k is reduced below the trained count ( ▲ ) with the corpus index fixed. (b) The effectiveness–efficiency view of Figure 1 , with each MoE retriever at its trained expert count ( ▲ ) and selected reduced query expert count ( ⋆ ), joined by an arrow; dense retrievers ( ∙ ) and the Pareto frontier (shaded) are as in Figure 1 .
Model
k
Active ↓
nDCG@10 retained
Query time ↓
OLMo-MoE-A1B
8 → 6
15.7%
99.5%
17.5%
LFM2-MoE-A1B
4 → 3
15.5%
98.4%
14.7%
Qwen3-MoE-A3B
8 → 6
13.5%
99.4%
19.4%
DeepSeek-MoE-A2B
6 → 3
25.4%
99.4%
21.0%
Gemma-4-MoE-A4B
8 → 4
18.7%
99.2%
26.2%
Table 3: Query-side expert reduction. We report the smallest query expert count retaining at least 99% of trained-count BEIR-13 nDCG@10, except for LFM2-MoE, for which we use one fewer expert. Reductions in active query parameters and query encoding time are relative to the trained count.
First stage
None
mxbai-v2
Qwen3-0.6B
RankLLaMA-v1-7B
LFM2-MoE-A1B
59.4
61.4 ( +3.4% )
60.4 ( +1.7% )
55.1 ( -7.2% )
OLMo-MoE-A1B
59.6
61.8 ( +3.7% )
60.3 ( +1.2% )
55.5 ( -7.0% )
DeepSeek-MoE-A2B
60.6
61.5 ( +1.5% )
60.4 ( -0.3% )
55.4 ( -8.5% )
Qwen3-MoE-A3B
62.3
61.9 ( -0.6% )
60.6 ( -2.7% )
55.5 ( -10.9% )
Gemma-4-MoE-A4B
62.5
61.4 ( -1.6% )
60.6 ( -3.0% )
55.1 ( -11.8% )
Table 4: Reranking the top 200 documents of each MoE first stage with three released pointwise rerankers (BEIR-13 nDCG@10). None is the unreranked first stage; parentheses give the relative change from it.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Checkpoint
Experts (total / active)
Total (B)
Active (B)
Dim.
OLMo-1B
allenai/OLMo-1B-0724-hf
–
1.28
1.28
2,048
OLMo-MoE-A1B
allenai/OLMoE-1B-7B-0125
64 / 8
6.92
1.28
2,048
OLMo-7B
allenai/OLMo-7B-0724-hf
–
6.89
6.89
4,096
LFM2-1.2B
LiquidAI/LFM2-1.2B
–
1.17
1.17
2,048
LFM2-MoE-A1B
LiquidAI/LFM2-8B-A1B
32 / 4
8.34
1.56
2,048
LFM2-2.6B
LiquidAI/LFM2-2.6B
–
2.57
2.57
2,048
Appendix
Table 5: Backbones used in our experiments. Experts gives the number of routed experts and the number selected per token. Shared or other always-active components are included in the reported active parameter counts. Parameter counts are in billions. Dim. is the hidden size of the backbone, which equals the embedding dimension.
Setting
Value
Optimizer
AdamW
Learning rate
1×10−4
Warmup
5%
Learning-rate schedule
Linear decay to zero
Maximum query length
192
Maximum document length
192
Appendix
Table 6: Training configuration shared across all retrievers.
Model
LoRA r
LoRA α
Epochs
OLMo-1B
32
64
2
OLMo-MoE-A1B
16
32
1
OLMo-7B
16
32
1
LFM2-1.2B
32
64
2
LFM2-MoE-A1B
16
32
1
LFM2-2.6B
32
64
2
Appendix
Table 7: Model-specific LoRA settings and number of training epochs.
Figure 3: Effect of router fine-tuning for OLMo-MoE on BEIR-13 nDCG@10. The pretrained routers are either frozen or updated together with the LoRA adapters during retrieval fine-tuning.
Model
k
Arg
Clm
DBP
FEV
FiQ
Hot
NFC
NQ
Quo
SCI
ScF
TRC
Tou
Avg.
OLMo-MoE-A1B
8 (trained)
74.8
38.3
46.1
89.7
48.3
79.6
41.0
66.7
82.6
26.2
76.3
80.7
24.4
59.6
6
73.7
38.6
46.1
89.6
48.3
79.1
40.8
66.6
82.7
25.9
76.3
79.8
23.4
59.3
4
71.9
38.3
45.2
88.7
47.0
77.5
40.6
65.4
82.7
25.6
77.4
78.1
24.9
58.7
1
56.6
29.6
34.7
70.1
25.3
50.5
34.4
43.6
77.5
17.1
67.3
65.0
14.8
45.1
LFM2-MoE-A1B
4 (trained)
73.9
39.8
46.5
87.5
46.3
77.3
40.6
66.4
78.2
26.7
77.5
82.0
29.1
59.4
3
73.9
40.4
45.5
87.0
44.9
75.6
39.8
64.9
78.3
26.3
77.2
80.7
25.2
58.4
Appendix
Table 8: Per-dataset BEIR nDCG@10 for the five MoE retrievers at different query expert counts k , with the BEIR-13 average (Avg.). The corpus index remains fixed at the trained expert count, and only query encoding is changed.
Model
k
Active (B)
Active ↓
nDCG@10
Retained
μ s/q
Query time ↓
OLMo-MoE-A1B
8 (trained)
1.28
–
59.6
100.0%
316
–
6
1.08
15.7%
59.3
99.5%
261
17.5%
4
0.88
31.4%
58.7
98.5%
216
31.6%
1
0.58
55.0%
45.1
75.7%
141
55.3%
LFM2-MoE-A1B
4 (trained)
1.56
–
59.4
100.0%
374
–
3
1.32
15.5%
58.4
98.4%
319
14.7%
Appendix
Table 9: Active query parameters, BEIR-13 nDCG@10, and query encoding time for the settings of Table 8 . Active ↓ and Query time ↓ are reductions relative to the trained expert count, and Retained is nDCG@10 relative to the trained count. Query encoding time is measured on one H100 at batch size 128, as in Section 6 .
Dense retrieval systems increasingly rely on multi-billion-parameter language models, whose memory and computational requirements make large-scale indexing, frequent corpus updates, and low-latency serving costly. We present a three-stage training pipeline for developing compact and efficient retrievers that remain competitive with substantially larger models. The pipeline combines cross-lingual alignment, relational knowledge distillation, and contrastive fine-tuning. It requires no original ground-truth relevance labels, relying exclusively on supervision generated by strong embedding models and rerankers utilised as teachers. Using this pipeline, we develop PolDense and EuroDense, both supporting contexts of up to 8,192 tokens. PolDense is a family of six Polish retrievers ranging from 17M to 1B parameters. EuroDense is a 435M-parameter retriever supporting nine European languages. We conduct an extensive evaluation covering 41 Polish and 150 multilingual retrieval tasks. The results demonstrate strong quality-efficiency trade-offs. PolDense-1B outperforms the evaluated retrievers with up to 9B parameters, while the PolDense family forms the Pareto frontier across model sizes. Among the evaluated models below 1B parameters, EuroDense ranks first in both task-averaged and language-averaged performance and leads in seven of nine languages. We release all models publicly.
Neural models, including dense retrievers, have been widely adopted in Information Retrieval (IR), often delivering state-of-the-art performance. Despite their effectiveness, these models operate as black boxes, limiting the interpretability of their ranking decisions. Existing post-hoc explainability methods for neural rankers primarily focus on feature-level attributions, which can be insufficient to capture the complexity of learned embedding spaces. In this work, we propose ExpertLens, a post-hoc explainability framework for Mixture-of-Experts (MoE)-enhanced dense retrievers that shifts focus from local scalar feature importance to representation-level global interpretability. ExpertLens leverages discriminative embedding space visualizations jointly with automatically extracted Concept Activation Vectors to reveal how expert routing drives embedding space formulation and retrieval effectiveness. Experiments across five IR benchmarks and two MoE-enhanced dense retrievers show that expert routing consistently improves embedding space structure, positioning queries and their relevant documents into better-defined geometric neighborhoods. Analysis of expert subspaces further reveals general-purpose dominant experts, along with minority experts exhibiting distinct linguistic specialization, with subspaces arranged according to multi-semantic concept similarity. Our code is publicly available.
Effrosyni Sokli, Isaac Roberts, Alexander Schulz +2
Department of Informatics, Systems and Communication, DISCo University of Milano-Bicocca Milan, Italy · CITEC - Center for Cognitive Interaction Technology Bielefeld University Bielefeld, Germany
Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.
Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand +3
Mila – Quebec AI Institute · Polytechnique Montréal · Université de Montréal +3