Retrieval Disagreement as a Signal: Diagnosing and Steering Multilingual Dense Embedders

Faeze Ghorbanpour; Shaghayegh Kolli; Jana Diesner; Alexander Fraser

cs.CLOct 13, 2026

Retrieval Disagreement as a Signal: Diagnosing and Steering Multilingual Dense Embedders

Authors: Faeze Ghorbanpour, Shaghayegh Kolli, Jana Diesner, Alexander Fraser

Organizations: School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning (MCML)

Abstract

Multilingual dense retrievers place all languages in one shared space so that a question finds relevant documents, regardless of which language they are written in. We show that the same question, asked in different languages, retrieves substantially different documents, and this disagreement is not because of weak language understanding. Each language performs well on its own and ranks candidates consistently with the other languages; divergence appears only when languages share an index. We trace the mechanism: each language keeps its own ranking, but in the shared index, its correct answers are pushed far down, buried under non-relevant documents that are either in the query's own language or in English, depending on the model. We introduce disagreement-guided steering: for each language, we contrast the documents that other languages agree on for a question against those that language retrieves idiosyncratically, and take the difference of their representations as a language-specific direction, added to that language's queries at inference. Against baselines of language-mean removal, subspace removal, and per-language rotation, it most strongly reduces same-language crowding while remaining competitive on recall, trading off against mean removal, which better preserves ranking quality.

Figures & tables

	Belebele			XQuAD
Model	nDCG ${}_{\text{all}}$	Rec ${}_{\text{all}}$	Agr	nDCG ${}_{\text{all}}$	Rec ${}_{\text{all}}$	Agr
mE5-large	95.0	99.0	53.6	97.2	99.9	66.3
mE5-base	93.5	98.6	50.7	96.1	99.8	63.1
bge-m3	91.1	97.6	53.8	90.3	98.0	62.5
Qwen3-0.6B	87.7	95.5	50.8	88.4	97.6	59.2
Qwen3-8B	92.7	98.1	52.5	90.8	99.0	59.9

Table 1: Isolation: monolingual nDCG@20, Recall@20, and among-language agreement (Agr) when each language retrieves only within its own documents.

	Belebele					XQuAD
Model	nDCG ${}_{\text{all}}$	Rec ${}_{\text{all}}$	Gold	Ord	Agr	nDCG ${}_{\text{all}}$	Rec ${}_{\text{all}}$	Gold	Ord	Agr
bge-m3	87.5	96.0	55	61.0	51.9	84.9	96.5	25	67.2	60.9
mE5-base	69.0	84.3	144	62.1	30.4	72.6	91.2	42	67.8	39.9
mE5-large	73.7	88.9	96	70.2	36.4	76.7	94.0	31	75.6	45.4
Qwen3-0.6B	80.7	91.3	108	55.7	41.5	79.6	94.5	34	61.5	53.7
Qwen3-8B	88.7	96.0	50	60.4	36.8	85.0	97.1	20	67.0	49.8

Table 2: Mixing: mixed-index nDCG@20, Recall@20, foreign-gold rank (Gold), preservation of each language’s monolingual order (Ord), and among-language agreement inside the shared index (Agr).

	Belebele					XQuAD
Model	Same	En	Other	Hub	Drop	Same	En	Other	Hub	Drop
mE5-large	87.7	0.1	3.8	8.5	23/24	89.6	0.2	5.1	8.7	22/24
mE5-base	77.7	0.5	9.7	7.3	11/12	79.5	1.1	11.2	7.1	11/12
e5-mistral	91.6	0.0	5.6	3.4	29/32	97.3	0.0	1.6	2.6	29/32
bge-m3	8.7	6.6	72.8	8.5	22/24	13.9	11.7	67.7	8.1	23/24
Qwen3-8B	38.4	9.7	33.3	7.5	34/36	43.5	12.4	31.3	5.8	30/36

Table 3: Composition of the non-relevant documents that replace a demoted cross-lingual gold, as a percentage of all entrants: same-language (Same), English (En), and other foreign languages (Other), with the hub share (Hub) and the average drop layer (Drop).

Belebele

XQuAD

Multi-EuP

Method

Rec

{}_{\text{xl}}

nDCG

{}_{\text{xl}}

Own

Rec

{}_{\text{xl}}

nDCG

{}_{\text{xl}}

Own

Rec

{}_{\text{xl}}

nDCG

{}_{\text{xl}}

Own

mE5-large

baseline

85.0

45.7

69.3

8.5

87.7

47.0

71.3

14.0

76.8

23.0

45.2

35.3

random

85.0

\pm

0.2

45.7

\pm

0.2

69.3

\pm

0.2

8.5

\pm

0.1

87.7

\pm

0.2

47.1

\pm

0.1

71.3

\pm

0.2

14.1

\pm

0.1

76.7

\pm

0.2

22.9

\pm

0.1

45.4

\pm

0.3

35.0

\pm

0.3

permuted

33.2

\pm

0.7

6.5

\pm

0.2

96.1

\pm

0.1

10.0

17.4

\pm

0.3

2.8

\pm

0.1

98.6

\pm

0.0

16.4

\pm

0.0

9.5

\pm

0.1

1.3

\pm

0.0

98.2

\pm

0.0

5.4

\pm

0.0

subspace

93.5

\pm

0.7

63.4

\pm

0.8

44.9

\pm

1.9

8.0

\pm

0.4

96.1

\pm

0.3

59.4

\pm

0.6

55.4

\pm

1.0

11.8

\pm

0.2

79.8

\pm

1.1

27.2

\pm

1.3

37.6

\pm

4.0

29.1

\pm

2.8

rotate

83.5

\pm

2.2

45.3

\pm

3.7

22.2

\pm

7.6

19.4

\pm

8.9

97.9

\pm

0.3

68.5

\pm

1.9

21.9

\pm

3.8

34.6

\pm

6.0

68.0

\pm

2.3

15.4

\pm

1.4

8.4

\pm

1.3

70.4

\pm

9.8

Table 4: Interventions: cross-lingual Recall@20, nDCG@20, own-language share (Own) and English share (En), averaged over query languages and five estimation seeds (

\pm

std; omitted when 0.0). Belebele directions are estimated on TED2020, XQuAD and Multi-EuP on FLORES-200. random and permuted are controls. The best per model and dataset is bold (highest Rec/nDCG, lowest Own/En). The final block averages across models.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Source	Params	Architecture	Multiling.	License
mE5-base	intfloat/multilingual-e5-base	278M	encoder	yes	MIT
mE5-large	intfloat/multilingual-e5-large	560M	encoder	yes	MIT
bge-m3	BAAI/bge-m3	568M	encoder	yes	MIT
Qwen3-0.6B	Qwen/Qwen3-Embedding-0.6B	0.6B	decoder	yes	Apache-2.0
Qwen3-8B	Qwen/Qwen3-Embedding-8B	8B	decoder	yes	Apache-2.0
e5-mistral	intfloat/e5-mistral-7b-instruct	7B	decoder	(en)	MIT

Appendix

Table 5: Embedding models evaluated, all obtained from Hugging Face and used in inference only. “Multiling.” indicates whether the model is explicitly trained and released for multilingual use; e5-mistral is primarily English-trained but applied multilingually. All listed licenses permit research use.

Query (it): Quale dei seguenti aspetti continuerà nonostante la presenza di stimoli esterni?
Rank	Lang	Gold	Document snippet
Baseline
1	it	✓	In molti hanno osservato che i ritmi fisiologici e il comportamento dipendono in maniera…
2	it		La stazione non ha cambiato approccio, nonostante la precedente perdita di un giroscopio…
3	it		L’avidità e l’egoismo faranno sempre parte di noi. È insito nella natura della cooperazione…
4	it		Internet riunisce in sé alcuni elementi sia della comunicazione di massa…

Appendix

Table 6: Top-5 retrievals for an Italian query under mE5-large. The baseline returns only Italian passages and no cross-lingual gold; steering keeps the Italian answer and surfaces the Russian and Hindi copies.

Query (pt): …qual dos seguintes é considerado uma solução de longo prazo viável para questões ambientais?
Rank	Lang	Gold	Document snippet
Baseline
1	pt	✓	Podemos começar a viver de maneira mais amigável com o ambiente, podemos nos juntar ao movimento…
2	pt		Outros assuntos na agenda em Bali incluem salvar as florestas remanescentes do mundo…
3	pt		Na savana, é difícil para um primata com um sistema digestivo como o dos humanos…
4	pt		Peter Costello, um tesoureiro da Austrália e o homem com mais chance de suceder…

Appendix

Table 7: Top-5 retrievals for a Portuguese query under mE5-large. The baseline returns only Portuguese passages; steering retains the Portuguese answer and surfaces the Spanish parallel copy at rank 5.

	Belebele		XQuAD
Model	R@20	R@100	R@20	R@100
mE5-base	56	77	71	90
mE5-large	63	82	78	93
Qwen3-0.6B	75	88	82	94
Qwen3-8B	84	94	88	97

Appendix

Table 8: Cross-lingual Recall at

K{=}20

vs.

K{=}100

on the parallel collections. Demoted golds reappear at the larger cutoff, confirming demotion rather than loss.

Model	RRF=20	RRF=60	RRF=100	no $w_{\ell}$
mE5-base	58.80	58.80	58.80	58.80
mE5-large	66.79	66.79	66.79	66.79
Qwen3-0.6B	69.22	69.22	69.21	69.22
Qwen3-8B	83.66	83.67	83.66	83.67

Appendix

Table 9: Steering nDCG@20 on Belebele under three rank-fusion constants and with the agreement weight removed. Values move by at most

0.01

across all conditions; cross-lingual recall is identical to within

0.01

. The main results use RRF=60.

Metric	Steering	Mean rem.	$p$
Own-lang share ( $\downarrow$ )	11.3	21.0	$<10^{-19}$
Recall	87.4	84.7	$0.28$
nDCG (Belebele)	65.8	67.1	$0.004$
nDCG (XQuAD)	67.2	64.3	$0.007$

Appendix

Table 10: Paired Wilcoxon signed-rank tests, steering vs. mean removal, over model

\times

dataset

\times

language cells (

n=184

). Steering’s reduction in own-language crowding is highly significant; recall is not significant; nDCG is significant in opposite directions by collection.

Figure 1: Per-language Recall@20 underbaseline, mean removal (centering), and disagreement-guided steering (caa), for mE5-large and Qwen3-8B on Belebele (all ten languages) and a subset of Multi-EuP. The Multi-EuP languages shown are a resource-balanced subset chosen to include cases where each correction wins, not to favor ours.

	Steering		Residual
Model	Rec	nDCG	Rec	nDCG	$\cos$
mE5-large	97.2	66.8	92.1	50.1	0.64
mE5-base	94.3	58.8	89.6	45.3	0.62
bge-m3	94.5	74.8	94.9	72.7	0.73
Qwen3-0.6B	92.3	69.2	91.2	63.4	0.65
Qwen3-8B	99.0	83.7	98.4	76.4	0.66

Appendix

Table 11: Steering vs. its mean-removed residual, cross-lingual Recall@20 / nDCG@20 on Belebele, and the cosine between the steering direction and the language-mean offset. Removing the mean-aligned component degrades steering for every model but bge-m3, confirming the orthogonal component carries signal.

	A1 (set)			A2 (rank)
Model	Base	Cent	Steer	Base	Cent	Steer
mE5-base	30.4	28.2	33.1	57.9	59.8	60.5
mE5-large	36.4	37.9	40.1	59.7	65.2	63.4
bge-m3	51.9	51.9	48.6	73.6	75.0	71.1
Qwen3-0.6B	39.1	43.7	39.6	64.9	70.6	62.2
Qwen3-8B	33.6	44.5	41.1	67.9	74.9	68.6

Appendix

Table 12: Cross-lingual agreement under interventions on Belebele, averaged over languages (

\times 100

): set agreement A1 and rank agreement A2 for baseline, mean removal (Cent), and steering (Steer).

Explore similar work

When Does Mixing Help? Analyzing Query Embedding Interpolation in Multilingual Dense Retrieval
Jun 11, 2026Tongyao Zhu, Chao-Ming Huang, Min-Yen KanMultilingual IR Dense Retrieval
National University of Singapore
Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval
Aug 3, 2026Chao Huang, Yufeng Chen, Changhao Guan +3Contrastive Learning Disentangled Representation Learning
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education · School of Computer Science and Technology, Beijing Jiaotong University
Parameter-Efficient Retrievers for Polish and European Languages
Sep 11, 2026Sławomir Dadas, Rafał Poświata, Małgorzata Grębowiec +1Multilingual IR Information Retrieval
National Information Processing Institute al. Niepodległości 188b, 00-608 Warsaw, Poland

Jun 11, 2026J/K move · Enter open · S save

When Does Mixing Help? Analyzing Query Embedding Interpolation in Multilingual Dense Retrieval

Tongyao Zhu, Chao-Ming Huang, Min-Yen Kan

National University of Singapore

Multilingual IR Dense Retrieval

Open paper AskPDF arXiv

While mixed-language querying is ubiquitous in multilingual communities, the sensitivity of dense retrievers to such queries remains poorly understood. We present a ratio-controlled study on mMARCO that systematically evaluates retrieval performance by varying the mixing proportion of parallel query translations via embedding-level mixing -- constructing mixed queries as an interpolation of monolingual embeddings. Experiments with BGE-M3 demonstrate that an optimal mixing ratio outperforms the best monolingual endpoint in 88/105 cases. We uncover a distinct asymmetry driven by English dominance: mixing is uniformly beneficial when retrieving from non-English document indices, whereas indices containing English are best served by pure English queries. Furthermore, English acts as the strongest mixing partner for every non-English document language. Finally, when controlling for English dominance, mixing gains correlate negatively with typological distance. We conclude that language-mix sensitivity is structured and predictable, and we validate the robustness of these patterns across model families and scales.

Abstract

Explore similar work

When Does Mixing Help? Analyzing Query Embedding Interpolation in Multilingual Dense Retrieval

Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval

Parameter-Efficient Retrievers for Polish and European Languages