cs.CLSep 29, 2026

Retrieval Sensitivity to Identity Signals in Queries

Authors: Andrew Tang, Nicholas Deas, Kathleen McKeown, Vishal Misra

Organizations: Columbia University, Department of Computer Science

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Dense retrievers decide which documents reach users and the language models that use them, yet they are typically evaluated with neutral queries. We ask whether the identity signals that real users express in their queries---political ideology and dialect---bias what a retriever returns. We design evaluations in two domains, political news and consumer-health questions, each pairing a controlled synthetic set that varies only the identity signal with naturalistic queries. Across five dense retrievers and a sparse baseline, every retriever (i) retrieves articles that align with the query's own political lean and (ii) performs worse for questions written in African American Language (AAL) than in White Mainstream English (WME). Two analyses tie these gaps to queries' identity signals beyond surface vocabulary: partialling out an aggregate lexical-asymmetry score leaves the synthetic gaps largely intact, and linear probes recover lean and dialect from the retrievers' query embeddings beyond token-level features. Left unaddressed, such retrieval biases risk contributing to polarization and reinforcing the health disparities already faced by AAL speakers. Code is available at https://github.com/Andrewtcr/bias-ret.

Figures & tables

Figure 1: Paper overview. We evaluate retrievers’ sensitivity to identity signals in queries, finding that retrievers favor documents aligning with the questioner’s signaled political ideology (top) and that lower-quality documents are retrieved for AAL health queries (bottom).
 FramesLeftRight
AllSides synthFairness & equalityIs raising the federal minimum wage to $10.10 a fair way to reduce inequality and lift nearly a million from poverty per CBO?Does the CBO report show raising the minimum wage to $10.10 unfairly distribute job losses , eliminating about 500,000 opportunities for workers?
EconomicHow would the CBO’s $10.10 minimum wage proposal economically benefit low income workers and reduce poverty despite some job losses?What are the economic costs to employment and small businesses of the CBO’s $10.10 minimum wage proposal, including projected job losses?
Reddit nat—Is anyone else tired of ground-level conservatives claiming they "agree" with news incidents while always supporting the system that created these injustices?Illegal aliens have always been subject to deportation. Why does the government now need the Alien Enemies Act to deport them?
Table 1: Select example queries from each political query dataset. Potential signals of the questioner’s political ideology are highlighted red and blue .
Figure 2: Per-retriever mean retrieval lean at k=10\displaystyle k{=}10 , gold-filtered ( section 2.2 ), shown separately for left and right query groups, with 95% cluster-bootstrap CIs ( Field and Welsh, 2007 ) ( B=5,000\displaystyle B{=}5{,}000 resamples; clusters =\displaystyle= queries from the same source article). Own-lean bias appears as the displacement between a retriever’s two markers; an unbiased retriever’s would coincide. The markers’ shared offset from zero reflects a corpus-level baseline common to both query groups. Stars mark the significance of the own-lean gap: * p<0.05\displaystyle p{<}0.05 , ** p<0.01\displaystyle p{<}0.01 , *** p<0.001\displaystyle p{<}0.001 . n\displaystyle n queries remaining after gold-filtering, in the right margin. Per-retriever gaps in appendix I .
 AALWME
HCMagic synthHi docta , how might can that renal functions be monitored in one patient undergoing hemodialysis 3 times per week?Hi doctor, how can the renal function be monitored in a patient undergoing hemodialysis 3 times per week?
HCMagic nat…about 2 weeks ago she notice she had a lump just below her left rib in the front…While standing if she twist her body you can feel it but once ahe straight again u can see it and feel it. What can this be.Every now and then I will feel my chest tighten and my heart starts pounding extremely hard. I cant breath and I begin feeling very lightheaded. It usually only lasts a few seconds and does not happen too often. Any ideas as to what could be causing this?
Table 2: Select example queries from each AAL/WME dataset. AAL surface markers are coloured purple .
Figure 3: Per-retriever mean MRR on the shared HealthCareMagic corpus, shown separately for WME and AAL queries, with 95% bootstrap CIs. Stars mark the significance of the WME-AAL gap: * p<0.05\displaystyle p{<}0.05 , ** p<0.01\displaystyle p{<}0.01 , *** p<0.001\displaystyle p{<}0.001 . n\displaystyle n queries in the right margin. Per-retriever gaps in appendix M .
 AllSides synth ( Δlean\displaystyle\Delta_{\mathrm{lean}} )Reddit nat (lean)HCMagic synth ( Δrank\displaystyle\Delta_{\mathrm{rank}} )HCMagic nat (MRR)
Retrieverrawresid.rawresid.rawresid.rawresid.
BGE-large0.027\displaystyle 0.0270.027\displaystyle 0.027∗∗∗0.024\displaystyle 0.0240.007\displaystyle 0.007 0.050\displaystyle 0.0500.047\displaystyle 0.047∗∗∗0.034\displaystyle 0.0340.042\displaystyle 0.042 
Qwen3-Emb-8B0.023\displaystyle 0.0230.031\displaystyle 0.031∗∗∗0.028\displaystyle 0.0280.015\displaystyle 0.015 0.023\displaystyle 0.0230.021\displaystyle 0.021∗∗∗0.077\displaystyle 0.0770.110\displaystyle 0.110∗∗
Llama-Nemotron-8B0.025\displaystyle 0.0250.025\displaystyle 0.025∗0.039\displaystyle 0.0390.022\displaystyle 0.022∗0.051\displaystyle 0.0510.046\displaystyle 0.046∗∗∗0.091\displaystyle 0.0910.085\displaystyle 0.085∗∗
Octen-Emb-8B0.023\displaystyle 0.0230.018\displaystyle 0.018∗∗0.048\displaystyle 0.0480.033\displaystyle 0.033∗∗∗0.027\displaystyle 0.0270.024\displaystyle 0.024∗∗∗0.062\displaystyle 0.0620.098\displaystyle 0.098∗∗
OpenAI te-3-large0.045\displaystyle 0.0450.029\displaystyle 0.029∗∗∗0.062\displaystyle 0.0620.036\displaystyle 0.036∗∗∗0.040\displaystyle 0.0400.039\displaystyle 0.039∗∗∗0.064\displaystyle 0.0640.081\displaystyle 0.081∗
Table 3: Per-retriever raw bias gap and residual gap after partialling out the lexical channel (definitions in table 3 ). Stars: ∗p<0.05\displaystyle{}^{*}p{<}0.05 , ∗∗p<0.01\displaystyle{}^{**}p{<}0.01 , ∗∗∗p<0.001\displaystyle{}^{***}p{<}0.001 . Stars test the residual against zero ( α^\displaystyle\hat{\alpha} for the paired datasets, γ^\displaystyle\hat{\gamma} for the unpaired); an unstarred residual is indistinguishable from zero. Raw columns carry no stars because they are not the quantity under test here: they correspond to the gaps of section 2.3 and section 3.3 . Full coefficients in appendix R .
Figure 4: Average F1 score (top) and V\displaystyle\mathcal{V} -information (bottom) for normal and conditional probes, respectively, at each model depth. F1 indicates how readily lean or dialect can be decoded at each layer; V\displaystyle\mathcal{V} -information indicates what that layer adds beyond the input embeddings. Both signals remain decodable across depth, with positive conditional gains at most layers except for BGE on AllSides.
   Query length (characters)
DatasetGroupn\displaystyle nMedianMeanMinMax
AllSides synthleft3,72313813998187
right3,72313913987192
Reddit natliberal1,32715216698302
conservative1,22515416897302
HCMagic synthWME4,999362437635,141
Table 4: Per-group query counts and query-length statistics (characters) for the four query datasets. Counts match the evaluated sets used throughout: HCMagic synth drops one WME source whose AAL augmentation was empty ( 4,999\displaystyle 4{,}999 WME), and HCMagic nat applies a ≥ ⁣20\displaystyle\geq\!20 -character noise filter.
Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Per-anchor L–R query cosine under three pairing schemes (cyan = same article and frame, yellow = same article different frame, grey = different article), grouped by retriever. Box = IQR with median; whiskers =1.5 ⁣× ⁣\displaystyle=1.5\!\times\! IQR; outliers suppressed. Inter-quartile boxes are cleanly separated within every retriever; whiskers cross substantially, so individual query pairs overlap across schemes even though the means do not.
Figure 6: Stage-A (frame extraction) and Stage-B (frame-conditioned query generation) prompts ( gpt-5-mini , medium reasoning effort, default temperature). ⟨⋅⟩\displaystyle\langle\cdot\rangle marks per-request substitutions: Stage A receives the 15 “key—definition” lines for the frames listed above; Stage B is instantiated once per (article, frame) cell. Both stages decode under strict JSON schemas—Stage A returns {frames: [{key, salience}]} , Stage B returns {neutral, left, right} . The neutral query is generated but unused in this paper’s left/right contrasts.
Figure 7: Prompt used for ideology classification. Blue text represents an example post, while green text represents the associated model prediction. Particular phrases that may signal the political ideology of the author are bolded .
ModelSizeEmb. dimCheckpoint
BGE-large0.3B1024BAAI/bge-large-en-v1.5
Qwen3-Embedding-8B8B4096Qwen/Qwen3-Embedding-8B
Llama-Nemotron-8B8B4096nvidia/llama-embed-nemotron-8b
Octen-Embedding-8B8B4096Octen/Octen-Embedding-8B
OpenAI te-3-large—3072text-embedding-3-large (API)
BM25-Okapi——bm25s (sparse)
Appendix
Table 5: Evaluated retrieval models. OpenAI te-3-large uses its default 3072\displaystyle 3072 dimensions (no Matryoshka truncation); BM25 is sparse/lexical and has no dense embedding.
 rawgpt-onlyQwen-onlygold
RetrieverR −\displaystyle- LR −\displaystyle- LR −\displaystyle- LR −\displaystyle- L ( p\displaystyle p )
BGE-large+0.020\displaystyle+0.020+0.069\displaystyle+0.069+0.088\displaystyle+0.088+0.109\displaystyle+0.109 ( 0.007\displaystyle 0.007 )
Qwen3-Emb-8B+0.025\displaystyle+0.025+0.065\displaystyle+0.065+0.080\displaystyle+0.080+0.069\displaystyle+0.069 ( 0.047\displaystyle 0.047 )
Llama-Nemotron+0.033\displaystyle+0.033+0.072\displaystyle+0.072+0.174\displaystyle+0.174+0.186\displaystyle+0.186 ( 10−3\displaystyle 10^{-3} )
Octen-Emb-8B+0.048\displaystyle+0.048+0.043\displaystyle+0.043+0.087\displaystyle+0.087+0.104\displaystyle+0.104 ( 0.002\displaystyle 0.002 )
OpenAI te-3-large+0.056\displaystyle+0.056+0.102\displaystyle+0.102+0.107\displaystyle+0.107+0.136\displaystyle+0.136 ( 10−4\displaystyle 10^{-4} )
Appendix
Table 6: Reddit right −\displaystyle- left mean lean@ 10\displaystyle 10 gap under four filter definitions: raw (no filter, n=2,552\displaystyle n=2{,}552 queries per encoder); gpt-5-mini-only; Qwen-only; and gold (both judges grade the article relevant; used in the main figure). All four agree on direction; the tighter the filter, the larger the gap. The gold column is the primary contrast used in fig. 2 b.
Figure 8: Binary relevance-judge prompts, run by both judges ( gpt-5-mini and Qwen3.5-35B-A3B , T=0\displaystyle T{=}0 ): the Reddit nat two-judge gold ( appendix F ) and the AllSides synth LLM-judge ablation ( appendix G ). Both share the binary 0/1\displaystyle 0/1 rubric and JSON {"grade"} schema; only the domain wording differs—Reddit nat asks whether the article helps the user answer their question, AllSides synth whether it is on-topic for the news story.
Figure 9: Per-retriever mean retrieval lean over the full top-10 (no relevance filter), 95% bootstrap CIs. (a) AllSides synth left / right queries; (b) Reddit nat liberal-/conservative-asker queries. Stars on retriever labels ( * p<0.05\displaystyle p{<}0.05 , ** p<0.01\displaystyle p{<}0.01 , *** p<0.001\displaystyle p{<}0.001 ); n\displaystyle n per group in the right margin. The own-lean gap ( right marker right of left ) holds without any filtering.
Figure 10: AllSides synth per-retriever lean over LLM-judged-relevant top-10 docs (the Reddit nat setting applied to AllSides synth ): the own-lean gap holds on all six retrievers ( p<10−10\displaystyle p<10^{-10} ), larger than under the editorial gold ( table 7 ). 95% bootstrap CIs; n\displaystyle n = retained left/right queries.
Figure 11: Reddit nat per-retriever lean over the full top-10, restricted to queries with ≥1\displaystyle\geq 1 relevant doc (the AllSides synth setting applied to Reddit nat ). Only three of six retrievers stay significant—the off-topic majority of the top-10 dilutes the signal ( fig. 12 ).
Figure 12: Number of relevant docs in the top-10 per (query, retriever), among queries with ≥1\displaystyle\geq 1 relevant. AllSides synth lists are far more on-topic than Reddit nat ’s ( 83%\displaystyle 83\% vs. 40%\displaystyle 40\% of lists contain any relevant doc).
Figure 13: AllSides synth LLM-judge relevance (both judges agree) vs. editorial same-roundup gold, over all 242,000\displaystyle 242{,}000 judged top-10 pairs. The judge is broader than the editorial set ( κ=0.17\displaystyle\kappa=0.17 ), yet the own-lean gap holds under both.
Retrievern\displaystyle n pairsΔlean‾\displaystyle\overline{\Delta_{\mathrm{lean}}} (R −\displaystyle- L)95% CIWilcoxon p\displaystyle pPerm p\displaystyle p
BGE-large2,422+0.027\displaystyle+0.027[+0.016,+0.038]\displaystyle[+0.016,+0.038]4.4 ⁣⋅ ⁣10−6\displaystyle 4.4\!\cdot\!10^{-6}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
Qwen3-Emb-8B2,508+0.023\displaystyle+0.023[+0.014,+0.033]\displaystyle[+0.014,+0.033]2.0 ⁣⋅ ⁣10−6\displaystyle 2.0\!\cdot\!10^{-6}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
Llama-Nemotron-8B1,416+0.025\displaystyle+0.025[+0.011,+0.038]\displaystyle[+0.011,+0.038]9.5 ⁣⋅ ⁣10−5\displaystyle 9.5\!\cdot\!10^{-5}2.8 ⁣⋅ ⁣10−4\displaystyle 2.8\!\cdot\!10^{-4}
Octen-Emb-8B2,605+0.023\displaystyle+0.023[+0.014,+0.032]\displaystyle[+0.014,+0.032]1.0 ⁣⋅ ⁣10−5\displaystyle 1.0\!\cdot\!10^{-5}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
OpenAI te-3-large2,663+0.045\displaystyle\mathbf{+0.045}[+0.035,+0.054]\displaystyle[+0.035,+0.054]3.9 ⁣⋅ ⁣10−21\displaystyle 3.9\!\cdot\!10^{-21}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
BM25 (Okapi)1,798+0.043\displaystyle\mathbf{+0.043}[+0.029,+0.057]\displaystyle[+0.029,+0.057]1.4 ⁣⋅ ⁣10−8\displaystyle 1.4\!\cdot\!10^{-8}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
Appendix
Table 7: AllSides synth gold-filtered paired contrast at k=10\displaystyle k{=}10 ; positive Δlean‾\displaystyle\overline{\Delta_{\mathrm{lean}}} means each query retrieves toward its own political lean. Both paired tests reject on every retriever: two-sided paired Wilcoxon and sign-flip permutation ( B=50,000\displaystyle B{=}50{,}000 , p\displaystyle p -floor 2 ⁣⋅ ⁣10−5\displaystyle 2\!\cdot\!10^{-5} ; appendix H ).
Retrievern\displaystyle n (lib/con)gap (con −\displaystyle- lib)95% CIMWU p\displaystyle p
BGE-large616/474\displaystyle 616/474+0.109\displaystyle+0.109[+0.028,+0.193]\displaystyle[+0.028,+0.193]7.1 ⁣⋅ ⁣10−3\displaystyle 7.1\!\cdot\!10^{-3}
Qwen3-Emb-8B698/539\displaystyle 698/539+0.069\displaystyle+0.069[−0.002,+0.139]\displaystyle[-0.002,+0.139]4.7 ⁣⋅ ⁣10−2\displaystyle 4.7\!\cdot\!10^{-2}
Llama-Nemotron-8B432/317\displaystyle 432/317+0.186\displaystyle\mathbf{+0.186}[+0.079,+0.297]\displaystyle[+0.079,+0.297]1.1 ⁣⋅ ⁣10−3\displaystyle 1.1\!\cdot\!10^{-3}
Octen-Emb-8B729/553\displaystyle 729/553+0.103\displaystyle+0.103[+0.037,+0.176]\displaystyle[+0.037,+0.176]1.5 ⁣⋅ ⁣10−3\displaystyle 1.5\!\cdot\!10^{-3}
OpenAI te-3-large747/570\displaystyle 747/570+0.136\displaystyle+0.136[+0.064,+0.205]\displaystyle[+0.064,+0.205]7.7 ⁣⋅ ⁣10−5\displaystyle 7.7\!\cdot\!10^{-5}
BM25 (Okapi)284/228\displaystyle 284/228+0.195\displaystyle\mathbf{+0.195}[+0.062,+0.323]\displaystyle[+0.062,+0.323]5.5 ⁣⋅ ⁣10−3\displaystyle 5.5\!\cdot\!10^{-3}
Appendix
Table 8: Reddit nat gold-filtered own-lean gap at k=10\displaystyle k{=}10 , parallel to table 7 : mean lean over conservative-asker queries minus mean lean over liberal-asker queries, computed over LLM-judged-relevant documents. Positive means each asker retrieves toward their own political lean. n\displaystyle n is the retained liberal/conservative query count; 95% bootstrap CI on the gap; two-sided Mann–Whitney U\displaystyle U p\displaystyle p (the test behind fig. 2 b). Qwen3-Emb-8B’s gap CI marginally includes 0\displaystyle 0 although its Mann–Whitney p<0.05\displaystyle p<0.05 .
Figure 14: AllSides synth gold-filtered paired lean across k∈{5,10,20,50,100}\displaystyle k\in\{5,10,20,50,100\} , per retriever (per-panel y\displaystyle y -axis). Whiskers: 95% cluster-bootstrap CIs.
Figure 15: Reddit nat mean retrieval lean by asker ideology across k∈{5,10,20,50,100}\displaystyle k\in\{5,10,20,50,100\} , per retriever (per-panel y\displaystyle y -axis; raw unfiltered, all retrievals). Whiskers: 95% cluster-bootstrap CIs.
HCMagic synth — real WME query, synthetic AAL rewrite, shared real gold document
AAL query (synthetic)Hi docta , how might can that renal functions be monitored in one patient undergoing hemodialysis 3 times per week?
WME query (real)Hi doctor, how can the renal function be monitored in a patient undergoing hemodialysis 3 times per week?
Gold document (real)Renal function is monitored by maintaining a chart of blood urea and serum creatinine levels…during each visit of dialysis you must…note down the…levels of renal parameters…regularly…show it to the nephrologist…Take care.
HCMagic nat — real AAL post and a separate real WME post, each with its own gold (unpaired)
AAL query (real)…about 2 weeks ago she notice she had a lump just below her left rib in the front…once ahe straight again u can see it and feel it. What can this be.
Gold document (real)Hi, Dear, thanks for the query to Chat Doctor…In my opinion your daughter has a third nipple or a Supernumerary nipple below the normal nipple on left side, or may be a puberty change…Have a good time.
Appendix
Table 9: Worked query–document examples for both dialect datasets (the queries from table 2 ), each scored against a real gold document—the doctor response indexed from its originating HealthCareMagic row (abridged with …). HCMagic synth is paired: a real WME query and its synthetic AAL rewrite share one gold document. HCMagic nat is unpaired, so we show a real AAL post and a separate real WME post—different patients, each with its own gold document ( section 3.1 ). AAL surface markers in purple .
idxchar lenquery (first 30 chars)
707zzxczxc
1449lipoprint
2677nothing
3864thtr
39812bhjkhjkhklhk
4568aesthema
Appendix
Table 10: The nine AAL rows dropped from the hcmagic source by the input-validity filter.
Retrievern\displaystyle n pairsΔrank‾\displaystyle\overline{\Delta_{\mathrm{rank}}} (WME −\displaystyle- AAL)95% CIWilcoxon p\displaystyle pPerm p\displaystyle p
BGE-large4,999+0.050\displaystyle\mathbf{+0.050}[+0.045,+0.055]\displaystyle[+0.045,+0.055]1.1 ⁣⋅ ⁣10−116\displaystyle 1.1\!\cdot\!10^{-116}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
Qwen3-Emb-8B4,999+0.023\displaystyle+0.023[+0.018,+0.027]\displaystyle[+0.018,+0.027]1.8 ⁣⋅ ⁣10−24\displaystyle 1.8\!\cdot\!10^{-24}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
Llama-Nemotron-8B4,999+0.051\displaystyle\mathbf{+0.051}[+0.046,+0.056]\displaystyle[+0.046,+0.056]2.6 ⁣⋅ ⁣10−192\displaystyle 2.6\!\cdot\!10^{-192}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
Octen-Emb-8B4,999+0.027\displaystyle+0.027[+0.022,+0.031]\displaystyle[+0.022,+0.031]1.3 ⁣⋅ ⁣10−24\displaystyle 1.3\!\cdot\!10^{-24}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
OpenAI te-3-large4,999+0.040\displaystyle+0.040[+0.035,+0.044]\displaystyle[+0.035,+0.044]1.1 ⁣⋅ ⁣10−96\displaystyle 1.1\!\cdot\!10^{-96}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
BM25 (Okapi)4,999+0.024\displaystyle+0.024[+0.020,+0.028]\displaystyle[+0.020,+0.028]8.3 ⁣⋅ ⁣10−50\displaystyle 8.3\!\cdot\!10^{-50}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
Appendix
Table 11: HCMagic synth paired dialect gap at k=10\displaystyle k{=}10 (the contrast behind fig. 3 a): mean per-sample Δrank=RRWME−RRAAL\displaystyle\Delta_{\mathrm{rank}}=\mathrm{RR}_{\text{WME}}-\mathrm{RR}_{\text{AAL}} , averaging the three synthetic AAL rewrites per WME source. Both paired tests (Wilcoxon, sign-flip permutation; appendix H ) reject on every retriever. Parallel to the AllSides synth table 7 .
Retrievern\displaystyle n (WME/AAL)MRR gap (WME −\displaystyle- AAL)95% CIMWU p\displaystyle p
BGE-large659/647+0.034\displaystyle+0.034[−0.010,+0.077]\displaystyle[-0.010,+0.077]0.10
Qwen3-Emb-8B659/647+0.077\displaystyle\mathbf{+0.077}[+0.028,+0.125]\displaystyle[+0.028,+0.125]2.3 ⁣⋅ ⁣10−3\displaystyle 2.3\!\cdot\!10^{-3}
Llama-Nemotron-8B659/647+0.091\displaystyle\mathbf{+0.091}[+0.056,+0.125]\displaystyle[+0.056,+0.125]8.7 ⁣⋅ ⁣10−8\displaystyle 8.7\!\cdot\!10^{-8}
Octen-Emb-8B659/647+0.062\displaystyle+0.062[+0.015,+0.111]\displaystyle[+0.015,+0.111]0.01
OpenAI te-3-large659/647+0.064\displaystyle+0.064[+0.017,+0.111]\displaystyle[+0.017,+0.111]0.01
BM25 (Okapi)659/647+0.040\displaystyle+0.040[+0.015,+0.065]\displaystyle[+0.015,+0.065]0.03
Appendix
Table 12: HCMagic nat unpaired dialect gap at k=10\displaystyle k{=}10 (the contrast behind fig. 3 b): difference in mean MRR between the naturally-occurring WME and AAL query pools. Parallel to the Reddit nat table 8 ; bold marks the two largest gaps. BGE’s CI includes 0\displaystyle 0 , consistent with its p=0.10\displaystyle p{=}0.10 .
Figure 16: HCMagic synth mean Recall@ k\displaystyle k across k∈{1,5,10,20,50,100}\displaystyle k\in\{1,5,10,20,50,100\} , per retriever (per-panel y-zoom to make CIs visible at this n\displaystyle n ). Whiskers: 95% bootstrap CIs.
Figure 17: HCMagic nat unpaired mean Recall@ k\displaystyle k across k∈{1,5,10,20,50,100}\displaystyle k\in\{1,5,10,20,50,100\} . Whiskers: 95% bootstrap CIs.
Figure 18: Translation-mitigation Recall@ k\displaystyle k across k∈{1,5,10,20,50,100}\displaystyle k\in\{1,5,10,20,50,100\} per retriever on the n=647\displaystyle n{=}647 paired AAL queries. Whiskers: 95% bootstrap CIs. Per-panel y-zoom to make CIs visible.
Figure 19: Translation-mitigation paired MRR on the n=647\displaystyle n{=}647 HCMagic nat AAL queries: original AAL vs. gpt-5-mini back-translation to WME . Same content (modulo translation noise), tested with paired Wilcoxon. Translation k\displaystyle k -sweep in appendix N .
Retrievern\displaystyle n pairsΔrank‾\displaystyle\overline{\Delta_{\mathrm{rank}}} (trans-WME −\displaystyle- AAL)95% CIWilcoxon p\displaystyle pPerm p\displaystyle p
BGE-large647+0.032\displaystyle\mathbf{+0.032}[+0.015,+0.048]\displaystyle[+0.015,+0.048]1.1 ⁣⋅ ⁣10−7\displaystyle 1.1\!\cdot\!10^{-7}2.0 ⁣⋅ ⁣10−4\displaystyle 2.0\!\cdot\!10^{-4}
Qwen3-Emb-8B647−0.005\displaystyle-0.005[−0.020,+0.010]\displaystyle[-0.020,+0.010]0.220.50
Llama-Nemotron-8B647+0.044\displaystyle\mathbf{+0.044}[+0.027,+0.061]\displaystyle[+0.027,+0.061]8.5 ⁣⋅ ⁣10−28\displaystyle 8.5\!\cdot\!10^{-28}<2 ⁣⋅ ⁣10−5\displaystyle<2\!\cdot\!10^{-5}
Octen-Emb-8B647−0.004\displaystyle-0.004[−0.019,+0.012]\displaystyle[-0.019,+0.012]0.430.64
OpenAI te-3-large647−0.000\displaystyle-0.000[−0.014,+0.014]\displaystyle[-0.014,+0.014]0.890.99
BM25 (Okapi)647+0.023\displaystyle+0.023[+0.010,+0.037]\displaystyle[+0.010,+0.037]1.2 ⁣⋅ ⁣10−9\displaystyle 1.2\!\cdot\!10^{-9}3.8 ⁣⋅ ⁣10−4\displaystyle 3.8\!\cdot\!10^{-4}
Appendix
Table 13: Translation-mitigation paired gap at k=10\displaystyle k{=}10 : each HCMagic nat AAL query back-translated to WME via gpt-5-mini , then Δrank=RRtrans-WME−RRAAL\displaystyle\Delta_{\mathrm{rank}}=\mathrm{RR}_{\text{trans-WME}}-\mathrm{RR}_{\text{AAL}} . Translation helps the dialect-sensitive retrievers (BGE, Llama-Nemotron, BM25: positive and significant) but not the rest (Qwen, Octen, OpenAI te-3-large : CIs span 0\displaystyle 0 ), and closes no gap fully.
Retrievern\displaystyle nΔlean‾\displaystyle\overline{\Delta_{\mathrm{lean}}}residual α\displaystyle\alphaslope β\displaystyle\beta ( p\displaystyle p )
BM25 (Okapi)1,798+0.043\displaystyle+0.043+0.038\displaystyle+0.038≈0 (p=0.56)\displaystyle\approx 0\ (p{=}0.56)
BGE-large2,422+0.027\displaystyle+0.027+0.027\displaystyle+0.027≈0 (p=0.98)\displaystyle\approx 0\ (p{=}0.98)
Qwen3-Emb-8B2,508+0.023\displaystyle+0.023+0.031\displaystyle+0.031+2.9 ⁣⋅ ⁣10−4 (p=0.11)\displaystyle+2.9\!\cdot\!10^{-4}\ (p{=}0.11)
Llama-Nemotron-8B1,416+0.025\displaystyle+0.025+0.025\displaystyle+0.025≈0 (p=0.94)\displaystyle\approx 0\ (p{=}0.94)
Octen-Emb-8B2,605+0.023\displaystyle+0.023+0.018\displaystyle+0.018≈0 (p=0.36)\displaystyle\approx 0\ (p{=}0.36)
OpenAI te-3-large2,663+0.045\displaystyle+0.045+0.029\displaystyle+0.029−5.7 ⁣⋅ ⁣10−4 (p=0.002)\displaystyle\mathbf{-5.7\!\cdot\!10^{-4}\ (p{=}0.002)}
Appendix
Table 14: Gold-filtered lex-residual fit. Each retriever’s regression runs on its own gold-filtered subset (so n\displaystyle n varies; cross-retriever α\displaystyle\alpha -comparison partly confounds with subset selection). OpenAI te-3-large is the only retriever with a non-zero β\displaystyle\beta ; the lex channel accounts for Δlean‾−α^=+0.016\displaystyle\overline{\Delta_{\mathrm{lean}}}-\hat{\alpha}=+0.016 of its gap at the mean.
Retrieverraw R −\displaystyle- L gapγ^\displaystyle\hat{\gamma} ( p\displaystyle p )γ^/raw\displaystyle\hat{\gamma}/\text{raw}β^\displaystyle\hat{\beta} ( p\displaystyle p )
BGE-large+0.024\displaystyle+0.024+0.007 (p=0.42)\displaystyle+0.007\ (p{=}0.42)0.31\displaystyle 0.31positive sig.
Qwen3-Emb-8B+0.028\displaystyle+0.028+0.015 (p=0.08)\displaystyle+0.015\ (p{=}0.08)0.55\displaystyle 0.55positive sig.
Llama-Nemotron-8B+0.039\displaystyle+0.039+0.022 (p=0.028)\displaystyle\mathbf{+0.022\ (p{=}0.028)}0.55\displaystyle 0.55positive sig.
Octen-Emb-8B+0.048\displaystyle+0.048+0.033 (p<10−3)\displaystyle\mathbf{+0.033\ (p<10^{-3})}0.70\displaystyle 0.70positive sig.
OpenAI te-3-large+0.062\displaystyle+0.062+0.036 (p<10−3)\displaystyle\mathbf{+0.036\ (p<10^{-3})}0.58\displaystyle 0.58positive sig.
BM25 (Okapi)+0.001\displaystyle+0.001+0.002\displaystyle+0.002 (NS)—NS
Appendix
Table 15: Reddit nat unpaired lex-residual at k=20\displaystyle k=20 . Bolded γ^\displaystyle\hat{\gamma} are detectable encoder-level residual gaps after partialling out per-query MCQ asymmetry; BGE’s collapse is consistent with topic-asymmetry exposure. β^\displaystyle\hat{\beta} is positive and significant ( p<0.05\displaystyle p<0.05 ) for all five dense retrievers, contrasting with AllSides synth where only OpenAI te-3-large had a detectable slope.
Retrievern\displaystyle n (lib/con)raw R −\displaystyle- L gapγ^\displaystyle\hat{\gamma} ( p\displaystyle p )γ^/raw\displaystyle\hat{\gamma}/\text{raw}
BGE-large616/474\displaystyle 616/474+0.109\displaystyle+0.109+0.020 (p=0.67)\displaystyle+0.020\ (p{=}0.67)0.18\displaystyle 0.18
Qwen3-Emb-8B698/539\displaystyle 698/539+0.069\displaystyle+0.069+0.017 (p=0.69)\displaystyle+0.017\ (p{=}0.69)0.24\displaystyle 0.24
Llama-Nemotron-8B432/317\displaystyle 432/317+0.186\displaystyle+0.186+0.148 (p=0.017)\displaystyle\mathbf{+0.148\ (p{=}0.017)}0.79\displaystyle 0.79
Octen-Emb-8B729/553\displaystyle 729/553+0.103\displaystyle+0.103+0.061 (p=0.13)\displaystyle+0.061\ (p{=}0.13)0.59\displaystyle 0.59
OpenAI te-3-large747/570\displaystyle 747/570+0.136\displaystyle+0.136+0.106 (p=0.009)\displaystyle\mathbf{+0.106\ (p{=}0.009)}0.78\displaystyle 0.78
BM25 (Okapi)284/228\displaystyle 284/228+0.195\displaystyle+0.195+0.138 (p=0.07)\displaystyle+0.138\ (p{=}0.07)0.71\displaystyle 0.71
Appendix
Table 16: Reddit nat lex-residual with the gold-filtered relevant-subset lean@10 of fig. 2 b as the outcome, on queries with ≥1\displaystyle\geq 1 both-judge-relevant article. Bolded γ^\displaystyle\hat{\gamma} are detectable residual gaps after partialling out per-query MCQ asymmetry.
Retrievern\displaystyle nΔrank‾\displaystyle\overline{\Delta_{\mathrm{rank}}}residual α\displaystyle\alphaslope β\displaystyle\beta ( p\displaystyle p )β ⁣⋅ ⁣asym‾\displaystyle\beta\!\cdot\!\overline{\text{asym}}
Llama-Nemotron-8B4,999+0.051\displaystyle+0.051+0.046\displaystyle+0.046+8.4 ⁣⋅ ⁣10−6 (p=0.21)\displaystyle+8.4\!\cdot\!10^{-6}\ (p{=}0.21)+0.005\displaystyle+0.005
BGE-large4,999+0.050\displaystyle+0.050+0.047\displaystyle+0.047+5.8 ⁣⋅ ⁣10−6 (p=0.43)\displaystyle+5.8\!\cdot\!10^{-6}\ (p{=}0.43)+0.003\displaystyle+0.003
OpenAI te-3-large4,999+0.040\displaystyle+0.040+0.039\displaystyle+0.039+5.9 ⁣⋅ ⁣10−7 (p=0.93)\displaystyle+5.9\!\cdot\!10^{-7}\ (p{=}0.93)+0.000\displaystyle+0.000
Octen-Emb-8B4,999+0.027\displaystyle+0.027+0.024\displaystyle+0.024+4.8 ⁣⋅ ⁣10−6 (p=0.48)\displaystyle+4.8\!\cdot\!10^{-6}\ (p{=}0.48)+0.003\displaystyle+0.003
BM25 (Okapi)4,999+0.024\displaystyle+0.024+0.024\displaystyle+0.024+4.2 ⁣⋅ ⁣10−7 (p=0.93)\displaystyle+4.2\!\cdot\!10^{-7}\ (p{=}0.93)+0.000\displaystyle+0.000
Qwen3-Emb-8B4,999+0.023\displaystyle+0.023+0.021\displaystyle+0.021+2.7 ⁣⋅ ⁣10−6 (p=0.68)\displaystyle+2.7\!\cdot\!10^{-6}\ (p{=}0.68)+0.002\displaystyle+0.002
Appendix
Table 17: Per-sample lex-residual fit on HCMagic synth (sorted by raw Δrank‾\displaystyle\overline{\Delta_{\mathrm{rank}}} ). All β^\displaystyle\hat{\beta} point estimates are positive (consistent direction) but none are statistically distinguishable from zero; the implied per-sample lex contribution β ⁣⋅ ⁣asym‾\displaystyle\beta\!\cdot\!\overline{\text{asym}} is at most 0.005\displaystyle 0.005 MRR (under 10%\displaystyle 10\% of the raw paired gap for the open-weights, ≈0\displaystyle\approx 0 for OpenAI te-3-large and BM25). Residual α\displaystyle\alpha ranking matches the raw Δrank‾\displaystyle\overline{\Delta_{\mathrm{rank}}} ranking up to two micro-swaps within noise (BGE/Llama by 10−3\displaystyle 10^{-3} ; BM25/Octen by 2 ⁣⋅ ⁣10−6\displaystyle 2\!\cdot\!10^{-6} ).
Retrieverraw WME −\displaystyle- AAL gapγ^\displaystyle\hat{\gamma} ( p\displaystyle p )γ^/raw\displaystyle\hat{\gamma}/\text{raw}β^\displaystyle\hat{\beta} ( p\displaystyle p )
Llama-Nemotron-8B+0.091\displaystyle+0.091+0.085 (p=0.0018)\displaystyle\mathbf{+0.085\ (p{=}0.0018)}0.93\displaystyle 0.93NS
Qwen3-Emb-8B+0.077\displaystyle+0.077+0.110 (p=0.0018)\displaystyle\mathbf{+0.110\ (p{=}0.0018)}1.42\displaystyle 1.42NS
OpenAI te-3-large+0.064\displaystyle+0.064+0.081 (p=0.023)\displaystyle\mathbf{+0.081\ (p{=}0.023)}1.26\displaystyle 1.26NS
Octen-Emb-8B+0.062\displaystyle+0.062+0.098 (p=0.0049)\displaystyle\mathbf{+0.098\ (p{=}0.0049)}1.59\displaystyle 1.59NS
BM25 (Okapi)+0.040\displaystyle+0.040+0.041 (p=0.043)\displaystyle\mathbf{+0.041\ (p{=}0.043)}1.01\displaystyle 1.01NS
BGE-large+0.034\displaystyle+0.034+0.042 (p=0.21)\displaystyle+0.042\ (p{=}0.21)1.21\displaystyle 1.21NS
Appendix
Table 18: HCMagic nat unpaired lex-residual. Bolded γ^\displaystyle\hat{\gamma} are significant at p<0.05\displaystyle p<0.05 . The residual gap survives partialling on every retriever with γ^/(raw gap)∈[0.93, 1.59]\displaystyle\hat{\gamma}/(\text{raw gap})\in[0.93,\,1.59] ; no retriever has a detectable within-group lex slope. Conclusion qualitatively matches the HCMagic synth result in table 17 .
More in this topic

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 471 papers

Explore similar work

CardsList

  1. Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors

    Jun 1, 2026Francisco Valentini, Edgar Altszyler, Martin FajcikIR EvaluationDense Retrieval

    CONICET-Universidad de Buenos Aires. Instituto de Ciencias de la Computación (ICC). Buenos Aires, Argentina · Quantit, Buenos Aires, Argentina · Brno University of Technology, Brno, Czech Republic

  2. OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries

    May 7, 2026Diane Tchuindjo, Devavrat Shah, Omar KhattabInformation RetrievalIR Evaluation

    Massachusetts Institute of Technology, Cambridge, MA

  3. Polar: A Benchmark for Evaluating Political Bias in LLMs

    Jun 11, 2026Sangho Kim, Heejin Kim, Yoonhee Park +2Multilingual Language Model EvaluationPolitical Bias in Language Models

    Graduate School of Data Science, Seoul National University · Dept. of Computer Science and Engineering, Seoul National University