Dense retrieval is central to modern search, question answering, and retrieval-augmented generation, where high similarity is often treated as evidence of semantic relevance. We examine how far this assumption can be stressed. NONSENSE is a black-box sparse-edit probe that constructs visibly non-semantic strings while forbidding query-term copying, optimizing only the dense similarity between a query and the string. Across five BEIR-style datasets and three popular single-vector retrievers, these strings frequently outscore the retriever's own best judged query-relevant passage (QRP), and often rank first in the full corpus. The result indicates a semanticity gap in the evaluated dense retrievers: nearest-neighbor status can be separated from passage-like meaning. These findings highlight the importance of evaluating dense retrievers not only by whether they retrieve relevant documents, but also by how reliably their similarity scores reflect semantic evidence.
Figures & tables
Query Relevant Passage (QRP)
Generated NONSENSE String
SciFact Query: Insomnia can be effectively treated with cognitive behavioral therapy.
A clinical approach to circadian rhythm sleep disorders.. Circadian rhythm sleep disorders are characterized by complaints of insomnia… Dense Similarity Score (E5): 0.854
NFCorpus Query: Do Vegetarians Get Enough Protein?
Figure 1: Representative queries illustrating the semanticity gap. Optimized NONSENSE strings frequently outscore the retriever’s best judged query-relevant passage (QRP) under the same scoring model (E5-base-v2). Starting from visible filler text, the probe optimizes only for dense similarity, showing that nearest-neighbor geometric status can separate from passage-like meaning. See Tables 8 – 10 for more qualitative examples.
Dataset
Retriever
Corpus Size
QRP Score
NONSENSE Score
Success Rate
Mean Rank
Median Rank
SciFact
E5
5,183
0.836
0.902
0.977
1.0
1
BGE
5,183
0.800
0.866
0.852
1.2
1
Contriever
5,183
0.541
0.721
0.956
1.1
1
SciDocs
E5
25,657
0.828
0.892
0.974
1.1
1
BGE
25,657
0.789
0.854
0.871
2.9
1
Contriever
25,657
0.510
0.646
0.928
2.3
1
Table 1: Main NONSENSE results over three seeds per dataset–retriever pair. Each run uses 256 sampled queries and 50k proposal generations. QRP denotes the highest-scoring judged query-relevant passage for a query under the evaluated retriever. Success Rate is the fraction of queries for which the optimized NONSENSE string receives a higher dense similarity score than QRP. Mean Rank and Median Rank are computed from the corpus rank of the inserted NONSENSE string after adding it to the full corpus of the stated size. Bold marks the higher score between QRP and NONSENSE in each row.
Figure 2: Mean dense similarity over proposal steps. Teal curves show the mean NONSENSE score over 256 sampled queries and three seeds; dashed purple lines show the corresponding mean query-relevant passage (QRP) score. Vertical dotted markers and boxed numbers indicate the first mean overtake step in units of 103 proposals. Early crossings show that the effect does not depend only on the final 50k-step endpoint.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Ablation
Retriever
Corpus Size
QRP Score
NONSENSE Score
Success Rate
Mean Rank
Median Rank
Default
E5
5,183
0.838
0.902
0.984
1.0
1
Default
BGE
5,183
0.802
0.865
0.852
1.2
1
Default
Contriever
5,183
0.540
0.718
0.953
1.1
1
Mutable spaces
E5
5,183
0.838
0.911
0.996
1.0
1
Mutable spaces
BGE
5,183
0.802
0.880
0.895
1.2
1
Mutable spaces
Contriever
5,183
0.540
0.745
0.977
1.0
1
Appendix
Table 2: Ablation results on SciFact over 256 sampled queries and 50k proposal steps for seed 4201. Default uses the main-experiment parameterization: length 256, frozen spaces, colon initialization, and the full character alphabet. Mutable spaces allows spaces to mutate instead of freezing them. Len 128 and Len 512 constrain the NONSENSE string length to 128 and 512 characters, respectively. Init aaaa and Init random replace the default colon initialization with repeated aaaa blocks or a random string. No letters / symbols+digits removes letters from the edit alphabet; in the current runner, both requested alphabet modes map to the same digits-plus-symbols alphabet, so they are reported once. QRP denotes the highest-scoring judged query-relevant passage for the same query under the evaluated retriever; Success Rate is the fraction of queries for which the optimized NONSENSE string receives a higher dense similarity score than QRP. Mean and median ranks are corpus ranks of the optimized NONSENSE strings. Bold marks the higher score between QRP and NONSENSE in each row.
Group
1k
5k
10k
50k
E5
0.288
0.724
0.854
0.966
BGE
0.162
0.465
0.591
0.783
Contriever
0.166
0.584
0.742
0.908
All
0.205
0.591
0.729
0.886
Appendix
Table 3: QRP-relative success at smaller optimization budgets. Entries report the fraction of query–seed cases where the snapshot NONSENSE score already exceeds QRP after the stated number of proposal steps. The 50k column corresponds to the final setting used in Table 1 , aggregated here by retriever and overall.
Group
Query-Term Rate
Exact Rate
BM25@1
Alpha Tokens
Dict. Words
Non-Alnum.
E5
0.000
0.000
0.022
50.9
0.00
0.621
BGE
0.000
0.000
0.018
53.6
0.01
0.624
Contriever
0.000
0.000
0.018
55.8
0.00
0.510
All
0.000
0.000
0.019
53.6
0.00
0.582
Appendix
Table 4: Lexical diagnostics for the main NONSENSE strings. Query-Term Rate is the fraction containing any banned query term as a substring; Exact Rate is the fraction containing a banned term as an exact alphanumeric span; BM25@1 is the fraction ranked first by BM25. Alpha Tokens and Dict. Words count alphabetic spans and common dictionary words in the generated string; Non-Alnum. is the fraction of characters that are not alphanumeric.
Dataset
Retriever
Median Margin
IQR Margin
Mean-Curve Overtake Step
SciFact
E5
0.065
[0.039, 0.090]
1,800
SciFact
BGE
0.061
[0.017, 0.114]
3,800
SciFact
Contriever
0.176
[0.090, 0.263]
2,400
SciDocs
E5
0.064
[0.039, 0.087]
1,400
SciDocs
BGE
0.065
[0.026, 0.107]
4,600
SciDocs
Contriever
0.132
[0.069, 0.203]
3,800
Appendix
Table 5: Score-margin and budget diagnostics. Margin is NONSENSE Score minus QRP Score per query, pooled over the three main seeds; the table reports the median and interquartile range (IQR). Mean-Curve Overtake Step is the first proposal step at which the mean NONSENSE score in Figure 2 exceeds the mean QRP baseline. ArguAna/BGE is the only pair whose mean curve does not cross QRP.
Retriever
Candidate
QRP Score
Candidate Score
Success Rate
Mean Rank
Query-Term Rate
BM25@1
E5
Optimized NONSENSE
0.838
0.902
0.984
1.0
0.000
0.000
E5
Blank init
0.838
0.737
0.004
1,802.6
0.000
0.000
E5
Random string
0.838
0.742
0.004
1,481.5
0.000
0.000
E5
Matched random edits
0.838
0.743
0.008
1,436
0.000
0.000
E5
Query-term repetition
0.838
0.903
0.969
1.0
1.000
0.988
BGE
Optimized NONSENSE
0.802
0.865
0.852
1.2
0.000
0.000
Appendix
Table 6: Control-baseline comparison on SciFact over 256 sampled queries for seed 4201. Optimized NONSENSE is the full search result from the main setting and is included as a reference. Blank init is the unoptimized colon string; Random string samples a same-length random string; Matched random edits applies the same number of edits as the optimizer but chooses edits at random; Query-term repetition repeats query terms to fill the 256-character budget. The key contrast is that random and blank controls have near-zero success, while query-term repetition succeeds only by explicitly copying query terms and becoming a BM25 top hit. Optimized NONSENSE attains high dense-retrieval success without query-term overlap, isolating the dense semanticity gap from trivial lexical matching. QRP denotes the highest-scoring judged query-relevant passage for the same query under the evaluated retriever. Success Rate is the fraction of queries for which the candidate receives a higher dense similarity score than QRP; Query-Term Rate is the fraction of candidates containing query terms; BM25@1 is the fraction ranked first by BM25.
Dataset
Source
Target-QRP Success Rate
Mean Target Rank
E5
BGE
Contriever
E5
BGE
Contriever
SciFact
E5
0.984
0.086
0.051
1.0
190.2
1,207.1
BGE
0.086
0.852
0.035
179.4
1.2
1,862.2
Contriever
0.156
0.066
0.953
60.1
278.0
1.1
SciDocs
E5
0.969
0.012
0.008
1.1
1,746.7
13,599.2
BGE
0.090
0.867
0.004
1,371.4
2.5
14,780.8
Appendix
Table 7: Cross-model transfer of optimized NONSENSE strings for seed 4201. Rows indicate the source retriever used to optimize the string; columns indicate the target retriever used only for evaluation. Target-QRP Success Rate is the fraction of transferred strings whose target-retriever dense score exceeds that target retriever’s highest-scoring judged query-relevant passage for the same query. Mean Target Rank is computed after inserting the transferred string into the full target corpus. Diagonal entries recover same-model evaluation; off-diagonal entries measure cross-retriever transfer.
Query-Relevant Passage (QRP)
Generated NONSENSE String
SciFact Query: Activator-inhibitor pairs are provided dorsally by Admpchordin.
Regulation of ADMP and BMP2/4/7 at Opposite Embryonic Poles Generates a Self-Regulating Morphogenetic Field. Embryos have the ability to self-regu … Dense Similarity Score (E5): 0.819
SciDocs Query: Subjectivity and Sentiment Analysis of Modern Standard Arabic
Appendix
Table 8: Representative E5 examples selected with sample seed 20260526. Each block shows one query per dataset, the model’s highest-scoring judged query-relevant passage (QRP), and the optimized NONSENSE string from the seed-4201 main run. Text is truncated for display; scores are dense similarities under the evaluated retriever.
Query-Relevant Passage (QRP)
Generated NONSENSE String
SciFact Query: Activator-inhibitor pairs are provided dorsally by Admpchordin.
Regulation of ADMP and BMP2/4/7 at Opposite Embryonic Poles Generates a Self-Regulating Morphogenetic Field. Embryos have the ability to self-regu … Dense Similarity Score (BGE): 0.731
SciDocs Query: Subjectivity and Sentiment Analysis of Modern Standard Arabic
Appendix
Table 9: Representative BGE examples selected with sample seed 20260526. Each block shows one query per dataset, the model’s highest-scoring judged query-relevant passage (QRP), and the optimized NONSENSE string from the seed-4201 main run. Text is truncated for display; scores are dense similarities under the evaluated retriever.
Query-Relevant Passage (QRP)
Generated NONSENSE String
SciFact Query: Activator-inhibitor pairs are provided dorsally by Admpchordin.
Regulation of ADMP and BMP2/4/7 at Opposite Embryonic Poles Generates a Self-Regulating Morphogenetic Field. Embryos have the ability to self-regu … Dense Similarity Score (Contriever): 0.485
SciDocs Query: Subjectivity and Sentiment Analysis of Modern Standard Arabic
Appendix
Table 10: Representative Contriever examples selected with sample seed 20260526. Each block shows one query per dataset, the model’s highest-scoring judged query-relevant passage (QRP), and the optimized NONSENSE string from the seed-4201 main run. Text is truncated for display; scores are dense similarities under the evaluated retriever.