cs.IRNov 17, 2026

NONSENSE: Non-Semantic Strings Expose Semanticity Gaps in Dense Retrieval

Authors: Ankit Gupta

Organizations: Department of Computer Science and Engineering, Michigan State University, East Lansing, United States

Abstract

Dense retrieval is central to modern search, question answering, and retrieval-augmented generation, where high similarity is often treated as evidence of semantic relevance. We examine how far this assumption can be stressed. NONSENSE is a black-box sparse-edit probe that constructs visibly non-semantic strings while forbidding query-term copying, optimizing only the dense similarity between a query and the string. Across five BEIR-style datasets and three popular single-vector retrievers, these strings frequently outscore the retriever's own best judged query-relevant passage (QRP), and often rank first in the full corpus. The result indicates a semanticity gap in the evaluated dense retrievers: nearest-neighbor status can be separated from passage-like meaning. These findings highlight the importance of evaluating dense retrievers not only by whether they retrieve relevant documents, but also by how reliably their similarity scores reflect semantic evidence.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Search-by-Answer Bias in Retrieval Models

    Nov 8, 2026Adiel Avigad, Shauli Ravfogel, Yoav GoldbergDense RetrievalIR Evaluation

  2. Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

    May 28, 2026Benjamin Clavié, Sean Lee, Aamir Shakir +1Learned Sparse RetrievalInformation Retrieval

  3. Training for Compositional Sensitivity Reduces Dense Retrieval Generalization

    Mar 16, 2026Radoslav Ralev, Aditeya Baral, Iliya Zhechev +2Information RetrievalDense Retrieval