Information Retrieval (IR) systems seek to identify relevant documents within a collection. In practical applications, collections are dynamic, with documents frequently added. We argue that ideally, a retriever's effectiveness should not decrease when non-relevant documents are added to a collection. This study formalises this concept and empirically evaluates it by merging two collections with negligible topic overlap. We hypothesise that the way an IR model conditions its ranking on other documents in a collection (e.g., the IDF component in BM25 or contextual documents in listwise rerankers) plays an important role in its robustness to the addition of non-relevant documents. We broadly classify models as those that do not depend on other documents (Multi-Document-Agnostic, MDA) and those that do (Multi-Document-Dependent, MDD). Our results show that neither MDD nor MDA models are fully robust to the addition of non-relevant documents, as all models exhibit some performance degradation. Interestingly, among the models we test, MDA is more effective than MDD for retrieval, whereas MDD and MDA rerankers are equally effective.
Figures & tables
Figure 1. Inter-Document Interaction Taxonomy. Models are organised by (i) how many documents influence scoring (single, top-k, all) and (ii) interaction type (text, vector, statistics), distinguishing MDA methods that score documents independently from MDD methods that model inter-document dependencies, with representative retrieval and re-ranking approaches shown.
Statistic
TREC-COVID
DL-2019
Hom
Het
Hom
Het
Het +
Total Documents
171K
9.1M
8.8M
9.1M
9.1M
Avg. Doc. Length
197.1
58.3
56.3
58.3
58.3
TREC-COVID Docs (%)
100.0
1.9
0.0
1.9
1.9
MS MARCO Docs (%)
0.0
98.1
100.0
98.1
98.1
Total Queries
200
200
50
50
50
Table 1. Statistics and relevance label distributions for TREC-COVID and DL-2019. Hom, Het, and Het + represent the original, combined, and augmented qrel configurations, respectively.
Figure 2. Relevance shifts under collection growth. Results are reported for nDCG@10 on TREC-COVID and DL-2019, evaluated on homogeneous (Circle) and heterogeneous (Cross, TREC-COVID + DL-2019) collections.
Retrieval robustness studies how search systems preserve relevant results when queries, data, collections, or evidence are noisy, shifted, or adversarial.