The de facto paradigm for applying dense retrieval (DR) to new tasks involves fine-tuning a pre-trained model for a specific task. However, this paradigm has two significant limitations: (1) It is difficult adapt the DR to a new domain if the training dataset is limited. (2) Old DR models are simply replaced by newer models that are trained from scratch when the former are no longer up to date. Especially for scenarios where the model needs to be updated frequently, this paradigm is prohibitively expensive. To address these challenges, we propose a novel dense retrieval approach, termed dynamic dense retrieval (DDR). DDR uses prefix tuning as a module specialized for a specific domain. These modules can then be compositional combined with a dynamic routing strategy, enabling highly flexible domain adaptation in the retrieval part. Extensive evaluation on six zero-shot downstream tasks demonstrates that this approach can surpass DR while utilizing only 2% of the training parameters, paving the way to achieve more flexible dense retrieval in IR. We see it as a promising future direction for applying dense retrieval to various tasks.
Figures & tables
Figure 1. Overview of DDR framework, built upon a dual-encoder architecture. The general prefix vector learns the general knowledge across tasks, while the domain prefix focuses on the specificity of each task domain. The frozen backbone incorporates both the general prefix and aspect prefix for query encoding, whereas the passage encoder only utilizes the general prefix.
Task
Instruction
NF Corpus
Retrieve scientific paper paragraph to answer this question.
SciFact
Retrieve a scientific paper sentence to verify if the following claim is true.
domains
Science , QA , Fact-Checking , Wikipedia
Table 1. Examples of instruction decomposition for retrieval tasks in the BERRI ( Asai et al., 2022 ) . Each task has diverse domains, which are denoted by experts.
Method
PLM (# Params)
TREC
NFC
SCD
SCF
CLI
DBP
Avg.
SCI QA
SCI QA
SCI SUM
SCI FC
FC
WIKI
DeepCT
BERT-base (110M)
40.6
28.3
12.4
63.0
6.6
17.7
28.1
SPARTA
DistilBERT (66M)
53.8
30.1
12.6
58.2
8.2
31.4
32.8
Contriever
BERT-base (110M)
27.4
31.7
14.9
64.9
15.5
29.2
30.6
ANCE
RoBERTa-base (110M)
65.4
23.7
12.2
50.7
19.8
28.1
33.3
DR
coCondenser (110M)
63.6
28.8
11.5
48.6
19.9
36.3
34.8
Table 2. Zero-shot retrieval results measured by NDCG@10. # Params denote the trainable parameters. The highest results among dense retrievers are bolded, and the second-best results are underlined . We report the results of baselines from their original papers. The labeled domains for each task are listed in colors: Science , QA , Fact-Checking , Wikipedia , Summarization .
Method
PLM (# Params)
TREC
NFC
SCD
SCF
CLI
DBP
Avg.
SCI QA
SCI QA
SCI SUM
SCI FC
FC
WIKI
DDR-prior
coCondenser (2.3M×8)
65.3
31.4
15.4
54.0
22.3
36.3
37.5
w/o routing
coCondenser (2.3M×8)
64.3
30.9
13.5
52.8
22.3
36.3
36.3
w/o prior
coCondenser (2.3M×8)
60.9
31.1
13.2
52.3
20.0
34.6
35.4
Table 3. Ablation study of DDR. We remove the trainable routing and assign uniform weights to all modules (w/o routing). We then remove the prior domain labels and let DDR activate all eight modules (w/o prior).