cs.CLFeb 19, 2026
SaveRPDR: A Round-trip Prediction-Based Data Augmentation Framework for Long-Tail Question Answering
Organizations: Zhejiang University · Nanyang Technological University · NYU Shanghai · Center for Data Science, New York University
Abstract
Long-tail question answering presents significant challenges for large language models (LLMs) due to their limited ability to acquire and accurately recall less common knowledge. Retrieval-augmented generation (RAG) systems have shown great promise in mitigating this limitation by integrating external retrieval mechanisms. However, dense retrieval models often face the same difficulties when generalizing to rare or niche knowledge. In this study, we introduce RPDR, a novel data augmentation framework that selects high-quality easy-to-learn training data, to enhance dense retrievers. Our approach is built around three core components: synthetic data generation, data selection with Round-Trip prediction to identify easy-to-learn instances, and retriever training with these instances. We evaluate RPDR on two long-tail retrieval benchmarks, PopQA and EntityQuestion, demonstrating substantial improvements over existing retrievers like BM25 and Contriver, especially on extremely long-tail categories. We identify the strengths and limitations of RPDR through detailed human analysis and propose a dynamic routing mechanism to dynamically route queries to specialized retrieval modules to further improve retrieval performance.
Figures & tables
Figure 1: (Top) Negative feedback loop in long-tail question answering in large language model-based systems. (Bottom) We challenge existing findings that dense retrievers struggle on long-tail questions, and argue that through appropriate training, dense retrieval based methods RPDR can surpass BM25 on long-tail retrieval.
| Property | PopQA | EntityQuestions |
| Knowledge Corpus | ||
| Source | Wikipedia | Wikipedia |
| Dataset Statistics | ||
| # Relations | 16 | 24 |
| Dataset Size (# Questions) | ||
| # Training Set | NA | 176.6k |
Table 1: Basic statistics of the PopQA and EntityQuestions used in our experiments. “NA" means no such set.
Figure 2: The RPDR framework consists of three main stages: (1) Synthetic Data Generation that generates a pool of long-tail QA pairs. (2) Data Selection with Round-trip Prediction that trains an inverse model to select easy-to-learn samples with reversibility. (3) Retriever Training that trains a dense retriever using the augmented samples.
| Method | PopQA | EntityQuestions | ||||||||||
| Long-tail | Infrequent | Frequent | Long-tail | Infrequent | Frequent | |||||||
| R@10 | R@20 | R@10 | R@20 | R@10 | R@20 | R@10 | R@20 | R@10 | R@20 | R@10 | R@20 | |
| BM 25 | 62.6 | 66.8 | 46.2 | 52.8 | 53.8 | 61.1 | 48.3 | 63.5 | 41.9 | 66.7 | 55.0 | 69.2 |
| Contriever | 52.3 | 59.3 | 53.3 | 61.1 | 80.0 | 85.8 | 42.9 | 53.3 | 53.4 | 61.0 | 57.3 | 67.6 |
| BGE | 53.9 | 61.2 | 57.4 | 65.3 | 82.4 | 87.9 | 44.7 | 60.3 | 55.1 | 69.8 | 60.9 | 75.2 |
| NV-Embed | 55.0 | 62.3 | 63.5 | 70.6 | 89.9 | 90.6 | 48.4 | 61.6 | 57.1 | 70.7 | 71.5 | 84.3 |
Table 2: Comparison of retrieval performance between RPDR and baseline retrievers for questions with long-tail, infrequent, and frequent entities in the PopQA and EntityQuestions datasets. “R@k” means Retrieval Accuracy in the top-k retrieved passages. “RPDR-random” is a model trained with randomly sampled data from the augmented long-tail distribution. More details refer to Appendix B .
| Generator | Retriever | PopQA | EntityQuestions | ||||
| Long-tail | Infrequent | Frequent | Long-tail | Infrequent | Frequent | ||
| GPT-j-6B | Parametric | 11.3 | 14.4 | 18.2 | 9.7 | 12.1 | 12.9 |
| BM25 | 24.9 | 19.4 | 24.5 | 23.2 | 27.1 | 29.8 | |
| Contriever | 19.2 | 24.7 | 39.1 | 16.5 | 26.6 | 30.2 | |
| RPDR | 30.2 | 29.7 | 40.0 | 28.9 | 32.8 | 29.2 | |
| GPT-neox 20B | Parametric | 13.5 | 21.2 | 28.9 | 10.1 | 12.8 | 14.8 |
Table 3: End-to-end QA exact match accuracy comparison for RAG systems with different retriever and reader models on the PopQA and the EntityQuestions datasets. “Parametric” means the generator models answer the query by their parametric knowledge.
| Category | Long-tail | Infrequent | Frequent | |||
| R@10 | R@20 | R@10 | R@20 | R@10 | R@20 | |
| Q correct & A correct | 74.5 | 75.7 | 76.8 | 78.2 | 81.3 | 87.1 |
| Q wrong & A correct | 66.8 | 74.9 | 68.6 | 76.6 | 77.6 | 85.4 |
| Q wrong & A wrong | 62.9 | 69.8 | 53.9 | 62.0 | 65.8 | 73.8 |
| RPDR -Random | 66.8 | 70.0 | 70.0 | 74.5 | 82.7 | 87.3 |
Table 4: Retrieval performance on the PopQA dataset for retrievers trained with different categories of augmented data. “Q” means the question entity, while “A” means the answer entity. “correct”/“wrong” means the entity can be recovered correctly/wrongly.
Figure 3: Case study of “John XIX”. “Original” means the top-1 retrieved passage from the off-the-shelf Contriever, while “Updated” means the one from RPDR .
| PopQA | ||||||
| Method | Long-tail | Infrequency | Frequency | |||
| R@10 | R@20 | R@10 | R@20 | R@10 | R@20 | |
| RPDR | 74.5 | 75.7 | 76.8 | 78.2 | 81.3 | 87.1 |
| RPDR + RM | 79.1 | 80.3 | 78.7 | 80.0 | 81.3 | 87.9 |
| EntityQuestions | ||||||
| Method | Long-tail | Infrequency | Frequency | |||
Table 5: Retrieval performance comparison of RPDR without and with the routing mechanism on the PopQA and the EntityQuestions datasets.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Generator | Retriever | PopQA | EntityQuestions | ||||
| Long-tail | Infrequent | Frequent | Long-tail | Infrequent | Frequent | ||
| LLaMA-3 8B | Contriever | 23.7 | 34.9 | 39.1 | 21.8 | 25.4 | 30.2 |
| FULL-random | 23.3 | 34.9 | 42.1 | 22.6 | 27.8 | 33.7 | |
| RPDR -random | 25.2 | 32.5 | 42.0 | 27.4 | 27.1 | 32.4 | |
| RPDR | 29.8 | 34.7 | 41.8 | 30.1 | 31.6 | 33.9 | |
Table 6: End-to-end QA exact match accuracy comparison for RAG systems with different retriever and LLaMA-3 8B on the PopQA and the EntityQuestions datasets. “FULL-random” means the retriever model trained on the examples randomly sampled from the same distribution as the data source. “ RPDR -random” means the retriever model trained on the examples randomly sampled from the same long-tail distribution as RPDR.
Figure 4: The Relationship between augmented data scale and retriever model performance. In the left figure, EQ represents EntityQuestions . The x-axis indicates the proportion of the dataset size relative to the full augmentated dataset.
| Method | PopQA | EntityQuestions | ||||
| Long-tail | Infrequent | Frequent | Long-tail | Infrequent | Frequent | |
| Recall@10 | Recall@10 | Recall@10 | Recall@10 | Recall@10 | Recall@10 | |
| BM 25 | 54.3 | 48.5 | 66.9 | 41.7 | 51.3 | 60.8 |
| Contriever | 61.2 | 67.4 | 87.3 | 56.9 | 63.2 | 73.1 |
| BGE | 66.4 | 77.5 | 89.7 | 61.1 | 69.8 | 77.1 |
| NV-Embed | 71.8 | 80.8 | 94.5 | 67.3 | 75.2 | 81.9 |
Table 7: Comparison of retrieval performance between RPDR and baseline retrievers for questions with long- tail, infrequent, and frequent entities in the POPQA and ENTITYQUESTIONS datasets. “Recall@10" means the gold passage being in the top-10 recall results.
| Model | Long-tail | Infrequent | Frequent |
| Contriever | 52.3 | 53.3 | 80.0 |
| RPDR-random | 66.8 | 70.0 | 82.7 |
| RPDR (Q wrong & A wrong) | 62.9 | 53.9 | 65.8 |
| RPDR-trained with full dataset | 72.1 | 71.9 | 73.4 |
| RPDR | 74.5 | 76.8 | 81.3 |
Table 8: Comparison across different training data.
| Cycle 1 | Cycle 2 | Cycle 3 | |
| Data scale | 22k | 23k | 23k |
Table 9: Comparison across different training data.
| Error Type | Query | Answer | Retrieved Passage |
| Hard-to-Embed Target Entities | What is Edwin Wallock ’s occupation? | actor, actress | Charles Edwin (died 1756): The Government could field no candidates at the rerun of the election and Edwin was returned unopposed as Member of Parliament (MP) for Westminster … Charles Edwin (died 1756) |
| Semantic Ambiguity | What genre is Milk ? | biographical film, biopic | Milk: … Freezing of milk can cause fat globule aggregation upon thawing, resulting in milky layers and butterfat lumps. … Milk is often served in coffee and tea. Steamed milk is used to prepare espresso-based drinks such as cafe latte. |
| Semantic Ambiguity | Who was the director of Finale ? | Ken Kwapis, Kenneth William Kwapis | Finale (software): Finale is the flagship program of a series of proprietary music notation software developed and released by MakeMusic for the Microsoft Windows and macOS operating systems. … including the score for an entire ensemble (e.g., orchestra, concert band, big band, etc.) and parts for the individual musicians. |
| Unanswerable Queries | What genre is Great Expectations ? | fiction, fictional | Great Expectations: The film adaptation in 1946 gained the greatest acclaim… Following are highlights of the adaptations for film and television, and for the stage, since the early 20th century. |
Table 10: Three major categories of RPDR errors.
More in this topic
Dense Retrieval
Dense retrieval uses learned vector representations to find relevant documents, passages, products, or tools by semantic similarity.
See all 152 papers