CLADE: Category-Level Answering from Deficient Evidence in Retrieval-Augmented Language Models
Organizations: Rensselaer Polytechnic Institute, Troy, NY, USA
Abstract
Retrieval-augmented systems often retrieve passages that are accurate and relevant but state the answer at a coarser level of granularity than the question requires. For example, a question may ask which specific disease a drug treats (familial hyperlipidemia), while the retrieved passage mentions the broader disease family (hyperlipidemia). Prior work on context sufficiency treats such coarse passages simply as insufficient. We argue that they form a distinct case, which we call resolvable insufficiency: text alone cannot settle the fine-grained answer, but the broader category it mentions maps to a short list of candidate answers in a domain ontology. When language models read these coarse, family-only passages, they exhibit a phenomenon we call Clade (Category-Level Answering from Deficient Evidence): they answer instead of abstaining or searching further. On biomedical questions from PrimeKG, family-only passages reduce abstention by 13.8 points compared to uninformative passages, and the resulting guesses are wrong about half the time. A counterfactual test confirms the cause: replacing an exact disease name with its broader family name lowers abstention by 17.4 points compared to deleting the sentence. Simple fixes like confidence filtering, scaling to a larger model, and explicitly instructing the model to abstain cannot resolve Clade, and this replicates on ecological species interactions in GloBI. Finally, we test whether an ontology can turn coarse passages into exact answers. Given the correct family, an ontology list includes the true answer in 98.5% of cases, but string matching finds that family on the page for only 19.8%. Letting the model choose the family raises this to 30.0% and end-to-end recovery from 10.0% to 26.8%. Overall, our results show when language models should stop guessing and consult an ontology, and highlight which steps of that process remain unsolved.
Figures & tables
| PrimeKG | GloBI | |
| Source records | 8,100,498 | 7,822,670 |
| Edges | indication 9,388 | eats 235,960 |
| contraind. 30,675 | preys on 87,678 | |
| off-label 2,568 | parasite of 117,894 | |
| pollinates 15,854 | ||
| Hierarchy | MONDO | GloBI taxonomy |
| Text names it | Text does not | ||||
|---|---|---|---|---|---|
| Relation | knows | not | knows | not | |
| Indication | 600 | 9.7 | 7.2 | 21.8 | 61.3 |
| Contraind. | 600 | 1.5 | 5.2 | 20.7 | 72.7 |
| Off-label | 600 | 4.3 | 6.2 | 24.2 | 65.3 |
| All | 1,800 | 5.2 | 6.2 | 22.2 | 66.4 |
| Relation | Text | Named | Family | Silent | |
|---|---|---|---|---|---|
| Indication | Wikipedia | 1,722 | 20.3 | 15.5 | 64.2 |
| PubMed | 1,722 | 26.3 | 18.6 | 55.1 | |
| both | 1,722 | 29.0 | 19.7 | 51.3 | |
| Contraind. | Wikipedia | 1,722 | 9.8 | 4.1 | 86.1 |
| PubMed | 1,722 | 11.4 | 5.4 | 83.2 | |
| both | 1,722 | 15.9 | 6.9 | 77.2 |
| Model | Passage Type | Correct | Wrong | Declines | Prec. |
|---|---|---|---|---|---|
| Qwen2.5 7B | named | 71.7 | 3.8 | 24.5 | 95.0 |
| family-only | 26.5 | 18.1 | 55.3 | 59.4 | |
| silent | 13.8 | 2.5 | 83.8 | 84.6 | |
| OLMo 2 7B | named | 77.4 | 15.1 | 7.5 | 83.7 |
| family-only | 37.2 | 44.7 | 18.1 | 45.5 | |
| silent | 30.6 | 24.4 | 45.0 | 55.7 |
| Metric | PrimeKG | GloBI |
| Questions in pooled set (all-neighbor) | 481 | 833 |
| Abstention page effect, family silent | ||
| Wrong-answer page effect, family silent | ||
| Precision on named passages (%) | 84–95 | 84–95 |
| Precision on family-only passages (%) | 45–59 | 22–28 |
| Confidence AUROC on named passages | .73–.87 | .82–.86 |
| Metric | Given | Page-built | Model-chosen |
|---|---|---|---|
| Usable lists (of 410) | 410 | 166 | 375 |
| Holds sampled answer | 98.5% | 19.8% | 30.0% |
| Holds any true answer | 99.5% | 24.6% | 44.9% |
| Median list size | 14 | 32 | 12 |
| End-to-end recovery | – | 10.0% | 26.8% |
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
| Main paper | Supplement |
|---|---|
| Data, page types, Tables 1 and 3 | Appendix B |
| Odds ratios, CIs, -values, Eq. 1 | Appendix C |
| Questions, prompts, Table 4 , Figure 2 | Appendix D |
| Page edits, Figure 3 | Appendix E |
| Confidence, scale, instruction, Figure 4 | Appendix F |
| GloBI, Table 5 | Appendix G |
| True disease | Unrelated diseases | |||
|---|---|---|---|---|
| Relation | named | + family | named | + family |
| Indication | 19.3 | 34.7 | 1.4 | 2.7 |
| Contraindication | 6.7 | 9.3 | 0.9 | 1.8 |
| Off-label use | 11.8 | 23.7 | 0.9 | 3.3 |
| Relation | Named | Family only | Silent |
|---|---|---|---|
| Indication | 84.2 (101) | 44.7 (94) | 44.0 (405) |
| Contraind. | 32.5 (40) | 18.8 (16) | 23.4 (544) |
| Off-label | 57.1 (63) | 37.5 (72) | 31.6 (465) |
| All | 65.7 (204) | 39.6 (182) | 32.0 (1,414) |
| Page type | Relation | All-neighbor (%) | |
|---|---|---|---|
| Named | both | 300 | 35.3 |
| Family-only | contraindication | 106 | 83.0 |
| Family-only | indication | 150 | 84.7 |
| Silent | contraindication | 150 | 45.3 |
| Silent | indication | 150 | 61.3 |
| Model | Group | Correct | Wrong | Declines |
|---|---|---|---|---|
| Qwen2.5 7B | named | 60.4 | 17.9 | 21.7 |
| family-only | 35.3 | 32.6 | 32.1 | |
| silent | 25.0 | 15.6 | 59.4 | |
| OLMo 2 7B | named | 65.1 | 31.1 | 3.8 |
| family-only | 36.7 | 56.7 | 6.5 | |
| silent | 37.5 | 51.2 | 11.2 |
| Model | Passage | Corr. | Wrong | Decl. | Prec. | Conf. |
|---|---|---|---|---|---|---|
| Qwen2.5 7B | named | 64.7 | 3.3 | 32.0 | 95.1 | 0.99 |
| family-only | 29.3 | 17.6 | 53.1 | 62.5 | 1.00 | |
| silent | 13.0 | 2.7 | 84.3 | 83.0 | 0.99 | |
| OLMo 2 7B | named | 71.3 | 20.7 | 8.0 | 77.5 | 0.80 |
| family-only | 39.1 | 42.6 | 18.4 | 47.8 | 0.84 | |
| silent | 32.0 | 28.0 | 40.0 | 53.3 | 0.76 |
| Declines (Abstention) | Correct Answers | Wrong Answers | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | Fam. | Sil. | Diff. [95% CI], | Fam. | Sil. | Diff. [95% CI], | Fam. | Sil. | Diff. [95% CI], |
| Qwen2.5 7B | +23.3 | +24.4 | [ , 8.4], .82 | +2.4 [ , 10.5], .57 | [ , 7.1], .76 | ||||
| OLMo 2 7B | +11.6 | +33.8 | [ , ], 6e-6 | +0.5 | +7.3 [ , 17.0], .15 | +14.8 [3.6, 26.1], .009 | |||
| Granite 3.1 8B | +9.3 | +27.5 | [ , ], .0006 | +8.5 [ , 17.8], .064 | +9.7 [ , 19.9], .056 | ||||
| Qwen2.5 0.5B | +9.8 | +18.1 | [ , ], .027 | +4.2 | 0.0 | +4.2 [ , 14.4], .42 | +4.2 [ , 14.6], .44 | ||
| Pooled 7–8B | +14.7 | +28.5 | [ , ], 4e-5 | +6.1 [ , 12.4], .058 | +7.7 [1.2, 14.0], .019 | ||||
| Indication | Contraindication | |||
|---|---|---|---|---|
| Model | Family-only | Silent | Family-only | Silent |
| Qwen2.5 7B | 26.0 | 2.2 | 6.8 | 2.9 |
| OLMo 2 7B | 38.6 | 21.7 | 53.4 | 27.9 |
| Granite 3.1 8B | 37.8 | 16.3 | 10.2 | 1.5 |
| Qwen2.5 0.5B | 46.5 | 46.7 | 69.3 | 55.9 |
| Family size | Median | Decline effect [95% CI] | ||
|---|---|---|---|---|
| 71 | 7 | [ , ] | .001 | |
| 11–20 | 69 | 16 | [ , ] | .001 |
| 68 | 44 | [ , ] | .174 |
| Model | Version | Decl. | Corr. | Wrong | Prec. |
|---|---|---|---|---|---|
| PrimeKG (130 questions) | |||||
| Qwen2.5-7B | original | 27.7 | 66.9 | 5.4 | 92.6 |
| family | 43.8 | 46.2 | 10.0 | 82.2 | |
| removed | 70.0 | 26.9 | 3.1 | 89.7 | |
| OLMo 2 7B | original | 6.2 | 74.6 | 19.2 | 79.5 |
| family | 7.7 | 58.5 | 33.8 | 63.3 | |
| Outcome | Family removed | Family original | |
|---|---|---|---|
| PrimeKG | Declines | [ , ] | [ , ] |
| Wrong | [ , ] | [ , ] | |
| Correct | [ , ] | [ , ] | |
| GloBI | Declines | [ , ] | [ , ] |
| Wrong | [ , ] | [ , ] | |
| Correct | [ , ] | [ , ] |
| Model | Page | Ans. | Prec. | Conf. R | Conf. W | AUROC |
|---|---|---|---|---|---|---|
| PrimeKG | ||||||
| Qwen2.5-0.5B | named | 97 | 84.5 | .83 | .49 | .85 |
| family | 194 | 38.1 | .71 | .60 | .64 | |
| Qwen2.5-7B | named | 80 | 95.0 | .98 | 1.00 | .73 |
| family | 96 | 59.4 | .98 | .90 | .72 | |
| OLMo 2 7B | named | 98 | 83.7 | .97 | .75 | .86 |
| Declines | Precision | ||||
|---|---|---|---|---|---|
| Model | Page | base | instr. | base | instr. |
| Qwen2.5-7B | named | 24.5 | 29.2 | 95.0 | 94.7 |
| family | 55.4 | 64.2 | 59.4 | 64.9 | |
| silent | 83.8 | 88.1 | 84.6 | 89.5 | |
| OLMo 2 7B | named | 7.5 | 27.4 | 83.7 | 89.6 |
| family | 18.1 | 56.3 | 45.5 | 47.9 | |
| Page type | eats | parasite of | pollinates | preys on |
|---|---|---|---|---|
| Named | 93.3 | 94.7 | 94.7 | 89.3 |
| Family-only | 100.0 | 98.7 | 100.0 | 93.3 |
| Silent | 90.7 | 85.3 | 92.0 | 78.7 |
| Correct | Wrong | Declines | |||||
|---|---|---|---|---|---|---|---|
| Model | Page | cl. | page | cl. | page | cl. | page |
| Qwen2.5-7B | named | 7.9 | 56.6 | 4.3 | 3.2 | 87.8 | 40.1 |
| family | 1.7 | 1.7 | 2.7 | 4.4 | 95.6 | 93.9 | |
| silent | 3.5 | 0.4 | 1.2 | 1.2 | 95.4 | 98.5 | |
| OLMo 2 7B | named | 20.8 | 69.9 | 24.7 | 12.9 | 54.5 | 17.2 |
| family | 10.9 | 12.6 | 22.8 | 34.4 | 66.3 | 53.1 | |
| Model | Outcome | [95% CI] | ||
| Qwen2.5-7B | declines | [ , ] | .022 | |
| wrong | [ , ] | .26 | ||
| OLMo 2 7B | declines | [ , ] | 2e-11 | |
| wrong | [ , ] | 1e-8 | ||
| Granite 3.1 8B | declines | [ , ] | .097 | |
| wrong | [ , ] | .019 |
| Wikipedia | + PubMed | |
| Matched families per page (median) | 5 | 17 |
| Usable lists (of 410) | 166 | 19 |
| Lists over the 64 cap | 209 | 391 |
| Usable list holds sampled answer | 19.8% | 3.7% |
| Usable list holds any true answer | 24.6% | 4.1% |
| Median usable list size | 32 | 59 |
| Signal | Picks | Ties | None | Acc./pick | Acc./all |
|---|---|---|---|---|---|
| Wikipedia lookup | 82 | 80 | 248 | 50.0 | 10.0 |
| PubMed co-mention | 324 | 12 | 74 | 45.4 | 35.9 |
| Combined evidence | 327 | 14 | 69 | 44.6 | 35.6 |
| Popularity prior | 409 | 0 | 1 | 46.9 | 46.8 |
| Normalized PubMed | 336 | 0 | 74 | 31.0 | 25.4 |
| Condition | Sampled | Any true |
|---|---|---|
| Model, given-family list | 18.3 | 35.9 |
| Model + page, given-family list | 17.8 | 32.4 |
| Model, random list | 30.5 | 34.9 |
| Model + page, random list | 29.0 | 32.7 |
| Drug | PrimeKG Target | Evidence Pick | Match? |
|---|---|---|---|
| Lovastatin | familial hyperlipidemia | familial hyperlipidemia | yes |
| Tetrabenazine | juvenile Huntington disease | juvenile Huntington disease | yes |
| Moxifloxacin | Klebsiella pneumonia | Legionnaires’ disease | no |
| Carboplatin | malignant germ cell tumor of ovary | ovarian carcinoma | no |
| Rule | Sampled answer | Any true answer |
|---|---|---|
| Qwen2.5-7B picks the family | 30.0 | 44.9 |
| First mentioned | 25.1 | 37.3 |
| Random | 13.9 | 22.0 |
| Smallest family | 7.8 | 13.2 |
| Page-built list (all families) | 19.8 | 24.6 |
| Any shown family (ceiling) | 50.0 | 64.6 |
| Model | Page | Corr. | Abst. | Family | Sibl. | Other |
|---|---|---|---|---|---|---|
| Qwen2.5-7B | named | 20.0 | 47.5 | 0.0 | 0.0 | 32.5 |
| family | 0.0 | 42.5 | 17.5 | 0.0 | 40.0 | |
| silent | 0.0 | 52.5 | 0.0 | 0.0 | 47.5 | |
| OLMo 2 7B | named | 25.0 | 37.5 | 0.0 | 0.0 | 37.5 |
| family | 0.0 | 32.5 | 20.0 | 2.5 | 45.0 | |
| silent | 0.0 | 52.5 | 0.0 | 0.0 | 47.5 |