Less but Sufficient: Relation Knowledge Selects Compact Knowledge-Graph Contexts, and What External Checks Can and Cannot Catch
Organizations: Department of Artificial Intelligence, Jeju National University · Department of Computer Science and Engineering, Konkuk University
Abstract
Large language models (LLMs) read knowledge-graph facts ranked by relevance, leaving open whether they suffice or conflict. Our checks use relation knowledge, written by hand or learned from training questions labelled with their relation paths. It selects a compact context of facts yielding an answer, and derives it; a derivability check tests whether an answer follows, a consistency check whether facts conflict, and a validation-chosen routing gate decides where the derived answer replaces the LLM's. Seven open LLMs read the compact context better in exact match than relevance-ranked facts; on ComplexWebQuestions (CWQ) they match SubgraphRAG's facts at a quarter of the prompt tokens, and stronger LLMs read SubgraphRAG's better. Where the routing gate passes, ours leads SubgraphRAG's in exact match in all 53 LLM, benchmark and prompt settings. A precision gate, returning the derived answer only when the check passes with exactly one derived entity, answered 18 and 5 percent of held-out WebQSP and CWQ questions in a pre-specified test, right for 90 and 96 percent. The consistency check judges facts by path evidence only where another path joins their entities, true of 2.5 percent of named-entity facts in our compact WebQSP and CWQ contexts.
Figures & tables
| PathQuestion | MetaQA | WebQSP | CWQ | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| context | CLUTRR | exact | Hit | F1 | tok. | exact | Hit | F1 | tok. | exact | Hit | F1 | tok. | exact | Hit | F1 | tok. |
| all facts or neighbourhood | 53.0 | 50.2 | 70.2 | 62.2 | 1,073 | 47.3 | 75.2 | 59.3 | 3,630 | 24.8 † | 51.6 † | 39.4 † | 6,557 † | – | – | – | – |
| similarity top- | 43.2 | 33.0 | 51.9 | 44.8 | 191 | 26.0 | 56.5 | 39.6 | 375 | 21.6 | 46.9 | 35.3 | 424 | – | – | – | – |
| KAPING | 33.7 | 19.0 | 33.4 | 26.2 | 172 | 31.3 | 50.9 | 38.7 | 224 | 28.5 | 52.6 | 41.6 | 311 | 10.4 | 23.7 | 17.7 | 318 |
| SubgraphRAG | – | 52.4 | 70.7 | 63.5 | 2,151 | 49.7 | 86.0 | 66.9 | 1,020 | 43.9 | 78.6 | 64.5 | 2,139 | 27.0 | 54.3 | 41.5 | 2,067 |
| Think-on-Graph | – | 44.5 | 65.8 | 55.8 | 3,658 | 33.5 | 59.8 | 44.4 | 2,906 | 29.0 | 63.3 | 46.5 | 5,676 | – | – | – | – |
| test | held-out | ||||
|---|---|---|---|---|---|
| benchmark | cov. | prec. | cov. | right / ret. | prec. |
| WebQSP | 19.0 | 91.3 [87.6, 93.9] | 17.6 | 79 / 88 | 89.8 [81.7, 94.5] |
| CWQ | 5.5 | 96.4 [93.1, 98.9] | 4.9 | 70 / 73 | 95.9 [87.0, 100.0] |
| routing gate passes | all questions | |||||
|---|---|---|---|---|---|---|
| knowledge | passes | derived | best LLM | GPT-5 | derived | routed |
| CLUTRR, rules | 99.8 | 99.4 | 73.3 | 91.7 ∗ | 99.2 | 99.2–99.3 |
| MetaQA, classifier | 100.0 | 99.9 | 89.0 | 97.0 | 99.9 | 99.9 |
| MetaQA, lexicon | 77.1 | 99.6 | 90.6 | 98.0 | 76.8 | 76.9–77.4 |
| PathQuestion, lexicon | 77.0 | 96.2 | 86.2 | 91.9 | 89.6 | 86.3–93.6 |
| WebQSP, chains | 85.1 | 65.1 | 66.0 | 69.8 | 56.3 | 57.8–60.9 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| benchmark | relation knowledge | question set | questions | check passes (%) | derived answer right where it passes (%) |
|---|---|---|---|---|---|
| CLUTRR | rules | stories from the benchmark’s generator, unused in testing | 257 | 100.0 | 100.0 |
| MetaQA | classifier, on the classifier’s context | dev split, 300 sampled per hop, 886 kept | 886 | 100.0 | 99.9 |
| MetaQA | classifier, on the lexicon’s context | dev split, 300 sampled per hop, 886 kept | 886 | 75.3 | 99.9 |
| PathQuestion | lexicon | held-out 20% of the training questions, at most 300 per subset | 719 | 78.6 | 95.2 |
| WebQSP | chains predicted from similar training questions | validation split | 246 | 40.7 | 85.0 |
| CWQ | chains voted from the SPARQL queries of similar training questions | validation split | 3,519 | 9.9 | 61.6 |
| policy | annotated | weak |
|---|---|---|
| LLM on SubgraphRAG’s 100 facts (no selection, no gate) | 43.9 | 43.9 |
| LLM on the compact context (selection) | 54.1 | 47.4 |
| derived answer where the routing gate passes, else LLM on the compact context (routing) | 59.1 | 47.8 |
| derived answer where the routing gate passes, else LLM on SubgraphRAG’s facts | 59.3 | 46.9 |
| LLM on the compact context where the routing gate passes, else on SubgraphRAG’s facts | 54.3 | 46.6 |
| routing gate instead of the vote share alone, as many derived answers returned: wrong derived answers 35.4 to 34.9% | ||
| routing gate passes: ours minus SubgraphRAG’s 100 facts | |||
| setting | exact match | Hit | F1 |
| WebQSP | +5.5 to +15.1 (7/0) | 1.3 to +3.5 (1/0) | +1.0 to +9.4 (6/0) |
| WebQSP, SubgraphRAG’s prompt | +3.2 to +16.9 (7/0) | 8.2 to 2.0 (0/5) | 2.7 to +4.3 (2/1) |
| WebQSP, GPT-5 without reasoning | +6.8 (1/0) | +0.9 (0/0) | +2.9 (1/0) |
| WebQSP, GPT-5 with reasoning | +1.2 (0/0) | 4.4 (0/1) | 2.2 (0/1) |
| CWQ, one-answer gate | +8.1 to +16.2 (7/0) | +4.4 to +13.0 (7/0) | +8.3 to +12.0 (7/0) |
| pipeline | Ll-3B | Qw-7B | Ll-8B | Ex-7.8B | Ge-27B | Qw-32B | Ex-32B |
|---|---|---|---|---|---|---|---|
| Balanced score, one necessary fact deleted (usually disconnects the queried people) | |||||||
| LLM decides alone | 0.303 | 0.644 | 0.386 | 0.399 | 0.770 | 0.715 | 0.747 |
| gate: queried people connected | 0.593 | 0.661 | 0.686 | 0.680 | 0.778 | 0.831 | 0.846 |
| gate: derivability, rules | 0.601 | 0.668 | 0.693 | 0.687 | 0.784 | 0.838 | 0.853 |
| Three-way score, one false fact added as well | |||||||
| LLM decides alone | 0.208 | 0.577 | 0.259 | 0.269 | 0.573 | 0.477 | 0.509 |
| contexts | counts | +GRU | rules |
|---|---|---|---|
| Contradicted compact contexts: share detected | |||
| CLUTRR, reversed edge / far | 99.9 / 87.6 | 99.9 / 99.9 | 99.9 / 99.8 |
| PathQuestion, reversed / two-step | 100.0 / 100.0 | 100.0 / 100.0 | – |
| WebQSP, reversed / via a record | 96.8 / 83.0 | 96.8 / 83.0 | – |
| WebQSP, second value | 96.4 | 96.4 | – |
| False fact in the graph before selection: share detected | |||
| (a) Named fact mentions (17,881) and contexts (2,895) | |||
|---|---|---|---|
| judgeable | % | flagged (%) | |
| mentions, second path | 451 | 2.5 | 27.6 |
| mentions, single-value rule only | 399 | 2.2 | 0.0 |
| mentions, either | 850 | 4.8 | 14.6 |
| contexts, second path | 107 | 3.7 | 29.9 |
| contexts, either | 406 | 14.0 | 7.9 |
| test | held-out | |||
| gate and benchmark | cov. | prec. | cov. | prec. |
| derivability check : an answer follows from the context | ||||
| WebQSP | 45.6 | 78.7 | 44.4 | 81.5 |
| CWQ | 11.8 | 52.0 | 9.7 | 62.1 |
| routing gate : where the derived answer replaces the LLM’s a | ||||
| WebQSP: the check or 20 answers | 85.1 | 65.1 | 85.4 | 66.0 |
| routing gate passes | routing gate fails | all questions | |||||||
|---|---|---|---|---|---|---|---|---|---|
| relation knowledge | passes | derived | LLMs | derived | LLMs | derived alone | routed | gain | derived |
| CLUTRR, rules | 99.8 | 99.4 | 36.3–73.3 | – | – | 99.2 | 99.2–99.3 | +26.0 to +63.0 | 0/7 |
| MetaQA, classifier | 100.0 | 99.9 | 58.4–89.0 | – | – | 99.9 | 99.9 | +10.9 to +41.4 | 0/7 |
| MetaQA, lexicon | 77.1 | 99.6 | 64.3–90.6 | 0.0 | 0.5–2.9 | 76.8 | 76.9–77.4 | +6.9 to +27.2 | 0/7 |
| PathQuestion, lexicon | 77.0 | 96.2 | 55.6–86.2 | 67.8 | 53.3–85.0 | 89.6 | 86.3–93.6 | +7.7 to +31.2 | 3/7 |
| WebQSP, chains | 85.1 | 65.1 | 50.5–66.0 | 6.2 | 16.0–36.6 | 56.3 | 57.8–60.9 | 0.7 to +12.5 | 7/7 |
| (a) contexts of Table 1 | PathQuestion, 782 questions | MetaQA, 900 questions | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5 | Qwen3-32B | open LLMs | GPT-5 | Qwen3-32B | open LLMs | |||||||||||||
| context | exact | Hit | F1 | exact | Hit | F1 | exact | Hit | F1 | exact | Hit | F1 | exact | Hit | F1 | exact | Hit | F1 |
| one-hop neighbourhood | 77.0 | 79.9 | 78.8 | 68.7 | 72.6 | 71.1 | 50.2 | 70.2 | 62.2 | 57.6 | 78.2 | 64.6 | 53.3 | 73.7 | 60.9 | 47.3 | 75.2 | 59.3 |
| similarity top- , as many facts as ours | 38.6 | 49.1 | 46.9 | 35.2 | 45.7 | 42.6 | 33.0 | 51.9 | 44.8 | 24.9 | 50.4 | 35.9 | 24.6 | 47.3 | 35.0 | 26.0 | 56.5 | 39.6 |
| KAPING | 19.3 | 27.4 | 21.6 | 20.5 | 29.3 | 23.1 | 19.0 | 33.4 | 26.2 | 32.6 | 45.4 | 36.6 | 31.1 | 41.4 | 34.8 | 31.3 | 50.9 | 38.7 |
| SubgraphRAG | 86.2 | 90.0 | 88.6 | 75.3 | 80.3 | 78.4 | 52.4 | 70.7 | 63.5 | 67.0 | 91.3 | 77.5 | 60.6 | 87.7 | 73.9 | 49.7 | 86.0 | 66.9 |
Knowledge Graph Question Answering
Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.
See all 173 papers