cs.CLNov 1, 2026
SaveLess but Sufficient: Relation Knowledge Selects Compact Knowledge-Graph Contexts, and What External Checks Can and Cannot Catch
Organizations: Department of Artificial Intelligence, Jeju National University · Department of Computer Science and Engineering, Konkuk University
Abstract
Large language models (LLMs) read knowledge-graph facts ranked by relevance, leaving open whether they suffice or conflict. Our checks use relation knowledge, written by hand or learned from training questions labelled with their relation paths. It selects a compact context of facts yielding an answer, and derives it; a derivability check tests whether an answer follows, a consistency check whether facts conflict, and a validation-chosen routing gate decides where the derived answer replaces the LLM's. Seven open LLMs read the compact context better in exact match than relevance-ranked facts; on ComplexWebQuestions (CWQ) they match SubgraphRAG's facts at a quarter of the prompt tokens, and stronger LLMs read SubgraphRAG's better. Where the routing gate passes, ours leads SubgraphRAG's in exact match in all 53 LLM, benchmark and prompt settings. A precision gate, returning the derived answer only when the check passes with exactly one derived entity, answered 18 and 5 percent of held-out WebQSP and CWQ questions in a pre-specified test, right for 90 and 96 percent. The consistency check judges facts by path evidence only where another path joins their entities, true of 2.5 percent of named-entity facts in our compact WebQSP and CWQ contexts.
Figures & tables
Figure 1: Overview. (a) Two of eleven facts form a compact sufficient context, from which the rules derive the answer. (b) The check’s result and the derived set’s size feed the routing gate, chosen on validation questions, which returns the derived answer or the LLM’s; the precision gate, a separate policy, returns the derived answer only where the check passes with one answer; the consistency check is a variant used in the contradiction experiments, where a flagged context ends in an abstention. (c) A deleted fact leaves no answer to derive, so the derivability check catches it. The false fact that is a parent of is caught by the consistency check because a second path, the true fact sister , also joins and and contradicts it; few selected facts have a second path (Section 6 ). A wrong fact already in the graph, such as a needed fact whose object we replaced before selection, almost always escapes both checks once selected. (d) The lift of a word for a relation is how many times as often the relation lies on the annotated paths of training questions containing the word as on those of all training questions: ethnicity lies on all 15 paths of questions with “ethnicity” and on 2.5 percent of all paths, a lift of 40.1. The path whose relations are exactly those the words name passes (✗: they differ); its shaded end is the derived answer. (e) Each of the 10 training questions most similar to the question votes for its annotated chain with its cosine similarity as weight; spouse_s, spouse gets 89 percent of the total weight (its vote share), above the 60 percent threshold, and is followed.
| PathQuestion | MetaQA | WebQSP | CWQ | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| context | CLUTRR | exact | Hit | F1 | tok. | exact | Hit | F1 | tok. | exact | Hit | F1 | tok. | exact | Hit | F1 | tok. |
| all facts or neighbourhood | 53.0 | 50.2 | 70.2 | 62.2 | 1,073 | 47.3 | 75.2 | 59.3 | 3,630 | 24.8 † | 51.6 † | 39.4 † | 6,557 † | – | – | – | – |
| similarity top- | 43.2 | 33.0 | 51.9 | 44.8 | 191 | 26.0 | 56.5 | 39.6 | 375 | 21.6 | 46.9 | 35.3 | 424 | – | – | – | – |
| KAPING | 33.7 | 19.0 | 33.4 | 26.2 | 172 | 31.3 | 50.9 | 38.7 | 224 | 28.5 | 52.6 | 41.6 | 311 | 10.4 | 23.7 | 17.7 | 318 |
| SubgraphRAG | – | 52.4 | 70.7 | 63.5 | 2,151 | 49.7 | 86.0 | 66.9 | 1,020 | 43.9 | 78.6 | 64.5 | 2,139 | 27.0 | 54.3 | 41.5 | 2,067 |
| Think-on-Graph | – | 44.5 | 65.8 | 55.8 | 3,658 | 33.5 | 59.8 | 44.4 | 2,906 | 29.0 | 63.3 | 46.5 | 5,676 | – | – | – | – |
Table 1: LLM answers per context, forced-choice prompt, mean over the seven open LLMs (%). A published method’s row is the context it selects, not its full pipeline; neighbourhood: the one-hop neighbourhood of the question’s entities. † Six LLMs, as WebQSP’s neighbourhoods exceed Gemma-3-27B’s window. Exact: the whole gold set (CLUTRR: the label); Hit: some gold answer; tok.: prompt tokens per question, summed over a method’s LLM calls (on CLUTRR 161 to 187 for every context). Similarity top- holds as many facts as ours (on MetaQA as the classifier’s context, 14.6 facts against the lexicon’s 12.6). Bold: the best context, excluding the annotated path, a reference that misses PathQuestion’s further answers and is empty for 125 WebQSP and 787 CWQ questions. Ours (on MetaQA the lexicon’s context) is read on every question (pass rates: Table 10 ). –: not run; CLUTRR’s stories are given whole, RoG released only WebQSP and CWQ paths, and the other CWQ cells are in Appendix G .
| test | held-out | ||||
|---|---|---|---|---|---|
| benchmark | cov. | prec. | cov. | right / ret. | prec. |
| WebQSP | 19.0 | 91.3 [87.6, 93.9] | 17.6 | 79 / 88 | 89.8 [81.7, 94.5] |
| CWQ | 5.5 | 96.4 [93.1, 98.9] | 4.9 | 70 / 73 | 95.9 [87.0, 100.0] |
Table 2: The precision gate, the derivability check passing with one derived answer: share of questions answered (cov., %) and share of those answers that are exact (prec., %) on the test sets and in the held-out test specified in advance, with its right and returned counts. Brackets: 95 percent intervals, Wilson on WebQSP, seed-cluster bootstrap on CWQ.
| routing gate passes | all questions | |||||
|---|---|---|---|---|---|---|
| knowledge | passes | derived | best LLM | GPT-5 | derived | routed |
| CLUTRR, rules | 99.8 | 99.4 | 73.3 | 91.7 ∗ | 99.2 | 99.2–99.3 |
| MetaQA, classifier | 100.0 | 99.9 | 89.0 | 97.0 | 99.9 | 99.9 |
| MetaQA, lexicon | 77.1 | 99.6 | 90.6 | 98.0 | 76.8 | 76.9–77.4 |
| PathQuestion, lexicon | 77.0 | 96.2 | 86.2 | 91.9 | 89.6 | 86.3–93.6 |
| WebQSP, chains | 85.1 | 65.1 | 66.0 | 69.8 | 56.3 | 57.8–60.9 |
Table 3: Who should answer: share of questions passing the routing gate of Section 3 (%); accuracy there of the derived answer, the best open LLM and GPT-5 with reasoning; and, on all questions, the derived answer alone and routed, that is, the derived answer where the routing gate passes, else the open LLM, with no consistency check, as a range over the seven LLMs. With the check instead, WebQSP’s derived answer is 78.7 and GPT-5 82.2, CWQ’s 52.0 and 70.5 (other CWQ LLMs: Appendix G ). ∗ CLUTRR’s 264-question subset, where the derived answer is 99.2.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| benchmark | relation knowledge | question set | questions | check passes (%) | derived answer right where it passes (%) |
|---|---|---|---|---|---|
| CLUTRR | rules | stories from the benchmark’s generator, unused in testing | 257 | 100.0 | 100.0 |
| MetaQA | classifier, on the classifier’s context | dev split, 300 sampled per hop, 886 kept | 886 | 100.0 | 99.9 |
| MetaQA | classifier, on the lexicon’s context | dev split, 300 sampled per hop, 886 kept | 886 | 75.3 | 99.9 |
| PathQuestion | lexicon | held-out 20% of the training questions, at most 300 per subset | 719 | 78.6 | 95.2 |
| WebQSP | chains predicted from similar training questions | validation split | 246 | 40.7 | 85.0 |
| CWQ | chains voted from the SPARQL queries of similar training questions | validation split | 3,519 | 9.9 | 61.6 |
Table 4: Relation knowledge on questions not used for testing: the question set and its size, the share that pass the derivability check and the share of these whose derived answer is exact (%).
Figure 2: Mean exact match (blue, solid), Hit (orange, dashed) and F1 (green, dotted) against mean prompt tokens per question (log scale; each LLM’s tokenizer, GPT-5’s as billed); lines join SubgraphRAG’s 10, 20, 50 and 100 facts, other markers as in the legend. Panels 1 to 3: forced-choice prompt, the seven open LLMs (six for WebQSP’s one-hop neighbourhood, Appendix A ). Panel 4: WebQSP, SubgraphRAG’s own prompt, the same LLMs; its tokens include SubgraphRAG’s second request for the answer format, sent for 4.5 to 8.0 percent of questions on average. Panel 5: WebQSP, GPT-5, forced-choice prompt; pentagons without reasoning, stars with it; our context filled, SubgraphRAG’s facts hollow.
| policy | annotated | weak |
|---|---|---|
| LLM on SubgraphRAG’s 100 facts (no selection, no gate) | 43.9 | 43.9 |
| LLM on the compact context (selection) | 54.1 | 47.4 |
| derived answer where the routing gate passes, else LLM on the compact context (routing) | 59.1 | 47.8 |
| derived answer where the routing gate passes, else LLM on SubgraphRAG’s facts | 59.3 | 46.9 |
| LLM on the compact context where the routing gate passes, else on SubgraphRAG’s facts | 54.3 | 46.6 |
| routing gate instead of the vote share alone, as many derived answers returned: wrong derived answers 35.4 to 34.9% | ||
Table 5: What each part contributes on WebQSP, chain predictor fixed: mean exact match of the seven open LLMs (%) per policy, with annotated and with weakly supervised chains, that is, shortest question-to-answer paths with their own check as the routing gate. Below, the routing gate against the vote share alone at equal coverage, and the consistency check added to routing as a variant. Per LLM: Appendices H , L and K .
| routing gate passes: ours minus SubgraphRAG’s 100 facts | |||
| setting | exact match | Hit | F1 |
| WebQSP | +5.5 to +15.1 (7/0) | 1.3 to +3.5 (1/0) | +1.0 to +9.4 (6/0) |
| WebQSP, SubgraphRAG’s prompt | +3.2 to +16.9 (7/0) | 8.2 to 2.0 (0/5) | 2.7 to +4.3 (2/1) |
| WebQSP, GPT-5 without reasoning | +6.8 (1/0) | +0.9 (0/0) | +2.9 (1/0) |
| WebQSP, GPT-5 with reasoning | +1.2 (0/0) | 4.4 (0/1) | 2.2 (0/1) |
| CWQ, one-answer gate | +8.1 to +16.2 (7/0) | +4.4 to +13.0 (7/0) | +8.3 to +12.0 (7/0) |
Table 6: Our context minus SubgraphRAG’s 100 facts (points) where the routing gate passes: the derivability check on PathQuestion and MetaQA, the check or at most 20 derived answers on WebQSP and one derived answer on CWQ, rules chosen on validation questions; the pooled row covers the 53 settings of Section 5 . Below it, CWQ split by the derivability check instead, and a supervision control split by its own check. Forced-choice prompt unless SubgraphRAG’s is named; under SubgraphRAG’s prompt its evaluators score exact match, Hit and macro-F1. Cells: range over the row’s LLMs, the seven open LLMs or the named LLM, and in parentheses those with ours / SubgraphRAG’s ahead at (Appendix A ’s tests, Holm-adjusted over a row’s seven open LLMs; F1: Wilcoxon signed-rank). Passing: WebQSP 1,385 of 1,628 (85.1%; weakly supervised chains 417, 25.6), CWQ 864 of 3,531 by the one-answer gate (24.5) and 417 by the check (11.8), MetaQA 694 and 900 of 900 (lexicon, classifier; 77.1, 100.0), PathQuestion 602 of 782 (77.0). The failing side: the supplementary material.
| pipeline | Ll-3B | Qw-7B | Ll-8B | Ex-7.8B | Ge-27B | Qw-32B | Ex-32B |
|---|---|---|---|---|---|---|---|
| Balanced score, one necessary fact deleted (usually disconnects the queried people) | |||||||
| LLM decides alone | 0.303 | 0.644 | 0.386 | 0.399 | 0.770 | 0.715 | 0.747 |
| gate: queried people connected | 0.593 | 0.661 | 0.686 | 0.680 | 0.778 | 0.831 | 0.846 |
| gate: derivability, rules | 0.601 | 0.668 | 0.693 | 0.687 | 0.784 | 0.838 | 0.853 |
| Three-way score, one false fact added as well | |||||||
| LLM decides alone | 0.208 | 0.577 | 0.259 | 0.269 | 0.573 | 0.477 | 0.509 |
Table 7: Faithful abstention on CLUTRR, abstention prompt (scores: Section 6 ; three-way: with one reversed-edge or far contradiction). Policies: the LLM alone, or the LLM answering where the named checks pass and abstaining where one fails, which is how the contradiction experiments use the consistency check; no derived answer is returned. With full detection a gated score is (accuracy + 1)/2 or (accuracy + 2)/3, from the LLM’s accuracy where the checks pass. Learned: Appendix J , with the GRU as derivability check in both rows; with the GRU added, the consistency check flags exactly the same compact contexts as one whose GRU replaces the counts. Ll: Llama, Qw-7B: Qwen2.5-7B, Qw-32B: Qwen3-32B, Ex: EXAONE, Ge: Gemma; billions of parameters.
Figure 3: The learned consistency check (Appendix J ) on Figure 1 ’s example. (a) Relations the graph stores between the ends of training walks along one path ( < : against a stored fact). (b) The false fact’s score from the other path, below every consistent calibration context’s; the GRU addition gives .
| contexts | counts | +GRU | rules |
|---|---|---|---|
| Contradicted compact contexts: share detected | |||
| CLUTRR, reversed edge / far | 99.9 / 87.6 | 99.9 / 99.9 | 99.9 / 99.8 |
| PathQuestion, reversed / two-step | 100.0 / 100.0 | 100.0 / 100.0 | – |
| WebQSP, reversed / via a record | 96.8 / 83.0 | 96.8 / 83.0 | – |
| WebQSP, second value | 96.4 | 96.4 | – |
| False fact in the graph before selection: share detected | |||
Table 8: Contexts flagged (%; higher is better for contradicted contexts, lower for consistent ones), thresholds giving 1% false alarms on consistent calibration contexts. Counts: the default learned check (Appendix J ); +GRU: the path classifier added; before selection: false facts that are selected, pass the derivability check and are flagged, as a share of the n entered (WebQSP substitution: 2 of 1,132; 14 counting contexts that fail the check, Appendix J ); TransE plausibility alone detects at most 1.2% of the PathQuestion and WebQSP probes; –: no rules; via a record: through an unnamed record. WebQSP: 126, 53 and 83 contradicted contexts, 1,132 and 83 false facts in the graph. The injected contradictions create the second path the path scores need, which 2.5% of the named fact mentions in clean compact contexts have (Table 9 ). MetaQA’s consistency is not tested, as all its relations lead from a film to an attribute.
| (a) Named fact mentions (17,881) and contexts (2,895) | |||
|---|---|---|---|
| judgeable | % | flagged (%) | |
| mentions, second path | 451 | 2.5 | 27.6 |
| mentions, single-value rule only | 399 | 2.2 | 0.0 |
| mentions, either | 850 | 4.8 | 14.6 |
| contexts, second path | 107 | 3.7 | 29.9 |
| contexts, either | 406 | 14.0 | 7.9 |
Table 9: What the consistency check can judge in clean compact WebQSP and CWQ test contexts. (a) Mention: one fact in one context; named: both entities named. Judgeable: a second path joins the mention’s entities, or the single-value rule applies; flagged: false alarms among mentions not confirmed as errors, or contexts with a flagged mention. (b) Errors confirmed against Wikidata by two LLM passes and a tie-break (consensus), by GPT-5 and by both; suspicion-score AUC against supported facts, chance 0.5 (Appendix J ).
| test | held-out | |||
| gate and benchmark | cov. | prec. | cov. | prec. |
| derivability check : an answer follows from the context | ||||
| WebQSP | 45.6 | 78.7 | 44.4 | 81.5 |
| CWQ | 11.8 | 52.0 | 9.7 | 62.1 |
| routing gate : where the derived answer replaces the LLM’s a | ||||
| WebQSP: the check or 20 answers | 85.1 | 65.1 | 85.4 | 66.0 |
Table 10: The derivability check and the two gates built on it: share of questions passing (cov.) and share of these whose derived answer is exact (prec.), on the test questions and on the held-out questions of the pre-specified plan (%; Section 6 ). a Picked on validation questions by a procedure fixed in advance among 43 rules defined after test comparisons; no LLM read PathQuestion or MetaQA validation questions, so their routing gate is the check. b Defined after its CWQ test precision had been seen.
| routing gate passes | routing gate fails | all questions | |||||||
|---|---|---|---|---|---|---|---|---|---|
| relation knowledge | passes | derived | LLMs | derived | LLMs | derived alone | routed | gain | derived |
| CLUTRR, rules | 99.8 | 99.4 | 36.3–73.3 | – | – | 99.2 | 99.2–99.3 | +26.0 to +63.0 | 0/7 |
| MetaQA, classifier | 100.0 | 99.9 | 58.4–89.0 | – | – | 99.9 | 99.9 | +10.9 to +41.4 | 0/7 |
| MetaQA, lexicon | 77.1 | 99.6 | 64.3–90.6 | 0.0 | 0.5–2.9 | 76.8 | 76.9–77.4 | +6.9 to +27.2 | 0/7 |
| PathQuestion, lexicon | 77.0 | 96.2 | 55.6–86.2 | 67.8 | 53.3–85.0 | 89.6 | 86.3–93.6 | +7.7 to +31.2 | 3/7 |
| WebQSP, chains | 85.1 | 65.1 | 50.5–66.0 | 6.2 | 16.0–36.6 | 56.3 | 57.8–60.9 | 0.7 to +12.5 | 7/7 |
Table 11: Who should answer, on compact contexts: questions passing the routing gate (%; on WebQSP the check or at most 20 derived answers, on CWQ one derived answer; WebQSP, lexicon: its check); accuracy of the derived answer and the LLMs where it passes and fails (–: under 1% fail); routed: the derived answer if it passes, else the LLM, with no consistency check; gain over the LLM alone (points); derived: LLMs whose routing beats the derived answer alone (Holm-adjusted ).
| (a) contexts of Table 1 | PathQuestion, 782 questions | MetaQA, 900 questions | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5 | Qwen3-32B | open LLMs | GPT-5 | Qwen3-32B | open LLMs | |||||||||||||
| context | exact | Hit | F1 | exact | Hit | F1 | exact | Hit | F1 | exact | Hit | F1 | exact | Hit | F1 | exact | Hit | F1 |
| one-hop neighbourhood | 77.0 | 79.9 | 78.8 | 68.7 | 72.6 | 71.1 | 50.2 | 70.2 | 62.2 | 57.6 | 78.2 | 64.6 | 53.3 | 73.7 | 60.9 | 47.3 | 75.2 | 59.3 |
| similarity top- , as many facts as ours | 38.6 | 49.1 | 46.9 | 35.2 | 45.7 | 42.6 | 33.0 | 51.9 | 44.8 | 24.9 | 50.4 | 35.9 | 24.6 | 47.3 | 35.0 | 26.0 | 56.5 | 39.6 |
| KAPING | 19.3 | 27.4 | 21.6 | 20.5 | 29.3 | 23.1 | 19.0 | 33.4 | 26.2 | 32.6 | 45.4 | 36.6 | 31.1 | 41.4 | 34.8 | 31.3 | 50.9 | 38.7 |
| SubgraphRAG | 86.2 | 90.0 | 88.6 | 75.3 | 80.3 | 78.4 | 52.4 | 70.7 | 63.5 | 67.0 | 91.3 | 77.5 | 60.6 | 87.7 | 73.9 | 49.7 | 86.0 | 66.9 |
Table 12: All 782 PathQuestion and 900 MetaQA questions, forced-choice prompt (%). GPT-5 (reasoning effort low) and Qwen3-32B read with reasoning; open LLMs: mean of the seven answering directly. (a) The contexts of Table 1 , for the published methods the facts each selects, not its full pipeline; exact: whole gold set; Hit: some gold answer. a Facts Think-on-Graph collected with Qwen3-32B without reasoning; open LLMs: own exploration. b The lexicon’s context. c Same prompts as the annotated path; open LLMs read them in separate runs (Appendix A : up to 1.0 point apart). –: no PathQuestion classifier. (b) Split by the derivability check, the routing gate on these benchmarks; passes: questions passing it; derived: the derived answer; LLM: the LLM on the compact context; routed: the derived answer if the check passes, else the LLM; best open LLM: of the seven where the check passes; –: no question fails. Differences in points, computed before rounding; : exact McNemar, Holm-adjusted over the two reasoning LLMs.
More in this topic
Knowledge Graph Question Answering
Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.
See all 173 papers