HiddenPathQA: A Benchmark for Knowledge Graph Question Answering When Questions Hide Their Paths
Organizations: KAIST
Abstract
Knowledge graph question answering (KGQA) systems are typically evaluated on questions whose surface tokens lexically expose the gold KG path. Users, however, may ask questions without knowing the underlying schema: "Which condition could Andrew have due to family history?" compresses a 4-hop chain "Andrew --[father]--> parent --[father]--> grandparent --[medical condition]--> disease --[subclass of]--> hereditary disorder" into a single phrase (family history) that names none of those relations. We identify this collapse of path-question alignment as a blind spot in existing KGQA evaluation. Three major LLM-KG paradigms, hop-by-hop exploration, plan-then-retrieve, and similarity-based retrieval, implicitly rely on such lexical cues, an assumption rarely tested in practice. We make three contributions. (i) We introduce HiddenPathQA, a manually verified benchmark of 1,055 questions spanning 2-6 hops and multiple domains, constructed so that question wording does not reveal the gold path's relations. (ii) We propose Holistic Path Selection, which compares questions against candidate relation chains in the local subgraph as complete units, replacing irreversible per-hop relation scoring with globally informed chain-level reasoning. We develop two implementations, TieredHolistic and FlatHolistic. (iii) We conduct a pseudoword stress test that replaces entity names with meaningless placeholders to distinguish KG-grounded reasoning from entity-level memorization. Holistic Path Selection outperforms six strong KGQA baselines (ToG, PoG, FiDeLiS, R2-KG, KAPING, and KARPA) by 8.8-17.9 accuracy points across two LLM backbones and both original and pseudoword settings. Its advantage under pseudoword replacement shows that the gains cannot be explained by entity-level memorization alone.
Figures & tables
| Type | Question / Reasoning Path | Choices (a,b,c,d,e) |
| (a) Compressed-only | Q. Betaxolol hydrochloride can treat which disease? Path. betaxolol HCl (physically interacts with) adrenoceptor 1 (encoded by) ADRB1 (genetic association) arterial hypertension. | … (d) keratosis follicularis (e) None |
| Q. If MBNL1 is altered, which gender could be affected more severely: men or women? Path. MBNL1 (genetic association) myotonic dystrophy (subclass of) male reproductive system disease. | (a) women (b) men (c) … | |
| (b) Cause-and-effect | ( Narrative ) Q. Through which environmental issue do greenhouse gases most plausibly lead to a social issue? Path. greenhouse gas (has effect) global warming (instance of) environmental issue (subclass of) social issue. | (a) …human health… (b) Greenhouse gas emissions have effect… (c) …disease outbreaks… |
| ( Counterfactual ) Q. If Robert of Beverley had never existed, which Catholic church building would most plausibly be missing or delayed? Path. R. of Beverley (notable work) Westminster Abbey (instance of) collegiate church (subclass of) Catholic church bldg. | (a) … (b) Westminster Abbey (c) … (d) conventual church (e) … | |
| (c) Intermediate-node | Q. Both Francesco Salvolini and Gustav Seyffarth are associated with which academic discipline? Path. Francesco Salvolini (student of) J.-F. Champollion (field of work) Egyptology; Gustav Seyffarth (field of work) Egyptology. | (a) … (b) Egyptology (d) sociolinguistics (e) None |
| Qwen3-32B | GPT-5-mini | ||||
| Paradigm | Method | Original | Pseudoword | Original | Pseudoword |
| Closed-book | Zero-shot | 63.8 | 25.2 | 66.6 | 25.7 |
| CoT [ 22 ] | 64.6 | 25.3 | 69.1 | 23.1 | |
| Plan-then-retrieve | KARPA [ 10 ] | 40.1 | 28.0 | 44.3 | 36.0 |
| Similarity retrieval | KAPING [ 4 ] | 69.1 † | 45.4 | 70.1 † | 41.7 † |
| hop-by-hop KG exploration | ToG [ 19 ] | 50.2 | 29.0 | 55.1 | 32.4 |
| Qwen3-32B | GPT-5-mini | |||||||||||
| Original | Pseudoword | Original | Pseudoword | |||||||||
| Method | Acc | AR | SA | Acc | AR | SA | Acc | AR | SA | Acc | AR | SA |
| R2-KG | 64.7 † | 85.6 † | 75.6 † | 47.5 † | 94.9 | 50.0 † | 66.6 † | 96.8 | 68.9 † | 40.9 † | 97.7 | 41.8 † |
| TieredHolistic | 74.9 | 97.2 | 77.1 | 56.0 | 91.9 † | 60.9 | 82.5 | 96.6 † | 85.4 | 59.6 | 92.2 † | 64.6 |
| FlatHolistic | 78.5 | 99.4 | 78.9 | 56.3 | 99.5 | 56.6 | 74.0 | 99.8 | 74.2 | 51.5 | 100.0 | 51.5 |
| Qwen3-32B | GPT-5-mini | |||
| Method | Path F1 | AC-Path F1 | Path F1 | AC-Path F1 |
| KARPA | 28.6 | 31.9 | 35.4 | 39.6 |
| KAPING | 43.8 | 47.2 | 41.7 | 45.5 |
| ToG | 31.7 | 35.8 | 34.9 | 39.3 |
| PoG | 45.9 | 50.0 | 40.5 | 44.7 |
| FiDeLiS | 32.9 | 36.6 | 34.0 | 38.1 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Label | a | b | c | d | e |
| Pct (%) | 21.4 | 26.0 | 17.4 | 21.5 | 13.7 |
| # | Subject | Relation | Object | Role |
| 1 | Luise von der Pfalz | given name | Luise | noise |
| 2 | Luise von der Pfalz | father | Charles I Louis | gold hop 1 |
| 3 | Luise | different from | Luise | noise |
| 4 | Charles I Louis | mother | Elizabeth Stuart | gold hop 2 |
| 5 | Luise | writing system | Latin script | noise |
| 6 | Charles I Louis | owner of | Portrait of a Man… | noise |
| # | Selected relation chain (depth 6) |
| 1 | father mother father medical condition subclass of age of onset |
| 2 | father mother father medical condition subclass of has cause |
| 3 | father mother father medical condition health specialty subclass of |
| Stage | Output (verbatim from log, abridged) |
| initial_planning | flat_relations : environmental_impact , economic_sector (free-form tokens not in KG schema). |
| relation_extraction | 25 KG relations retained; both has cause (gold) and has effect included. |
| re_planning | Length-2 plan: relates to SDG, target or indicator industry . Length-3 variant prepends has effect . Neither plan uses has cause . |
| matching | 8 grounded chains; see Table 9 . None uses has cause . |
| reasoning | Selects path 7 ( has effect urbanization ); emits { construction and services sectors } (not in option list). |
| mcq_mapping | Resolves off-list answer to d. restoration . |
| # | Score | Grounded chain (verbatim, head tail entity) |
| 1 | 0.631 | h.f. class of object(s) of occurrence different from relates to SDG, Target 6.6 |
| 2 | 0.627 | … relates to SDG facet of, SDG 6 |
| 3 | 0.614 | … relates to SDG instance of, SDG Target |
| 4 | 0.595 | … facet of instance of, SDG |
| 5 | 0.563 | … instance of subclass of, goal |
| 6 | 0.561 | … instance of subclass of subclass of, condition |
| Method | GPT-5-mini | Claude-Sonnet-5 | Gemini-2.5-Pro | |||
| Original | Transformed | Original | Transformed | Original | Transformed | |
| Zero-shot | 63 | 56 | 77 | 74 | 85 | 80 |
| KARPA | 70 | 46 | 72 | 52 | 77 | 65 |
| ToG | 63 | 60 | 89 | 89 | 83 | 83 |
| FiDeLiS | 83 | 73 | 90 | 87 | 85 | 78 |
| PoG | 78 | 57 | 80 | 71 | 86 | 69 |
| Method | GPT-5-mini | Claude-Sonnet-5 | Gemini-2.5-Pro | |||
| Original | Transformed | Original | Transformed | Original | Transformed | |
| Zero-shot | 70 | 66 | 81 | 77 | 85 | 83 |
| KARPA | 63 | 59 | 85 | 78 | 81 | 74 |
| ToG | 61 | 54 | 90 | 82 | 79 | 70 |
| FiDeLiS | 83 | 71 | 97 | 88 | 80 | 72 |
| PoG | 83 | 74 | 91 | 82 | 78 | 65 |
Knowledge Graph Question Answering
Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.
See all 181 papers