cs.CLNov 9, 2026

HiddenPathQA: A Benchmark for Knowledge Graph Question Answering When Questions Hide Their Paths

Authors: Sumin Jo, Edward Choi

Organizations: KAIST

Abstract

Knowledge graph question answering (KGQA) systems are typically evaluated on questions whose surface tokens lexically expose the gold KG path. Users, however, may ask questions without knowing the underlying schema: "Which condition could Andrew have due to family history?" compresses a 4-hop chain "Andrew --[father]--> parent --[father]--> grandparent --[medical condition]--> disease --[subclass of]--> hereditary disorder" into a single phrase (family history) that names none of those relations. We identify this collapse of path-question alignment as a blind spot in existing KGQA evaluation. Three major LLM-KG paradigms, hop-by-hop exploration, plan-then-retrieve, and similarity-based retrieval, implicitly rely on such lexical cues, an assumption rarely tested in practice. We make three contributions. (i) We introduce HiddenPathQA, a manually verified benchmark of 1,055 questions spanning 2-6 hops and multiple domains, constructed so that question wording does not reveal the gold path's relations. (ii) We propose Holistic Path Selection, which compares questions against candidate relation chains in the local subgraph as complete units, replacing irreversible per-hop relation scoring with globally informed chain-level reasoning. We develop two implementations, TieredHolistic and FlatHolistic. (iii) We conduct a pseudoword stress test that replaces entity names with meaningless placeholders to distinguish KG-grounded reasoning from entity-level memorization. Holistic Path Selection outperforms six strong KGQA baselines (ToG, PoG, FiDeLiS, R2-KG, KAPING, and KARPA) by 8.8-17.9 accuracy points across two LLM backbones and both original and pseudoword settings. Its advantage under pseudoword replacement shows that the gains cannot be explained by entity-level memorization alone.

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Bounded Path Context: A Controlled Study of Visible Path History in LLM-Based Knowledge Graph Question Answering

    May 26, 2026Xihang Shan, Ye LuoMulti-Hop QALLM Reasoning with Graphs

  2. Theseus in the Graph: Towards Traceable Multi-Hop Graph Navigation

    Sep 13, 2026Eduin E. Hernandez, Luis F. Garcia, Nurassyl Askar +2Multi-Hop QAKnowledge Graph Question Answering

  3. RouterKGQA: Specialized--General Model Routing for Constraint-Aware Knowledge Graph Question Answering

    Mar 20, 2026Bo Yuan, Hexuan Deng, Xuebo Liu +1Knowledge Graph Question AnsweringLLM Reasoning with Graphs