Knowledge graph question answering (KGQA) systems are typically evaluated on questions whose surface tokens lexically expose the gold KG path. Users, however, may ask questions without knowing the underlying schema: "Which condition could Andrew have due to family history?" compresses a 4-hop chain "Andrew --[father]--> parent --[father]--> grandparent --[medical condition]--> disease --[subclass of]--> hereditary disorder" into a single phrase (family history) that names none of those relations. We identify this collapse of path-question alignment as a blind spot in existing KGQA evaluation. Three major LLM-KG paradigms, hop-by-hop exploration, plan-then-retrieve, and similarity-based retrieval, implicitly rely on such lexical cues, an assumption rarely tested in practice. We make three contributions. (i) We introduce HiddenPathQA, a manually verified benchmark of 1,055 questions spanning 2-6 hops and multiple domains, constructed so that question wording does not reveal the gold path's relations. (ii) We propose Holistic Path Selection, which compares questions against candidate relation chains in the local subgraph as complete units, replacing irreversible per-hop relation scoring with globally informed chain-level reasoning. We develop two implementations, TieredHolistic and FlatHolistic. (iii) We conduct a pseudoword stress test that replaces entity names with meaningless placeholders to distinguish KG-grounded reasoning from entity-level memorization. Holistic Path Selection outperforms six strong KGQA baselines (ToG, PoG, FiDeLiS, R2-KG, KAPING, and KARPA) by 8.8-17.9 accuracy points across two LLM backbones and both original and pseudoword settings. Its advantage under pseudoword replacement shows that the gains cannot be explained by entity-level memorization alone.
Figures & tables
Figure 1: Overview of KGQA benchmarks alongside HiddenPathQA . Red = entities/relations appearing (almost) verbatim in both the question and the inference graph ( low lexical gap). Blue = entities/relations appearing in only one of the two ( high lexical gap). Low † : questions derived from a logical form, where surface overlap is reduced but path–question alignment is preserved at the logical-form level. ▲ : HiddenPathQA does not test commonsense, but uses compressed expressions (e.g., aunt-in-law ) that the LLM must interpret to recover the correct relation chain.
Q. If MBNL1 is altered, which gender could be affected more severely: men or women? Path. MBNL1 → (genetic association) → myotonic dystrophy → (subclass of) → male reproductive system disease.
(a) women (b) men (c) …
(b) Cause-and-effect
( Narrative ) Q. Through which environmental issue do greenhouse gases most plausibly lead to a social issue? Path. greenhouse gas → (has effect) → global warming → (instance of) → environmental issue → (subclass of) → social issue.
(a) …human health… (b) Greenhouse gas emissions have effect… (c) …disease outbreaks…
( Counterfactual ) Q. If Robert of Beverley had never existed, which Catholic church building would most plausibly be missing or delayed? Path. R. of Beverley → (notable work) → Westminster Abbey → (instance of) → collegiate church → (subclass of) → Catholic church bldg.
Q. Both Francesco Salvolini and Gustav Seyffarth are associated with which academic discipline? Path. Francesco Salvolini → (student of) → J.-F. Champollion → (field of work) → Egyptology; Gustav Seyffarth → (field of work) → Egyptology.
Table 1: Representative examples of HiddenPathQA three question types. The correct option is highlighted in green ; gold paths are shown in blue.
Qwen3-32B
GPT-5-mini
Paradigm
Method
Original
Pseudoword
Original
Pseudoword
Closed-book
Zero-shot
63.8
25.2
66.6
25.7
CoT [ 22 ]
64.6
25.3
69.1
23.1
Plan-then-retrieve
KARPA [ 10 ]
40.1
28.0
44.3
36.0
Similarity retrieval
KAPING [ 4 ]
69.1 †
45.4
70.1 †
41.7 †
hop-by-hop KG exploration
ToG [ 19 ]
50.2
29.0
55.1
32.4
Table 2: Accuracy on HiddenPathQA for both Original split and Pseudoword split. Bold = best, underline = second-best, † = third-best per column (excluding the Oracle).
Qwen3-32B
GPT-5-mini
Original
Pseudoword
Original
Pseudoword
Method
Acc
AR
SA
Acc
AR
SA
Acc
AR
SA
Acc
AR
SA
R2-KG
64.7 †
85.6 †
75.6 †
47.5 †
94.9
50.0 †
66.6 †
96.8
68.9 †
40.9 †
97.7
41.8 †
TieredHolistic
74.9
97.2
77.1
56.0
91.9 †
60.9
82.5
96.6 †
85.4
59.6
92.2 †
64.6
FlatHolistic
78.5
99.4
78.9
56.3
99.5
56.6
74.0
99.8
74.2
51.5
100.0
51.5
Table 3: Accuracy ( Acc ), answer rate ( AR ), selective accuracy ( SA ) for methods with explicit abstention mechanisms on HiddenPathQA and R2-KG. Bold = best, underline = second-best, † = third-best per column.
Qwen3-32B
GPT-5-mini
Method
Path F1
AC-Path F1
Path F1
AC-Path F1
KARPA
28.6
31.9
35.4
39.6
KAPING
43.8
47.2
41.7
45.5
ToG
31.7
35.8
34.9
39.3
PoG
45.9
50.0
40.5
44.7
FiDeLiS
32.9
36.6
34.0
38.1
Table 4: Path-retrieval performance on HiddenPathQA . AC-Path F1 = Answer-Correct Path F1, computed only over correctly answered examples. Bold = best, underline = second-best.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Label
a
b
c
d
e
Pct (%)
21.4
26.0
17.4
21.5
13.7
Appendix
Table 5: Distribution of correct labels in HiddenPathQA ( N=1,055 ).
#
Subject
Relation
Object
Role
1
Luise von der Pfalz
given name
Luise
noise
2
Luise von der Pfalz
father
Charles I Louis
gold hop 1
3
Luise
different from
Luise
noise
4
Charles I Louis
mother
Elizabeth Stuart
gold hop 2
5
Luise
writing system
Latin script
noise
6
Charles I Louis
owner of
Portrait of a Man…
noise
Appendix
Table 6: KAPING top-6 retrieval. Three of six slots are consumed by entity-name surface-overlap distractors; gold hops 3–6, including the answer entity gout , are not retrieved.
#
Selected relation chain (depth 6)
1
father → mother → father → medical condition → subclass of → age of onset
2
father → mother → father → medical condition → subclass of → has cause
3
father → mother → father → medical condition → health specialty → subclass of
Appendix
Table 7: Relation chains selected by FlatHolistic in a single step. All three share the gold 5-hop prefix ( father, mother, father, medical condition, subclass of ), which no relation alone matches lexically with the question phrase “family history”—the alignment is recoverable only at the chain level.
Stage
Output (verbatim from log, abridged)
initial_planning
flat_relations : environmental_impact , economic_sector (free-form tokens not in KG schema).
relation_extraction
25 KG relations retained; both has cause (gold) and has effect included.
re_planning
Length-2 plan: relates to SDG, target or indicator → industry . Length-3 variant prepends has effect . Neither plan uses has cause .
matching
8 grounded chains; see Table 9 . None uses has cause .
reasoning
Selects path 7 ( has effect → urbanization ); emits { construction and services sectors } (not in option list).
mcq_mapping
Resolves off-list answer to d. restoration .
Appendix
Table 8: KARPA pipeline trace. Although the gold relation has cause is retained at relation_extraction , re_planning commits to a plan that does not include it, and the failure cascades through matching, reasoning, and option mapping.
#
Score
Grounded chain (verbatim, head → tail entity)
1
0.631
h.f. → class of object(s) of occurrence → different from → relates to SDG, Target 6.6
2
0.627
… → relates to SDG → facet of, SDG 6
3
0.614
… → relates to SDG → instance of, SDG Target
4
0.595
… → facet of → instance of, SDG
5
0.563
… → instance of → subclass of, goal
6
0.561
… → instance of → subclass of → subclass of, condition
Appendix
Table 9: KARPA matching top-8 grounded chains. (h.f. = habitat fragmentation). The six top-scored chains all enter SDG-related sub-graphs through class of object(s) of occurrence → different from ; chains 7–8 use has effect but score lower. Has cause (gold first-hop) does not appear in any chain because the re_planning step did not include it.
Figure 4: Trace comparison , a representative compressed_only question. The surface phrase “family history” silently compresses two father hops; only the path’s tail ( medical condition ) carries a lexical anchor in the question (top, gold chain). Each panel below reproduces one method’s trace from the log. ToG reaches a 1-hop neighbor at depth 1; the LLM judge correctly defers with {No} . At depth 2, after relation scoring and entity_search , the candidate beam is empty ; half_stop fires and the empty-chain MCQ adapter falls back to “None”. The specific relations selected by the depth-2 LLM scorer are not recorded in the existing trace; the failure is observable only as the empty intersection of relation pruning and entity expansion. OneChance ranks all 859 length-6 relation paths jointly; in the first batch, the top-5 LLM-selected paths are all variants of [father, father, medical condition, subclass of, …] , despite father having no surface counterpart in the question. StepByDepth keeps father at depth 1 from a 3-path candidate set, defers via unsure while the disease tail is not yet visible, and at depth 3 ranks [father, father, medical condition] #1 of 55 candidates, grounding to gout . The contrast isolates the failure mode discussed in § 4.2 : locally-scored, hop-by-hop pruning is non-recoverable when path–question alignment is concentrated at the path’s tail, while methods that defer commitment until later (per-depth unsure ) or until the entire path is visible (global path ranking) survive the same alignment gap.
Method
GPT-5-mini
Claude-Sonnet-5
Gemini-2.5-Pro
Original
Transformed
Original
Transformed
Original
Transformed
Zero-shot
63
56
77
74
85
80
KARPA
70
46
72
52
77
65
ToG
63
60
89
89
83
83
FiDeLiS
83
73
90
87
85
78
PoG
78
57
80
71
86
69
Appendix
Table 10: Accuracy (%) on the original and transformed MetaQA questions. Bold and underline denote the best and second-best results, respectively.
Method
GPT-5-mini
Claude-Sonnet-5
Gemini-2.5-Pro
Original
Transformed
Original
Transformed
Original
Transformed
Zero-shot
70
66
81
77
85
83
KARPA
63
59
85
78
81
74
ToG
61
54
90
82
79
70
FiDeLiS
83
71
97
88
80
72
PoG
83
74
91
82
78
65
Appendix
Table 11: Accuracy (%) on the original and transformed CWQ questions. Bold and underline denote the best and second-best results, respectively.
Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.