Knowledge graph question answering (KGQA) and knowledge-graph-based retrieval-augmented generation (KG-RAG) aim to ground answers in explicit graph evidence, but real-world knowledge graphs are often sparse, outdated, and incomplete. Existing robustness evaluations usually report aggregate changes in answer quality after evidence is removed or perturbed, which measures sensitivity to incomplete support but leaves the source of degradation under-specified: the same score change can conflate the type of missing evidence, the response of the evaluated system, and the sensitivity of the answer-matching protocol. To address this gap, we propose MissDiag, a diagnostic evaluation framework for incomplete-knowledge robustness in KGQA and KG-RAG. MissDiag keeps the question and gold answer fixed while applying structurally typed missingness interventions to benchmark-provided support graphs, enabling paired comparisons that decompose robustness changes by evidence type, system response, and evaluation protocol rather than reducing them to a single aggregate score drop. Experiments across multiple system families show that incomplete-knowledge robustness is better understood as a typed degradation phenomenon than as a uniform property: answer-adjacent evidence loss produces the largest observed degradation, source-context removal is often neutral and can be beneficial, and semantic answer matching changes absolute scores while preserving the main typed degradation patterns. By transforming aggregate robustness measurement into typed diagnostic attribution, MissDiag provides a more interpretable basis for comparing, diagnosing, and stress-testing KGQA and KG-RAG systems under incomplete knowledge.
Figures & tables
Figure 1: Motivation for typed degradation analysis. The same QA instance can show similar aggregate degradation under structurally different missingness conditions. Aggregate score drops alone do not reveal which evidence type was removed, whether answer-local evidence was affected, or whether the effect is systematic.
Figure 2: Overview of MissDiag. (1) Input : an instance contains a question, gold answer set, and local support graph. (2) Typed Missingness Operators : typed operators select removable support edges for random, source-context, relation-level, and answer-adjacent missingness. (3) Severity-Controlled Missingness : a shared severity budget produces an incomplete support graph. (4) Paired Evaluation : the same system is evaluated under complete and incomplete support to compute paired degradation. (5) Typed Degradation Profile : degradation values are summarized across missingness type, severity, system, and metric.
Category
System
Complete
Paired degradation Δ F1
F1
Random
Source-context
Relation-level
Answer-adjacent
Trained KGQA Models
ReaRev
80.8
4.5
-1.2
2.7
11.9
NuTrea
77.2
1.9
-9.9
-2.7
12.2
NSM
46.7
3.0
-24.1
-5.2
12.1
Graph Prompting Methods
MindMap
46.2
10.1
0.6
11.2
12.7
StructGPT
52.9
9.6
-0.7
8.0
13.8
Table 1: Main typed degradation profiles across system families at α=0.3 . Scores are reported as complete-support macro F1 and paired Δ F1 under each missingness type. Positive Δ F1 indicates degradation; negative values indicate improvement.
Figure 3: Severity effects on paired degradation. Curves show paired Δ F1 across missingness types for Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct as α increases.
System
Metric
Paired degradation Δ F1
Rand
Src
Rel
Ans-adj
Qwen2.5-7B
Exact F1
11.1
3.0
7.4
21.3
Semantic F1
11.6
3.2
8.0
21.4
Qwen2.5-14B
Exact F1
6.7
1.2
5.5
17.0
Semantic F1
6.9
1.2
5.4
16.1
Llama-3.1-8B
Exact F1
6.1
-0.5
4.0
15.8
Table 2: Metric sensitivity of typed degradation profiles for direct LLM baselines at α=0.3 . Rand, Src, Rel, and Ans-adj denote random, source-context, relation-level, and answer-adjacent missingness.
Slice
Group
N
Full
Rand
Ans-adj
Gap
Answer cardinality
Single-answer
888
90.8
6.8
14.4
7.6
Multi-answer
163
54.5
10.4
30.2
19.9
Support size
Small support
353
80.5
12.6
26.7
14.2
Medium support
527
87.0
5.2
13.3
8.0
Large support
171
89.2
2.9
7.4
4.5
Table 3: Structural slices for direct LLM baselines at α=0.3 . Rand and Ans-adj report paired Δ F1 under random and answer-adjacent missingness. Gap is the difference between Ans-adj and Rand degradation.
Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.