cs.CLAug 19, 2026

MissDiag: Diagnostic Evaluation of Incomplete-Knowledge Robustness in KGQA and KG-RAG

Authors: Hang Wang, Hang Dong, Lu Liu, Chuanru Ren

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Knowledge graph question answering (KGQA) and knowledge-graph-based retrieval-augmented generation (KG-RAG) aim to ground answers in explicit graph evidence, but real-world knowledge graphs are often sparse, outdated, and incomplete. Existing robustness evaluations usually report aggregate changes in answer quality after evidence is removed or perturbed, which measures sensitivity to incomplete support but leaves the source of degradation under-specified: the same score change can conflate the type of missing evidence, the response of the evaluated system, and the sensitivity of the answer-matching protocol. To address this gap, we propose MissDiag, a diagnostic evaluation framework for incomplete-knowledge robustness in KGQA and KG-RAG. MissDiag keeps the question and gold answer fixed while applying structurally typed missingness interventions to benchmark-provided support graphs, enabling paired comparisons that decompose robustness changes by evidence type, system response, and evaluation protocol rather than reducing them to a single aggregate score drop. Experiments across multiple system families show that incomplete-knowledge robustness is better understood as a typed degradation phenomenon than as a uniform property: answer-adjacent evidence loss produces the largest observed degradation, source-context removal is often neutral and can be beneficial, and semantic answer matching changes absolute scores while preserving the main typed degradation patterns. By transforming aggregate robustness measurement into typed diagnostic attribution, MissDiag provides a more interpretable basis for comparing, diagnosing, and stress-testing KGQA and KG-RAG systems under incomplete knowledge.

Figures & tables

Figure 1: Motivation for typed degradation analysis. The same QA instance can show similar aggregate degradation under structurally different missingness conditions. Aggregate score drops alone do not reveal which evidence type was removed, whether answer-local evidence was affected, or whether the effect is systematic.
Figure 2: Overview of MissDiag. (1) Input : an instance contains a question, gold answer set, and local support graph. (2) Typed Missingness Operators : typed operators select removable support edges for random, source-context, relation-level, and answer-adjacent missingness. (3) Severity-Controlled Missingness : a shared severity budget produces an incomplete support graph. (4) Paired Evaluation : the same system is evaluated under complete and incomplete support to compute paired degradation. (5) Typed Degradation Profile : degradation values are summarized across missingness type, severity, system, and metric.
CategorySystemCompletePaired degradation Δ\boldsymbol{\Delta} F1
F1RandomSource-contextRelation-levelAnswer-adjacent
Trained KGQA ModelsReaRev80.84.5-1.22.711.9
NuTrea77.21.9-9.9-2.712.2
NSM46.73.0-24.1-5.212.1
Graph Prompting MethodsMindMap46.210.10.611.212.7
StructGPT52.99.6-0.78.013.8
Table 1: Main typed degradation profiles across system families at α=0.3\alpha=0.3 . Scores are reported as complete-support macro F1 and paired Δ\Delta F1 under each missingness type. Positive Δ\Delta F1 indicates degradation; negative values indicate improvement.
Figure 3: Severity effects on paired degradation. Curves show paired Δ\Delta F1 across missingness types for Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct as α\alpha increases.
SystemMetricPaired degradation Δ\boldsymbol{\Delta} F1
RandSrcRelAns-adj
Qwen2.5-7BExact F111.13.07.421.3
Semantic F111.63.28.021.4
Qwen2.5-14BExact F16.71.25.517.0
Semantic F16.91.25.416.1
Llama-3.1-8BExact F16.1-0.54.015.8
Table 2: Metric sensitivity of typed degradation profiles for direct LLM baselines at α=0.3\alpha=0.3 . Rand, Src, Rel, and Ans-adj denote random, source-context, relation-level, and answer-adjacent missingness.
SliceGroupNFullRandAns-adjGap
Answer cardinalitySingle-answer88890.86.814.47.6
Multi-answer16354.510.430.219.9
Support sizeSmall support35380.512.626.714.2
Medium support52787.05.213.38.0
Large support17189.22.97.44.5
Table 3: Structural slices for direct LLM baselines at α=0.3\alpha=0.3 . Rand and Ans-adj report paired Δ\Delta F1 under random and answer-adjacent missingness. Gap is the difference between Ans-adj and Rand degradation.
More in this topic

Knowledge Graph Question Answering

Knowledge Graph Question Answering answers natural-language questions by finding and reasoning over entities and relations in structured knowledge graphs.

See all 164 papers

Explore similar work

CardsList
Sep 30, 2026

Explore-over-Graph: Hybrid Embedding-LLM Reasoning for Knowledge Graph Question Answering under Incompleteness

Large language models (LLMs) are increasingly combined with knowledge graphs (KGs) to ground reasoning in structured evidence. However, most LLM-based KGQA methods rely on traversing existing graph edges and become unreliable when reasoning paths are broken by missing facts. Alternatives that ask LLMs to generate missing knowledge risk introducing hallucinated evidence. We introduce XoG (eXplore-over-Graph), a framework for multi-hop question answering over incomplete KGs that recovers missing reasoning paths from learned graph structure rather than LLM parametric knowledge. XoG combines type-level entity-relation statistics to identify candidate relations with KG embeddings to retrieve plausible missing entities, using the LLM as a semantic selector and reasoner. These mechanisms are integrated into an iterative planning-exploration-reasoning process. Experiments on WebQSP, CWQ, and the Wikidata-based BRINK benchmark show that XoG remains competitive on complete KGs and consistently outperforms comparable methods without task-specific KGQA training under KG incompleteness. These gains persist across multiple LLM backbones, indicating that stronger LLMs alone do not resolve missing graph evidence. XoG also reduces LLM token consumption by up to 33% compared with a closely related planning-based approach.

Ola El Khatib, Djellel Difallah

New York University Abu Dhabi, UAE

Knowledge Graph Question AnsweringMulti-Hop QALLM Reasoning with Graphs
AskPDF
Jan 28, 2026

UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAG

Large language models (LLMs) frequently generate confident yet factually incorrect content when used for language generation (a phenomenon often known as hallucination). Retrieval augmented generation (RAG) tries to reduce factual errors by identifying information in a knowledge corpus and putting it in the context window of the model. While this approach is well-established for document-structured data, it is non-trivial to adapt it for Knowledge Graphs (KGs), especially for queries that require multi-node/multi-hop reasoning on graphs. We introduce UltRAG, a training-free KG-RAG recipe that combines LLM query generation, a fully inductive neural query executor, and LLM arbitration. This off-the-shelf composition achieves state-of-the-art results on Knowledge Graph Question Answering (KGQA) tasks without retraining the LLM or executor, while enabling language models to interface with Wikidata-scale graphs (116M entities, 1.6B relations) at comparable or lower costs. Our ablation studies indicate that these gains come from the full system design rather than from any single component.

Dobrik Georgiev, Kheeran K. Naidu, Alberto Cattaneo +3

Graphcore Research

Multi-Hop QALLM Reasoning with GraphsGraph Retrieval-Augmented GenerationKnowledge Graph Question AnsweringKnowledge Graphs
AskPDF
Mar 20, 2026

RouterKGQA: Specialized--General Model Routing for Constraint-Aware Knowledge Graph Question Answering

Knowledge graph question answering (KGQA) is a promising approach for mitigating LLM hallucination by grounding reasoning in structured and verifiable knowledge graphs. Existing approaches fall into two paradigms: retrieval-based methods utilize small specialized models, which are efficient but often produce unreachable paths and miss implicit constraints, while agent-based methods utilize large general models, which achieve stronger structural grounding at substantially higher cost. We propose RouterKGQA, a framework for specialized--general model collaboration, in which a specialized model generates reasoning paths and a general model performs KG-guided repair only when needed, improving performance at minimal cost. We further equip the specialized with constraint-aware answer filtering, which reduces redundant answers. In addition, we design a more efficient general agent workflow, further lowering inference cost. Experimental results show that RouterKGQA outperforms the previous best by 3.57 points in F1 and 0.49 points in Hits@1 on average across benchmarks, while requiring only 1.15 average LLM calls per question. Codes and models are available at https://github.com/Oldcircle/RouterKGQA.

Bo Yuan, Hexuan Deng, Xuebo Liu +1

Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Zhongguancun Academy, Beijing, China

Knowledge Graph Question AnsweringLLM Reasoning with GraphsLLM Routing
AskPDF