cs.CLSep 3, 2026

GRACE: Graph-Grounded Reflective Agent Copilot Engine for Expert-in-the-Loop Knowledge Expansion

Authors: John Seon Keun Yi, Joshua R. Minot, Dokyun Lee

Organizations: Boston University · MassMutual

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models deployed in high-stakes settings frequently generate plausible but ungrounded claims. Standard retrieval-augmented generation (RAG) pipelines offer limited remedy, since they retrieve isolated passages without tracking cross-document evidence relationships or quantifying uncertainty. We introduce GRACE (Graph-grounded Reflective Agent Copilot Engine), a framework that deconstructs LLM responses into atomic claims and grounds them against trusted knowledge priors within a weighted bipartite graph. Edge weights encode the closeness of each claim to the priors, enabling weighted centrality analysis that classifies claims as Grounded, Refuted, or Boundary. Such classification identifies not just hallucinations but also novel or contested claims at the frontier of the model's knowledge. To efficiently allocate human or agent resources, we formulate a Return on Attention (RoA) objective that defers a claim to expert review only when its priority-weighted uncertainty exceeds the cost of verification. Claims verified by experts are promoted to new evidence anchors, closing a validator-LLM evolutionary loop that expands the knowledge base across iterations. We evaluate GRACE across multiple language models and on datasets spanning both general and domain-specific knowledge. Our results show that our knowledge base serves as a reliable foundation for retrieval that outperforms RAG baselines, and that the RoA framework efficiently selects valuable boundary knowledge for expert verification. These findings demonstrate that graph-structured representations combined with expert-in-the-loop verification can mitigate hallucination at the system level rather than at the generation level. Code available at https://github.com/johnsk95/grace_code

Figures & tables

Figure 1: Overview of the GRACE framework. Given user queries and source documents, priors P\mathcal{P} and LLM responses R\mathcal{R} are decomposed into atomic claims and organized into a weighted bipartite graph. Closeness centrality CC(v)CC(v) classifies each claim as Grounded, Boundary, or Refuted. The Return on Attention (RoA) framework routes only boundary claims whose expected utility exceeds verification cost to experts (human or LLM agent). Confirmed claims are promoted to new priors, closing one knowledge expansion loop.
 QASPER [ Dasigi et al. (2021) ]QuALITY [ Pang et al. (2022) ]
 Accuracy (%)Answerable (%)Accuracy (%)
GPT-4o-mini [ OpenAI (2024) ]
Zero-shot + Part10.826.656.2
Zero-shot + Full29.053.679.6
RAG19.845.266.5
GraphRAG27.890.485.3
Table 1: GRACE serves as a reliable retrieval tool over other retrieval and zero-shot methods. Numbers indicate average accuracy (%) and percentage of questions answerable (%) across two QA benchmarks. GRACE displays competitive QA performance, while not being overly confident in question answerability. Underlined numbers indicate the full context ceiling.
ContextAccuracy (%)Δ\Delta
Abstract only0.1413—
Round 0 (Abstract + Intro)0.2225+8.1+8.1
Round 1 (Batch 1)0.2447+10.3+10.3
Round 2 (Batch 2)0.3716+23.0+23.0
Round 3 (Batch 3)0.4913+35.0+35.0
Full paper0.3360+19.5+19.5
Table 2: Downstream QA accuracy improves monotonically with each expansion round. Δ\Delta is relative to the abstract-only baseline.
Figure 2: GRACE claim classification align with ground truth. Confusion matrix of system-assigned labels versus oracle labels.
RoundContextPCR ↑\uparrowKF ↓\downarrowYieldCascade
0Abstract + Intro63.1%10.02——
1+ Batch 177.3%6.2556.3%3.77
2+ Batch 280.6%5.3058.4%0.95
3+ Batch 383.4%4.5769.5%0.73
Table 3: Knowledge coverage grows and the boundary shrinks with each expansion round. PCR rises while verification yield increases across rounds.
RoundAccuracyPCR ↑\uparrowKF ↓\downarrowYieldAbstention
Round 016.67%89.55%5.33——
Round 1 (LLM validator)18.75%93.58%2.9334.48%44.83%
Round 1 (Human)18.75%93.63%2.0737.93%3.45%
Table 4: Human validators match the LLM judge on knowledge coverage but shrink the knowledge frontier substantially further. One expansion round on a 15-paper QASPER subset, verified by 15 human experts compared with LLM validators.
Figure 3: GRACE displays reliable retrieval performance on large knowledge bases with multiple long documents. QA accuracy drops initially at 1→51\rightarrow 5 documents, but plateaus when more documents are added.
More in this topic

LLM Grounding

Large language model (LLM) grounding studies how models anchor answers and actions in evidence, structured knowledge, and real-world context.

See all 320 papers

Explore similar work

CardsList
Jun 4, 2026

Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection

Retrieval-Augmented Generation (RAG) reduces but does not eliminate hallucination in large language models. Existing detection methods rely on flat similarity between generated answers and retrieved passages, ignoring structural relationships among evidence pieces and answer claims. We propose Evidence Graph Consistency (EGC), a framework that constructs a local evidence graph per response and computes five structural consistency measures as hallucination indicators. Evaluated on the full question answering split of RAGTruth across six LLMs (5,767 responses), EGC reveals a consistent model-family split: graph consistency features show the expected diagnostic direction for hallucinations in Llama-2 models but exhibit systematic reversal in GPT-4, GPT-3.5, and Mistral-7B. This reversal suggests qualitatively different hallucination patterns across model families and indicates that embedding-based graph consistency cannot serve as a model-independent hallucination detection signal.

Jianru Shen

University of Montana Missoula, MT, USA

Retrieval-Augmented GenerationHallucination in Language ModelsLLM Hallucination DetectionHallucination in RAG
AskPDF
Mar 10, 2026

Reason and Verify: A Framework for Faithful Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) significantly improves the factuality of Large Language Models (LLMs), yet standard pipelines often lack mechanisms to verify inter- mediate reasoning, leaving them vulnerable to hallucinations in high-stakes domains. To address this, we propose a domain-specific RAG framework that integrates explicit rea- soning and faithfulness verification. Our architecture augments standard retrieval with neural query rewriting, BGE-based cross-encoder reranking, and a rationale generation module that grounds sub-claims in specific evidence spans. We further introduce an eight-category verification taxonomy that enables fine-grained assessment of rationale faithfulness, distinguishing between explicit and implicit support patterns to facilitate structured error diagnosis. We evaluate this framework on the BioASQ and PubMedQA benchmarks, specifically analyzing the impact of dynamic in-context learning and rerank- ing under constrained token budgets. Experiments demonstrate that explicit rationale generation improves accuracy over vanilla RAG baselines, while dynamic demonstration selection combined with robust reranking yields further gains in few-shot settings. Using Llama-3-8B-Instruct, our approach achieves 89.1% on BioASQ-Y/N and 73.0% on Pub- MedQA, competitive with systems using significantly larger models. Additionally, we perform a pilot study combining human expert assessment with LLM-based verification to explore how explicit rationale generation improves system transparency and enables more detailed diagnosis of retrieval failures in biomedical question answering.

Eeham Khan, Luis Rodriguez, Marc Queudot

Concordia University, Montréal, QC, Canada · Centre de recherche informatique de Montréal (CRIM), Montréal, QC, Canada

RAG Citation FaithfulnessRetrieval-Augmented GenerationRAG RobustnessFaithfulness of Language Model ExplanationsBiomedical QARetrieval-Augmented ICL
AskPDF
Jun 4, 2026

Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)

Large language models (LLMs) have fundamentally transformed the landscape of Natural Language Processing. Despite these advances, LLMs and LLM-based systems remain prone to a variety of failure modes. Retrieval-augmented generation (RAG) systems have emerged as a common deployment scenario seeking to both avoid the well known risk of the LLM "hallucinating" information, and to enable reasoning and question answering over proprietary information that the LLM did not have access to during training without resorting to expensive model fine-tuning. In this work, we explore the idea of using a lightweight graph structure with a relatively simple graph schema, to support the RAG subsystem via a dedicated toolset. We design an agentic system with a variety of vector search and graph query tools operating over a structured dataset based on a curated subset of English Wikipedia articles, and evaluate its performance on questions from MoNaCo, a challenging Wikipedia QA benchmark of complex query answering tasks. Our results show that the introduction of graph-based tools can significantly increase the precision and recall of factual correctness, can halve the number of hallucinated answers, and achieves the highest fine-grained truthfulness score among the three evaluated scenarios. All this with a modest increase in token usage.

Christopher J. Wedge, Joshua Stutter, Danny Dixon +1

National Innovation Centre for Data, Newcastle University

Multi-Hop QATool-Augmented Language Model AgentsLLM Hallucination MitigationGraph Retrieval-Augmented GenerationAgentic RAGHallucination in RAG
AskPDF