Industrial advertising question answering (QA) is a high-stakes task in which hallucinated content, particularly fabricated URLs, can lead to financial loss, compliance violations, and legal risk. Although Retrieval-Augmented Generation (RAG) is widely adopted, deploying it in production remains challenging because industrial knowledge is inherently relational, frequently updated, and insufficiently aligned with generation objectives. We propose a reinforced co-adaptation framework that jointly optimizes retrieval and generation through two components: (1) Graph-aware Retrieval (GraphRAG), which models entity-relation structure over a high-citation knowledge subgraph for multi-hop, domain-specific evidence selection; and (2) evidence-constrained reinforcement learning via Group Relative Policy Optimization (GRPO) with multi-dimensional rewards covering faithfulness, style compliance, safety, and URL validity. Experiments on an internal advertising QA dataset show consistent gains across expert-judged dimensions including accuracy, completeness, and safety, while reducing the hallucination rate by 72%. A two-week online A/B test demonstrates a 28.6% increase in like rate, a 46.2% decrease in dislike rate, and a 92.7% reduction in URL hallucination. The system has been running in production for over half a year and has served millions of QA interactions.
Figures & tables
Figure 1: Traditional QA vs. our approach over a shared knowledge base. Given the same user query and knowledge items A, B, C, D, traditional methods often yield incomplete , hallucinated , over-generated , or verbose answers. Our method produces an exact answer that remains complete, faithful, and concise.
Figure 2: System overview. Given a user query q and a private knowledge base K , the retrieval system constructs an evidence set D via two parallel channels: a GraphRAG channel over a high-citation knowledge base Kh and a traditional RAG channel with query rewriting and BGE + BM25 hybrid retrieval. Results are merged and deduplicated. The RL-tuned generator then produces a response optimized by GRPO with multi-dimensional rewards for faithfulness, style compliance, safety, and URL validity.
Metric
DeepSeek-V3.2
GPT-5.2
Qwen3-32B
Qwen3-32B-SFT
Qwen3-32B-RL
Base RAG
GraphRAG
Base RAG
GraphRAG
Base RAG
GraphRAG
Base RAG
GraphRAG
Base RAG
Ours
ROUGE-L ↑
33.27
37.00
32.82
35.88
29.39
32.96
30.79
33.82
31.40
35.49
Accuracy ↑
7.82
8.37
7.94
8.39
7.25
7.82
7.50
8.10
7.85
8.26
Completeness ↑
6.78
7.10
6.70
7.08
6.20
6.66
6.43
6.91
6.46
6.99
Clarity ↑
8.96
8.99
8.92
8.97
8.52
8.74
8.82
8.94
8.83
8.95
Style ↑
8.19
8.25
8.14
8.25
7.82
8.03
8.07
8.19
8.27
8.33
Table 1: Main experimental results. Ours refers to Qwen3-32B-RL with GraphRAG. Best results in bold , second best underlined .
Figure 3: Knowledge recall enhancement across Base RAG, GraphRAG, and Parallel retrieval. Effective chunks pre query and recall effectiveness in percent.
Figure 4: Training dynamics of multi-dimensional reward components during RL.
Metric
Base RAG
Ours
Δ
Like Rate (%)
0.21
0.27
+28.6%
Dislike Rate (%)
0.26
0.18
− 30.8%
Avg. Conv. Turns
1.54
1.81
+17.5%
Table 2: Online A/B testing at 50% traffic.
Figure 5: FaithEval generalization: accuracy (%) on Inconsistent, Unanswerable, Counterfactual, and Overall.
Figure 6: Offline evaluation comparison: win/tie/lose distribution across four dimensions.
Figure 7: Accuracy on the factual QA evaluation set. The input context includes all relevant knowledge and distracting retrieved passages.
Figure 8: Comparison of dialogue responses: the previous online answer (left, red border) contains hallucinated links marked in red, while our answer (right, green border) uses validated links with platform-specific terms replaced by placeholders.
Figure 9: A real dialogue case from our deployed system: the user inquires about restrictions on mini-program landing pages in the financial credit industry; the system responds with a structured, policy-grounded answer covering qualifications, content standards, interaction design, prohibited scenarios, and compliant production suggestions.
Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.
Retrieval-augmented generation (RAG) can improve factual grounding without guaranteeing that every generated claim is supported. Previous work has studied claim-level conformal factuality in single-stage RAG and, more recently, multi-hop RAG. We examine its behavior in agentic RAG, where a language-model policy adaptively issues follow-up retrieval queries or terminates retrieval before response generation. We retain the retrieval-based relevance-scoring formulation and post-generation conformal filtering and evaluate six model-dataset configurations: GPT-6 Luna and Llama 3.1 8B on HotpotQA, Natural Questions, and TriviaQA. Each configuration uses 60 fixed questions, 100 seeded 30/30 calibration-test partitions, five conformal target levels, and a no-conformal-prediction baseline. Across the six configurations, higher target levels are accompanied by higher observed full-support rates and reduced answer availability. At the 95% target, non-empty response coverage ranges from 17.23% to 60.17%, claim retention from 10.72% to 43.16%, and full support among non-empty outputs from 89.14% to 95.56%. We formulate the adaptive retrieval trajectory explicitly and characterize the reliability-informativeness trade-off when conformal factuality is applied after agent termination.
Muhammad Aimal Rehman, Chi-Kuang Yeh
Department of Mathematics & Statistics, Georgia State University, Atlanta, GA, USA
Incorporating specific knowledge into large language models via retrieval-augmented generation (RAG) is a widespread technique that fuels many of today's industry AI applications. A fundamental problem is to assess if the context retrieved by some similarity search provides indeed supporting facts, or instead misguides the generator with irrelevant information. It is critical to associate meaningful confidence measures about the factuality of the retrieval process with the generated answers. We present a new, two-staged approach to predict fact faithfulness of the output of retrieval-augmented generations. First, we employ conformal prediction to select only those retrieved chunks who have a high chance to come from the correct source. This approach in itself can improve answer quality by up to 6% in some of the studied datasets, however, the associated statistical guarantees do not hold generally, since the assumption of sample exchangeability depends on the retriever setup. We present diagnostic metrics to assess whether a setup is suitable. Second, we quantify confidence in the consistency of a generated final answer with a given retrieved context, using an attention-based factuality classifier. This approach can detect inconsistent answers with a chance of up to 77%. Our work helps to establish a novel type of certified RAG systems for a broad range of natural language industry applications.
Florian Geissler, Francesco Carella, Laura Fieback +1
Fraunhofer Institute for Cognitive Systems (IKS), Munich, Germany · Volkswagen AG, Wolfsburg, Germany
Retrieval-Augmented Generation (RAG) models frequently produce answers grounded in parametric memory rather than the retrieved context, undermining the core promise of retrieval augmentation. A fundamental obstacle to fixing this unfaithfulness is the lack of training data that explicitly requires models to prefer context over internal knowledge. We introduce Faithfulness-QA, a large-scale dataset of 99,094 samples constructed through counterfactual entity substitution. Starting from two established extractive QA benchmarks--SQuAD and TriviaQA--we automatically identify answer-bearing named entities in each context, replace them with type-consistent alternatives drawn from a curated bank of 76,953 entities, and thereby manufacture controlled knowledge conflicts between context and parametric memory. Rigorous quality filtering ensures 100% pass rates across four automated checks on random 200-sample audits. We release the full dataset, the construction pipeline, and a typed entity bank covering eight named entity categories. Faithfulness-QA is designed as a training resource for attention-based faithfulness objectives and as an evaluation benchmark for measuring context-grounding behavior in RAG systems. Data and code are available at https://github.com/qzhangFDU/faithfulness-qa-dataset.
Li Ju, Junzhe Wang, Qi Zhang
College of Computer Science and Artificial Intelligence, Fudan University