Hallucination remains a critical bottleneck for large language models (LLMs), undermining their reliability in real-world applications, especially in Retrieval-Augmented Generation (RAG) systems. While existing hallucination detection methods employ LLM-as-a-judge to verify LLM outputs against retrieved evidence, they suffer from inherent confirmation bias, where the verifier inadvertently reproduces the errors of the original generation. To address this, we introduce Multi-Agent Reinforced Self-Check for Hallucination (MARCH), a framework that enforces rigorous factual alignment by leveraging deliberate information asymmetry. MARCH orchestrates a collaborative pipeline of three specialized agents: a Solver, a Proposer, and a Checker. The Solver generates an initial RAG response, which the Proposer decomposes into claim-level verifiable atomic propositions. Crucially, the Checker validates these propositions against retrieved evidence in isolation, deprived of the Solver's original output. This well-crafted information asymmetry scheme breaks the cycle of self-confirmation bias. By training this pipeline with multi-agent reinforcement learning (MARL), we enable the agents to co-evolve and optimize factual adherence. Extensive experiments across hallucination benchmarks demonstrate that MARCH substantially reduces hallucination rates. Notably, an 8B-parameter LLM equipped with MARCH achieves performance competitive with powerful closed-source models. MARCH paves a scalable path for factual self-improvement of LLMs through co-evolution. The code is at https://github.com/Qwen-Applications/MARCH.
Figures & tables
Figure 1: Overview of the MARCH framework. The Solver conducts retrieval-augmented generation base on the input query and related documents. The Proposer then decomposes this response into atomic propositions (the orange line in the response block) and formulates verifiable question-answer pairs. The Checker performs isolated verification by re-answering questions solely based on the retrieved evidence, without access to the Solver ’s original output. This information-asymmetric pipeline, where the policy model plays all three roles, is optimized via Multi-Agent Reinforcement Learning (MARL) to achieve robust factual alignment.
Dataset
Original Dataset
#Samples
Avg. Query Len
#Docs per Query
Total Doc Len (words)
Irrelevant Doc Ratio
Avg
Max
Min
Avg
Max
Min
STEM
BioASQ
4721
8.62
9.03
157
1
1314.86
22135
1
30.14%
General
2WikiMultiHopQA
4500
12.87
10
10
10
575.65
2099
95
24.4%
MuSiQue
4500
15.85
19.99
20
16
11221.72
20741
4112
88.35%
Table 1: Statistics of the training dataset, where we report the number of training samples, average query length, the number of retrieved documents per query, the average document length, and the proportion of documents that are irrelevant to the query. The large variance in document counts and lengths, together with the high ratio of irrelevant documents, highlights the realism and difficulty of our training setup.
Model
RAGTruth
FaithBench
Average
Summary
Data2Txt
QA
o4-mini-high-2025-04-16
76.67
64.00
82.01
45.83
67.13
o4-mini-medium-2025-04-16
76.67
67.33
88.49
55.56
71.01
o4-mini-low-2025-04-16
70.00
71.33
83.45
55.56
70.09
GLM-4-9B-Chat
87.77
52.67
90.67
61.11
73.06
Phi-4-mini-Instruct
78.67
24.67
94.96
38.89
59.30
Table 2: Consistency Rate (%) on RAGTruth and FaithBench benchmark. Baseline results are cited from https://github.com/vectara/FaithJudge/tree/main/eval_results .
Figure 2: Comparison of performance on the Facts Grounding. MARCH (highlighted in blue and purple) demonstrates competitive performance against a range of leading open and proprietary models.
Model
Refusal
Refusal
Faithfulness
Faithfulness
Completeness
Completeness
Conciseness
Conciseness
Avg.
(Ans.)
(Unans.)
(QA)
(Summ.)
(QA)
(Summ.)
(QA)
(Summ.)
Small Judge Model
Glider-3.8B ( Deshpande et al., 2024 )
12.0
8.8
45.6
9.2
20.8
28.7
5.1
4.1
16.8
Promtheus-2-7b ( Kim et al., 2024 )
12.4
44.0
27.2
32.0
24.0
42.6
6.7
29.5
27.3
Llama-3-OffsetBias-8B ( Park et al., 2024 )
64.8
11.2
34.0
26.4
33.2
21.1
46.3
23.0
32.6
Skywork-8B ( Liu et al., 2024a )
60.8
12.0
38.8
31.6
38.4
26.7
29.4
21.3
32.4
Table 3: Accuracy (%) on ContextualJudgeBench. Baseline results are cited from Xu et al. (2025a) .
Method
Backbone
HotpotQA
MuSiQue
2WikiMQA
Advanced RAG Frameworks with Proprietary Models
CoT ( Wei et al., 2022 )
GPT-4o
56.2
26.8
58.4
RAG-Standard ( Lewis et al., 2020 )
GPT-4o
64.0
29.8
57.8
IRCoT ( Trivedi et al., 2022 )
GPT-4o
66.4
44.2
78.0
Iter-RetGen ( Shao et al., 2023 )
GPT-4o
62.6
42.0
71.4
RAG-Star ( Jiang et al., 2024 )
GPT-4o
57.0
40.0
63.0
Table 4: Comparison of MARCH-Best against RAG baselines. MARCH-Best denotes our best-performing model. Underline indicates the best performance among 8B-class models, and Bold indicates the best overall performance. Baseline results are cited from Xu et al. (2025b) .
Training Dataset
STEM
General
Method
Solver
Checker
MuSiQue
2Wiki
Hotpot
Average
MuSiQue
2Wiki
Hotpot
Average
Compare with Instruct Baseline
Baseline
5.60
17.40
35.00
19.33
5.60
17.40
35.00
19.33
+MARCH
✔
-
12.80 ↑7.20
25.40 ↑8.00
49.40 ↑14.40
29.20 ↑9.87
12.20 ↑6.60
27.80 ↑10.40
54.80 ↑19.80
31.60 ↑12.27
✔
✔
16.60 ↑3.80
35.40 ↑10.00
55.20 ↑5.80
35.73 ↑6.53
18.40 ↑6.20
38.60 ↑10.80
56.40 ↑1.60
37.80 ↑6.20
Compare with RLHF Baseline
Table 5: Main results on Multi-Hop Question Answering benchmarks across two training datasets based on Llama3.1-8B-Instruct. We report the accuracy rate (%) by comparing the model output with the corresponding ground-truth label. We produce baseline results on our own environment.
Reward
Reward
Solver
Checker
RAGTruth
Facts Grounding
MuSiQue
2Wiki
Hotpot
Average
Function
Scalar
Summary
Data2Txt
QA
Llama3.1-8B Baseline
71.33
28.67
63.31
57.09
5.60
17.40
35.00
39.77
Ablation Study on Reward Function, General Training Dataset
ERR
-
✔
-
80.00
49.33
74.82
71.98
10.60
24.80
52.00
51.93
ZTR
-1/0
✔
-
88.67
66.67
82.73
80.12
12.20
27.80
54.80
59.00
ERR
-
✔
✔
87.33
62.67
76.26
76.98
9.80
25.60
49.60
55.46
Table 6: Ablation study on reward functions, scalars and base LLMs. We compare Error Rate Reward (ERR) and Zero-Tolerance Reward (ZTR) under different training configurations.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Data & Model Configuration
Max Prompt Length
24,567
Max Response Length
8,192
Filter Overlong Prompts
True
Optimizer & Learning Rates
Actor Learning Rate
1×10−6
Critic Learning Rate
1×10−5
Actor LR Warmup Steps
5
Critic LR Warmup Steps
10
Appendix
Table 7: Key hyperparameters for our RL training, derived from the execution script.
Figure 3: Average number of proposed questions per step on the STEM training dataset. The dashed lines represent the MARCH framework without maintaining the question number, while the solid lines (w/ Constraint) indicate the results after incorporating instructional constraints to maintain informational density.
Figure 4: Visualization of training efficiency and dynamics. (a)-(d) illustrate the convergence of accuracy and rewards across general and STEM datasets. (e) Comparison of cumulative training time on the General dataset.
Hallucination detection is crucial for large language models (LLMs), as hallucinated content creates significant barriers in applications requiring factual accuracy. Current detection methods mainly depend on internal signals like uncertainty and self-consistency checks, using the model's pre-trained knowledge to identify unreliable outputs. However, pre-trained knowledge may become outdated and has coverage limitations, especially for specialized or recent information. To address these limitations, retrieval-augmented generation (RAG) has emerged as a promising solution by retrieving relevant evidence at inference time, grounding outputs beyond the model's parametric knowledge. In this paper, we target a critical and practical learning problem RAG-based hallucination detection (RHD), where RAG is employed to enhance hallucination detection by addressing information updating challenges. To address RHD, we propose a novel method Evidence-Aligned Entity Verification (EAEV), which detects entity-level hallucinations by leveraging RAG to align generated entities with retrieved evidence contexts. Specifically, EAEV evaluates entity-evidence alignment through three complementary dimensions and introduces counterfactual stability analysis to ensure robust alignments under evidence perturbations. Experiments across multiple RAG benchmarks demonstrate that EAEV achieves consistent improvements over existing methods with strong generalization capabilities.
Runsong Jia, Zhen Fang, Mengjia Wu +2
University of Technology Sydney, Sydney, Australia
Retrieval-Augmented Generation (RAG) is widely used to augment the input to Large Language Models (LLMs) with external information, such as recent or domain-specific knowledge. Nonetheless, current models still produce closed-domain hallucinations and generate content that is unsupported by the retrieved context. Current detection approaches typically treat hallucination as a post-hoc problem, relying on black-box consistency checks or probes over frozen internal representations. In this work, we demonstrate that hallucination detection based on internal state representation can also serve as a direct training signal. We introduce RAGognize, a dataset of naturally occurring closed-domain hallucinations with token-level annotations, and RAGognizer, a hallucination-aware fine-tuning approach that integrates a lightweight detection head into an LLM, allowing for the joint optimization of language modeling and hallucination detection. This joint objective forces the model to improve the separability of its internal states regarding hallucinations while simultaneously learning to generate well-formed and meaningful responses. Across multiple benchmarks, RAGognizer achieves state-of-the-art token-level hallucination detection while substantially reducing hallucination rates during generation, without degrading language quality or relevance.
Fabian Ridder, Laurin Lessel, Malte Schilling
Computer Science Department, University of Münster, Münster, Germany
Identifying faithfulness hallucinations in LLM-generated outputs remains challenging due to the scarcity of high-quality annotated data. Recent work relies on advanced LLMs to synthesize training data, including rationales, labels, and hallucinated claims. However, these methods treat the generator as a static component, limiting iterative improvement of the detector. To address this limitation, we introduce Hallucination Self-Play (HSP), a novel framework that enables the detector to bootstrap with an evolved generator. HSP involves two roles initialized from the same base model, a detector that assesses the faithfulness of model outputs, and a generator that produces increasingly hard-to-detect hallucinated responses. Specifically, the detector is first fine-tuned on human-labeled data and then employed as a reward model to train the generator via reinforcement learning from AI feedback (RLAIF). In turn, the evolved generator synthesizes hallucination data to further optimize the detector through rule-based reinforcement learning. Experiments on RAGTruth and LLM-AggreFact across three model families demonstrate that the proposed framework can progressively enhance a small LLM to match or even outperform advanced LLMs without external supervision. Our code is available at https://github.com/maybenotime/Hallucination_Self-Play.
Shiping Yang, Shining Liang, Weihao Liu +4
Simon Fraser University · Microsoft · Work done during an internship at Microsoft +1