KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning

Cheng Gao; Cheng Huang; Kangyang Luo; Ziqing Qiao; Shuzheng Si; Huimin Chen; Chaojun Xiao; Maosong Sun

cs.LGApr 3, 2026

KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning

Authors: Cheng Gao, Cheng Huang, Kangyang Luo, Ziqing Qiao, Shuzheng Si, Huimin Chen, Chaojun Xiao, Maosong Sun

Organizations: Tsinghua University

Abstract

Enabling large language models (LLMs) to appropriately abstain from answering questions beyond their knowledge is crucial for mitigating hallucinations. While existing reinforcement learning methods foster autonomous abstention, they often compromise answer accuracy because their static reward mechanisms, agnostic to models' knowledge boundaries, drive models toward excessive caution. In this work, we propose KARL, a novel framework that continuously aligns an LLM's abstention behavior with its evolving knowledge boundary. KARL introduces two core innovations: a Knowledge-Boundary-Aware Reward that performs online knowledge boundary estimation using within-group response statistics, dynamically rewarding correct answers or guided abstention; and a Two-Stage RL Training Strategy that first explores the knowledge boundary and bypasses the "abstention trap", and subsequently converts incorrect answers beyond the knowledge boundary into abstentions without sacrificing accuracy. Extensive experiments on multiple benchmarks demonstrate that KARL achieves a superior accuracy-hallucination trade-off, effectively suppressing hallucinations while maintaining high accuracy across both in-distribution and out-of-distribution scenarios.

Figures & tables

Figure 1: Hallucination rate vs. accuracy on NaturalQuestions (NQ) ( Kwiatkowski et al., 2019 ) . KARL achieves a better trade-off by increasing accuracy while reducing hallucination compared to existing non-abstention and abstention-aware methods. Dashed isocontours represent constant Rely scores.

Figure 2: Training dynamics of response distributions for Llama-3.1-8B-Instruct on NQ under different reward schemes. The plots show the proportions of Correct (green), Abstention (yellow), and Incorrect (red) responses during early training for (a) Binary Reward, (b) Static Ternary Reward, and (c) our method. The red dashed region in (b) highlights a rapid increase in abstention accompanied by declining accuracy.

Figure 3: Distribution of rollout groups across different models and datasets. We categorize sampled groups based on the composition of Correct ( T ), Incorrect ( F ), and Abstention ( U ) responses.

Figure 4: An overview of the KARL framework. KARL utilizes a Knowledge-Boundary-Aware Reward ( KAR ) and employs a two-stage strategy to first explore knowledge boundary and subsequently calibrate abstention behavior.

	NQ				TriviaQA				BioASQ				ARC-C
Method	T ( $\uparrow$ )	U ( $-$ )	F ( $\downarrow$ )	Rely ( $\uparrow$ )	T ( $\uparrow$ )	U ( $-$ )	F ( $\downarrow$ )	Rely ( $\uparrow$ )	T ( $\uparrow$ )	U ( $-$ )	F ( $\downarrow$ )	Rely ( $\uparrow$ )	T ( $\uparrow$ )	U ( $-$ )	F ( $\downarrow$ )	Rely ( $\uparrow$ )
Llama3.1-8B-Inst
Prompting	38.7	6.0	55.3	44.3	71.6	1.3	27.1	72.9	55.8	0.3	43.9	56.1	81.7	0.3	18.0	82.0
SFT	40.0	0.0	60.0	40.0	70.2	0.0	29.8	70.2	53.4	0.0	46.6	53.4	77.6	0.0	22.4	77.6
R-Tuning	24.8	56.8	18.4	49.3	55.4	35.9	8.7	78.4	27.9	65.6	6.5	50.5	10.3	87.8	1.9	21.0
RLKF	28.7	30.7	40.6	50.0	57.6	23.2	19.2	75.4	43.3	27.6	29.1	63.3	54.9	5.2	39.9	59.8

Table 1: Experimental results (%) on in-distribution and three out-of-distribution datasets. T , U , and F denote the rates of correct answers , abstentions , and hallucinations , respectively. Rely is a unified reliability score that balances accuracy and hallucination rate. The

\Delta

row reports the performance difference between our method and the Prompting baseline. Green numbers indicates improvement. Best performance is highlighted in bold .

	Llama3.1-8B-Instruct						Qwen2.5-7B-Instruct
Method	GSM8K			BBH			GSM8K			BBH
Method	T ( $\uparrow$ )	U ( $-$ )	F ( $\downarrow$ )	T ( $\uparrow$ )	U ( $-$ )	F ( $\downarrow$ )	T ( $\uparrow$ )	U ( $-$ )	F ( $\downarrow$ )	T ( $\uparrow$ )	U ( $-$ )	F ( $\downarrow$ )
Prompting	83.7	0.5	15.8	64.3	2.1	33.6	89.1	0.5	10.4	67.5	5.8	26.7
Binary	85.2	0.0	14.8	63.2	0.0	36.8	91.4	0.0	8.6	71.5	0.1	28.4
Ternary	85.5	3.0	11.5	53.4	28.1	18.5	88.5	3.0	8.5	61.5	15.6	22.9
Ours	86.1	1.3	12.6	62.5	11.3	26.2	90.4	0.4	9.2	67.8	7.4	24.8

Table 2: Performance (%) comparison on GSM8K and BBH using Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Question: What type of database is library literature and information science?

[0.8pt/2pt] Prompting Prompting : Metadata database ( F )

[0.8pt/2pt] Ternary Ternary : Library Literature and Information Science is a database that indexes and abstracts articles and other publications in the field of library and information science. I don’t know its specific database type. (U)

[0.8pt/2pt] Ours Ours : Library Literature and Information Science is a database that indexes articles and publications related to library science, often a bibliographic or index database. ( T )

ARC-C

Question: Farmers in Wyoming were concerned because some of their chickens were being preyed upon by hawks that lived in areas around their ranches. The farmers grouped together and hunted the hawks until they were no longer in their area. Which would most likely happen next? Options: A. The chicken population would go down. B. Populations of mice and rats would increase. C. Another bird of prey would replace the hawk. D. The chickens would have a lower rate of disease.

Appendix

Table 4: Case examples from the NQ and ARC-C datasets comparing responses generated by the Prompting baseline, Ternary baseline, and our method. ( T ) , ( F ) , and ( U ) denote correct answers, incorrect answers, and abstentions, respectively. Models are implemented on Llama-3.1-8B-Instruct.

Table 9

Ablation on Transition Timing
Phase I Duration	T ( $\uparrow$ )	U ( $-$ )	F ( $\downarrow$ )	Rely ( $\uparrow$ )
0%	36.0	50.0	14.0	61.0
25%	40.6	40.9	18.5	64.8
50%	42.7	40.6	16.7	66.8
75%	43.9	32.1	24.0	65.6
100%	50.6	18.0	31.4	65.4

Appendix

Table 7: Ablation study on the transition timing from Stage I (Exploration) to Stage II (Abstention Calibration).

Ablation on Strategy
	NQ				TriviaQA				BioASQ				ARC-C
Setting	T( $\uparrow$ )	U( $-$ )	F( $\downarrow$ )	Rely	T( $\uparrow$ )	U( $-$ )	F( $\downarrow$ )	Rely	T( $\uparrow$ )	U( $-$ )	F( $\downarrow$ )	Rely	T( $\uparrow$ )	U( $-$ )	F( $\downarrow$ )	Rely
w/o Phase II ( $\alpha=50\%$ )	50.6	18.0	31.4	65.4	74.9	10.6	14.5	84.4	65.6	13.8	20.6	77.5	81.5	0.8	17.8	82.2
Ablation on Mixing Ratio $\alpha$ in Stage I
$\alpha=0\%$ (Pure KAR)	36.0	50.0	14.0	61.0	62.7	29.7	7.6	83.6	52.7	39.0	8.3	76.5	79.4	6.4	14.2	85.4
$\alpha=25\%$	42.4	35.5	22.1	65.3	68.2	22.2	9.6	85.5	61.0	22.7	16.3	78.6	82.7	2.7	14.6	85.3

Appendix

Table 8: Extended ablation results across all datasets (NQ, TriviaQA, BioASQ, and ARC-C).

Panel of Figure 6 — Figure 6: Inference prompt without CoT.

Panel of Figure 8 — Figure 8: LLM-as-a-Judge Prompt.

LLM Hallucination Mitigation

Large language model hallucination mitigation develops ways to detect and reduce unsupported, fabricated, or inconsistent outputs.

See all 179 papers

To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning
Jul 12, 2026Fengji Zhang, Tianyu Fan, Yuxiang Zheng +4Question Answering RL for LLM Agents
City University of Hong Kong · Alibaba Group
Towards Lightweight Reliability: Using Soft Prompts for Hallucination Mitigation in Large Language Models
May 30, 2026S M Tahmid Siddiqui, Akib Jawad Ononto, Anoop Singhal +1LLM Hallucination Mitigation Question Answering
The University of Texas at Dallas, Richardson, TX 75080, USA · National Institute of Standards and Technology, Gaithersburg, USA
Chain-based Adaptive Reconfiguration Over Lattices for Hallucination Reduction
May 26, 2026Joan Vendrell Gallart, Solmaz Kia, Russell Bent +1LLM Hallucination Mitigation Hallucination in Language Models
Department of Mechanical and Aerospace University of California Irvine Irvine, CA 92617-4322, USA · T-5 Los Alamos National Laboratory Los Alamos, NM 88220, USA · CAI-4 Los Alamos National Laboratory Los Alamos, NM 88220, USA

Jul 12, 2026J/K move · Enter open · S save

To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning

Fengji Zhang, Tianyu Fan, Yuxiang Zheng, Xinyao Niu+3

City University of Hong Kong · Alibaba Group

Question Answering RL for LLM Agents LLM Hallucination Mitigation AI Agent Reliability Agentic RAG Selective QA+1

Open paper AskPDF arXiv

Recent advances in equipping Large Language Models (LLMs) with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's query-specific prior capabilities and continuous on-policy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at https://github.com/zfj1998/AWA-RL.

Abstract

LLM Hallucination Mitigation

Explore similar work

To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning

Towards Lightweight Reliability: Using Soft Prompts for Hallucination Mitigation in Large Language Models

Chain-based Adaptive Reconfiguration Over Lattices for Hallucination Reduction