Organizations: School of Artificial Intelligence, Beihang University, Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Blockchain and Edge Computing, Beijing, China
Large language models (LLMs) often produce plausible but incorrect answers, motivating reinforcement learning methods that reward correct answers, penalize incorrect answers, and allow abstention. However, fixed outcome-based rewards do not distinguish incorrect responses with different confidence levels or account for differences in answering reliability across queries. Consequently, such rewards may inadequately penalize overconfident errors and fail to discourage unnecessary abstention. We propose CalibraRL, a reinforcement learning framework that incorporates response confidence and query-level answering evidence into reward design. For non-abstaining responses, CalibraRL combines correctness-based utility with a Brier-based penalty that encourages agreement between confidence and correctness. For abstentions, it estimates conditional answering reliability using a Beta posterior derived from the correctness of non-abstaining responses within each sampled group. The abstention reward increases with the posterior probability that the model's accuracy on non-abstaining responses is below a predefined threshold. CalibraRL reuses training rollouts without requiring additional annotations indicating whether the model knows the answer to each question. Experiments on three QA datasets demonstrate that CalibraRL improves answer precision and reduces hallucinations compared with strong baselines. Further analysis shows that CalibraRL reduces mean expected calibration error (ECE) by 30.8% across two backbone models and three datasets, highlighting its potential to improve the reliability of retrieval-augmented generation (RAG) in web search.
Figures & tables
Figure 1. Reward design motivation. IDK denotes abstention. (A) Unlike fixed penalties, our method penalizes higher-confidence errors more strongly. (B) Unlike fixed abstention rewards, our posterior-adaptive reward reflects the correctness of non-abstaining rollouts for the same query.
Figure 2. Empirical motivation using LLaMA3-8B-Instruct on NQ. (A) Empirical accuracy generally increases with confidence. (B) Incorrect responses exhibit heterogeneous confidence. (C) Questions eliciting abstention differ in the accuracy of non-abstaining responses obtained through independent reference sampling. IDK denotes “I don’t know” , indicating abstention. Reference groups containing only abstentions are excluded from (C).
Table 1. Main results (%) on NQ, TriviaQA, and HotpotQA using LLaMA3-8B-Instruct and Qwen2.5-7B-Instruct as base models. Results are reported in terms of Prec. , Truth. , and ER , with higher values indicating better performance for all metrics. Avg. denotes the unweighted mean across the three benchmarks. The best results are in bold and the second are underlined .
Variant
NQ
TriviaQA
HotpotQA
Avg.
Prec. ↑
Truth. ↑
ER ↑
Prec. ↑
Truth. ↑
ER ↑
Prec. ↑
Truth. ↑
ER ↑
Prec. ↑
Truth. ↑
ER ↑
Base Model: LLaMA3-8B-Instruct
CalibraRL (Ours)
68.65
87.89
14.40
80.27
85.03
45.94
57.10
84.46
5.14
68.67
85.79
21.83
w/o Answer Confidence
66.97
86.50
13.87
78.97
83.30
46.01
56.05
83.48
4.55
67.33
84.43
21.48
w/o Adaptive Abstention
65.81
84.65
14.20
79.17
83.58
45.99
56.26
83.52
4.71
67.08
83.92
21.63
Base Model: Qwen2.5-7B-Instruct
Table 2. Ablation results (%) on three QA benchmarks. w/o Answer Confidence removes the confidence reward term; w/o Adaptive Abstention replaces the adaptive abstention reward with a fixed reward of zero. Avg. denotes the unweighted mean across the three benchmarks. Best results are shown in bold .
Strategy
NQ
TriviaQA
HotpotQA
Avg.
Prec. ↑
Truth. ↑
ER ↑
Prec. ↑
Truth. ↑
ER ↑
Prec. ↑
Truth. ↑
ER ↑
Prec. ↑
Truth. ↑
ER ↑
Base Model: LLaMA3-8B-Instruct
SFT + CalibraRL
58.92
74.68
11.00
74.15
77.20
42.59
47.22
72.41
−2.91
60.10
74.76
16.89
CalibraRL
68.65
87.89
14.40
80.27
85.03
45.94
57.10
84.46
5.14
68.67
85.79
21.83
Base Model: Qwen2.5-7B-Instruct
SFT + CalibraRL
50.41
67.87
0.53
77.74
81.43
46.27
56.39
77.45
6.61
61.51
75.58
17.80
Table 3. Effect of SFT initialization. CalibraRL applies reinforcement learning directly to the Instruct backbone, whereas SFT + CalibraRL first performs SFT following R-Tuning and then applies CalibraRL. All scores are reported as percentages, with higher values indicating better performance. Avg. denotes the unweighted mean across the three benchmarks. Best results are shown in bold .
Figure 4. Cross-dataset generalization results of CalibraRL using LLaMA3-8B-Instruct. Rows indicate source datasets used for training, and columns indicate target datasets used for evaluation. All scores are reported as percentages, with higher values indicating better performance.
Stage
NQ
TriviaQA
HotpotQA
Base Model: LLaMA3-8B-Instruct
Original Model
0.318
0.147
0.433
Ours
0.193
0.133
0.348
Base Model: Qwen2.5-7B-Instruct
Original Model
0.444
0.239
0.513
Ours
0.266
0.172
0.338
Table 4. ECE performance before and after CalibraRL training on three datasets. Lower values indicate better calibration.
Figure 5. Sensitivity of CalibraRL to the reliability threshold τ using LLaMA3-8B-Instruct. The bold tick label indicates the threshold used in the main experiments. All other hyperparameters are held fixed.
Figure 6. Effect of the confidence penalty weight λc using LLaMA3-8B-Instruct. Setting λc=0 removes the confidence penalty while retaining the posterior-adaptive abstention reward. The bold tick label indicates the weight used in the main experiments. All other hyperparameters are held fixed.
Figure 7. Effect of training data size using LLaMA3-8B-Instruct on NQ. The bold tick label indicates the training size of 10k examples used in the main experiments.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Type
# Train
# Dev
# Test
NQ
single-hop
79.1k
8.7k
3.6k
TriviaQA
single-hop
78.7k
11.3k
8.8k
HotpotQA
multi-hop
88.9k
5.6k
5.6k
Appendix
Table 5. Dataset statistics, including reasoning types and the number of examples in each split.
Figure 8. Empirical motivation using LLaMA3-8B-Instruct on TriviaQA. (A) Empirical accuracy generally increases with confidence. (B) Incorrect responses exhibit heterogeneous confidence. (C) Questions eliciting abstention differ in the accuracy of non-abstaining responses obtained through independent reference sampling.
Figure 9. Empirical motivation using LLaMA3-8B-Instruct on HotpotQA. (A) Empirical accuracy generally increases with confidence. (B) Incorrect responses exhibit heterogeneous confidence. (C) Questions eliciting abstention differ in the accuracy of non-abstaining responses obtained through independent reference sampling.
Table 6. Prompt for the base models.
Figure 10. Effect of rollout group size G on (a) precision, (b) truthfulness, and (c) effective reliability (ER) across NQ, TriviaQA, and HotpotQA. The vertical dashed lines indicate the default setting G=32 .