Can Language Models Learn When to Abstain? Policy Learning and Calibrated Answer Selection

Juan Terven; Diana-Margarita Córdova-Esparza; Ruben Gomez; Carlos Miranda Rocha; Julio-Alejandro Romero-González; Edgar Chavez-Urbiola

cs.CLOct 9, 2026

Can Language Models Learn When to Abstain? Policy Learning and Calibrated Answer Selection

Authors: Juan Terven, Diana-Margarita Córdova-Esparza, Ruben Gomez, Carlos Miranda Rocha, Julio-Alejandro Romero-González, Edgar Chavez-Urbiola

Organizations: CICATA Qro., Instituto Politécnico Nacional, México · Facultad de Informática, Universidad Autónoma de Querétaro, México

Abstract

A useful language model should know when to answer and when to say "I don't know." Can we improve that decision by training the model itself or by training a separate component to check its answers? We study both approaches on Qwen3-14B for factual questions, using a score that rewards correct answers, penalizes mistakes, and allows abstention. Across three training seeds and two held-out confidence thresholds, a learned answer selector improves threshold-averaged utility over the original model's calibrated confidence filter. On TriviaQA, mean coverage increases from 20.2% to 30.9%, while the pooled error rate among scored answers decreases from 23.0% to 14.5% under exact-match grading. The tested reinforcement-learning changes do not establish consistent gains. On NQ-Open, the selector improves threshold-averaged utility over the filter but still performs worse than always abstaining. A partial human-reviewed audit preserves the TriviaQA finding. Our results support learning to select answers in this setting while showing that reliable abstention across datasets remains unresolved.

Figures & tables

Figure 1: Two routes to selective factual question answering. (a) A Qwen3-14B LoRA policy learns to answer or emit IDK ; the training reward uses $\kappa=\tau$ in arms A–C and $\kappa=0.75$ in arm D. (b) A frozen generator supplies identical candidates $\hat{y}$ to three alternative scorers. Each score is calibrated on held-out TriviaQA data, and a valid candidate is released only if $\hat{p}>\tau$ ; calibration transfers unchanged to NQ. Reference answers provide training, calibration, and evaluation labels; they are absent from generation and scoring inputs.

Arm	Reward groups per exposure	Threshold exposure	Reward cost
A	Two groups of four	One threshold, repeated	Requested $\tau$
B	One group of eight	One threshold, repeated	Requested $\tau$
C	One group of eight	All three thresholds	Requested $\tau$
D	One group of eight	All three thresholds	Fixed 0.75

Table 1: Twelve RL runs. B–A tests reward-group size, C–B tests the exposure schedule, and C–D tests reward conditioning. All arms sample eight responses per exposure.

	TriviaQA ( $n=2{,}000$ )			NQ ( $n=500$ )
Method	Utility	Cov.%	Risk%	Utility	Cov.%	Risk%
Original + threshold prompt	-0.1412	79.4	42.9	-0.2786	57.1	74.1
A: two G4, single threshold	-0.0094	27.5	28.7	-0.0381	12.8	55.6
B: G8, single threshold	-0.0012	31.0	25.7	-0.0356	12.5	54.1
C: G8, all thresholds	+0.0091	27.6	22.1	-0.0225	10.0	48.5
D: G8, fixed cost	-0.0267	46.3	31.0	-0.0769	23.4	58.5

Table 2: Primary exact-match results at thresholds 0.65 and 0.85. Utility and coverage average the two thresholds and, for trained methods, three seeds. Risk is the total number of errors divided by the total number of scored answers, including format errors from direct generation. Coverage uses all questions. The original model and logistic filters are deterministic. All selection methods use identical candidates.

	TriviaQA ( $M=1{,}784$ )		NQ ( $M=349$ )
Scoring method	Brier $\downarrow$	AUC $\uparrow$	Brier $\downarrow$	AUC $\uparrow$
Original raw score	0.314	0.779	0.544	0.648
Original + calibration	0.189	0.779	0.237	0.648
TRAIN logistic	0.188	0.779	0.247	0.648
TRAIN logistic + calibration	0.189	0.779	0.237	0.648
Learned selector raw score	0.161	0.852	0.179	0.787

Table 3: Confidence scores evaluated on the same valid, non-abstaining candidates before deciding which answers to return. Selector values average three seeds; the other scorers are deterministic. Raw scores are evaluated before calibration on held-out data. TRAIN logistic already uses the training labels; its second row adds calibration. These diagnostics describe score quality on the stated candidate sets.

Panel of Figure 2 — Figure 2: The five planned TriviaQA comparisons and their descriptive NQ counterparts, using the original exact-match labels. TriviaQA uses 99% intervals to account for multiple comparisons; NQ uses descriptive 95% intervals. The RL intervals are wide and include zero. Both selector comparisons favor the learned selector on average, although its utility remains negative on NQ.

Dataset (interval level)	Original EM $\Delta U$	Partial $\Delta U$	Interval envelope
TriviaQA (99%)	+0.01830	+0.01797	[0.01027, 0.02559]
NQ (95%)	+0.01878	[0.01478, 0.01578]	[0.00277, 0.02730]

Table 4: Utility difference between the selector and the original calibrated filter. Partial results apply the reviewed labels and retain exact matching for all other outputs. Ranges cover the eight assignments of unclear labels. Each interval envelope spans the lowest lower bound and highest upper bound of the corresponding conditional bootstrap intervals; it is a sensitivity summary rather than a new confidence interval.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Dataset	Conditioned utility	Conditioned $-$ binary
Qwen3-1.7B	TriviaQA	-0.0039	+0.1324
SmolLM2-1.7B	TriviaQA	-0.0336	+0.3374
Qwen3-14B	TriviaQA	+0.0338	+0.1670
Qwen3-1.7B	NQ	-0.0036	+0.1820
SmolLM2-1.7B	NQ	-0.0430	+0.4545
Qwen3-14B	NQ	-0.0374	+0.3093

Appendix

Table 5: Historical evaluation on fresh questions at thresholds 0.60 and 0.80, using a different test population and training recipe from the main study. Binary denotes the historical training reward defined above; both columns report cost-sensitive evaluation utility. These values are retained as antecedents and are not pooled with the follow-up.

Dataset	Contrast	Seed 17	Seed 29	Seed 43	Mean	99% interval
TriviaQA	B – A	-0.0195	+0.0150	+0.0293	+0.0083	[-0.0233, +0.0343]
TriviaQA	C – B	+0.0506	+0.0004	-0.0201	+0.0103	[-0.0245, +0.0572]
TriviaQA	C – D	+0.0510	+0.0747	-0.0185	+0.0358	[-0.0224, +0.0842]
TriviaQA	Selector – original filter	+0.0189	+0.0196	+0.0164	+0.0183	[+0.0104, +0.0259]
TriviaQA	Selector – TRAIN logistic	+0.0190	+0.0196	+0.0164	+0.0183	[+0.0104, +0.0260]
NQ	B – A	-0.0299	-0.0060	+0.0435	+0.0025	[-0.0378, +0.0530]

Appendix

Table 6: All three prespecified training seeds. The five TriviaQA intervals form the nominal 95% family; NQ intervals remain descriptive despite using the same percentile level here. Pairing is preserved across questions and thresholds.

Stage	Jobs	Hours
Technical pilots	2	0.478
Shared candidate preparation	2	1.754
RL training	12	23.676
Selector training	3	1.880
Selector development/calibration	3	0.242
Direct test evaluation	13	7.065

Appendix

Table 7: Sum of outer job durations, including loading. Nested ledgers are not added again. Rounded components may differ from the rounded total.

Dataset	Method	$\tau$	Utility	Cov.%	Risk%
TriviaQA	Original calibrated filter	0.65	$+0.04820$	40.2	23.0
	Learned selector + calibration	0.65	$+0.07716$	41.0	16.2
	Original calibrated filter	0.85	$+0.00015$	0.1	0.0
	Learned selector + calibration	0.85	$+0.00780$	20.9	11.3
NQ	Original calibrated filter	0.65	$-0.05980$	21.2	63.2
	Learned selector + calibration	0.65	$-0.01577$	14.7	45.7

Appendix

Table 8: Results by primary threshold on the full test populations. Selector utility and coverage average three seeds, while risk pools their scored answers; the original calibrated filter is deterministic. Coverage and risk are percentages. Risk uses scored answers, including format errors under the evaluation rules. The original filter releases two TriviaQA answers and one NQ answer at

\tau=0.85

		Requested threshold $\tau$
Method	Metric	0.60	0.65	0.75	0.85	0.90	0.95
Original + threshold prompt	$U_{\tau}$	$-0.0330$	$-0.0753$	$-0.1500$	$-0.2295$	$-0.2495$	$-0.2875$
	Cov.%	78.0	78.5	78.0	77.0	75.5	75.0
	Risk%	44.2	44.6	44.2	44.8	43.0	43.3
A: two G4, single threshold	$U_{\tau}$	$+0.0287$	$+0.0131$	$-0.0146$	$-0.0402$	$-0.0447$	$-0.0528$
	Cov.%	28.0	28.5	28.2	27.7	25.3	24.5

Appendix

Table 9: TriviaQA diagnostic panel: the same 200 questions at every threshold. Trained methods aggregate three seeds; deterministic methods are evaluated once.

U_{\tau}

is mean utility; coverage and pooled selective risk are percentages. All results use the original exact-match labels. A dash indicates that selective risk is undefined because no answers are released.

		Requested threshold $\tau$
Method	Metric	0.60	0.65	0.75	0.85	0.90	0.95
Original + threshold prompt	$U_{\tau}$	$-0.1640$	$-0.1935$	$-0.2513$	$-0.3010$	$-0.3210$	$-0.3335$
	Cov.%	59.0	59.0	57.5	56.0	54.0	53.0
	Risk%	67.8	67.8	68.7	68.8	69.4	67.9
A: two G4, single threshold	$U_{\tau}$	$-0.0037$	$-0.0122$	$-0.0288$	$-0.0358$	$-0.0292$	$-0.0306$
	Cov.%	13.7	13.7	13.2	11.7	9.2	8.8

Appendix

Table 10: NQ-Open diagnostic panel: the same 200 questions at every threshold. Trained methods aggregate three seeds; deterministic methods are evaluated once.

U_{\tau}

Explore similar work

Geometry-Calibrated Conformal Abstention for Language Models
Apr 30, 2026Rui Xu, Yi Chen, Sihong Xie +1Uncertainty Quantification Selective Prediction
Information Hub, AI Thrust Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China
Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL
Apr 18, 2026Skylar Zhai, Jingcheng Liang, Dongyeop KangLLM Hallucination Mitigation RL for Language Model Reasoning
University of Minnesota
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
Jul 9, 2026Benedikt J. WagnerSelective Prediction Confidence Estimation in Language Models
City St George’s, University of London

Apr 30, 2026, cs.CLJ/K move · Enter open · S save

Geometry-Calibrated Conformal Abstention for Language Models

Rui Xu, Yi Chen, Sihong Xie, Hui Xiong

Information Hub, AI Thrust Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China

Uncertainty Quantification Selective Prediction Confidence Estimation in Language Models Hallucination in Language Models Conformal Prediction Language Model Calibration+3

Open paper AskPDF arXiv

When language models lack relevant knowledge for a given query, they frequently generate plausible responses that can be hallucinations, rather than admitting being agnostic about the answer. Retraining models to reward admitting ignorance can lead to overly conservative behaviors and poor generalization due to scarce evaluation benchmarks. We propose a post hoc framework, Conformal Abstention (CA), adapted from conformal prediction (CP) to determine whether to abstain from answering a query. CA provides finite-sample guarantees on both the probability of participation (i.e., not abstaining) and the probability that the generated response is correct. Importantly, the abstention decision relies on prediction confidence rather than the non-conformity scores used in CP, which are intractable for open-ended generation. To better align prediction confidence with the model's ignorance, we introduce a calibration strategy using representation geometry within the model to measure knowledge involvement in shaping the response. Experiments demonstrate that we improve selective answering significantly with 75 percent conditional correctness.

Abstract

Explore similar work

Geometry-Calibrated Conformal Abstention for Language Models

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

Two Axes of LLM Abstention: Answer Correctness and Question Answerability