What Models Know, How Well They Know It: Knowledge-Weighted Fine-Tuning for Learning When to Say "I Don't Know"

Joosung Lee; Hwiyeol Jo; Donghyeon Ko; Kyubyung Chae; Cheonbok Park; Jeonghoon Kim

cs.CLApr 7, 2026

What Models Know, How Well They Know It: Knowledge-Weighted Fine-Tuning for Learning When to Say "I Don't Know"

Authors: Joosung Lee, Hwiyeol Jo, Donghyeon Ko, Kyubyung Chae, Cheonbok Park, Jeonghoon Kim

Organizations: NAVER CLOUD · Seoul National University · KAIST

Abstract

While large language models (LLMs) demonstrate strong capabilities across diverse user queries, they still suffer from hallucinations, often arising from knowledge misalignment between pre-training and fine-tuning. To address this misalignment, we reliably estimate a fine-grained, instance-level knowledge score via multi-sampled inference. Using the knowledge score, we scale the learning signal according to the model's existing knowledge, while encouraging explicit "I don't know" responses for out-of-scope queries. Experimental results show that this approach allows the model to explicitly express uncertainty when it lacks knowledge, while maintaining accuracy on questions it can answer. Furthermore, we propose evaluation metrics for uncertainty, showing that accurate discrimination between known and unknown instances consistently improves performance.

Figures & tables

Question	SFT Response	KWT Response
Which genus contains more species: Fallopia or Nicandra?	Fallopia	Fallopia
Where was the host of Pressure Pad born?	Glasgow, Scotland	Glasgow, Scotland <IDK>

Table 1: Comparison of response behaviors with and without instance-level knowledge awareness. SFT generates confident but incorrect answers, whereas KWT generates <IDK> to express uncertainty. Blue and red text indicate correct and incorrect answer content, respectively.

Metric	KP-Acc	KP-F1	KNP-Acc	KNP-F1
EM	63.2	57.2	71.2	55.8
Rouge	82.4	85.8	83.8	81.6
LLM-as-a-judge	93.4	94.8	93.8	93.4

Table 2: Agreement of automatic metrics (EM, ROUGE, LLM) with human evaluation under knowledge-provided (KP) and knowledge-not-provided (KNP) settings.

	EM			Rouge			LLM
KnowledgeScore (KS)	HaluEval	MedQA	SciQ	HaluEval	MedQA	SciQ	HaluEval	MedQA	SciQ
0	66.75	87.98	40.00	49.83	80.28	25.84	47.74	67.34	14.38
0.2	9.34	6.60	12.66	12.59	10.26	11.97	13.13	15.74	9.07
0.4	7.24	2.63	9.58	9.41	4.45	9.80	8.80	7.69	9.21
0.6	5.85	1.56	9.21	8.48	2.63	10.75	8.55	4.75	10.79
0.8	5.68	0.87	10.52	8.76	1.58	13.21	9.34	2.77	15.93

Table 3: Distribution (%) of questions across knowledge score bins, computed from multi-sampled model responses generated by Llama 3.2-3B and evaluated using EM, ROUGE, and LLM-based metrics.

	Correct	Incorrect
Response w/ IDK	A	B
Response w/o IDK	C	D

Table 4: Confusion matrix of responses based on correctness and IDK usage.

Model	HaluEval	MedQA	SciQ
SFT	31.0	20.4	62.7
FT-TOP	29.5	17.2	63.9
R-Tuning	30.3	19.7	64.6
KWT (EM)	30.7	16.9	62.1
KWT (Rouge)	30.3	16.9	63.8
KWT (LLM)	29.8	18.5	62.6

Table 5: Accuracy comparison across datasets, where accuracy is defined as

(A+C)/(A+B+C+D)

Dataset	Model	nAUPC $\uparrow$	A-FPR $\downarrow$	IDK Precision $\uparrow$
HaluEval	R-Tuning	2.7	50.2	80.4
	KWT (EM)	8.8	25.4	86.7
	KWT (Rouge)	10.2	14.5	88.9
	KWT (LLM)	11.3	11.41	92.1
MedQA	R-Tuning	-2.0	75.3	82.9
	KWT (EM)	1.3	49.3	87.6

Table 6: Comparison of uncertainty-aware behavior on the in-domain datasets. Bold indicates the best performance in each column.

Dataset	Model	nAUPC $\uparrow$	A-FPR $\downarrow$	IDK Precision $\uparrow$
RefuNQ	R-Tuning	-1.9	73.3	81.6
	KWT (EM)	2.2	45.8	86.0
	KWT (Rouge)	5.4	22.1	90.7
	KWT (LLM)	5.7	19.7	91.0
SelfAware	R-Tuning	-2.0	70.7	74.7
	KWT (EM)	4.6	42.3	80.5

Table 7: Comparison of uncertainty-aware behavior on the out-of-domain datasets: RefuNQ and SelfAware.

Domain	Dataset	Acc $\uparrow$	nAUPC $\uparrow$
ID	HaluEval	27.9 (–1.9)	11.3 (0.0)
	MedQA	18.3 (–0.2)	5.8 (+1.5)
	SciQ	62.6 (0.0)	40.2 (+2.1)
OOD	RefuNQ	17.5 (–4.0)	5.4 (-0.3)
OOD	SelfAware	27.0 (–3.9)	11.1 (+1.2)

Table 8: Performance of SEAL compared to KWT on in-domain (ID) and out-of-domain (OOD) datasets. Values in parentheses indicate differences from KWT (LLM).

Dataset	Model	nAUPC $\uparrow$	A-FPR $\downarrow$	IDK Precision $\uparrow$
HaluEval	KWT-RF	9.2 (-2.1)	25.5 (+14.1)	87.1 (–5.0)
HaluEval	KWT-U	9.9 (-1.4)	22.3 (+10.9)	88.1 (–4.0)
MedQA	KWT-RF	2.1 (-2.2)	45.9 (+24.3)	86.5 (–4.7)
MedQA	KWT-U	2.6 (-1.7)	41.8 (+20.2)	86.5 (–4.7)
SciQ	KWT-RF	33.9 (-4.2)	11.1 (+7.4)	70.3 (–8.0)
SciQ	KWT-U	38.1 (0.0)	8.0 (+4.3)	71.9 (–6.4)

Table 9: Comparison of alternative weighting strategies (KWT-RF and KWT-U) relative to the KWT baseline.

Dataset	Model	nAUPC $\uparrow$	A-FPR $\downarrow$	IDK Precision $\uparrow$
HaluEval	KWT + k	57.8 (+46.5)	3.5 (–7.9)	68.6 (–23.5)
SciQ	KWT + k	62.9 (+24.8)	1.4 (–2.3)	81.4 (+3.1)

Table 10: Effect of knowledge-based prompting on KWT. Here,

k

denotes external knowledge provided in the input.

Figure 1: Relationship between instance-level knowledge scores and the frequency of <IDK> responses on the test dataset.

Backbone	Method	HaluEval	MedQA	SciQ	Avg
Llama3.2-3B	SFT	31.0	20.4	62.7	38.0
	KWT (non-IDK)	30.5	21.1	65.9	39.2
	KWT	29.8	18.5	62.6	37.0
Qwen2.5-3B	SFT	29.5	19.3	70.7	39.8
	KWT (non-IDK)	30.6	21.3	71.8	41.2
	KWT	29.8	17.8	72.4	40.0

Table 11: Standard accuracy of SFT, KWT, and KWT (non-IDK). Removing <IDK> supervision while keeping familiarity-based weighting improves average accuracy across both backbones.

Model	HaluEval	MedQA	SciQ	Avg
SFT	3.54	3.06	5.14	3.91
SEAL	3.71	3.20	5.11	4.01
KWT	3.30	2.56	5.20	3.69

Table 12: Token-level KL divergence between the base model and each fine-tuned model. Lower values indicate closer alignment to the base model’s predictive distribution.

Panel of Figure 2 — Figure 2: Token-level probability of generating the <IDK> token across relative response positions on HaluEval . The left panel shows the prepend-IDK setting, while the right panel corresponds to the append-IDK setting.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Model	NEC			RefuNQ			SelfAware
	$\boldsymbol{\mathrm{IR}_{\text{ans}}\downarrow}$	$\boldsymbol{\mathrm{IR}_{\text{unans}}\uparrow}$	IDK Score $\uparrow$	$\boldsymbol{\mathrm{IR}_{\text{ans}}\downarrow}$	$\boldsymbol{\mathrm{IR}_{\text{unans}}\uparrow}$	IDK Score $\uparrow$	$\boldsymbol{\mathrm{IR}_{\text{ans}}\downarrow}$	$\boldsymbol{\mathrm{IR}_{\text{unans}}\uparrow}$	IDK Score $\uparrow$
R-Tuning	94.45	100	52.78	87.82	95.44	53.81	86.95	77.71	45.38
KWT (EM)	79.92	81.67	50.88	68.62	81.96	56.67	68.46	43.02	37.28
KWT (Rouge)	38.18	49.76	55.79	50.57	69.86	59.65	39.54	21.22	40.84
KWT (LLM)	50.05	67.13	58.54	47.13	66.77	59.82	41.98	16.18	37.10

Appendix

Table 13: Comparison of IDK ratios on the out-of-domain datasets: NEC, RefuNQ, and SelfAware.

Dataset	Model	Acc $\uparrow$	nAUPC $\uparrow$	A-FPR $\downarrow$	IDK Precision $\uparrow$
HaluEval	R-Tuning	30.4	5.7	38.3	83.5
	SEAL	27.8	10.6	0.0 †	100.0 †
	KWT (EM)	30.9	9.1	25.6	87.0
	KWT (Rouge)	30.9	10.9	14.9	89.5
	KWT-RF (LLM)	30.1	11.0	17.0	90.7
	KWT-U (LLM)	30.1	10.9	15.6	90.5

Appendix

Table 14: Comparison of uncertainty-aware behavior on the in-domain datasets using the Qwen 2.5 model. † indicates a deterministic outcome induced by token-level abstention, where any output containing <IDK> is counted as incorrect.

Model	HaluEval	MedQA	SciQ
R-Tuning	89.5	89.6	73.7
SEAL	35.0	33.8	12.4
KWT (LLM)	56.1	51.1	22.2

Appendix

Table 15: Hallucination recall (

B/(B+D)

) across datasets.

Dataset	Model	Acc $\uparrow$	nAUPC $\uparrow$	A-FPR $\downarrow$	IDK Precision $\uparrow$
HaluEval	KWT (prepend-IDK)	29.7	8.3	26.8	87.2
	KWT (only-IDK)	21.2	11.2	0.0 †	100.0 †
	KWT (append-IDK)	29.8	11.3	11.4	92.1
MedQA	KWT (prepend-IDK)	16.3	1.7	45.2	89.6
	KWT (only-IDK)	7.5	3.6	0.0 †	100.0 †
	KWT (append-IDK)	18.5	4.3	21.6	91.2

Appendix

Table 16: Comparison of KWT variants across datasets with both standard accuracy and nAUPC. KWT (prepend-IDK) prepends the <IDK> token to the response, KWT (only-IDK) replaces the entire response with <IDK>, and KWT (append-IDK) appends <IDK> to the end of the response, which is the default setting in our experiments. † indicates a deterministic outcome induced by token-level abstention, where any output containing <IDK> is counted as incorrect.

Panel of Figure 3 — Figure 3: Token-level probability of generating the <IDK> token at each relative position in the response under KWT. The top row corresponds to the prepend-IDK setting, while the bottom row shows the append-IDK setting.

LLM Abstention

Large language model abstention studies when AI systems should answer, ask for more information, defer, or withhold unsupported claims.

See all 99 papers

Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
Apr 19, 2026Jingyi Ren, Ante Wang, Yunghwei Lai +5LLM Evaluation Selective Prediction
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
Calibration of Structured Ignorance Certificates for Diagnosing Unknown Unknowns in Reasoning Models
Jun 7, 2026Subramanyam SahooHallucination in Language Models Language Model Calibration
Independent Researcher.
MoRFI: Monotonic Sparse Autoencoder Feature Identification
Apr 29, 2026Dimitris Dimakopoulos, Shay B. Cohen, Ioannis KonstasLLM Hallucination Mitigation LLM Interpretability
University of Edinburgh, UK · Heriot-Watt University, UK

Apr 19, 2026J/K move · Enter open · S save

Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty

Jingyi Ren, Ante Wang, Yunghwei Lai, Xiaolong Wang+4

Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China · Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China

LLM Evaluation Selective Prediction Confidence Estimation in Language Models Language Model Self-Assessment LLM Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

Reliable Large Language Models (LLMs) should abstain when confidence is insufficient. However, prior studies often treat refusal as a generic "I don't know'', failing to distinguish input-level ambiguity (data uncertainty) from capability limitations (model uncertainty). This lack of distinction limits downstream action decisions like requesting clarification or invoking external tools. In this work, we introduce UA-Bench, a benchmark of over 3,500 questions drawn from six datasets spanning knowledge-intensive and reasoning-intensive tasks, designed to evaluate explicit uncertainty attribution. An evaluation of 18 frontier LLMs shows that even state-of-the-art models struggle to reliably discriminate between data uncertainty and model uncertainty, and that high answer accuracy does not necessarily imply strong uncertainty attribution ability. To narrow this gap, we propose a lightweight data synthesis and reinforcement learning strategy. Experiments on both Qwen3-4B-Instruct-2507 and Qwen3-8B in thinking mode show that the proposed method improves uncertainty attribution while preserving answer accuracy. Our code and data are publicly available now.

Abstract

LLM Abstention

Explore similar work

Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty

Calibration of Structured Ignorance Certificates for Diagnosing Unknown Unknowns in Reasoning Models

MoRFI: Monotonic Sparse Autoencoder Feature Identification