Purging the Gray Zone: Latent-Geometric Denoising for Precise Knowledge Boundary Awareness

Hao An; Yibin Lou; Jiayi Guo; Yang Xu

cs.CLApr 15, 2026

Purging the Gray Zone: Latent-Geometric Denoising for Precise Knowledge Boundary Awareness

Authors: Hao An, Yibin Lou, Jiayi Guo, Yang Xu

Organizations: Computational Linguistics and Consciousness Sciences Lab, Southern University of Science and Technology

Abstract

Large language models (LLMs) often exhibit hallucinations due to their inability to accurately perceive their own knowledge boundaries. Existing abstention fine-tuning methods typically partition datasets directly based on response accuracy, causing models to suffer from severe label noise near the decision boundaries and consequently exhibit high rates of abstentions or hallucinations. This paper adopts a latent space representation perspective, revealing a "gray zone" near the decision hyperplane where internal belief ambiguity constitutes the core performance bottleneck. Based on this insight, we propose the GeoDe (Geometric Denoising) framework for abstention fine-tuning. This method constructs a truth hyperplane using linear probes and performs "geometric denoising" by employing geometric distance as a confidence signal for abstention decisions. This approach filters out ambiguous boundary samples while retaining high-fidelity signals for fine-tuning. Experiments across multiple models (Llama3, Qwen3) and benchmark datasets (TriviaQA, NQ, SciQ, SimpleQA) demonstrate that GeoDe significantly enhances model truthfulness and demonstrates strong generalization in out-of-distribution (OOD) scenarios. Code is available at https://github.com/Notbesidemoon/GeoDe.

Figures & tables

Figure 1: Visualization of the hidden states of questions that are known and unknown to the model. “Gray zone” refers to the overlapping area.

Figure 2: Overview of our method. GeoDe contains three steps: (1) Identify the knowledge boundary by dividing the source data into two subsets ( $\text{D0}_{\text{ik}}$ and $\text{D0}_{\text{idk}}$ ), and train a probe using these two subsets. (2) Calculate the distance from test data D1 to the hyperplane learned by the probe, and select the subset of samples that are farthest from the hyperplane $D1_{\text{ik}}^{\text{selected}}$ and $D1_{\text{idk}}^{\text{selected}}$ . (3) Adjust the target answers based on the probe’s prediction results (retain correct answers for $D1_{\text{ik}}^{\text{selected}}$ and replace incorrect answers with “I don’t know” for $D1_{\text{idk}}^{\text{selected}}$ ), then conduct abstention fine-tuning on the selected subset.

Panel of Table 1 — Table 1: Abstention confusion matrix. R denotes the answer of the refined (fine-tuned) model. GT denotes the initial model. “Known” denotes that the initial model answered correctly, while “unknown” indicates that the initial model answered wrongly.

Dataset	TriviaQA			NQ			SciQ			SimpleQA
Method	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$
Llama3-8B-Instruct
IDK	78.7	35.3	48.8	61.4	56.3	58.7	83.3	5.6	10.4	14.6	64.0	23.8
Uncertainty	67.7	46.4	55.0	45.9	18.0	25.9	64.7	17.9	28.0	10.0	52.2	16.8
R-Tuning	77.3	71.6	74.4	47.0	78.1	58.7	69.9	58.2	63.5	14.6	96.1	25.4
Probe-Tuning TBG	78.8	72.8	75.7	51.6	7 9.1	62.5	81.4	53.4	64.5	12.8	96.1	22.6

Table 2: Performance on in-domain and out-of-domain question answering benchmarks. All results are multiplied by 100. The best result is bolded. The second best is underlined.

	Golden					Golden & RRN
	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	Acc.	Hallu.	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	Acc.	Hallu.
ICL	-	-	-	71.2	28.8	-	-	-	63.8	36.2
IDK	76.1	45.5	57.0	58.1	17.9	71.6	45.6	55.7	52.7	23.6
R-Tuning	80.8	52.5	63.7	61.0	11.8	74.4	53.3	62.1	50.4	15.3
Probe-tuning TBG	83.8	47.9	61.0	67.3	13.0	81.1	48.6	60.8	63.3	16.8
GeoDe TBG	75.2	52.5	61.8	51.5	9.2	74.3	56.9	64.5	49.1	11.4

Table 3: RAG results. Acc. is accuracy. Hallu. is hallucination rate.

Figure 3: Probing performance vs. distance on TriviaQA. Bin0 denotes the nearest subset, and bin9 the farthest subset. Accuracy and F1 score increase as distance increases.

Dataset	TriviaQA			NQ			SciQ			SimpleQA
$X$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$
6.25%	66.9	69.2	68.1	42.2	70.0	52.6	71.0	40.5	51.6	11.2	63.8	19.1
12.5%	72.8	74.0	73.4	48.2	70.9	57.4	85.6	48.5	61.9	12.1	64.8	20.4
25%	77.7	77.4	77.6	54.3	71.3	61.6	81.8	47.2	59.8	13.0	67.3	21.7
50%	75.6	75.6	75.6	51.1	70.9	59.4	85.4	45.7	59.6	12.5	64.1	20.9
75%	75.8	75.0	75.4	50.7	70.4	58.9	86.0	45.9	59.9	12.3	63.3	20.6

Table 4: Ablation study on the selection threshold

X

(percentage of samples retained by geometric distance) on Qwen3-8B. The optimal setting

X{=}25\%

balances data quantity and label noise. Lower

X

causes underfitting due to insufficient data; higher

X

reintroduces boundary noise.

Dataset	TriviaQA			NQ
Metric	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$
Ours TBG	77.7	77.4	77.6	54.3	71.3	61.6
-middle	74.9	74.2	74.6	50.0	70.2	58.4
-nearest	75.2	71.2	73.2	49.7	69.6	58.0
Ours SLT	75.6	75.6	75.6	51.6	71.7	60.0
-middle	75.4	72.8	74.1	50.1	69.2	58.1

Table 5: Ablation study on distance-based data partitioning for abstention fine-tuning. Samples are partitioned into three tiers based on their geometric distance

|d(x)|

to the probing hyperplane.

Figure 4: Accuracy and hallucination rate according to different positive proportions in the training set.

Method	Alcuna	FalseQA	SA
Llama3-8b-Instruction
IDK	78.4	49.8	50.9
Uncertainty	19.2	8.5	15.4
R-Tuning	98.3	91.8	98.1
Probe-Tuning TBG	99.2	91.8	99.7
GeoDe TBG	98.2	92.4	99.2

Table 6: Abstention rates (%) across specialized benchmarks. High scores indicate effective identification of unanswerable or deceptive queries.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	TriviaQA			NQ			SciQ			SimpleQA
Method	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$	$\text{F1}_{\text{ans}}$	$\text{F1}_{\text{abs}}$	$\text{F1}_{\text{rel}}$
Qwen3-1.7B
R-Tuning	56.7	87.0	68.7	38.8	89.4	54.1	66.5	73.6	69.9	5.9	63.3	10.8
GeoDe	56.9	89.0	69.5	38.0	92.2	53.8	77.0	73.2	75.0	6.8	98.5	12.7
Qwen3-4B
R-Tuning	69.4	80.6	74.6	44.1	71.2	54.5	83.0	63.3	71.8	10.9	65.2	18.7

Appendix

Table 7: Performance of R-Tuning and GeoDe across different Qwen3 model sizes. All results are multiplied by 100. The best result per model size is bolded.

	Size	Llama Acc.	Qwen Acc.
TriviaQA Train	87622	66.17	54.93
TriviaQA Val	11313	65.55	54.65
NQ	3610	40.86	34.04
SciQ	1000	72.50	81.60
SimpleQA	4326	6.80	5.59
Alcuna	2001	-	-

Appendix

Table 8: Dataset details. Llama (Llama3-8B-Instruct) and Qwen (Qwen3-8B) are the initial models used in experiments. Acc. (Accuracy) is for reference.

Dataset	Cohen’s Kappa
TriviaQA	0.9558
NQ	0.9119
SciQ	0.8281
SimpleQA	0.9318

Appendix

Table 9: Inter-judge agreement (Cohen’s Kappa) between Llama3.1-8B-Instruct and Gemma2-9B-IT across evaluation datasets.

Input	Content	Conf.
Case 1 — GT: Italic languages ; Model output: River Tiber
Question	Where did the Latin language originate from?	0.605
Golden	Latin is a member of the broad family of Italic languages . Its alphabet, the Latin alphabet, emerged from the Old Italic alphabets, which in turn were derived from the Greek and Phoenician scripts. Historical Latin came from the prehistoric language of the Latium region, specifically around the River Tiber , where Roman civilization first developed.	0.456
Noise	The solecisms and non-Classical usages occasionally found in late Latin texts also shed light on the spoken language. A windfall source lies in the chance finds of wax tablets such as those found at Vindolanda on Hadrian’s Wall. The Romance languages, a major branch of the Indo-European language family, comprise all languages that descended from Latin, the language of the Roman Empire.	0.721
Case 2 — GT: Darth Vader ; Model output: Jake Lloyd
Question	Who was Anakin Skywalker?	0.983

Appendix

Table 10: Hallucination cases in the Golden & RRN RAG setting. Confidence is the probe’s predicted probability that the model knows the answer (higher = more confident). The model hallucinates despite initially high confidence due to interference from retrieval noise.

LLM Hallucination Mitigation

Large language model hallucination mitigation develops ways to detect and reduce unsupported, fabricated, or inconsistent outputs.

See all 178 papers

Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs
Apr 2, 2026Abinitha Gourabathina, Inkit Padhi, Manish Nagireddy +2LLM Abstention LLM Hallucination Mitigation
MIT · IBM Research · MIT-IBM Watson AI Lab
KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning
Apr 3, 2026Cheng Gao, Cheng Huang, Kangyang Luo +5LLM Hallucination Mitigation LLM Abstention
Tsinghua University
Strong yet Controllable LLM Abstention without Learning to Abstain
Oct 12, 2026Junwei Zhang, Yifei Wang, Wanli Yang +6Factual QA LLM Abstention
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Beijing Institute of Technology

Apr 2, 2026J/K move · Enter open · S save

Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs

Abinitha Gourabathina, Inkit Padhi, Manish Nagireddy, Subhajit Chaudhury+1

MIT · IBM Research · MIT-IBM Watson AI Lab

LLM Abstention LLM Hallucination Mitigation LLM Reasoning

Open paper AskPDF arXiv

For Large Language Models (LLMs) to be reliably deployed, models must effectively know when not to answer: abstain. Reasoning models, in particular, have gained attention for impressive performance on complex tasks. However, reasoning models have been shown to have worse abstention abilities. Taking the vulnerabilities of reasoning models into account, we propose our Query Misalignment Framework. Hallucinations resulting in failed abstention can be reinterpreted as LLMs answering the wrong question (rather than answering a question incorrectly). Based on this framework, we develop a new class of state-of-the-art abstention methods called Trace Inversion. First, we generate the reasoning trace of a model. Based on only the trace, we then reconstruct the most likely query that the model responded to. Finally, we compare the initial query with the reconstructed query. Low similarity score between the initial query and reconstructed query suggests that the model likely answered the question incorrectly and is flagged to abstain. Extensive experiments demonstrate that Trace Inversion effectively boosts abstention performance in four frontier LLMs across nine abstention QA datasets, beating competitive baselines in 33 out of 36 settings.

Abstract

LLM Hallucination Mitigation

Explore similar work

Answering the Wrong Question: Reasoning Trace Inversion for Abstention in LLMs

KARL: Mitigating Hallucinations in LLMs via Knowledge-Boundary-Aware Reinforcement Learning

Strong yet Controllable LLM Abstention without Learning to Abstain