Towards Reliable Long-Context Reasoning: Detecting Unanswerable Questions via FactGuard

Qian-Wen Zhang; Biao Liu; Fang Li; Jie Wang; Lingfeng Qiao; Yifei Yu; Di Yin; Xing Sun

cs.CLApr 8, 2025

Towards Reliable Long-Context Reasoning: Detecting Unanswerable Questions via FactGuard

Authors: Qian-Wen Zhang, Biao Liu, Fang Li, Jie Wang, Lingfeng Qiao, Yifei Yu, Di Yin, Xing Sun

Organizations: Tencent YouTu Lab, Beijing, China

Abstract

Despite the success of Large Language Models (LLMs) in long-context tasks, their reliability in recognizing unanswerable questions remains a critical challenge. In information-seeking scenarios, models often suffer from overconfidence and hallucinations when evidence is missing or misleading. In this paper, we propose FACTGUARD, a systematic multi-task workflow for generating evidence-grounded answerable and unanswerable questions. We introduce FACTGUARD-BENCH, a benchmark comprising 25,220 examples with context lengths up to 128K, specifically targeting Lack of Evidence and Misleading Evidence scenarios. Evaluations of eleven representative open-source and proprietary LLMs reveal a significant reliability gap: models struggle to reject unanswerable questions as context scales, with the leading model achieving only 71.62% accuracy. By training with FACTGUARD-BENCH, we improve overall accuracy to 81.17% and significantly enhance models' reasoned rejection capabilities. Our work provides a scalable path toward building more faithful and trustworthy long-context LLMs. Our source is publicly available at https://github.com/biaoliu-kiritsugu/FactGuard.

Figures & tables

Figure 1: Illustration of FactGuard for data synthesis in a collaborative multi-task workflow framework.

Lack of Evidence. The question is related to the article, but the factual basis is deleted.

Fragment: …There had been a lack of confidence in Murray since Romani, and the two failed Gaza battles increased his unpopularity among both the infantry and the mounted troops. After the war Allenby acknowledged Murray’s achievements in a June 1919 despatch in which he summed up his campaigns … Question: According to this article, in what year did Allenby recognize Murray’s accomplishments in his circular? Answer: The question cannot be answered. The article mentions Murray’s performance in the battle, but does not mention what year Allenby recognized his accomplishments.

Misleading Evidence. The key information of the question is misaligned against the facts of the article.

Fragment: Global and Local Mixture Consistency Cumulative Learning (GLMC) for Long-Tailed Visual Recognition…The paper introduces GLMC , a one-stage training strategy designed to improve long-tailed visual recognition by enhancing the robustness of the feature extractor and reducing the bias of the classifier towards head classes. GLMC uses a global and local mixture consistency loss … Raw Question: What are the core ideas behind the Global and Local Mixture Consistency cumulative learning (GLMC) framework and how does it improve long-tailed visual recognition? Question 1 with Entity Substitution: What are the core ideas behind the Global and Local Augmentation Consistency Learning (GLACL) framework and how does it improve long-tailed visual recognition? Answer 1 : The article focuses on GLMC and does not mention GLACL. The core ideas of GLACL cannot be answered, but about GLMC… Question 2 with Impossible Condition Insertion: What are the core ideas behind the Global and Local Mixture Consistency cumulative learning (GLMC) framework and how does it improve long-tailed visual recognition on CIFAR-100-LT ? Answer 2 : The article does not mention CIFAR-100-LT. The question of how GLMC improves long-tailed visual recognition on CIFAR-100-LT cannot be answered, but the article mentioned GLMC improve long-tailed visual recognition by enhancing…

Table 1: A detailed categorization of unanswerable examples in FactGuard - Bench .

		English (En)				Chinese (Zh)
Split	Overall	0-16k	16-32k	32-64k	64-128k	0-16k	16-32k	32-64k	64-128k
Train	19100	2043	2508	3077	3071	4065	3141	826	369
Dev	1920	300	300	270	270	300	120	300	60
Test	4200	600	600	600	600	600	300	300	600

Table 2: Detailed statistics of FactGuard - Bench .

Dataset	Large-scale (#Num)	Long Context	GT w/ Reason	Low Human Cost	Language
SQuAD 2.0 ( Rajpurkar et al., 2018 )	✓ (151k)	✗	✗	✗	en
QnotA ( Agarwal et al., 2023 )	✗ (320)	✗	✗	✗	en
KUQP ( Deng et al., 2024 )	✗ (320)	✗	✗	✗	en
FactGuard - Bench	✓ (25k)	✓	✓	✓	en&zh

Table 3: Distinguishing features of unanswerable question datasets.

		English (En)			Chinese (Zh)
Model	Overall	Ans.	Lack.	Mis.	Ans.	Lack.	Mis.
Proprietary & Open-Source Models
Gemini-3-Pro-Preview	71.62	90.50	53.00	74.71	89.00	51.33	74.67
Gemini1.5-Pro	58.20	86.25	54.60	59.61	83.05	45.45	50.81
GPT-4o	45.90	89.89	41.57	40.78	87.89	37.06	30.30
DeepSeek-V4-Pro	56.83	87.75	53.50	60.07	84.67	42.22	40.95

Table 4: Comprehensive results on the FactGuard - Bench test set. The table compares SOTA baselines with the Qwen2.5 series before and after SFT / LoRA. Subscripts denote the absolute change versus the corresponding Qwen2.5-{3,7,14,32}B-Instruct baseline (

\uparrow

red for gains,

\downarrow

green for drops). Ans., Lack., and Mis. denote Answerable questions, Lack of evidence, and Misleading evidence.

Figure 2: Distributions of FactGuard - Bench in terms of domain, question type and length.

Figure 3: Percentage breakdown of answering unanswerable questions in the FactGuard - Bench test set.

Panel of Figure 4 — Figure 4: Model performance under different experimental settings. Panels (a) and (b) report prediction accuracy across input-length intervals for general LLMs and SFT-tuned Qwen models, respectively, while panel (c) shows model accuracy under different data ratios.

Model	Tuning	Overall	Answerable	Unanswerable
Qwen2.5 Series (Base Instruct)
Qwen2.5-7B-Instruct	-	69.43	94.16	44.77
Qwen2.5-32B-Instruct	-	78.66	94.43	62.93
Qwen2.5 Series (After SFT)
Qwen2.5-7B-Ins-SFT	SFT	83.20 ↑20%	86.10 ↓8%	80.30 ↑79%
Qwen2.5-32B-Ins-SFT	SFT	79.47 ↑1%	90.55 ↓4%	68.41 ↑9%

Table 5: Prediction accuracy on the dev set of SQuAD 2.0. We compare the baseline Instruct models with their SFT and LoRA variants. Subscripts denote the percentage change relative to the corresponding Qwen2.5-{7,32}B-Instruct (

\uparrow

red for gains,

\downarrow

green for drops).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Category	#Samples	Quality (%)
Answerable	120	94.17
Unanswerable
Lack of evidence	120	96.67
Misleading evidence	240	92.50
Sub-total	360	93.89
Overall	480	93.96

Appendix

Table 6: Manual review of synthetic data quality.

Category	#Samples	Quality (%)
Task 1: Answer Consistency Evaluation
Answerable	80	95.00
Lack of evidence	60	93.33
Misleading evidence	160	93.75
Overall	300	94.00
Task 2: Reasoning Ability for Unanswerable Questions

Appendix

Table 7: Manual review results of judgment quality by the discriminative model on Qwen2.5-72B response answers.

Question Type	Agreement Rate	Cohen’s $\kappa$
Lack of Evidence	96.3%	0.925
Misleading Evidence	93.0%	0.738
Answerable	93.5%	0.648

Appendix

Table 8: Agreement and Cohen’s

\kappa

between Qwen2.5-72B-Instruct and DeepSeek-V3-0324

Model	Rej.	Rej.+Reason	Reason Acc.
Ministral-3B-Instruct
Base	1453	962	66.21
+ LoRA	2131	1469	68.93
+ SFT	2704	2377	87.91
Ministral-8B-Instruct
Base	1880	1390	73.94

Appendix

Table 9: Reason-grounded rejection on the 3,500 unanswerable test questions of FactGuard - Bench . Rej. counts correct rejections (the surface metric used in the main results); Rej.+Reason counts rejections whose stated reason is also grounded in the document; Reason Acc. (%) is the fraction of correct rejections that carry a correct reason.

Figure 5: Case study. An example of lack of evidence in English on the left, and an example of misleading evidence in Chinese on the right (translated below). Red underlined text indicates hallucinatory content and green italicized text indicates useful explanations.

	FactGuard - Bench							SQuAD 2.0
		English (En)			Chinese (Zh)
Model	Overall	Ans.	Lack.	Mis.	Ans.	Lack.	Mis.	Overall	Ans.	Unans.
Llama-3.1-8B-Instruct
Base	41.21	82.00	58.35	41.20	82.67	31.28	13.14	73.22	93.55	52.95
+ SFT	77.91	83.25	72.08	83.32	69.67	86.31	74.19	80.34	80.83	79.85
Ministral-3B-Instruct

Appendix

Table 10: Cross-family generalization. Performance of Llama-3.1-8B-Instruct and the Ministral series on FactGuard - Bench (test set) and SQuAD 2.0 (dev set) before and after fine-tuning with FactGuard - Bench . Ans., Lack., Mis., and Unans. denote Answerable, Lack of evidence, Misleading evidence, and Unanswerable, respectively. Bold marks the best overall accuracy within each model family.

Dataset	Samples	Mean	Min / Max
SQuAD 2.0	11,873	167	30 / 780
PeerQA	579	11,645	2,670 / 52,360
FactGuard - Bench	4,200	15,554	935 / 127,665

Appendix

Table 11: Context length of the evaluation datasets, measured in Llama-3 tokenizer tokens. Sample counts refer to the split used for evaluation ( FactGuard - Bench test set).

	Binary classification			Gen.
Model	Unans. P/R	Ans. P/R	M-F1	R-L
Ministral-3B-Instruct
Base	0.54 / 0.12	0.80 / 0.97	0.54	0.2825
+ LoRA	0.28 / 0.52	0.83 / 0.64	0.54	0.3073
+ SFT	0.25 / 0.59	0.83 / 0.53	0.50	0.2803
Ministral-8B-Instruct

Appendix

Table 12: PeerQA under the binary-classification protocol of Baumgärtner et al. (2025) (414 answerable / 112 unanswerable). P/R are per-class precision and recall, M-F1 is Macro-F1 over the two classes, and R-L is ROUGE-L against the GPT-4-augmented reference answer on the answerable subset.

Model	Ans.	Unans.	Overall
Ministral-3B-Instruct
Base	26.81	28.57	27.19
+ LoRA	38.41	66.07	44.30
+ SFT	46.86	72.32	52.28
Ministral-8B-Instruct
Base	35.02	40.18	36.12

Appendix

Table 13: PeerQA under the LLM-as-judge protocol used throughout this paper (Qwen2.5-72B-Instruct), which scores the generated response rather than an explicit label. Accuracy (%) on the same 526 samples.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 787 papers

Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspective
Oct 6, 2026Zizhuo Zhang, Xiong Peng, Jingwei Sun +3Contextual Faithfulness LLM Evaluation
TMLR Group, Department of Computer Science, Hong Kong Baptist University · Huawei Noah’s Ark Lab.
Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs
Jan 5, 2026Amirali Ebrahimzadeh, Seyyed M. SaliliLong-Context Retrieval LLM Hallucination Mitigation
Department of Electrical Engineering & Computer Science, University of Michigan, Ann Arbor, MI 48109, USA · Independent Researcher, Ann Arbor, MI 48106, USA
ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Jul 2, 2026Yanjun Zhao, Ruizhong Qiu, Tianxin Wei +6Long-Context Inference Long-Context Reasoning in Language Models
University of Illinois Urbana-Champaign

Oct 6, 2026J/K move · Enter open · S save

Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspective

Zizhuo Zhang, Xiong Peng, Jingwei Sun, Rong Yao+2

TMLR Group, Department of Computer Science, Hong Kong Baptist University · Huawei Noah’s Ark Lab.

Contextual Faithfulness LLM Evaluation Hallucination in Language Models LLM Abstention

Open paper AskPDF arXiv

Large language models (LLMs) are expected to answer questions faithfully based on the provided context, abstaining when the context information is insufficient to answer the questions. Existing faithfulness evaluations typically assess each question-context instance in isolation; however, such instance-level evaluation fails to capture a fundamental requirement of faithful behavior: the ability to adapt model responses to changes in available contexts. In particular, a model should provide correct answers when sufficient evidence is present and abstain when it is not. In this work, we propose a Pairwise Faithfulness Benchmark (PFaithBench) that evaluates whether a model can switch between answering and abstaining for the same question under supporting versus non-supporting contexts. Our evaluations across thirty-nine models with seven model families demonstrate that faithfulness fundamentally involves a trade-off between answering and abstaining, and that most current models exhibit a strong bias toward answering, with most faithfulness errors arising from over-answering, i.e., models tend to fabricate a response even when the provided context is insufficient. We further conduct a series of studies on faithfulness training under different data constructions. Our results show that training outcomes are highly sensitive to the specific composition of answering and abstaining data. Constructing answering and abstaining data from mismatched sources can cause models to rely on dataset-specific shortcuts rather than actual context sufficiency. Moreover, increasing answer-supervised data improves answering performance but exacerbates over-answering, while increasing abstaining data reduces hallucination but leads to over-abstention. The code and data are released at https://github.com/tmlr-group/PFaithBench.

Abstract

LLM Reliability

Explore similar work

Rethinking Faithfulness in LLMs: A Pairwise Context-Sensitive Perspective

Not All Needles Are Found: How Fact Distribution and Don't Make It Up Prompts Shape Retrieval, Reasoning, and Hallucination in Long-Context LLMs

ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning