PassiveQA: A Three-Action Framework for Epistemically Calibrated Question Answering via Supervised Finetuning

Madhav S Baidya

cs.CLApr 6, 2026

PassiveQA: A Three-Action Framework for Epistemically Calibrated Question Answering via Supervised Finetuning

Authors: Madhav S Baidya

Organizations: Indian Institute of Technology (BHU) Varanasi

Abstract

Large Language Models (LLMs) have achieved strong performance in question answering and retrieval-augmented generation (RAG), yet they implicitly assume that user queries are fully specified and answerable. In real-world settings, queries are often incomplete, ambiguous, or missing critical variables, leading models to produce overconfident or hallucinated responses. In this work, we study decision-aware query resolution under incomplete information, where a model must determine whether to Answer, Ask for clarification, or Abstain. We show that standard and enhanced RAG systems do not reliably exhibit such epistemic awareness, defaulting to answer generation even when information is insufficient. To address this, we propose PassiveQA, a three-action framework that aligns model behaviour with information sufficiency through supervised finetuning. Our approach integrates structured information-state representations, knowledge graph-grounded context, and a finetuned planner that explicitly models missing variables and decision reasoning. Experiments across multiple QA datasets show that the finetuned planner achieves significant improvements in macro F1 and abstention recall while reducing hallucination rates, under a compute-constrained training regime. These results provide strong empirical evidence that epistemic decision-making must be learned during training rather than imposed at inference time.

Figures & tables

Figure 1: Full PassiveQA pipeline. Left to right: the four source datasets are merged into a unified 61K-sample schema with explicit variable-state fields (§ 4 ); a knowledge base of 105,420 chunks is constructed and indexed (§ 5 ); three progressive RAG architectures are evaluated on the KB alone (§ 5 ); the KB is simultaneously processed through a three-phase KG construction pipeline producing the decision-weighted graph $G_{2}$ (§ 6 ); $G_{2}$ and the unified dataset jointly generate the 34K KG-grounded finetuning dataset (§ 7 ), which trains the LoRA planner (§ 8 ); at inference the planner receives the query and KG context and routes to one of three specialised agents (§ 9 ). The dashed feedback arrow models multi-turn state update (Eq. 20 ): a resolved variable from the Ask agent transitions from $V_{\mathrm{missing}}$ to $V_{\mathrm{known}}$ before the next planner call.

Equation	Status	Implementation
$I(q)$ (Eq. 2 )	Direct	Variable population via GPT-4o-mini (§ 4 ); threshold $\tau_{i}$ in hard gate (Eq. 21 )
$\operatorname{Conf}(q)$ (Eq. 10 )	Direct	Max sigmoid cross-encoder logit, Architecture 3
$\operatorname{Coverage}(q)$ (Eq. 11 )	Direct	Non-stopword term overlap, Architecture 3
$\operatorname{Amb}(q)$ (Eq. 12 )	Direct	Five heuristic signals, Architecture 3
$\operatorname{Conflict}(q)$ (Eq. 13 )	Direct	Pairwise chunk dissimilarity, Architecture 3
$A(q)$ (Eq. 14 )	Approximated	The hard gate uses per-signal thresholds rather than the product form; $A(q)$ provides the theoretical motivation for why all four signals are jointly necessary

Table 1: Mapping from formal signals to implementation components.

Source	Total	Answer	Ask	Abstain
ShARC	14,000	–	Primary	1,256
QuAC	26,000	Primary	Followup	CANNOTANSWER
HotpotQA	14,000	All	–	–
ContractNLI	7,000	60.8%	–	39.2%
Merged	61K	49.9%	28.9%	21.2%

Table 2: Final merged dataset composition across sources and action labels.

Property	ShARC	QuAC	HotpotQA	ContractNLI
Avg. question length (words)	7.5	6.5	17.8	18.2
Avg. context length (words)	45.5	400+	varies	1,674
Multi-turn	Yes (up to 6)	Yes (avg 7.2)	No	No
Ask signal source	Evidence chain	followup=y	None	None
Abstain signal source	Irrelevant label	CANNOTANSWER	None	NotMentioned
Reasoning type	Policy	Wikipedia	Multi-hop	NLI

Table 3: Dataset characteristics across the four sources.

Metric	Baseline	Enhanced	v3
Decision Accuracy	34.0%	34.0%	38.0%
Macro F1	26.7%	26.7%	35.3%
Hallucination Rate	42.7%	51.7%	33.8%
Ask Recall	2.0%	12.0%	40.0%
Abstain Recall	26.0%	9.0%	13.3%
Coverage	67.3%	78.7%	54.0%

Table 4: Quantitative comparison of three RAG architectures on the 900-sample evaluation set.

Panel of Figure 2 — Figure 2: Overview of the PassiveQA pipeline. After hybrid retrieval and evidence scoring, either the hard gate (Architecture 3) or the finetuned planner routes the query to one of three specialised agents. Dashed arrows indicate the planner path used in the full three-agent architecture.

Phase	Nodes	Edges	Avg. Weight	Notes
$G_{0}$	27,189	24,491	0.726	Raw extraction
$G_{1}$	15,468	11,965	0.728	Semantic validation
$G_{2}$	19,763	17,715	0.819	Incl. 4,295 variable nodes

Table 5: Knowledge graph statistics across construction phases.

Figure 3: Three-phase construction of the PassiveQA knowledge graph. Phase 1 ( $G_{0}$ ): dependency parsing extracts SVO triples, including low-confidence edges (e.g., pronoun-linked relations). Phase 2 ( $G_{1}$ ): semantic filtering using SBERT cosine similarity removes edges below a threshold ( $\tau=0.50$ ), pruning noisy connections. Phase 3 ( $G_{2}$ ): graph refinement via action-based reinforcement, where Answer signals strengthen edges and Abstain signals penalize them; additionally, variable nodes are introduced via requires relations to capture missing information.

Property	Train	Val	Test
Total samples	24,456	5,157	5,218
Answer	5,663 (23.2%)	1,213 (23.5%)	1,265 (24.2%)
Ask	9,726 (39.8%)	1,987 (38.5%)	2,025 (38.8%)
Abstain	9,067 (37.1%)	1,957 (37.9%)	1,928 (36.9%)
Multi-turn	59.4%	58.9%	58.5%
Avg. KG triples	2.31	2.35	2.33

Table 6: Planner finetuning dataset statistics across splits.

Hyperparameter	Value
Base model	Mistral-7B-Instruct-v0.3
LoRA rank ( $r$ )	32
LoRA alpha ( $\alpha$ )	64
LoRA dropout	0.05
Trainable parameters	$\sim$ 83M (1.15%)
Sequence length	512 tokens

Table 7: Planner finetuning hyperparameters.

Panel of Figure 4 — Figure 4: Three-agent architecture. The finetuned planner parses the <decision> tag and delegates execution to one of three specialised agents. The Ask agent reuses the <clarification_question> tag directly.

Selective QA

See all 10 papers

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL
Apr 18, 2026Skylar Zhai, Jingcheng Liang, Dongyeop KangLLM Hallucination Mitigation RL for Language Model Reasoning
University of Minnesota
Gradually Excavating External Knowledge for Implicit Complex Question Answering
Mar 9, 2026Chang Liu, Xiaoguang Li, Lifeng Shang +4Iterative RAG Question Answering
The University of Hong Kong · Huawei Noah’s Ark Lab
When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain
Sep 21, 2026Yunxiang Li, Xixin Wu, Helen MengClarification Question Generation LLM Agent Training
The Chinese University of Hong Kong, Hong Kong SAR, China

Apr 18, 2026J/K move · Enter open · S save

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

Skylar Zhai, Jingcheng Liang, Dongyeop Kang

University of Minnesota

LLM Hallucination Mitigation RL for Language Model Reasoning LLM Reliability Reinforcement Learning with Verifiable Rewards Refusal Behavior in Language Models LLM Abstention

Open paper AskPDF arXiv

Reinforcement fine-tuning improves the reasoning ability of large language models, but it can also encourage them to answer unanswerable queries by guessing or hallucinating missing information. Existing abstention methods either train models to produce generic refusals or encourage follow-up clarifications without verifying whether those clarifications identify the key missing information. We study queries that are clear in meaning but cannot be reliably resolved from the given information, and argue that a reliable model should not only abstain, but also explain what is missing. We propose a clarification-aware RLVR reward that, while rewarding correct answers on answerable queries, jointly optimizes explicit abstention and semantically aligned post-refusal clarification on unanswerable queries. Using this reward, we train Abstain-R1, a 3B model that improves abstention and clarification on unanswerable queries while preserving strong performance on answerable ones. Experiments on Abstain-Test, Abstain-QA, and SelfAware show that Abstain-R1 substantially improves over its base model and achieves unanswerable-query behavior competitive with larger systems including DeepSeek-R1, suggesting that calibrated abstention and clarification can be learned through verifiable rewards rather than emerging from scale alone.

Abstract

Selective QA

Explore similar work

Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RL

Gradually Excavating External Knowledge for Implicit Complex Question Answering

When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain