Stop or Clarify? Experience-Conditioned Active Fact Verification

Yuhang Dong; Jinghong Li; Xinyue Wang; Yuanning Cui

cs.AIOct 26, 2026

Stop or Clarify? Experience-Conditioned Active Fact Verification

Authors: Yuhang Dong, Jinghong Li, Xinyue Wang, Yuanning Cui

Organizations: Nanjing University of Information Science and Technology, Nanjing, China

Abstract

Fact-verification benchmarks score the final label, while an interactive verifier must first decide whether the available evidence is sufficient. We focus on that intermediate Stop-or-Clarify Decision. Uncertainty alone is insufficient, because the benefit of another evidence-acquisition step is unknown before the step is taken. Sequential Experience-Adaptive Clarification (SEAC) wraps a fixed verifier with a lightweight evidence-acquisition policy. For each claim, SEAC retrieves evidence, runs the verifier, forms a label-free decision state, and combines stop risk with an observational clarification-utility estimate from earlier executed trajectories. Clarification is requested only when this score exceeds the action cost and decision threshold. On SciFact, LIAR, and FEVER, SEAC improves mean accuracy and macro-F1 over the compared fact-verification and active-retrieval baselines. Equal-budget and formula-level diagnostics support selective clarification as a contributor to the gain, while the small SciFact split leaves its paired improvement inconclusive.

Figures & tables

Figure 1: SEAC framework overview. A fixed verifier predicts a label from the current evidence. The stop-or-clarify decision module estimates stop risk and clarification value, then either accepts the current label or executes one focused clarification step. Completed trajectories update later gain estimates.

Group	Method	SciFact	LIAR	FEVER
		Accuracy / Macro-F1
Control	Random	.310 / .295	.323 / .301	.333 / .333
	Retrieval-only	.400 / .288	.517 / .356	.332 / .242
	Direct LLM	.453 / .439	.363 / .341	.556 / .544
Fact verification	MiniCheck	.363 / .310	.592 / .481	.553 / .489
	BEVERS	.313 / .262	.602 / .481	.508 / .443

Table 1: Evaluation-set accuracy/macro-F1 comparison. Higher is better. This table reports final-label quality under the shared verifier and retrieval interface.

Table 2: Paired significance against the strongest non-oracle baseline. The table tests whether the main comparison reflects reliable paired prediction changes.

Data	$B$	SEAC Acc.	Best Active	Turn
SciFact	2	.713	.390	1.00 / .28
LIAR	1	.654	.592	1.00 / .50
FEVER	1	.640	.416	.64 / .60

Table 3: Equal-budget active comparison.

B

is the shared cap; Turn reports SEAC / best active. This table separates action allowance from action selection.

Figure 2: Focused SciFact diagnostics. Subfigure (a): clarification budget reaches early saturation rather than steady gains from repeated retrieval. Dashed budget points mark saturated reuse runs. Subfigure (b): threshold sensitivity exposes the role of the stop-or-clarify boundary; the best observed SciFact accuracy is reached without increasing average turn use.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Variant	Avg Turn	Guard changes	Arbitration	Refresh	Memory hits	Accuracy
Full SEAC	1.00	8%	3%	15%	68%	.713
No clarification	0.00	35%	22%	0%	62%	.340
No graph memory	0.03	28%	18%	0%	0%	.707
No uncertainty	0.00	12%	8%	14%	70%	.710
No experience	0.38	18%	10%	12%	0%	.297
No retrieval refresh	0.11	10%	5%	0%	66%	.700

Appendix

Table 5: Ablation-path audit. Percentages report how often non-clarification paths changed or supported the final decision.

Table 6: Extended equal-budget visualization. The main text reports the compact numerical comparison; this version adds the Pareto view and budget-effect summary.

$\delta_{i}$ bin	Mean $\delta_{i}$	Acc. gain	$n$
Lowest 20%	-.12	-.04	120
20–40%	-.03	+.01	120
40–60%	+.05	+.03	120
60–80%	+.15	+.08	120
Highest 20%	+.32	+.14	120

Appendix

Table 7: Offline association between label-free trajectory utility and post-hoc label changes for 600 executed clarifications. Labels are used only for analysis, never for decision updates; the bins do not identify a causal clarification effect.

Transition	Count	Percent
Correct $\rightarrow$ Correct	145	48.3%
Wrong $\rightarrow$ Correct	52	17.3%
Correct $\rightarrow$ Wrong	18	6.0%
Wrong $\rightarrow$ Wrong	85	28.3%

Appendix

Table 8: SciFact clarification transition matrix used only for offline utility validation.

Dataset	Acc.	Std	Turn	Std
SciFact	.708	.018	1.00	.00
FEVER	.635	.015	0.64	.02

Appendix

Table 9: Online order robustness over five shuffled evaluation orders.

Progress	Accuracy	Mean $g_{t}$	Turn
0–50	.58	.35	.95
50–100	.65	.48	.90
100–150	.70	.55	.88
150–200	.72	.58	.85
200–250	.74	.60	.83
250–300	.75	.62	.82

Appendix

Table 10: SciFact online progression over the evaluation stream.

Error type	SciFact	FEVER	LIAR
Wrong stop	28%	22%	35%
Unhelpful clarification	25%	28%	20%
Retrieval failure	20%	18%	15%
Experience mismatch	15%	18%	18%
Arbitration/guard error	12%	14%	12%

Appendix

Table 11: Quantitative error decomposition.

Dataset	Latency (s)	Retrieval calls	LLM calls	Tokens	Memory size	Turn
SciFact	2.8	1.8	2.0	820	3.8K	1.00
FEVER	2.4	1.5	1.9	750	4.1K	0.64
LIAR	2.6	1.9	2.0	890	4.5K	1.00

Appendix

Table 12: Runtime and interaction-cost diagnostics.

Automated Fact-Checking

Automated fact-checking detects checkable claims, retrieves evidence, and assesses whether that evidence supports or refutes them.

See all 139 papers

OpenFC: Learning Verification Policies towards Open-Search Fact Checking
Sep 27, 2026Xinming Wang, Kaixiang Qiu, Yansong Lin +5RL for Language Model Reasoning Automated Fact-Checking
Institute of Automation, Chinese Academy of Sciences · University of Electronic Science and Technology of China · Beijing Institute of Technology +1
Calibrated Selective Fact-Checking via Evidence Chain Evaluation
Apr 15, 2026Dekun YangAutomated Fact-Checking Selective Prediction
Zhejiang University
EAVer: Long-Form Factuality Verification as an End-to-End Agentic Policy
Sep 2, 2026Kening Zheng, Aoying Zheng, Zhigang Chang +21LLM Answer Verification Automated Fact-Checking
University of Illinois Chicago · Taobao & Tmall Group · Shandong University +3

Sep 27, 2026J/K move · Enter open · S save

OpenFC: Learning Verification Policies towards Open-Search Fact Checking

Xinming Wang, Kaixiang Qiu, Yansong Lin, Chunji Lv+4

Institute of Automation, Chinese Academy of Sciences · University of Electronic Science and Technology of China · Beijing Institute of Technology+1

RL for Language Model Reasoning Automated Fact-Checking LLM Tool Use

Open paper AskPDF arXiv

Open-search fact checking is not merely retrieval followed by classification, but a sequential decision problem in which every query, source visit, and stopping decision reshapes the evidence available for verification. Yet existing systems often distribute these decisions across predefined pipelines or separately prompted modules rather than learning them as a unified task-specific policy. We introduce OpenFC, a unified verification-policy training framework that post-trains Qwen3-8B as a compact next-action controller over reasoning, evidence acquisition, and stopping. OpenFC learns this policy in two stages. Stepwise-Calibrated Cold Start (SCCS) uses a strong training-time supervisor to review post-initial reasoning, tool-use, and stopping proposals before execution, producing reliable trajectories for supervised fine-tuning without access to gold verdicts. Verification-Aware Reinforcement Learning (VA-RL) then improves the cold-start policy on unresolved claims through budget-aware tool rewards, label-aware advantage reweighting, and localized response masking. Across six fact-checking benchmarks, OpenFC achieves 70.39% average accuracy and 63.30% macro-F1, the highest overall averages among the evaluated methods. Stage-wise ablations further show that SCCS and VA-RL provide complementary gains, supporting the design of the two-stage training framework. These results position OpenFC as a strong and effective framework for open-search fact-checking. We will open-source our code and release the model checkpoints to support reproducibility.

Abstract

Automated Fact-Checking

Explore similar work

OpenFC: Learning Verification Policies towards Open-Search Fact Checking

Calibrated Selective Fact-Checking via Evidence Chain Evaluation

EAVer: Long-Form Factuality Verification as an End-to-End Agentic Policy