Global Consistency Repair for Natural-Language Claim Sets

Paul He; Elke Kirschbaum; Shiva Kasiviswanathan

cs.AIJan 20, 2026

Global Consistency Repair for Natural-Language Claim Sets

Authors: Paul He, Elke Kirschbaum, Shiva Kasiviswanathan

Organizations: Nanyang Technological University · Work done while at Amazon · Amazon Web Services

Abstract

NLP systems increasingly aggregate claims from retrieved passages, documents, and generated reasoning steps. Yet current factuality methods mostly verify claims individually or compare pairs of claims, leaving open the question of whether the entire aggregated claim set can be retained coherently. This misses a fundamental failure mode: claims may be locally plausible and pairwise compatible while still being jointly inconsistent. We formalize global consistency repair: given an inconsistent natural-language claim set, preserve the largest coherent subset while localizing the compact sources of inconsistency. We introduce QXR, an adaptive diagnosis-and-repair framework that treats an LLM as a noisy set-level consistency judge, localizes minimal conflicting subsets, and repairs the claim set by deleting a small hitting set of claims. Across LLM judges and benchmarks derived from VitaminC, FEVER, and conflicting-evidence RAG, QXR preserves substantially more valid claims than direct all-at-once judging while maintaining high precision. These results suggest that factuality in claim aggregation systems should be treated as a set-level repair problem, not only as local verification.

Figures & tables

Figure 1: Why global consistency requires structured reasoning. Left: pairwise checks can all succeed even when the full claim set is globally inconsistent. Middle: direct all-at-once LLM judging may over-prune, removing more claims than necessary to restore consistency. Right: adaptive decomposition localizes a minimal unsatisfiable subset and repairs the set while preserving more valid claims.

	VitaminC	FEVER
MUSes / inst.	4.05 [2, 6]	4.77 [0, 28]
Min. HS size	4.05 [2, 6]	4.01 [0, 8]
Facts / inst.	28.11 [24, 32]	33.73 [20, 62]

Table 1: Conflict-structure statistics. Values report mean [min, max] across instances.

	VitaminC						FEVER
	Direct (baseline)			Qxr (ours)			Direct (baseline)			Qxr (ours)
Model	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Sonnet 3.7	0.979	0.854	0.912	0.956	0.975	0.965	0.992	0.805	0.889	0.983	0.977	0.980
Sonnet 4	0.956	0.877	0.915	0.938	0.983	0.960	0.995	0.833	0.907	0.981	0.977	0.978
Sonnet 4.5	0.997	0.617	0.762	0.975	0.957	0.966	0.990	0.901	0.944	0.978	0.947	0.962
Opus 4.6	0.654	0.574	0.611	0.985	0.968	0.976	0.995	0.984	0.989	0.971	0.958	0.964

Table 2: Evaluation of consistent fact sets

F^{\prime}

on two datasets (VitaminC Schuster et al. (2021) / FEVER Thorne et al. (2018) ). Precision (P), recall (R), and F1 are computed with respect to gold consistent subsets. Qxr yields cleaner and more complete

F^{\prime}

than direct all-at-once LLM judging.

Model	Direct			Qxr
Model	P	R	F1	P	R	F1
Qwen2.5 7B	.883	.639	.741	.902	.743	.815
Qwen2.5 32B	.852	.839	.845	.854	.998	.920
Haiku 4.5	.979	.575	.724	.925	.810	.863
Sonnet 4.5	.962	.814	.882	.887	.854	.870
DS-v4-Pro	.905	.822	.861	.860	.952	.904

Table 3: RAMDocs consistency repair results.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Illustration of bounded conflict-graph treewidth. (a) A bag contains a subset of conflict-graph vertices. (b) Bags form a tree satisfying the running-intersection property. (c) Every MUS induces a clique in $G_{\mathcal{A}}$ and is therefore contained in some bag. Here $\mathcal{A}_{\mathrm{incons}}=\{\{a,b,d\},\{b,c,d\},\{c,d,e\}\}$ .

Method	Description	Complexity
Pairwise NLI Graph	Run NLI on every pair of claims, removing any node involved in a contradiction edge.	$\mathcal{O}(N^{2})$
Transitive Closure / Entailment Graph	Build a full entailment–contradiction graph and perform reasoning, e.g., SAT solving or closure.	$>\mathcal{O}(N^{2})$
LLM-as-NLI	Prompt an LLM with two sentences, e.g., “Does A contradict B?”	$\mathcal{O}(N^{2})$
Multi-premise NLI	Treat the entire set of claims as premises and ask if they are jointly consistent.	$\mathcal{O}(1)$
Clustered Pairwise	For each claim, compare only to its top- $K$ nearest neighbors.	$\mathcal{O}(NK)$

Appendix

Table 4: Common NLI-style baselines for consistency checking and their computational complexity. Pairwise and entailment-graph methods grow quadratically with the number of claims, while multi-premise NLI corresponds to the direct all-at-once baseline.

	Claude-3.7			Claude-4			DeepSeek-R1			GPT-OSS-120B			Mistral Large 2
Prompting Style	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Zero-shot	.986	.681	.806	.989	.928	.958	.993	.860	.922	.994	.876	.931	.947	.568	.710
Chain-of-Thought	.973	.684	.803	.987	.908	.946	.996	.850	.917	.995	.851	.917	.997	.570	.725
Decomposition	.987	.626	.766	.677	.608	.641	.996	.882	.936	.954	.816	.880	.991	.535	.695
Few-shot	.975	.658	.786	.992	.891	.939	.882	.761	.817	.997	.896	.944	.238	.142	.178
Self-consistency	.986	.547	.704	.992	.768	.866	.987	.718	.831	.996	.475	.643	.990	.522	.684

Appendix

Table 5: Prompt-sensitivity results across models. We compare direct judging under alternative prompting strategies with the original direct baseline and Qxr .

Factual Consistency Evaluation

Factual consistency evaluation measures whether generated claims are accurate, supported by evidence, and consistent across documents or interactions.

See all 72 papers

CREDENCE: Claim Reduction for Decomposition & Enhanced Credibility -- Semantic Metrics and Convergence Analysis
Jun 18, 2026Phuong Huu Vu Tran, Thuan Duc Mai, Bach Xuan LeLLM Self-Correction Automated Fact-Checking
Vietnamese-German University, Ho Chi Minh City, Vietnam · Ho Chi Minh University of Technology, Ho Chi Minh City, Vietnam
Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval
Mar 5, 2026Artem Vazhentsev, Maria Marina, Daniil Moskovskiy +8Automated Fact-Checking Factual Knowledge in Language Models
MBZUAI · S-NLP Group · AXXX +3
Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
Aug 12, 2026Sen Xu, Wei Wang, Shixi Liu +5Test-Time Scaling LLM Reliability
Sina Weibo Inc.

Jun 18, 2026J/K move · Enter open · S save

CREDENCE: Claim Reduction for Decomposition & Enhanced Credibility -- Semantic Metrics and Convergence Analysis

Phuong Huu Vu Tran, Thuan Duc Mai, Bach Xuan Le

Vietnamese-German University, Ho Chi Minh City, Vietnam · Ho Chi Minh University of Technology, Ho Chi Minh City, Vietnam

LLM Self-Correction Automated Fact-Checking

Open paper AskPDF arXiv

Decomposing compound sentences into atomic, verifiable claims is a prerequisite for reliable automated fact-checking. Prior work has relied on token-overlap (Jaccard) metrics that systematically underestimate decomposition quality for paraphrastic claims, and has lacked formal termination analysis for the repair loop. We present Credence, a revised claim decomposition and evaluation framework addressing both shortcomings. Our contributions are: (1) Semantic-F1: we use BGE-large cosine similarity fidelity metric that resolves Jaccard's penalisation and improves downstream fact-checking accuracy; (2) Convergence theorems: we formally characterise four properties of the repair pipeline, establishing that rule-based repair is monotone and finitely terminating under an oracle parser assumption; LLM-based self-repair is provably non-monotone and requires an early-exit guard; (3) Three evaluation benchmarks spanning social-media, encyclopaedic, and news domains for cross-domain generalisation measurement; (4) Multi-model benchmarking across four decomposer models (3.8B-12B) and a closed API model. Experiments on SocialClaimSplit, WikiSplitBench, and ClaimDecompBench show that Semantic-F1 outperforms Jaccard-F1 by +15-32pp. EPR ranges from 0.94 to 1.00 on SocialClaimSplit and WikiSplitBench, while ClaimDecompBench includes lower base EPR cases (down to 0.824) due to harder news-domain constructions, and rule-repair reduces the Atomicity Violation Rate (AVR) by 47-100% relative to the base model without degrading fidelity.

Abstract

Factual Consistency Evaluation

Explore similar work

CREDENCE: Claim Reduction for Decomposition & Enhanced Credibility -- Semantic Metrics and Convergence Analysis

Leveraging LLM Parametric Knowledge for Fact Checking without Retrieval

Claim-Level Reliability Assessment for Efficient Test-Time Reasoning