When Verification Fails: How Compositionally Infeasible Claims Escape Rejection

Muxin Liu; Delip Rao; Grace Kim; Chris Callison-Burch

cs.CLApr 13, 2026

When Verification Fails: How Compositionally Infeasible Claims Escape Rejection

Authors: Muxin Liu, Delip Rao, Grace Kim, Chris Callison-Burch

Organizations: University of Pennsylvania

Abstract

Scientific claim verification, the task of determining whether claims are entailed by scientific evidence, is fundamental to establishing discoveries in evidence while preventing misinformation. This process involves evaluating each asserted constraint against validated evidence. Under the Closed-World Assumption (CWA), a claim is accepted if and only if all asserted constraints are positively supported. We show that existing verification benchmarks cannot distinguish models enforcing this standard from models applying a simpler shortcut called salient-constraint checking, which applies CWA's rejection criterion only to the most salient constraint and accepts when that constraint is supported. Because existing benchmarks construct infeasible claims by perturbing a single salient element they are insufficient at distinguishing between rigorous claim verification and simple salient-constraint reliance. To separate the two, we construct compositionally infeasible claims where the salient constraint is supported but a non-salient constraint is contradicted. Across model families and modalities, models that otherwise saturate existing benchmarks consistently over-accept these claims, confirming the prevalence of such shortcut reasoning. Via model context interventions, we show that different models and prompting strategies occupy distinct positions on a shared ROC curve, indicating that the gap between model families reflects differences in verification threshold rather than underlying reasoning ability, and that the compositional inference bottleneck is a structural property of current verification behavior that strategy guidance alone cannot overcome.

Figures & tables

Figure 1: CWA verification checks all constraints; salient-constraint checking only finds the most prominent constraint. On existing benchmarks both agree since violated constraints are always salient. On compositionally infeasible claims, salient-constraint checking accepts while CWA rejects.

Figure 2: Averaged accuracy across models. Compositionally feasible claims score higher (

\uparrow

) than standard positives; compositionally infeasible claims score lower (

\downarrow

). Compositional structure shifts behavior toward acceptance.

Figure 3: Graph-based adversarial construction. Observation nodes ground directly retrievable facts; context and interpretation nodes combine them into higher-level conclusions. Corrupting one non-salient node produces a compositionally infeasible claim while preserving all observation-layer support.

	NLI4CT					SCITAB					SciVer
Model	Standard			Adv	$\bm{\Delta}$ (%)	Standard			Adv	$\bm{\Delta}$ (%)	Standard			Adv	$\bm{\Delta}$ (%)
Model	Pos.(%)	Neg.(%)	$\bm{F_{1}}$	Neg.(%)	$\bm{\Delta}$ (%)	Pos.(%)	Neg.(%)	$\bm{F_{1}}$	Neg.(%)	$\bm{\Delta}$ (%)	Pos.(%)	Neg.(%)	$\bm{F_{1}}$	Neg.(%)	$\bm{\Delta}$ (%)
GPT-5.4	80.2	94.1	0.86	88.7	$\downarrow$ 5.4	75.5	93.0	0.83	91.0	$\downarrow$ 2.0	55.9	92.5	0.68	87.7	$\downarrow$ 4.8
GPT-5-mini	83.7	91.1	0.87	84.7	$\downarrow$ 6.4	79.0	92.0	0.84	82.5	$\downarrow$ 9.5	78.4	84.5	0.81	73.4	$\downarrow$ 11.2
GPT-o4-mini	86.1	91.1	0.88	76.8	$\downarrow$ 14.2	84.0	88.0	0.86	77.0	$\downarrow$ 11.0	78.9	87.8	0.83	63.4	$\downarrow$ 24.4
GPT-4o-mini	71.3	85.1	0.77	73.4	$\downarrow$ 11.7	58.5	81.0	0.66	67.0	$\downarrow$ 14.0	66.7	59.2	0.64	50.7	$\downarrow$ 8.4

Table 1: Evaluation across three domains. Pos./Neg. = accuracy (%) on standard positive/negative claims;

F_{1}

= macro-

F_{1}

on standard negatives; Adv Neg. = accuracy (%) on adversarial negatives;

\Delta

= Adv Neg.

-

Standard Neg. (%).

\Delta

values shown as

\downarrow

; bold entries indicate substantial drop (

|\Delta|>10\%

Figure 4: Scaling behavior across Qwen3 (VL) model sizes for three domains. Curves show accuracy for positive, negative, and adversarial negative claims.

Figure 5: Adversarial negative accuracy under four evidence conditions across eight models. Graph-O provides no consistent improvement over no graph, while Graph-O+C produces intermediate gains and Graph-All approaches ceiling on NLI4CT and SCITAB.

Panel of Figure 6 — Figure 6: (a) Verification trajectories in ROC space (TPR vs. FPR) under different prompting conditions for Gemini-3.0-Flash. Arrows indicate the ordering of prompting variants (DCP+OWA $\to$ OWA $\to$ Baseline $\to$ CWA $\to$ DCP+CWA); stars denote the Sonnet-4.5 reference. (b) Adversarial negative accuracy under each prompting condition; dashed lines indicate the Sonnet-4.5 reference.

	Standard Negative			Adversarial Neg
	Correct (TN) $\uparrow$	Incorrect (FN) $\uparrow$	Avg $\uparrow$	TN $\uparrow$	FN $\uparrow$	Avg $\uparrow$
GPT-5-mini	86.2	61.7	84.0	92.3	64.3	88.0
Gemini-3.0-Flash	83.0	61.9	80.1	91.4	65.1	84.9
$+$ OWA	87.7	59.2	83.6	92.0	63.6	84.5
$+$ OWA	88.0	63.2	85.9	93.9	63.4	87.1
Claude-Sonnet-4.5	89.1	71.4	87.6	92.1	70.9	89.2

Table 2: CWA trace scores (0–100) on NLI4CT. Correct (TN) denotes traces where the model successfully rejected the claim (True Negative); Incorrect (FN) denotes traces where the model falsely accepted it (False Negative).

+

CWA and

+

OWA are prompt intervention.

	GPQA		AIME
	$F_{1}\uparrow$	Prec@Rej $\uparrow$	$F_{1}\uparrow$	Prec@Rej $\uparrow$
Baseline	.218	.253	.287	.256
OWA	.165	.206	.238	.226
CWA	.262	.273	.347	.290

Table 3: Self-verification on GPQA Diamond and AIME. A rejection is valid if the solver response is incorrect and the verifier reasoning correctly identifies the exact solver error.

F_{1}

: macro-

F_{1}

over valid rejections and valid acceptances. Prec@Rej: precision over valid rejections only. Bold : best.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Original source	Subset used	Generated	Validated	Notes
NLI4CT	Clinical trial claims	Single-trial examples only	242	203	We exclude multi-trial claims and generate hard negatives only from single-trial evidence.
SciTab	Scientific table claims	Randomly selected table subset	230	200	Candidate pool was sufficient to retain 200 validated hard negatives after manual filtering.
SciVer	Scientific figure claims	Single-chart direct-reasoning subset	213	180	SciVer includes multiple reasoning types and multimodal evidence settings (e.g., chart, text, table, or multiple evidence pieces). We restrict to single-chart examples, which yields 213 source items; after validation, fewer than 200 hard negatives remain.

Appendix

Table 4: Dataset subsets and annotation outcomes for adversarial hard-negative construction.

	Evidence	Feasible claims	Standard infeasible claims	Our hard-negative perturbations
NLI4CT	Breast cancer clinical trial report excerpts; single section per instance.	Expert-written claims requiring biomedical, numerical, and commonsense reasoning.	Contrast-set perturbations, including numerical, lexical, semantic, and syntactic changes.	Compositional clinical misreadings such as qualifier omission, scope/population shifts, conditional or boundary flips, cross-arm or proxy misattribution, and surfaced assumptions.
SciTab	Scientific tables with captions from NLP/ML papers.	Sentences from the paper grounded in the table; two-annotator verified.	InstructGPT-generated negatives with wrong calculations, value mismatches, and incorrect approximations.	Structural table perturbations such as inconsistent baselines, part–whole and averaging errors, derived-metric overextension, null-as-zero aggregation, and occasional causal or categorical overreach.
SciVer	Chart screenshots from CS/arXiv papers; direct-reasoning subset only.	Expert-written claims grounded in chart content.	Expert-rewritten negatives with hallucinated values, value misattribution, or relationship reversal.	Structural visual misreadings such as local-to-global overreach, ignored panel or layout boundaries, visual analogy or element conflation, prominence-based over-interpretation, and unsupported comparison classes.

Appendix

Table 5: Benchmark datasets used in this work, including both their original infeasible-claim construction and our compositional hard-negative perturbations.

Figure 7: Qwen3 model families performance of SCITAB with decoding at different temperatures

Figure 8: Longer negatives show no improvement or degradation over standard negatives, ruling out surface form as the reason for performance drop on adversarial negatives

Domain	Evaluator	Rephrased (%)	Original (%)
NLI4CT	GPT-5-mini	86.0	86.0
	Gemini-3.0-Flash	76.0	76.0
SCITAB	GPT-5-mini	74.0	82.0
	Gemini-3.0-Flash	72.0	70.0
SciVer	GPT-5-mini	78.0	66.0
	Gemini-3.0-Flash	66.0	70.0

Appendix

Table 6: Hard negative accuracy on Gemini-rephrased claims (rephrased) vs. raw GPT-5-mini-generated claims (original), evaluated by two independent model families. Differences are small and inconsistent in direction, ruling out systematic generation-verifier bias.

Dimension	Salient-constraint checking ( $-2$ )	Full CWA ( $+2$ )
Constraint enumeration	Checks one prominent constraint	Lists all constraints before checking any
Non-salient coverage	Stops at salient constraint	Checks non-salient and composed constraints
Evidence boundary	Accepts on local consistency	Rejects when scope exceeds evidence
Verdict warrant	Notes gaps but accepts	Verdict follows from all gaps

Appendix

Table 7: Rubric dimensions for scoring reasoning traces on the salient-constraint checking vs. full CWA axis. Each dimension is scored

\{-2,\ldots,+2\}

with anchored examples.

Solver	Verifier	Reasoning	Label	Description	Count as
Incorrect	Reject	Valid	TP	Verifier correctly identifies the actual error	True Positive
Correct	Accept	—	TN	Verifier correctly accepts a correct solution	True Negative
Incorrect	Reject	Invalid	Lucky Catch	Verifier rejects but for wrong/unrelated reason	False Positive
Correct	Reject	—	FP	Verifier incorrectly rejects a correct solution	False Positive
Incorrect	Accept	—	FN	Verifier incorrectly accepts an incorrect solution	False Negative

Appendix

Table 8: Case taxonomy for self-verification evaluation.

Dataset	Variant	n	Acc	F1	TP	TN	FP	FN	Prec@Rej
GPQA	baseline	198	0.313	0.218	19	43	82	54	19/75 $\approx$ 0.253
	CWA	198	0.318	0.262	24	39	94	41	24/88 $\approx$ 0.273
	OWA	198	0.283	0.165	14	42	81	61	14/68 $\approx$ 0.206
AIME	baseline	200	0.255	0.287	30	21	93	56	30/117 $\approx$ 0.256
	CWA	200	0.285	0.347	38	19	101	42	38/131 $\approx$ 0.290
	OWA	200	0.230	0.238	24	22	87	67	24/106 $\approx$ 0.226

Appendix

Table 17

Figure 11: Illustrative hard-negative generation for SciVer. A single interpretation-node corruption turns a supported local reading at level_3 into an unsupported global claim about performance across the plotted noise spectrum.

Explore similar work

What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis
Apr 2, 2026Delip Rao, Chris Callison-BurchClaim Verification LLM Evaluation
University of Pennsylvania, Philadelphia, PA, USA
ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models
Apr 2, 2026Delip Rao, Feijiang Han, Chris Callison-BurchClaim Verification LLM Answer Verification
University of Pennsylvania, Philadelphia, PA, USA
DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation
May 26, 2026Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari +1Selective Prediction Citation Verification
FirstPrinciples · School of Computing Science, Simon Fraser University

Apr 2, 2026J/K move · Enter open · S save

What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis

Delip Rao, Chris Callison-Burch

University of Pennsylvania, Philadelphia, PA, USA

Claim Verification LLM Evaluation Scientific Claim Verification

Open paper AskPDF arXiv

Despite rapid progress in claim verification, we lack a systematic understanding of what reasoning these benchmarks actually exercise. We generate structured reasoning traces for 24K claim-verification examples across 9 datasets using GPT-4o-mini and find that direct evidence extraction dominates, while multi-sentence synthesis and numerical reasoning are severely under-represented. A dataset-level breakdown reveals stark biases: some datasets almost exclusively test lexical matching, while others require information synthesis in roughly half of cases. Using a compact 1B-parameter reasoning verifier, we further characterize five error types and show that error profiles vary dramatically by domain -- general-domain verification is dominated by lexical overlap bias, scientific verification by overcautiousness, and mathematical verification by arithmetic reasoning failures. Our findings suggest that high benchmark scores primarily reflect retrieval-plus-entailment ability. We outline recommendations for building more challenging evaluation suites that better test the reasoning capabilities verification systems need.

Abstract

Explore similar work

What Do Claim Verification Datasets Actually Test? A Reasoning Trace Analysis

ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation