Scientific claim verification, the task of determining whether claims are entailed by scientific evidence, is fundamental to establishing discoveries in evidence while preventing misinformation. This process involves evaluating each asserted constraint against validated evidence. Under the Closed-World Assumption (CWA), a claim is accepted if and only if all asserted constraints are positively supported. We show that existing verification benchmarks cannot distinguish models enforcing this standard from models applying a simpler shortcut called salient-constraint checking, which applies CWA's rejection criterion only to the most salient constraint and accepts when that constraint is supported. Because existing benchmarks construct infeasible claims by perturbing a single salient element they are insufficient at distinguishing between rigorous claim verification and simple salient-constraint reliance. To separate the two, we construct compositionally infeasible claims where the salient constraint is supported but a non-salient constraint is contradicted. Across model families and modalities, models that otherwise saturate existing benchmarks consistently over-accept these claims, confirming the prevalence of such shortcut reasoning. Via model context interventions, we show that different models and prompting strategies occupy distinct positions on a shared ROC curve, indicating that the gap between model families reflects differences in verification threshold rather than underlying reasoning ability, and that the compositional inference bottleneck is a structural property of current verification behavior that strategy guidance alone cannot overcome.
Figures & tables
Figure 1: CWA verification checks all constraints; salient-constraint checking only finds the most prominent constraint. On existing benchmarks both agree since violated constraints are always salient. On compositionally infeasible claims, salient-constraint checking accepts while CWA rejects.
Figure 3: Graph-based adversarial construction. Observation nodes ground directly retrievable facts; context and interpretation nodes combine them into higher-level conclusions. Corrupting one non-salient node produces a compositionally infeasible claim while preserving all observation-layer support.
NLI4CT
SCITAB
SciVer
Model
Standard
Adv
Δ (%)
Standard
Adv
Δ (%)
Standard
Adv
Δ (%)
Pos.(%)
Neg.(%)
F1
Neg.(%)
Pos.(%)
Neg.(%)
F1
Neg.(%)
Pos.(%)
Neg.(%)
F1
Neg.(%)
GPT-5.4
80.2
94.1
0.86
88.7
↓ 5.4
75.5
93.0
0.83
91.0
↓ 2.0
55.9
92.5
0.68
87.7
↓ 4.8
GPT-5-mini
83.7
91.1
0.87
84.7
↓ 6.4
79.0
92.0
0.84
82.5
↓ 9.5
78.4
84.5
0.81
73.4
↓ 11.2
GPT-o4-mini
86.1
91.1
0.88
76.8
↓ 14.2
84.0
88.0
0.86
77.0
↓ 11.0
78.9
87.8
0.83
63.4
↓ 24.4
GPT-4o-mini
71.3
85.1
0.77
73.4
↓ 11.7
58.5
81.0
0.66
67.0
↓ 14.0
66.7
59.2
0.64
50.7
↓ 8.4
Table 1: Evaluation across three domains. Pos./Neg. = accuracy (%) on standard positive/negative claims; F1 = macro- F1 on standard negatives; Adv Neg. = accuracy (%) on adversarial negatives; Δ = Adv Neg. − Standard Neg. (%). Δ values shown as ↓ ; bold entries indicate substantial drop ( ∣Δ∣>10% ).
Figure 4: Scaling behavior across Qwen3 (VL) model sizes for three domains. Curves show accuracy for positive, negative, and adversarial negative claims.
Figure 5: Adversarial negative accuracy under four evidence conditions across eight models. Graph-O provides no consistent improvement over no graph, while Graph-O+C produces intermediate gains and Graph-All approaches ceiling on NLI4CT and SCITAB.
Figure 6: (a) Verification trajectories in ROC space (TPR vs. FPR) under different prompting conditions for Gemini-3.0-Flash. Arrows indicate the ordering of prompting variants (DCP+OWA → OWA → Baseline → CWA → DCP+CWA); stars denote the Sonnet-4.5 reference. (b) Adversarial negative accuracy under each prompting condition; dashed lines indicate the Sonnet-4.5 reference.
Standard Negative
Adversarial Neg
Correct (TN) ↑
Incorrect (FN) ↑
Avg ↑
TN ↑
FN ↑
Avg ↑
GPT-5-mini
86.2
61.7
84.0
92.3
64.3
88.0
Gemini-3.0-Flash
83.0
61.9
80.1
91.4
65.1
84.9
+ OWA
87.7
59.2
83.6
92.0
63.6
84.5
+ OWA
88.0
63.2
85.9
93.9
63.4
87.1
Claude-Sonnet-4.5
89.1
71.4
87.6
92.1
70.9
89.2
Table 2: CWA trace scores (0–100) on NLI4CT. Correct (TN) denotes traces where the model successfully rejected the claim (True Negative); Incorrect (FN) denotes traces where the model falsely accepted it (False Negative). + CWA and + OWA are prompt intervention.
GPQA
AIME
F1↑
Prec@Rej ↑
F1↑
Prec@Rej ↑
Baseline
.218
.253
.287
.256
OWA
.165
.206
.238
.226
CWA
.262
.273
.347
.290
Table 3: Self-verification on GPQA Diamond and AIME. A rejection is valid if the solver response is incorrect and the verifier reasoning correctly identifies the exact solver error. F1 : macro- F1 over valid rejections and valid acceptances. Prec@Rej: precision over valid rejections only. Bold : best.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Original source
Subset used
Generated
Validated
Notes
NLI4CT
Clinical trial claims
Single-trial examples only
242
203
We exclude multi-trial claims and generate hard negatives only from single-trial evidence.
SciTab
Scientific table claims
Randomly selected table subset
230
200
Candidate pool was sufficient to retain 200 validated hard negatives after manual filtering.
SciVer
Scientific figure claims
Single-chart direct-reasoning subset
213
180
SciVer includes multiple reasoning types and multimodal evidence settings (e.g., chart, text, table, or multiple evidence pieces). We restrict to single-chart examples, which yields 213 source items; after validation, fewer than 200 hard negatives remain.
Appendix
Table 4: Dataset subsets and annotation outcomes for adversarial hard-negative construction.
Evidence
Feasible claims
Standard infeasible claims
Our hard-negative perturbations
NLI4CT
Breast cancer clinical trial report excerpts; single section per instance.
Expert-written claims requiring biomedical, numerical, and commonsense reasoning.
Contrast-set perturbations, including numerical, lexical, semantic, and syntactic changes.
Compositional clinical misreadings such as qualifier omission, scope/population shifts, conditional or boundary flips, cross-arm or proxy misattribution, and surfaced assumptions.
SciTab
Scientific tables with captions from NLP/ML papers.
Sentences from the paper grounded in the table; two-annotator verified.
InstructGPT-generated negatives with wrong calculations, value mismatches, and incorrect approximations.
Structural table perturbations such as inconsistent baselines, part–whole and averaging errors, derived-metric overextension, null-as-zero aggregation, and occasional causal or categorical overreach.
SciVer
Chart screenshots from CS/arXiv papers; direct-reasoning subset only.
Expert-written claims grounded in chart content.
Expert-rewritten negatives with hallucinated values, value misattribution, or relationship reversal.
Structural visual misreadings such as local-to-global overreach, ignored panel or layout boundaries, visual analogy or element conflation, prominence-based over-interpretation, and unsupported comparison classes.
Appendix
Table 5: Benchmark datasets used in this work, including both their original infeasible-claim construction and our compositional hard-negative perturbations.
Figure 7: Qwen3 model families performance of SCITAB with decoding at different temperatures
Figure 8: Longer negatives show no improvement or degradation over standard negatives, ruling out surface form as the reason for performance drop on adversarial negatives
Domain
Evaluator
Rephrased (%)
Original (%)
NLI4CT
GPT-5-mini
86.0
86.0
Gemini-3.0-Flash
76.0
76.0
SCITAB
GPT-5-mini
74.0
82.0
Gemini-3.0-Flash
72.0
70.0
SciVer
GPT-5-mini
78.0
66.0
Gemini-3.0-Flash
66.0
70.0
Appendix
Table 6: Hard negative accuracy on Gemini-rephrased claims (rephrased) vs. raw GPT-5-mini-generated claims (original), evaluated by two independent model families. Differences are small and inconsistent in direction, ruling out systematic generation-verifier bias.
Dimension
Salient-constraint checking ( −2 )
Full CWA ( +2 )
Constraint enumeration
Checks one prominent constraint
Lists all constraints before checking any
Non-salient coverage
Stops at salient constraint
Checks non-salient and composed constraints
Evidence boundary
Accepts on local consistency
Rejects when scope exceeds evidence
Verdict warrant
Notes gaps but accepts
Verdict follows from all gaps
Appendix
Table 7: Rubric dimensions for scoring reasoning traces on the salient-constraint checking vs. full CWA axis. Each dimension is scored {−2,…,+2} with anchored examples.
Solver
Verifier
Reasoning
Label
Description
Count as
Incorrect
Reject
Valid
TP
Verifier correctly identifies the actual error
True Positive
Correct
Accept
—
TN
Verifier correctly accepts a correct solution
True Negative
Incorrect
Reject
Invalid
Lucky Catch
Verifier rejects but for wrong/unrelated reason
False Positive
Correct
Reject
—
FP
Verifier incorrectly rejects a correct solution
False Positive
Incorrect
Accept
—
FN
Verifier incorrectly accepts an incorrect solution
False Negative
Appendix
Table 8: Case taxonomy for self-verification evaluation.
Dataset
Variant
n
Acc
F1
TP
TN
FP
FN
Prec@Rej
GPQA
baseline
198
0.313
0.218
19
43
82
54
19/75 ≈ 0.253
CWA
198
0.318
0.262
24
39
94
41
24/88 ≈ 0.273
OWA
198
0.283
0.165
14
42
81
61
14/68 ≈ 0.206
AIME
baseline
200
0.255
0.287
30
21
93
56
30/117 ≈ 0.256
CWA
200
0.285
0.347
38
19
101
42
38/131 ≈ 0.290
OWA
200
0.230
0.238
24
22
87
67
24/106 ≈ 0.226
Appendix
Table 17
Figure 18Figure 19
Figure 11: Illustrative hard-negative generation for SciVer. A single interpretation-node corruption turns a supported local reading at level_3 into an unsupported global claim about performance across the plotted noise spectrum.
Despite rapid progress in claim verification, we lack a systematic understanding of what reasoning these benchmarks actually exercise. We generate structured reasoning traces for 24K claim-verification examples across 9 datasets using GPT-4o-mini and find that direct evidence extraction dominates, while multi-sentence synthesis and numerical reasoning are severely under-represented. A dataset-level breakdown reveals stark biases: some datasets almost exclusively test lexical matching, while others require information synthesis in roughly half of cases. Using a compact 1B-parameter reasoning verifier, we further characterize five error types and show that error profiles vary dramatically by domain -- general-domain verification is dominated by lexical overlap bias, scientific verification by overcautiousness, and mathematical verification by arithmetic reasoning failures. Our findings suggest that high benchmark scores primarily reflect retrieval-plus-entailment ability. We outline recommendations for building more challenging evaluation suites that better test the reasoning capabilities verification systems need.
We present ThinknCheck, a 1B-parameter verifier for grounded claim verification that first produces a short, structured rationale and then a binary verdict. We construct LLMAggreFact-Think, a 24.1k reasoning-augmented training set derived from LLMAggreFact, and fine-tune a 4-bit Gemma3 model to follow this format. On LLMAggreFact, ThinknCheck attains 78.1 balanced accuracy (BAcc), surpassing MiniCheck-7B (77.4) with 7x fewer parameters; removing the reasoning step reduces BAcc to 57.5. On SciFact, ThinknCheck reaches 64.7 BAcc, a +14.7 absolute gain over MiniCheck-7B. By contrast, zero-shot chain-of-thought on the base Gemma3-1B harms accuracy relative to direct answers, and preference optimization with a simple format+accuracy reward underperforms supervised reasoning. To probe the latter, we introduce GSMClaims and a domain-specialized variant, ThinknCheck-Science, which improves across benchmarks, including 61.0% accuracy on GSMClaims. Overall, explicit, supervised reasoning enables compact verifiers that are competitive while remaining resource-efficient and interpretable.
Misalignment between claims and their cited evidence is a common failure mode in reports generated by large language models, limiting their reliability in scientific and other high-stakes settings. We present DeepSciVerify, a two-stage pipeline for scientific claim-citation verification that combines abstract-level reasoning with selective escalation to passage-level evidence. The system first verifies claims using the abstract and defers uncertain cases, retrieving and analyzing full-text passages only when necessary. This design leverages complementary behaviors across LLMs, as some models are more conservative while others are more decisive under uncertainty. On the SCitance benchmark, DeepSciVerify achieves 86.7 Micro-F1, outperforming strong abstract-only baselines by +4.5 points while resolving 67% of instances without full-text retrieval. These results suggest that selective evidence escalation improves both accuracy and efficiency in claim-citation verification.