Misinformation that cites legitimate papers can be especially harmful when it distorts what those studies actually report. While existing automatic fact-checking systems based on large language models (LLMs) can assess whether a model assigns an Incorrect verdict and can gen- erate explanations for that decision, they typi- cally do not indicate whether the model follows the same reasoning path as human experts or arrives at the verdict through a different but still valid path. In this work, we introduce a graph- based framework (typed reasoning graph) for comparing human and LLM reasoning paths in scientific fact-checking. Building on prior work on fallacious reasoning in biomedical misinformation, MISSCIPLUS (Glockner et al., 2025), we model each explanation as a rea- soning graph that links the false claim to the relevant study context, study findings, fallacy- supporting premises, and fallacy labels. This representation enables one-to-one alignment of human and LLM reasoning at the level of fallacy-specific sub-graphs. For non-human- aligned LLM paths, we validate grounding in the cited study, relevance to the claim, and suf- ficiency for the verdict. Using 84 false claims from MISSCIPLUS, we evaluate GPT-5, Claude Opus 4.7, and Qwen3-32B across prompt and evidence settings. Results show distinct perfor- mance dimensions: Qwen3-32B has the lowest verdict failure rate, GPT-5 the highest human alignment, and Claude Opus 4.7 weak verdict prediction but often valid reasoning in success- ful cases
Figures & tables
Figure 1: Example reasoning-path comparison for a misleading scientific claim: Both explanations reach the same verdict, Incorrect . The LLM aligns with the human reference on the upper reasoning path labeling it as Hasty Generalization . In addition, the LLM extracts a second reasoning path labeling it as Fallacy of Exclusion
Figure 2: Overview of our graph-based framework for comparing human and LLM reasoning in scientific fact-checking. For Incorrect -verdict cases, human and LLM explanations are converted into reasoning graphs and compared at the level of fallacy-specific sub-graphs. Non-aligned LLM reasoning paths are further evaluated for study grounding, relevance to the claim, and sufficiency for the verdict.
Model
Evidence
Prompt
Correct
Incorrect
NEI
No Response
Consistency
Qwen3-32B
Internal Knowledge (IK)
IK_Prompt
4.76
82.14
8.33
4.76
100.00
Full Study
Short Prompt
30.56
39.68
29.76
0.00
80.95
Detailed Prompt
23.81
73.41
2.78
0.00
85.71
Selected Passages
Short Prompt
27.78
31.75
40.48
0.00
79.76
Detailed Prompt
19.84
76.98
3.17
0.00
86.90
GPT-5
Internal Knowledge (IK)
IK_Prompt
7.54
84.92
7.54
0.00
78.57
Table 1: LLM-based fact-checking results across prompting strategies and evidence settings. Values report average verdict distribution (%) over three runs and model consistency (agreement across runs), except Claude Sonnet 4.6 full-study results, which are from seed 1 only. The internal-knowledge setting uses no external evidence.
Figure 3: Claim-level outcome breakdown under the full-study evidence setting with the detailed prompt ( n=84 per model).
LLM ( n )
Human-
Not Human-Aligned
Final
Aligned
Grounded (QA)
Grounded (final)
Ungrounded
Relevance
Sufficiency
Rel.+Suff.
Accepted
Not Accepted
2
1
0
2
1
0
2
1
0
GPT-5 (158)
19.0
62.7
74.7
6.3
74.1
0.6
0.0
63.3
11.4
0.0
62.7
12.0
0.0
81.7
18.3
Qwen3-32B (127)
10.2
49.6
81.1
8.7
79.5
0.8
0.8
70.1
10.2
0.8
70.1
10.2
0.8
80.3
19.7
Claude Opus 4.7 (84)
11.9
69.0
82.1
6.0
81.0
1.2
0.0
75.0
6.0
1.2
75.0
6.0
1.2
86.9
13.1
Table 2: Alignment and validation of LLM reasoning sub-graphs, reported as % of each model’s total sub-graph set. Human-Aligned + Grounded (final) + Ungrounded sum to 100% per row; Accepted + Not Accepted also sum to 100%. Grounded (QA) is the subset of Grounded (final) admitted by the automatic QA layer alone; QA over-flagged premises as not-grounded due to overly generic questions and granularity mismatches with the document ; human re-verification corrected these cases, raising the Grounded (final) number . Relevance, Sufficiency, and Rel.+Suff. columns report the three-LLM-judge consensus score distribution (2/1/0) over the Grounded (final) pool. A sub-graph is Accepted if it is human-aligned, or if its Rel.+Suff. score is 2; otherwise Not Accepted . A high accepted rate alone does not imply strong overall performance, since Claude performs poorly at verdict prediction.
Figure 4: Fallacy-label distributions across human and model reasoning graphs. Black outlines mark the highest percentage within each fallacy type. The most frequent labels are Fallacy of Exclusion (FOE) , Causal Oversimplification (CO) , and Hasty Generalization (HG) . Fallacy abbreviations are listed in Appendix Table 5 .
Model pair
Cohen’s κ
GPT-5 ↔ Qwen3-32B
0.511
GPT-5 ↔ Claude Opus 4.7
0.466
Claude Opus 4.7 ↔ Qwen3-32B
0.428
Table 3: Cross-model agreement in fallacy assignment. Cohen’s κ measures whether two models assign the same fallacy label to the same claim, showing moderate agreement across model pairs.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Claude Opus 4.7
GPT-5
Qwen3-32B
Veracity
Correct
30.0
10.0
0.0
Incorrect
70.0
70.0
80.0
Not Enough Information
0.0
20.0
20.0
No Response
0.0
0.0
0.0
Consistency
90.0
90.0
90.0
Appendix
Table 4: Contamination check on 10 false claims published in 2025–2026 that are not part of the dataset used in this study. Values are reported as %.
Figure 5: Agreement between human and GPT-5 annotations for the same 84 GPT-5 fact-checking explanations. Each explanation was independently converted into a reasoning graph by a human annotator and by GPT-5. The figure summarizes the extent to which the GPT-5-generated reasoning graphs align with the human reference graphs. Left: fallacy-type agreement at the argument level, including at-least-one type match, mean Jaccard similarity, precision, recall, and F1 under type-equal multiset matching. Middle: total fallacy labels produced by each annotator and the subset matched between them. Right: mean cosine similarity using all-mpnet-base-v2 for corresponding textual components: Study Context , Study Result , and Fallacy-Supporting Premise . For fallacy-supporting premises, each GPT-5 premise is scored against its best-matching human premise and then averaged at the argument level.
Abbr.
Full name
FOE
Fallacy of Exclusion
CO
Causal Oversimplification
HG
Hasty Generalization
FE
False Equivalence
AM
Ambiguity
BSP
Biased Sample Fallacy
Appendix
Table 5: Abbreviations for the nine fallacy categories used in Figure 4 .
Figure 6: Illustration of human and LLM reasoning paths for a scientific fact-checking example. The example shows that an LLM reasoning path can align with the human reasoning path, while it can also take a different but valid reasoning path.