Same Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claims

Delip Rao; Ahmed Muharram; Chris Callison-Burch

cs.CLOct 14, 2026

Same Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claims

Authors: Delip Rao, Ahmed Muharram, Chris Callison-Burch

Organizations: University of Pennsylvania

Abstract

Co-scientists and agentic scientific claim verifiers are increasingly used to assess claims across the sciences. Natural-language scientific claims, written for human experts, are often underspecified and so admit several defensible interpretations. Such verifiers usually pick one, often tacitly, and benchmarks score their verdicts against expert labels without recording the expert's interpretation or the verifier's in a comparable form. This leads to disagreements with expert labels that cannot be traced to the evidence or the interpretation, and to compute spent on unchecked interpretations. We introduce Claim Cards to make such interpretations explicit in the reasoning itself. A Claim Card records a verifier's interpretation as nine fields before evidence gathering, each marked stated, inferred, or default. To test whether an agent's interpretation can change its verdict, we ran a GPT-5.4 agent five times on each of 60 expert-labeled materials-science and biomedical claims with a card, without one, and with one card frozen across runs (900 runs). Over 40% of card fields were not marked stated, and a card added about a tenth to a run's GPT-5.4 tokens and time without measurably changing agreement with expert labels. The agent's own interpretations varied between runs, although within a claim this variation did not measurably track the verdict. A deliberate change to one field of the frozen card, however, making the quantifier universal or the threshold for support stricter, changed the verdict on about a third of the claims, against about one in seven expected by chance (p < 0.001 for each edit). On 24 materials-science claims run with computational tools, the stricter threshold also changed what the agent asked those tools to compute. We therefore argue that verdicts should be reported with their interpretations and that benchmarks could publish the interpretation behind each label.

Figures & tables

Figure 1: (a) The opening claim, assessed by its labeling expert and by the agent, whose cards required a single solid phase at every temperature below 700 K. The expert’s boundaries are rounded (Appendix F.1 ). In this card, from a run with the computational tools, the parser marked the quantifier and threshold stated , though the expert judged the temperature range unspecified. (b) A run in card mode and a re-run with an edited, frozen card.

Field	What it records	Value on this card	Prov.
System specification
SystemEntity	The object of study	Organisms undergoing aging	stated
StatePhase	Its phase or state before any intervention	Aged versus young/adult	stated
Conditions	The conditions in which the claim is purported to hold	Matched measurement conditions; taxa and tissues must be identified when evaluating generality	stated
Intervention
Manipulation	How the system is modified, processed, or intervened upon	Aging; no experimental intervention specified	inferred

Table 1: The nine fields of an interpretation, and the Claim Card that ClaimSpy produced for “Autophagy declines in aged organisms” in the first card run, with the provenance mark of each value. It was also the claim’s frozen card.

			Did not	Below /
Edit	$n$	Shift	increase	above
Without computational libraries (60 claims)
Quantifier	56	$-0.62$ [ $-0.91$ , $-0.35$ ]	47	16 / 0
Threshold	58	$-0.54$ [ $-0.79$ , $-0.32$ ]	50	14 / 1
With the computational tools (24 claims)
Quantifier	20	$-0.55$ [ $-1.04$ , $-0.10$ ]	16	4 / 1

Table 2: Score shifts after one card field is edited to make the claim harder to satisfy (Section 3 ), with bootstrap intervals. The last two columns count claims whose edited score did not exceed the unedited mean and claims whose edited score fell below every unedited run or rose above the unedited range. The two identical shifts on the 24 claims are separate estimates.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: The Claim Card that ClaimSpy produced for the GaN claim, with field values and provenance marks as output by the parser. The margin notes are ours; gap numbers refer to the list in Appendix A.1 .

Conditions	Claims	Setting	Runs	Frozen or edited card
Card, no card, frozen	60	Without computational libraries a	900	Frozen: the card of each claim’s first card run †
Edited cards	56 + 58	Without computational libraries a	114	The first card run’s card with one field edited, the quantifier (56 claims) or the threshold (58 claims) †
Card, no card, frozen	24	With the computational tools	360	Frozen: a card from a separate preliminary card run, not one of the reported runs
Frozen	24	Search only	120	The same frozen card
Edited cards	20 + 24	With the computational tools and with search only	88	The frozen card with one field edited, the quantifier or the threshold, each run once in each setting
Pilot (Appendix I )	20	Three LLM backbones, card and no card, one run each	120	–

Appendix

Table 3: The runs in the study, by condition and setting. We ran 60 claims without computational libraries (30 materials science, 30 biomedical) and 24 materials-science claims with the computational tools and with search only, six of them among the 60. A frozen run injects a stored card into the agent’s prompt. An edited card is a stored card with one field changed, run once in the same way. a The CALPHAD and CHGNet tools could be called, but their libraries were not installed. Biomedical claims had a general coding agent instead (Appendix B.2 ). † A search outage affected some of these runs (Section 3 ).

Contrast	Bootstrap over groups	Sign-flip $p$
Quantifier edit, shift
with the tools	$-0.55$ [ $-0.93$ , $-0.20$ ]	0.039
search only	$-0.48$ [ $-0.68$ , $-0.28$ ]	0.004
Threshold edit, shift
with the tools	$-0.48$ [ $-0.75$ , $-0.23$ ]	0.016
search only	$-0.14$ [ $-0.42$ , $+0.15$ ]	0.26

Appendix

Table 4: Contrasts on the 24 materials-science claims, with the claims grouped by chemical system. The interval is a percentile interval from a bootstrap that resamples groups of claims, and

p

is from an exact sign-flip test over the per-group mean differences. The edit shifts are defined as in Table 9 , and the other rows are paired differences as in Tables 15 , 17 and 19 .

Problem id	Label
matsci_db_002	$+2$
matsci_db_003	$+2$
matsci_db_006	$-2$
mof_alloys_0007	$+2$
mof_batteries_0010	$-1$
vis_reas_alloys_0002	$+2$

Appendix

Table 5: The 30 materials-science claims run without computational libraries, from SFBench ( Costello et al., 2026 ) . Each is listed by its SFBench problem id, with its gold feasibility label (

-2

+2

). Table 7 names the six computational_tools claims. Of the others, the text refers to matsci_db_003 as the MgO surface-energy claim, str_dft_08 as the rock-salt cathode capacity claim and str_speculative_11 as the toluene polysulfide claim.

Claim id	Label	Claim id	Label	Claim id	Label
56	$+2$	532	$-2$	1021	$-2$
118	$+2$	539	$+2$	1062	$+2$
148	$+2$	540	$+2$	1140	$-2$
239	$+2$	589	$+2$	1144	$-2$
248	$+2$	636	$+2$	1221	$-2$
249	$-2$	770	$+2$	1241	$+2$

Appendix

Table 6: The 30 SciFact claims ( Wadden et al., 2020 ) , by claim id in the SciFact development set, with the gold label mapped to our scale ( support

=+2

, contradict

=-2

). Claim 148 is the autophagy claim of Section 1 .

Id	Short name	Label	Library	Tool- calling
0002	Al–Zn eutectic	$+2$	pycalphad	$\bullet$
0003 †	Al–Zn single-phase (the opening claim)	$0$	pycalphad	$\bullet$
0006	Al–Mg anode potential	$-2$	pycalphad
0009	Li–CuAl capacity	$-1$	pycalphad	$\bullet$
0010	Li–AlCu reaction	$+2$	pycalphad	$\bullet$
0011	silver bulk-modulus	$+2$	CHGNet

Appendix

Table 7: The 24 materials-science claims run with the computational tools, by problem id ( computational_tools_ id ), with the short name by which the text refers to the claim, the expert label and the library the claim is tagged for. A bullet marks the 12 tool-calling claims, the post hoc group of claims on which at least three of the five frozen runs with the computational tools called a tool (Appendix J.1 ). † Also among the 60 claims run without computational libraries (Table 5 ).

Figure 3: Share of cards that do not mark a field stated, that is, mark it inferred or default, in the 300 cards of the card condition without computational libraries (150 per domain) and the 118 cards parsed in the card runs with the computational tools. Figure 4 splits each bar into inferred and default. Fields are labeled with short forms of the names in Table 1 .

Figure 4: Where each card field’s value comes from. The left and middle panels show the 300 cards from the card runs on the 60 claims (150 materials-science, 150 biomedical), and the right panel the 118 cards parsed in the card runs on the 24 claims. The dark segment of each bar is the share of cards that mark the field stated, and the segments above it the shares that mark it inferred or default. The parser assigned the marks (Section 2.2 ). The dotted line is a 50% reference. Fields are labeled as in Figure 3 .

Figure 5: Agreement across five runs, by field, for explicit Claim Cards and for implicit interpretations recovered from runs without a card, in the setting without computational libraries. A judge model grouped the explicit interpretations of 51 of the 60 claims and the recovered ones of 56, and the paired comparison uses the 50 claims in both. Fields are labeled with short forms of the names in Table 1 .

		Same sign			Absolute score difference
Field compared	Claims	Alike	Differ	Alike $-$ differ	Alike	Differ	Differ $-$ alike
Without computational libraries (60 claims)
Threshold	31	85%	83%	$+0.03$ [ $-0.03$ , $+0.09$ ]	0.40	0.43	$+0.02$ [ $-0.09$ , $+0.17$ ]
Quantifier’s value	20	83%	84%	$-0.01$ [ $-0.08$ , $+0.07$ ]	0.48	0.52	$+0.03$ [ $-0.11$ , $+0.19$ ]
Judge’s quantifier groups	17	82%	87%	$-0.04$ [ $-0.11$ , $+0.01$ ]	0.42	0.50	$+0.08$ [ $-0.06$ , $+0.25$ ]
All nine fields	30	88%	82%	$+0.06$ [ $-0.03$ , $+0.16$ ]	0.27	0.34	$+0.07$ [ $-0.07$ , $+0.21$ ]

Appendix

Table 8: Whether a claim’s card runs that recorded the same interpretation also reached the same verdict. For each field, pairs of card runs whose values fell in the same consistency group ( alike ) are compared with pairs whose values fell in different groups ( differ ), on the claims that have both kinds of pair. Same sign is the share of pairs whose two scores have the same sign, and absolute score difference is the mean absolute difference between the two scores. Shares and means are averaged within each claim and then over claims, and both differences are paired by claim and oriented so that a positive value is what tracking would predict. Quantifier’s value compares the quantifier written on the card, and judge’s quantifier groups the consistency judge’s grouping of it. Under all nine fields , alike pairs agree in every field and the other pairs differ in at least one. The rows for runs whose answer stated a score use that score, which leaves the scorer out, and give sign agreement only. Brackets give bootstrap intervals over claims.

			Did not	Below /	Runs scoring 0			$\|$ Score $\|$
Edit and claims	$n$	Shift	increase	above	Ed.	Un.	Diff.	change
Without computational libraries (60 claims)
Quantifier, all claims	56	$-0.62$ [ $-0.91$ , $-0.35$ ]	84%	16 / 0	11	54	0.00 [ $-0.09$ , $+0.10$ ]	$-0.18$ [ $-0.35$ , $-0.03$ ]
materials science	28	$-0.44$ [ $-0.77$ , $-0.13$ ]	82%	8 / 0	9	54	$-0.06$ [ $-0.22$ , $+0.11$ ]	$-0.01$ [ $-0.21$ , $+0.19$ ]
biomedical	28	$-0.80$ [ $-1.26$ , $-0.37$ ]	86%	8 / 0	2	0	$+0.07$ [0.00, $+0.18$ ]	$-0.36$ [ $-0.60$ , $-0.14$ ]
no outage-hit runs	54	$-0.61$ [ $-0.91$ , $-0.34$ ]	83%	15 / 0	–	–	–	–

Appendix

Table 9: Score shifts when one card field is edited to make the claim harder to satisfy. The quantifier edit sets the quantifier to universal, and the threshold edit makes the threshold stricter. Each edited card was run once with the card frozen, and its score is compared with the five unedited frozen runs of the same claim in the same setting. Shift is the edited score minus the mean unedited score, averaged over the

n

claims. Did not increase counts claims whose edited score was at or below that mean (as a percentage of claims in the rows without computational libraries). Below and above count claims whose edited score fell below every unedited run or rose above the unedited range. Runs scoring 0 counts the edited runs that abstained (Ed.), out of

n

, and the unedited runs that abstained (Un.), out of five per claim, and Diff. is the paired difference in the share abstaining.

|

Score

|

change is the change in the absolute score, so a negative value means that scores moved toward 0. Rows marked no outage-hit runs leave out the edited runs that lost searches to the search outage (Section 3 ). Rows marked without leave out claims whose edited run in that setting was irregular. With the computational tools, both such rows leave out the Li–AlCu reaction claim, whose quantifier-edited run lost its CALPHAD result and whose threshold-edited run replaced an attempt that had lost one, and the threshold row also leaves out the Al–Mg–Zn

\tau

-phase claim, whose threshold-edited run read a mole fraction as a percentage. With search only, the row leaves out the Fe–Al equiatomic strengthening claim, whose threshold-edited run found only empty searches. Brackets give bootstrap intervals over claims, and dashes mark values not computed.

Figure 6: Change in the feasibility score after editing the card, relative to the mean of five runs with the unedited card, on the 60 claims. Dark bars are claims whose edited score is below all five unedited runs; light bars are all other claims, including the one claim whose threshold-edited run scored above the unedited range.

Claims	$n$	Quantifier edit	$n$	Threshold edit
Without computational libraries (60 claims)
All claims	56	$-0.62$ [ $-0.91$ , $-0.35$ ]	58	$-0.54$ [ $-0.79$ , $-0.32$ ]
Without the claim whose unedited runs all scored $-2$	55	$-0.63$ [ $-0.92$ , $-0.35$ ]	57	$-0.55$ [ $-0.81$ , $-0.33$ ]
Unedited mean above 0	33	$-1.01$ [ $-1.43$ , $-0.60$ ]	34	$-0.99$ [ $-1.32$ , $-0.66$ ]
Unedited mean 0 or below	23	$-0.05$ [ $-0.21$ , $+0.09$ ]	24	$+0.08$ [ $-0.01$ , $+0.20$ ]
Card existential before the edit	30	$-0.65$ [ $-1.01$ , $-0.31$ ]		–

Appendix

Table 10: Edit shifts in subsets of claims. As in Table 9 , a shift is the edited score minus the mean of the claim’s five unedited frozen runs, averaged over the

n

claims, and the rows for all claims repeat that table. The unedited mean is the mean score of those five runs. a By the keyword rules of Appendix E.2 , applied by one coder, which differ between the 60 claims and the 24. b Whether the quantifier-edited run’s first reasoning step contains the universal marker (Appendix E.2 ). c These rows select on what happened after the edit and are descriptive. d Computed from the agent’s own stated scores, on claims where the edited run and at least three unedited runs stated one, a selection that the edit itself may influence. Brackets give bootstrap intervals over claims, and dashes mark splits that concern only the other edit.

	Without computational libraries (60 claims)				With the computational tools (24 claims)				Search only (24 claims)
	Quantifier		Threshold		Quantifier		Threshold		Quantifier		Threshold
Text that mentions the value	Ed.	Un.	Ed.	Un.	Ed.	Un.	Ed.	Un.	Ed.	Un.	Ed.	Un.
Runs	56	280	56	280	20	100	24	120	20	100	24	120
First reasoning step	34	1	23	0	15	0	18	1	15	1	18	1
Any reasoning step	43	6	34	5	16	4	22	3	16	2	21	2
Tool inputs a	22	1	20	2	9	0	22	0	8	0	16	1

Appendix

Table 11: Where the edited value appears, in the edited-card runs (Ed.) and in the five unedited frozen runs of the same claims (Un.). Each cell counts runs whose text mentions the value, and the first row gives the number of runs. The agent’s own outputs come first and the scorer’s rationale last. A quantifier edit is matched on universal outside its CHGNet sense (“universal potential”). On the 60 claims, a threshold edit is matched on word trigrams of the edited threshold that carry a number absent from the claim and the unedited card, so those threshold counts cover only the edits with such a trigram. On the 24 claims, a threshold edit is matched on a pattern for the values it introduced, written per claim, including phrases that also occur without the edit (Appendix E.2 ). a Questions passed to the computational tools, and search queries together with the agent-written context passed to the search agent; in runs with search only or without computational libraries, only the search queries with that context. b Among runs that asked a calculation question. c The agent’s own final answer, not the composed assessment. d The scorer is given the card, so its mentions show only that the edit reached the score, not that the agent used it. Dashes mark values not computed.

Contrast	Same calculation	What differed	Scores	Gold
Al–Zn single-phase (the opening claim)
Agent vs expert, every condition with the tools	A single solid phase only above about 600 K in 15 of 15 runs, as in the expert’s calculation, and the full phase sequence in 11 of 15	Every card-run card and the frozen card were universal, requiring one solid phase at every temperature in the claimed range, while the expert labeled the claim 0 because the temperature range is not specified	$-1$ in 15 of 15 runs	$0$
Li–AlCu reaction
Between conditions	The same three-phase field, $\eta$ -AlCu + $\zeta$ + T1, at the onset of lithiation, in six card and no-card runs (all scoring $-1$ ) and in 5 of 5 frozen runs	Whether a two-phase reaction means two coexisting phases (card and no-card runs) or coexistence with a Li-containing phase rather than a single-phase solid solution (frozen card)	Card $+2$ , $-1$ , $+1$ , $-1$ , $-1$ No card $-1$ , $-1$ , $-1$ , $+1$ , $-1$ Frozen $+2$ , $+2$ , $+2$ , $+2$ , $+2$	$+2$
Al–Mg–Zn $\tau$ -phase
Within the card condition	The same $\tau$ -phase fractions in 5 of 5 card runs, 0.12 at 2 at.% Zn and 0.22 at 3 at.% Zn with $x(\mathrm{Mg})=0.345$ , at 298–300 K	Not established for the runs that scored $+1$ and $+2$ . All five cards were existential with the same threshold, and the first run’s report interpreted the claim as typical although its card was existential	Card $-1$ , $+1$ , $+1$ , $+2$ , $+2$ Frozen $+2$ , $+2$ , $+2$ , $+2$ , $+2$	$+2$

Appendix

Table 12: Runs with the computational tools that obtained the same calculation but reached different scores, on five of the 24 claims. The cases illustrate how verdicts can diverge and do not estimate how often this happens. Contrast says whether the runs compared share a condition or one frozen card, come from different conditions, or are compared with the expert. The playbooks ask the coding agent for a score of its own, and some tool answers returned to the agent carry one (Appendix B.2 , Table 17 ). Scores are listed in run order, and gold is the expert label. The cases were coded by a single coder and checked by a second reader.

	Materials	Biomed.
Fields judged faithful	64%	49%
Fields judged reasonable default	33%	46%
Fields judged unsupported	1%	4%
Fields judged contradicting	2%	0%
Cards with no flagged field	25 / 30	25 / 30
Mark agrees with judge	77%	59%

Appendix

Table 13: Judge audit of the first-run cards of the 60 claims (30 per domain). † With the expert rationale, on cards where the judge returned a ruling; it returned none for the other 19 materials-science cards.

Per run	Card	No card	Card $-$ no card	Rel.
Without computational libraries (60 claims)
Input tokens	40.4K	35.6K	$+4.8$ K [ $+3.9$ K, $+5.6$ K]	$+13$ %
Output tokens	11.1K	10.3K	$+0.7$ K [ $+0.5$ K, $+1.0$ K]	$+7$ %
Cost (US$)	$0.27	$0.24	$+0.023$ [ $+0.018$ , $+0.028$ ]	$+9$ %
Prompt tokens served from the cache	65%	65%	–	–
Wall-clock time (min)	3.8	3.5	$+16$ s [ $+9$ s, $+24$ s]	$+8$ %

Appendix

Table 14: What a card adds to a run. Rows give per-run means for card and no-card runs. Tokens and cost count only GPT-5.4, which runs the agent, the report step and the scorer. Cost is at list price and does not apply the discount for cached prompt tokens, which were most of the prompt tokens. The difference is paired over claims, with bootstrap intervals in brackets, and Rel. is the relative difference in means. Searches are reported by domain because the tool-use policies allow different numbers of searches in each domain (Appendix B ). The 24 claims are all in materials science. Dashes mark values not computed.

	Sign accuracy	Abstention	SD	Sign disagreement	Runs calling a tool
Without computational libraries (60 claims)
Card	0.68	0.14	0.32	16/60	–
No card	0.71	0.10	0.29	15/60	–
Frozen	0.62	0.18	0.28	15/60	–
Card $-$ no card	$-0.03$ [ $-0.07$ , $+0.01$ ]	$+0.04$ [0.00, $+0.08$ ]	$+0.03$ [ $-0.05$ , $+0.11$ ]	$+0.02$ [ $-0.10$ , $+0.13$ ]
Frozen $-$ card	$-0.06$ [ $-0.13$ , 0.00]	$+0.04$ [ $-0.01$ , $+0.10$ ]	$-0.04$ [ $-0.16$ , $+0.07$ ]	$-0.02$ [ $-0.15$ , $+0.12$ ]

Appendix

Table 15: Outcomes by condition, with paired differences. Sign accuracy is the share of runs whose score has the sign of the gold label, with 0 counted as its own sign, and abstention is the share of runs scoring 0. Always predicting a positive score would give a sign accuracy of 0.58 over all 60 claims, 0.50 on their materials-science claims and 0.67 on their biomedical claims, and 0.58 on the 24 claims. SD is the standard deviation of a claim’s five scores, averaged over claims, and sign disagreement counts claims whose five runs do not all share a sign. Runs calling a tool counts runs with at least one call to the CALPHAD or CHGNet tool, and Table 17 gives the paired difference for the runs with the computational tools. Only materials-science claims could call these tools, and in the runs on the 60 claims their libraries were not installed, so 89 of the 90 calls returned no calculation. For the 60 claims, the frozen card is the card of the first card run, and for the 24 it is a card from a separate preliminary run. Difference rows are paired over claims, with bootstrap intervals in brackets; for sign disagreement they give the difference in the share of claims. Tool calls are counted for the materials-science claims only, so the rows for all 60 claims show dashes.

		Materials		Biomed.
LLM backbone	Card	Sign	Abst.	Sign	Abst.
GPT-6 Sol	yes	0.20	0.70	0.70	0.30
	no	0.20	0.70	0.80	0.20
GPT-5.4	yes	0.40	0.20	1.00	0.00
	no	0.30	0.40	1.00	0.00
Gemini 3.1 Pro	yes	0.60	0.20	0.90	0.00

Appendix

Table 16: Sign accuracy and abstention rate on a 20-claim pilot (10 per dataset, one run each) for three LLM backbones, with and without a card.

	Card	No card	Frozen	Card $-$ no card
Runs calling a computational tool	64/120	72/120	64/120	$-0.07$ [ $-0.17$ , $+0.02$ ]
Calls, CALPHAD + CHGNet a	66 (56 + 10)	72 (54 + 18)	64 (49 + 15)	$-0.05$ [ $-0.16$ , $+0.05$ ]
Calls returning a calculation b	64/66	66/72	63/64	–
Runs receiving a calculation	62/120	66/120	63/120	$-0.03$ [ $-0.14$ , $+0.07$ ]
First reasoning step names CALPHAD	97/120	97/120	105/120	–
First reasoning step names CHGNet	41/120	42/120	52/120	–

Appendix

Table 17: How the agent used the computational tools on the 24 claims. The upper part compares the card, no-card and frozen-card conditions, and its last column gives card minus no card, paired over claims (per-claim shares or per-run means), with bootstrap intervals in brackets. An empty search is a search call that returned no results. The lower part compares each edited-card run with the five unedited frozen runs of the same claim, and its difference is in the share of runs calling a tool. a The difference is in calls per run. b A call returns a calculation if the agent receives the result, in full or in part, of a calculation that the coding agent ran (Appendix B ). c A tool answer carries its own score when it states one on the agent’s scale. The playbooks ask the coding agent for one (Appendix B.2 ). d Among runs that asked a calculation question (Table 11 ). Dashes mark values not computed.

	With the tools			Search only
Claims	Gold	Abst.	Wrong	Gold	Abst.	Wrong
All claims	82	14	24	68	28	24
Tool-calling	43	3	14	37	10	13
Other claims	39	11	10	31	18	11
SD	0.31			0.31
Sign disagr.	6/24			8/24

Appendix

Table 18: Outcomes of frozen-card runs with the computational tools and with search only, on all 24 claims (120 runs per setting) and on two post hoc groups of 12 claims. The tool-calling claims are those on which at least three of the five frozen runs with the tools called a computational tool, and the other claims are the rest. Gold counts runs whose score has the sign of the expert label, Abst. runs scoring 0 on claims whose label is not 0, and Wrong the remaining runs. SD and sign disagreement ( Sign disagr. ) are over all claims and defined as in Table 15 .

Figure 7: Outcomes of the frozen-card runs with the computational tools and with search only, on all 24 claims, on the 12 tool-calling claims and on the other 12. Each bar splits the runs into those with the gold sign, those scoring 0 on a claim whose gold label is not 0, and those with the wrong sign, which include nonzero scores on the Al–Zn single-phase claim, the only claim labeled 0. The numbers in the bars are run counts, out of 120 per bar for all claims and 60 for each group of 12. A claim is tool-calling if at least three of its five frozen runs with the tools called a computational tool.

Tools $-$ search only	Claims	Difference
Sign accuracy
All claims	24	$+0.12$ [ $+0.03$ , $+0.23$ ]
without the rerun claims a	22	$+0.06$ [0.00, $+0.15$ ]
without nine claims b	15	$+0.08$ [ $-0.01$ , $+0.19$ ]
without two claims c	22	$+0.06$ [0.00, $+0.14$ ]
Tool-calling claims	12	$+0.10$ [ $-0.05$ , $+0.28$ ]

Appendix

Table 19: Paired differences between the frozen-card runs with the computational tools and those with search only, with bootstrap intervals over claims. Sign accuracy, abstention and wrong signs are the shares of runs counted in Table 18 , and the edit shifts are those of Table 9 . a Without the claims on which a run with the tools was rerun. b Without the nine claims whose gold entries have documented problems (Section 7 ). c Without the Li–AlCu reaction and Al–Mg–Zn

\tau

-phase claims. d On the 15 claims with at least one frozen run with the tools that made no computational call, those runs against the search-only runs. e Search only minus tools. The rows marked without one claim leave out the Li–AlCu reaction claim for the quantifier edit and the Fe–Al equiatomic strengthening claim for the threshold edit.

LLM Agent Verification

LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.

See all 147 papers

May 26, 2026

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation

Misalignment between claims and their cited evidence is a common failure mode in reports generated by large language models, limiting their reliability in scientific and other high-stakes settings. We present DeepSciVerify, a two-stage pipeline for scientific claim-citation verification that combines abstract-level reasoning with selective escalation to passage-level evidence. The system first verifies claims using the abstract and defers uncertain cases, retrieving and analyzing full-text passages only when necessary. This design leverages complementary behaviors across LLMs, as some models are more conservative while others are more decisive under uncertainty. On the SCitance benchmark, DeepSciVerify achieves 86.7 Micro-F1, outperforming strong abstract-only baselines by +4.5 points while resolving 67% of instances without full-text retrieval. These results suggest that selective evidence escalation improves both accuracy and efficiency in claim-citation verification.

Shaghayegh Sadeghi, Khashayar Khajavi, Rise Adhikari +1

FirstPrinciples · School of Computing Science, Simon Fraser University

Selective Prediction Citation Verification Evidence Retrieval Scientific Claim Verification

Ask PDF

Oct 21, 2026

Real PMIDs, Wrong Papers: Harness Design for Biomedical Claim-Verification Agents

Literature-searching language agents are becoming a standard component of scientific assistants for the life sciences, yet there is little controlled evidence on which parts of their harness matter. We run a controlled ablation on biomedical claim verification (SciFact) with the live PubMed API as the only tool. Holding the backbone and output format fixed, we compare four harnesses: closed-book answering, single-shot retrieval using the claim as the query, a ReAct-style agent that writes its own PubMed queries, and an oracle given the gold abstracts, each with two efficient frontier backbones. Naive single-shot retrieval finds the gold paper for only 6% of claims and turns overconfident errors into abstentions without improving accuracy. Letting the model formulate its own queries triples gold recall and yields the best verdicts (62% accuracy vs. 49-59% for the other realistic harnesses), although these differences are not significant at our sample size and agents use fewer than half of their search budget. With gold abstracts, the same models reach 79-86% accuracy, a significant 17-24 point gap showing that retrieval, not reasoning, is the main bottleneck. Finally, a citation audit shows that closed-book models cite PubMed IDs that almost always exist (327/335) but are almost never relevant: a cross-family LLM judge rates only 3 of 327 existing citations as relevant to the claim, versus 67% for citations produced by the search agent. Existence checks are therefore insufficient safeguards for scientific agents. We release code, prompts, and all cached tool calls at https://github.com/saqlain2204/real-pmids-wrong-papers.

Mohammed Saqlain

Independent Researcher, Bangalore, India

LLM Agent Evaluation Scientific Claim Verification Biomedical IR RAG Citation Faithfulness

Ask PDF

Aug 24, 2026

Beyond Verdicts: A Graph-Based Analysis of Human and LLM Reasoning in Scientific Fact-Checking

Misinformation that cites legitimate papers can be especially harmful when it distorts what those studies actually report. While existing automatic fact-checking systems based on large language models (LLMs) can assess whether a model assigns an Incorrect verdict and can gen- erate explanations for that decision, they typi- cally do not indicate whether the model follows the same reasoning path as human experts or arrives at the verdict through a different but still valid path. In this work, we introduce a graph- based framework (typed reasoning graph) for comparing human and LLM reasoning paths in scientific fact-checking. Building on prior work on fallacious reasoning in biomedical misinformation, MISSCIPLUS (Glockner et al., 2025), we model each explanation as a rea- soning graph that links the false claim to the relevant study context, study findings, fallacy- supporting premises, and fallacy labels. This representation enables one-to-one alignment of human and LLM reasoning at the level of fallacy-specific sub-graphs. For non-human- aligned LLM paths, we validate grounding in the cited study, relevance to the claim, and suf- ficiency for the verdict. Using 84 false claims from MISSCIPLUS, we evaluate GPT-5, Claude Opus 4.7, and Qwen3-32B across prompt and evidence settings. Results show distinct perfor- mance dimensions: Qwen3-32B has the lowest verdict failure rate, GPT-5 the highest human alignment, and Claude Opus 4.7 weak verdict prediction but often valid reasoning in success- ful cases

Abdul Ghafoor, Muhammad Arslan Manzoor, Yufang Hou

Interdisciplinary Transformation University Austria (ITU)

Automated Fact-Checking Scientific Reasoning in Language Models LLM Reasoning with Graphs

Ask PDF

Abstract

LLM Agent Verification

Explore similar work

DeepSciVerify: Verifying Scientific Claim--Citation Alignment via LLM-Driven Evidence Escalation

Real PMIDs, Wrong Papers: Harness Design for Biomedical Claim-Verification Agents

Beyond Verdicts: A Graph-Based Analysis of Human and LLM Reasoning in Scientific Fact-Checking