cs.CLOct 14, 2026
SaveSame Claim, Different Interpretations: How LLM Agents Resolve Ambiguity in Scientific Claims
Organizations: University of Pennsylvania
Abstract
Co-scientists and agentic scientific claim verifiers are increasingly used to assess claims across the sciences. Natural-language scientific claims, written for human experts, are often underspecified and so admit several defensible interpretations. Such verifiers usually pick one, often tacitly, and benchmarks score their verdicts against expert labels without recording the expert's interpretation or the verifier's in a comparable form. This leads to disagreements with expert labels that cannot be traced to the evidence or the interpretation, and to compute spent on unchecked interpretations. We introduce Claim Cards to make such interpretations explicit in the reasoning itself. A Claim Card records a verifier's interpretation as nine fields before evidence gathering, each marked stated, inferred, or default. To test whether an agent's interpretation can change its verdict, we ran a GPT-5.4 agent five times on each of 60 expert-labeled materials-science and biomedical claims with a card, without one, and with one card frozen across runs (900 runs). Over 40% of card fields were not marked stated, and a card added about a tenth to a run's GPT-5.4 tokens and time without measurably changing agreement with expert labels. The agent's own interpretations varied between runs, although within a claim this variation did not measurably track the verdict. A deliberate change to one field of the frozen card, however, making the quantifier universal or the threshold for support stricter, changed the verdict on about a third of the claims, against about one in seven expected by chance (p < 0.001 for each edit). On 24 materials-science claims run with computational tools, the stricter threshold also changed what the agent asked those tools to compute. We therefore argue that verdicts should be reported with their interpretations and that benchmarks could publish the interpretation behind each label.
Figures & tables
Figure 1: (a) The opening claim, assessed by its labeling expert and by the agent, whose cards required a single solid phase at every temperature below 700 K. The expert’s boundaries are rounded (Appendix F.1 ). In this card, from a run with the computational tools, the parser marked the quantifier and threshold stated , though the expert judged the temperature range unspecified. (b) A run in card mode and a re-run with an edited, frozen card.
| Field | What it records | Value on this card | Prov. |
| System specification | |||
| SystemEntity | The object of study | Organisms undergoing aging | stated |
| StatePhase | Its phase or state before any intervention | Aged versus young/adult | stated |
| Conditions | The conditions in which the claim is purported to hold | Matched measurement conditions; taxa and tissues must be identified when evaluating generality | stated |
| Intervention | |||
| Manipulation | How the system is modified, processed, or intervened upon | Aging; no experimental intervention specified | inferred |
Table 1: The nine fields of an interpretation, and the Claim Card that ClaimSpy produced for “Autophagy declines in aged organisms” in the first card run, with the provenance mark of each value. It was also the claim’s frozen card.
| Did not | Below / | |||
| Edit | Shift | increase | above | |
| Without computational libraries (60 claims) | ||||
| Quantifier | 56 | [ , ] | 47 | 16 / 0 |
| Threshold | 58 | [ , ] | 50 | 14 / 1 |
| With the computational tools (24 claims) | ||||
| Quantifier | 20 | [ , ] | 16 | 4 / 1 |
Table 2: Score shifts after one card field is edited to make the claim harder to satisfy (Section 3 ), with bootstrap intervals. The last two columns count claims whose edited score did not exceed the unedited mean and claims whose edited score fell below every unedited run or rose above the unedited range. The two identical shifts on the 24 claims are separate estimates.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: The Claim Card that ClaimSpy produced for the GaN claim, with field values and provenance marks as output by the parser. The margin notes are ours; gap numbers refer to the list in Appendix A.1 .
| Conditions | Claims | Setting | Runs | Frozen or edited card |
|---|---|---|---|---|
| Card, no card, frozen | 60 | Without computational libraries a | 900 | Frozen: the card of each claim’s first card run † |
| Edited cards | 56 + 58 | Without computational libraries a | 114 | The first card run’s card with one field edited, the quantifier (56 claims) or the threshold (58 claims) † |
| Card, no card, frozen | 24 | With the computational tools | 360 | Frozen: a card from a separate preliminary card run, not one of the reported runs |
| Frozen | 24 | Search only | 120 | The same frozen card |
| Edited cards | 20 + 24 | With the computational tools and with search only | 88 | The frozen card with one field edited, the quantifier or the threshold, each run once in each setting |
| Pilot (Appendix I ) | 20 | Three LLM backbones, card and no card, one run each | 120 | – |
Table 3: The runs in the study, by condition and setting. We ran 60 claims without computational libraries (30 materials science, 30 biomedical) and 24 materials-science claims with the computational tools and with search only, six of them among the 60. A frozen run injects a stored card into the agent’s prompt. An edited card is a stored card with one field changed, run once in the same way. a The CALPHAD and CHGNet tools could be called, but their libraries were not installed. Biomedical claims had a general coding agent instead (Appendix B.2 ). † A search outage affected some of these runs (Section 3 ).
| Contrast | Bootstrap over groups | Sign-flip |
| Quantifier edit, shift | ||
| with the tools | [ , ] | 0.039 |
| search only | [ , ] | 0.004 |
| Threshold edit, shift | ||
| with the tools | [ , ] | 0.016 |
| search only | [ , ] | 0.26 |
Table 4: Contrasts on the 24 materials-science claims, with the claims grouped by chemical system. The interval is a percentile interval from a bootstrap that resamples groups of claims, and is from an exact sign-flip test over the per-group mean differences. The edit shifts are defined as in Table 9 , and the other rows are paired differences as in Tables 15 , 17 and 19 .
| Problem id | Label |
|---|---|
| matsci_db_002 | |
| matsci_db_003 | |
| matsci_db_006 | |
| mof_alloys_0007 | |
| mof_batteries_0010 | |
| vis_reas_alloys_0002 |
Table 5: The 30 materials-science claims run without computational libraries, from SFBench ( Costello et al., 2026 ) . Each is listed by its SFBench problem id, with its gold feasibility label ( to ). Table 7 names the six computational_tools claims. Of the others, the text refers to matsci_db_003 as the MgO surface-energy claim, str_dft_08 as the rock-salt cathode capacity claim and str_speculative_11 as the toluene polysulfide claim.
| Claim id | Label | Claim id | Label | Claim id | Label |
|---|---|---|---|---|---|
| 56 | 532 | 1021 | |||
| 118 | 539 | 1062 | |||
| 148 | 540 | 1140 | |||
| 239 | 589 | 1144 | |||
| 248 | 636 | 1221 | |||
| 249 | 770 | 1241 |
Table 6: The 30 SciFact claims ( Wadden et al., 2020 ) , by claim id in the SciFact development set, with the gold label mapped to our scale ( support , contradict ). Claim 148 is the autophagy claim of Section 1 .
| Id | Short name | Label | Library | Tool- calling |
|---|---|---|---|---|
| 0002 | Al–Zn eutectic | pycalphad | ||
| 0003 † | Al–Zn single-phase (the opening claim) | pycalphad | ||
| 0006 | Al–Mg anode potential | pycalphad | ||
| 0009 | Li–CuAl capacity | pycalphad | ||
| 0010 | Li–AlCu reaction | pycalphad | ||
| 0011 | silver bulk-modulus | CHGNet |
Table 7: The 24 materials-science claims run with the computational tools, by problem id ( computational_tools_ id ), with the short name by which the text refers to the claim, the expert label and the library the claim is tagged for. A bullet marks the 12 tool-calling claims, the post hoc group of claims on which at least three of the five frozen runs with the computational tools called a tool (Appendix J.1 ). † Also among the 60 claims run without computational libraries (Table 5 ).
Figure 3: Share of cards that do not mark a field stated, that is, mark it inferred or default, in the 300 cards of the card condition without computational libraries (150 per domain) and the 118 cards parsed in the card runs with the computational tools. Figure 4 splits each bar into inferred and default. Fields are labeled with short forms of the names in Table 1 .
Figure 4: Where each card field’s value comes from. The left and middle panels show the 300 cards from the card runs on the 60 claims (150 materials-science, 150 biomedical), and the right panel the 118 cards parsed in the card runs on the 24 claims. The dark segment of each bar is the share of cards that mark the field stated, and the segments above it the shares that mark it inferred or default. The parser assigned the marks (Section 2.2 ). The dotted line is a 50% reference. Fields are labeled as in Figure 3 .
Figure 5: Agreement across five runs, by field, for explicit Claim Cards and for implicit interpretations recovered from runs without a card, in the setting without computational libraries. A judge model grouped the explicit interpretations of 51 of the 60 claims and the recovered ones of 56, and the paired comparison uses the 50 claims in both. Fields are labeled with short forms of the names in Table 1 .
| Same sign | Absolute score difference | ||||||
| Field compared | Claims | Alike | Differ | Alike differ | Alike | Differ | Differ alike |
| Without computational libraries (60 claims) | |||||||
| Threshold | 31 | 85% | 83% | [ , ] | 0.40 | 0.43 | [ , ] |
| Quantifier’s value | 20 | 83% | 84% | [ , ] | 0.48 | 0.52 | [ , ] |
| Judge’s quantifier groups | 17 | 82% | 87% | [ , ] | 0.42 | 0.50 | [ , ] |
| All nine fields | 30 | 88% | 82% | [ , ] | 0.27 | 0.34 | [ , ] |
Table 8: Whether a claim’s card runs that recorded the same interpretation also reached the same verdict. For each field, pairs of card runs whose values fell in the same consistency group ( alike ) are compared with pairs whose values fell in different groups ( differ ), on the claims that have both kinds of pair. Same sign is the share of pairs whose two scores have the same sign, and absolute score difference is the mean absolute difference between the two scores. Shares and means are averaged within each claim and then over claims, and both differences are paired by claim and oriented so that a positive value is what tracking would predict. Quantifier’s value compares the quantifier written on the card, and judge’s quantifier groups the consistency judge’s grouping of it. Under all nine fields , alike pairs agree in every field and the other pairs differ in at least one. The rows for runs whose answer stated a score use that score, which leaves the scorer out, and give sign agreement only. Brackets give bootstrap intervals over claims.
| Did not | Below / | Runs scoring 0 | Score | |||||
| Edit and claims | Shift | increase | above | Ed. | Un. | Diff. | change | |
| Without computational libraries (60 claims) | ||||||||
| Quantifier, all claims | 56 | [ , ] | 84% | 16 / 0 | 11 | 54 | 0.00 [ , ] | [ , ] |
| materials science | 28 | [ , ] | 82% | 8 / 0 | 9 | 54 | [ , ] | [ , ] |
| biomedical | 28 | [ , ] | 86% | 8 / 0 | 2 | 0 | [0.00, ] | [ , ] |
| no outage-hit runs | 54 | [ , ] | 83% | 15 / 0 | – | – | – | – |
Table 9: Score shifts when one card field is edited to make the claim harder to satisfy. The quantifier edit sets the quantifier to universal, and the threshold edit makes the threshold stricter. Each edited card was run once with the card frozen, and its score is compared with the five unedited frozen runs of the same claim in the same setting. Shift is the edited score minus the mean unedited score, averaged over the claims. Did not increase counts claims whose edited score was at or below that mean (as a percentage of claims in the rows without computational libraries). Below and above count claims whose edited score fell below every unedited run or rose above the unedited range. Runs scoring 0 counts the edited runs that abstained (Ed.), out of , and the unedited runs that abstained (Un.), out of five per claim, and Diff. is the paired difference in the share abstaining. Score change is the change in the absolute score, so a negative value means that scores moved toward 0. Rows marked no outage-hit runs leave out the edited runs that lost searches to the search outage (Section 3 ). Rows marked without leave out claims whose edited run in that setting was irregular. With the computational tools, both such rows leave out the Li–AlCu reaction claim, whose quantifier-edited run lost its CALPHAD result and whose threshold-edited run replaced an attempt that had lost one, and the threshold row also leaves out the Al–Mg–Zn -phase claim, whose threshold-edited run read a mole fraction as a percentage. With search only, the row leaves out the Fe–Al equiatomic strengthening claim, whose threshold-edited run found only empty searches. Brackets give bootstrap intervals over claims, and dashes mark values not computed.
Figure 6: Change in the feasibility score after editing the card, relative to the mean of five runs with the unedited card, on the 60 claims. Dark bars are claims whose edited score is below all five unedited runs; light bars are all other claims, including the one claim whose threshold-edited run scored above the unedited range.
| Claims | Quantifier edit | Threshold edit | ||
| Without computational libraries (60 claims) | ||||
| All claims | 56 | [ , ] | 58 | [ , ] |
| Without the claim whose unedited runs all scored | 55 | [ , ] | 57 | [ , ] |
| Unedited mean above 0 | 33 | [ , ] | 34 | [ , ] |
| Unedited mean 0 or below | 23 | [ , ] | 24 | [ , ] |
| Card existential before the edit | 30 | [ , ] | – | |
Table 10: Edit shifts in subsets of claims. As in Table 9 , a shift is the edited score minus the mean of the claim’s five unedited frozen runs, averaged over the claims, and the rows for all claims repeat that table. The unedited mean is the mean score of those five runs. a By the keyword rules of Appendix E.2 , applied by one coder, which differ between the 60 claims and the 24. b Whether the quantifier-edited run’s first reasoning step contains the universal marker (Appendix E.2 ). c These rows select on what happened after the edit and are descriptive. d Computed from the agent’s own stated scores, on claims where the edited run and at least three unedited runs stated one, a selection that the edit itself may influence. Brackets give bootstrap intervals over claims, and dashes mark splits that concern only the other edit.
| Without computational libraries (60 claims) | With the computational tools (24 claims) | Search only (24 claims) | ||||||||||
| Quantifier | Threshold | Quantifier | Threshold | Quantifier | Threshold | |||||||
| Text that mentions the value | Ed. | Un. | Ed. | Un. | Ed. | Un. | Ed. | Un. | Ed. | Un. | Ed. | Un. |
| Runs | 56 | 280 | 56 | 280 | 20 | 100 | 24 | 120 | 20 | 100 | 24 | 120 |
| First reasoning step | 34 | 1 | 23 | 0 | 15 | 0 | 18 | 1 | 15 | 1 | 18 | 1 |
| Any reasoning step | 43 | 6 | 34 | 5 | 16 | 4 | 22 | 3 | 16 | 2 | 21 | 2 |
| Tool inputs a | 22 | 1 | 20 | 2 | 9 | 0 | 22 | 0 | 8 | 0 | 16 | 1 |
Table 11: Where the edited value appears, in the edited-card runs (Ed.) and in the five unedited frozen runs of the same claims (Un.). Each cell counts runs whose text mentions the value, and the first row gives the number of runs. The agent’s own outputs come first and the scorer’s rationale last. A quantifier edit is matched on universal outside its CHGNet sense (“universal potential”). On the 60 claims, a threshold edit is matched on word trigrams of the edited threshold that carry a number absent from the claim and the unedited card, so those threshold counts cover only the edits with such a trigram. On the 24 claims, a threshold edit is matched on a pattern for the values it introduced, written per claim, including phrases that also occur without the edit (Appendix E.2 ). a Questions passed to the computational tools, and search queries together with the agent-written context passed to the search agent; in runs with search only or without computational libraries, only the search queries with that context. b Among runs that asked a calculation question. c The agent’s own final answer, not the composed assessment. d The scorer is given the card, so its mentions show only that the edit reached the score, not that the agent used it. Dashes mark values not computed.
| Contrast | Same calculation | What differed | Scores | Gold |
|---|---|---|---|---|
| Al–Zn single-phase (the opening claim) | ||||
| Agent vs expert, every condition with the tools | A single solid phase only above about 600 K in 15 of 15 runs, as in the expert’s calculation, and the full phase sequence in 11 of 15 | Every card-run card and the frozen card were universal, requiring one solid phase at every temperature in the claimed range, while the expert labeled the claim 0 because the temperature range is not specified | in 15 of 15 runs | |
| Li–AlCu reaction | ||||
| Between conditions | The same three-phase field, -AlCu + + T1, at the onset of lithiation, in six card and no-card runs (all scoring ) and in 5 of 5 frozen runs | Whether a two-phase reaction means two coexisting phases (card and no-card runs) or coexistence with a Li-containing phase rather than a single-phase solid solution (frozen card) | Card , , , , No card , , , , Frozen , , , , | |
| Al–Mg–Zn -phase | ||||
| Within the card condition | The same -phase fractions in 5 of 5 card runs, 0.12 at 2 at.% Zn and 0.22 at 3 at.% Zn with , at 298–300 K | Not established for the runs that scored and . All five cards were existential with the same threshold, and the first run’s report interpreted the claim as typical although its card was existential | Card , , , , Frozen , , , , | |
Table 12: Runs with the computational tools that obtained the same calculation but reached different scores, on five of the 24 claims. The cases illustrate how verdicts can diverge and do not estimate how often this happens. Contrast says whether the runs compared share a condition or one frozen card, come from different conditions, or are compared with the expert. The playbooks ask the coding agent for a score of its own, and some tool answers returned to the agent carry one (Appendix B.2 , Table 17 ). Scores are listed in run order, and gold is the expert label. The cases were coded by a single coder and checked by a second reader.
| Materials | Biomed. | |
|---|---|---|
| Fields judged faithful | 64% | 49% |
| Fields judged reasonable default | 33% | 46% |
| Fields judged unsupported | 1% | 4% |
| Fields judged contradicting | 2% | 0% |
| Cards with no flagged field | 25 / 30 | 25 / 30 |
| Mark agrees with judge | 77% | 59% |
Table 13: Judge audit of the first-run cards of the 60 claims (30 per domain). † With the expert rationale, on cards where the judge returned a ruling; it returned none for the other 19 materials-science cards.
| Per run | Card | No card | Card no card | Rel. |
| Without computational libraries (60 claims) | ||||
| Input tokens | 40.4K | 35.6K | K [ K, K] | % |
| Output tokens | 11.1K | 10.3K | K [ K, K] | % |
| Cost (US$) | $0.27 | $0.24 | [ , ] | % |
| Prompt tokens served from the cache | 65% | 65% | – | – |
| Wall-clock time (min) | 3.8 | 3.5 | s [ s, s] | % |
Table 14: What a card adds to a run. Rows give per-run means for card and no-card runs. Tokens and cost count only GPT-5.4, which runs the agent, the report step and the scorer. Cost is at list price and does not apply the discount for cached prompt tokens, which were most of the prompt tokens. The difference is paired over claims, with bootstrap intervals in brackets, and Rel. is the relative difference in means. Searches are reported by domain because the tool-use policies allow different numbers of searches in each domain (Appendix B ). The 24 claims are all in materials science. Dashes mark values not computed.
| Sign accuracy | Abstention | SD | Sign disagreement | Runs calling a tool | |
| Without computational libraries (60 claims) | |||||
| Card | 0.68 | 0.14 | 0.32 | 16/60 | – |
| No card | 0.71 | 0.10 | 0.29 | 15/60 | – |
| Frozen | 0.62 | 0.18 | 0.28 | 15/60 | – |
| Card no card | [ , ] | [0.00, ] | [ , ] | [ , ] | |
| Frozen card | [ , 0.00] | [ , ] | [ , ] | [ , ] | |
Table 15: Outcomes by condition, with paired differences. Sign accuracy is the share of runs whose score has the sign of the gold label, with 0 counted as its own sign, and abstention is the share of runs scoring 0. Always predicting a positive score would give a sign accuracy of 0.58 over all 60 claims, 0.50 on their materials-science claims and 0.67 on their biomedical claims, and 0.58 on the 24 claims. SD is the standard deviation of a claim’s five scores, averaged over claims, and sign disagreement counts claims whose five runs do not all share a sign. Runs calling a tool counts runs with at least one call to the CALPHAD or CHGNet tool, and Table 17 gives the paired difference for the runs with the computational tools. Only materials-science claims could call these tools, and in the runs on the 60 claims their libraries were not installed, so 89 of the 90 calls returned no calculation. For the 60 claims, the frozen card is the card of the first card run, and for the 24 it is a card from a separate preliminary run. Difference rows are paired over claims, with bootstrap intervals in brackets; for sign disagreement they give the difference in the share of claims. Tool calls are counted for the materials-science claims only, so the rows for all 60 claims show dashes.
| Materials | Biomed. | ||||
|---|---|---|---|---|---|
| LLM backbone | Card | Sign | Abst. | Sign | Abst. |
| GPT-6 Sol | yes | 0.20 | 0.70 | 0.70 | 0.30 |
| no | 0.20 | 0.70 | 0.80 | 0.20 | |
| GPT-5.4 | yes | 0.40 | 0.20 | 1.00 | 0.00 |
| no | 0.30 | 0.40 | 1.00 | 0.00 | |
| Gemini 3.1 Pro | yes | 0.60 | 0.20 | 0.90 | 0.00 |
Table 16: Sign accuracy and abstention rate on a 20-claim pilot (10 per dataset, one run each) for three LLM backbones, with and without a card.
| Card | No card | Frozen | Card no card | |
| Runs calling a computational tool | 64/120 | 72/120 | 64/120 | [ , ] |
| Calls, CALPHAD + CHGNet a | 66 (56 + 10) | 72 (54 + 18) | 64 (49 + 15) | [ , ] |
| Calls returning a calculation b | 64/66 | 66/72 | 63/64 | – |
| Runs receiving a calculation | 62/120 | 66/120 | 63/120 | [ , ] |
| First reasoning step names CALPHAD | 97/120 | 97/120 | 105/120 | – |
| First reasoning step names CHGNet | 41/120 | 42/120 | 52/120 | – |
Table 17: How the agent used the computational tools on the 24 claims. The upper part compares the card, no-card and frozen-card conditions, and its last column gives card minus no card, paired over claims (per-claim shares or per-run means), with bootstrap intervals in brackets. An empty search is a search call that returned no results. The lower part compares each edited-card run with the five unedited frozen runs of the same claim, and its difference is in the share of runs calling a tool. a The difference is in calls per run. b A call returns a calculation if the agent receives the result, in full or in part, of a calculation that the coding agent ran (Appendix B ). c A tool answer carries its own score when it states one on the agent’s scale. The playbooks ask the coding agent for one (Appendix B.2 ). d Among runs that asked a calculation question (Table 11 ). Dashes mark values not computed.
| With the tools | Search only | |||||
| Claims | Gold | Abst. | Wrong | Gold | Abst. | Wrong |
| All claims | 82 | 14 | 24 | 68 | 28 | 24 |
| Tool-calling | 43 | 3 | 14 | 37 | 10 | 13 |
| Other claims | 39 | 11 | 10 | 31 | 18 | 11 |
| SD | 0.31 | 0.31 | ||||
| Sign disagr. | 6/24 | 8/24 | ||||
Table 18: Outcomes of frozen-card runs with the computational tools and with search only, on all 24 claims (120 runs per setting) and on two post hoc groups of 12 claims. The tool-calling claims are those on which at least three of the five frozen runs with the tools called a computational tool, and the other claims are the rest. Gold counts runs whose score has the sign of the expert label, Abst. runs scoring 0 on claims whose label is not 0, and Wrong the remaining runs. SD and sign disagreement ( Sign disagr. ) are over all claims and defined as in Table 15 .
Figure 7: Outcomes of the frozen-card runs with the computational tools and with search only, on all 24 claims, on the 12 tool-calling claims and on the other 12. Each bar splits the runs into those with the gold sign, those scoring 0 on a claim whose gold label is not 0, and those with the wrong sign, which include nonzero scores on the Al–Zn single-phase claim, the only claim labeled 0. The numbers in the bars are run counts, out of 120 per bar for all claims and 60 for each group of 12. A claim is tool-calling if at least three of its five frozen runs with the tools called a computational tool.
| Tools search only | Claims | Difference |
| Sign accuracy | ||
| All claims | 24 | [ , ] |
| without the rerun claims a | 22 | [0.00, ] |
| without nine claims b | 15 | [ , ] |
| without two claims c | 22 | [0.00, ] |
| Tool-calling claims | 12 | [ , ] |
Table 19: Paired differences between the frozen-card runs with the computational tools and those with search only, with bootstrap intervals over claims. Sign accuracy, abstention and wrong signs are the shares of runs counted in Table 18 , and the edit shifts are those of Table 9 . a Without the claims on which a run with the tools was rerun. b Without the nine claims whose gold entries have documented problems (Section 7 ). c Without the Li–AlCu reaction and Al–Mg–Zn -phase claims. d On the 15 claims with at least one frozen run with the tools that made no computational call, those runs against the search-only runs. e Search only minus tools. The rows marked without one claim leave out the Li–AlCu reaction claim for the quantifier edit and the Fe–Al equiatomic strengthening claim for the threshold edit.
More in this topic
LLM Agent Verification
LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.
See all 147 papers