cs.AIOct 19, 2026

Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy

Authors: Ke Zhang, Sahchit Chundur, Jerry Peprah Owusu, Mohammad Javad Abdolhosseini Qomi, Maziar Raissi

Organizations: University of California, Riverside · University of California, Irvine

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM agents that compute the right answer can still answer wrongly. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads on demand. We ask where this costs accuracy with a real scientific tool: PHREEQC, a widely used geochemistry simulator from the USGS, whose reports reach 846.8k characters and whose determinism lets every printed value be traced. Agents use it to answer 158 execution-verified multiple-choice items. When agents read the simulator's report through a section index rather than in context, GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost. Re-running all 11,007 logged simulator calls shows that both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from. Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution. With the simulation held fixed, retrieval still costs GPT-5.2 9.4 points. For GPT-5.4, more frequent correct simulations mask an evidence-use loss of about ten points, leaving a net change indistinguishable from zero. Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models; for GPT-5.1 and GPT-5.2, inlining outputs that fit works as well, while asking for correct rounding does not. Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.

Figures & tables

Figure 1: Same simulation, different evidence (item 121, GPT-5.1). The agent’s simulation writes the keyed value to the report, at the options’ precision, and to its own output file, with more digits. Under TOC the agent reads only its own file and truncates 2.75065 to the distractor 2.750 (evidence substitution), even when told how to round; told to answer from the report, it answers correctly. Right: for GPT-5.1 and GPT-5.2, report skipping over all 158 items, and wrong answers once the keyed value is seen (100 minus closure) on the 137 numeric items.
ModelDirectCoTAgent-TOCAgent-Raw100kTOC −- inline, pooled (pp)pHolmp_{\text{Holm}}
Claude Opus 4.643.0%39.9%87.3%88.0%−- 0.6 pp [ −- 3.8, ++ 2.5]1.000
Claude Sonnet 4.638.0%46.2%91.8%91.1%−- 0.5 pp [ −- 3.8, ++ 3.2]1.000
GPT-5.425.9%34.2%75.3%70.9%++ 0.2 pp [ −- 5.5, ++ 5.7]1.000
GPT-5.227.2%32.9%57.0%65.8%−- 7.9 pp [ −- 14.9, −- 0.9]0.128
GPT-5.141.1%36.1%57.0%67.7%−- 11.3 pp [ −- 18.0, −- 4.9]0.005
Gemini 3 Flash Preview41.1%40.5%27.8%50.0%––
Table 1: Accuracy on the 158 verified items. Direct, CoT, Agent-TOC, and Agent-Raw100k are the headline runs; unparseable or missing answers count as incorrect. The last two columns report the primary contrast: per-item accuracy under TOC minus inline, with every run of each condition pooled per item (runs in Section 2.2 ), a 95% item-bootstrap interval, and a Holm-corrected sign-flip permutation pp over the five models.
 Retrieval Inline  Input tokens per item
ModelTOCGrep Raw100k HybridTOCRaw100kHybrid
Opus 4.687.3%– 88.0% ––––
Sonnet 4.689.9%–91.8%89.2% 91.1% –100.1k––
GPT-5.469.0%–76.6%– 70.9%, 76.6% –55.7k64.3k–
GPT-5.255.7%–60.1%57.6% 65.8%, 67.7% 66.5%–69.0%61.8k65.7k59.7k
GPT-5.153.2%–59.5%48.1% 67.7%, 65.8% 60.1%–63.9%52.8k60.9k28.3k †
Table 2: Accuracy over runs on the 158 verified items. TOC ranges cover the headline run and three reruns, which share one run on the 33 long-output items (Section 2.2 ); Raw100k lists the headline and the logged run (GPT-5.1, GPT-5.2, GPT-5.4); Hybrid ranges cover three runs. Tokens are mean input tokens per item from the runs with full logs; full-set logs are unavailable for Opus and for Sonnet inline. † Mostly a change in behavior unrelated to the condition (Appendix B ).
  Stage rates (%) Change vs. inline (pp)
ModelConditionppsscc TotalBuildExposureClosureUnsupp.
Sonnet 4.6TOC86.998.998.0 –––––
GPT-5.4Raw100k73.796.099.0 (reference)
TOC80.091.289.0 −- 4.4++ 4.1−- 2.6−- 7.2++ 1.3
GPT-5.2Raw100k65.096.695.3 (reference)
TOC62.590.782.4 −- 11.4−- 1.5−- 2.5−- 7.7++ 0.3
Table 3: Where accuracy is lost, on numeric items. Left: stage rates ( pp = the agent’s own simulation printed the keyed value; ss = the value reached the agent, given produced; cc = correct, given seen). Right: Shapley attribution of each condition’s accuracy change relative to the same model’s logged inline run, in pp (95% item-bootstrap intervals in Table 9 ); unsupported covers answers given without the keyed value in view. TOC averages the three logged reruns. The intervention conditions are in Table 5 .
ModelCond.SkipOwnDigitsTrapCorr. PCorr. D
Sonnet 4.6TOC10.562.98.51/3599.783.3
GPT-5.4TOC41.861.040.315/3199.475.2
GPT-5.2TOC60.386.767.026/35100.072.5
GPT-5.1TOC53.067.547.619/27100.066.3
GPT-5.4Raw0.013.33.10/098.9100.0
GPT-5.2Raw0.029.70.01/295.3–
Table 4: Which evidence the agents use (%, except Trap counts). Skip : no part of result.out reached the agent. Own : the agent read one of its own selected-output files. Digits : of trajectories that saw the keyed value, those that saw it only with more digits than the report. Trap : of trajectories that saw a value rounding to the key but truncating to a distractor, those that chose the distractor (each trajectory once). Corr. P / Corr. D : accuracy when the keyed value was seen as the report prints it / only with more digits.
   AccuracyEvidence use (%)Stage rates (%)Closure
ModelConditionRuns%Δ\DeltaSkipDef.OwnDigitsppssccterm
GPT-5.2Inline167.7%–0.0100.029.70.065.096.695.3(ref.)
TOC control163.3%–57.098.787.363.567.292.483.5−- 7.4
TOC+Report266.8%++ 3.50.042.11.30.663.195.498.8++ 2.1
TOC+Round259.5%−- 3.855.198.480.759.966.489.081.5−- 8.5
Hybrid367.9%++ 4.65.399.834.88.363.597.394.9−- 0.3
Table 5: Intervention runs on the 158 verified items. Δ\Delta : per-item accuracy difference from the TOC control run (GPT-5.4: from all four TOC runs; pp). Skip, Own, and Digits as in Table 4 ; Def. : the agent defined a selected-output file. Stage rates and the closure term (Shapley, pp relative to the logged inline run) as in Table 3 . The inline rows are the logged inline runs, shown for reference; GPT-5.4’s TOC row averages its three TOC reruns.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

ReasonnnItems
Incorrect key122
Unstable redox-derived value171, 8, 48, 108, 115, 118, 119, 125, 128, 130, 133, 136, 138, 140, 145, 166, 192
Missing question sentence34, 37, 78
Malformed option or species name23, 186
Asks for moles, key is molality354, 64, 66
Wrong process or step in stem356 (precipitated, not dissolved); 5, 6 (key matches step 48)
Appendix
Table 6: Excluded items by reason (dataset item IDs).
 Agent-TOC acc.Agent-Raw100k acc.TOC −- Raw100k
Model200158134200158134200158134
Claude Opus 4.683.5%87.3%86.6%83.0%88.0%88.1%++ 0.5 pp−- 0.6 pp−- 1.5 pp
Claude Sonnet 4.684.0%91.8%91.0%83.5%91.1%90.3%++ 0.5 pp++ 0.6 pp++ 0.7 pp
GPT-5.472.5%75.3%75.4%70.0%70.9%70.9%++ 2.5 pp++ 4.4 pp++ 4.5 pp
GPT-5.254.0%57.0%59.7%61.5%65.8%69.4%−- 7.5 pp−- 8.9 pp−- 9.7 pp
GPT-5.154.0%57.0%54.5%63.5%67.7%68.7%−- 9.5 pp−- 10.8 pp−- 14.2 pp
Appendix
Table 7: Headline-run results on the full pool (200), the verified items (158), and the strict subset (134).
StepAgent call →\rightarrow tool result (abridged)
1write_file(calc.inp) : SOLUTION 1 Pure water / pH 7 charge / REACTION 1 / H2SO4 1 / 0.001 / SELECTED_OUTPUT -file sel.out -pH true -solution true / END
2execute_phreeqc(calc.inp) →\rightarrow ok; output_files: phreeqc.log, result.out (6,957 bytes), sel.out (315 bytes); result_out_toc: 19 entries, among them {line 92: Reaction step 1}, {line 112: Solution composition}, {line 118: Description of solution}; hint: Use read_file with start_line/end_line to read specific sections.
3read_file(sel.out, 1, 200) →\rightarrow sim state soln … step pH pe / 1 i_soln 1 … 6.99738 4 / 1 react 1 … 1 2.75065 14.6924
4write_file(final_answer.txt) : <<< A >>>
–Not read: result.out , line 120 (in “Description of solution” of reaction step 1): pH = 2.751
Appendix
Figure 4: A GPT-5.1 TOC trajectory on item 121 (pH after the first reaction step; options A 2.750, B 2.751, C 2.748, D 2.745; key B), with tool results abridged and “/” marking line breaks. The agent answers A, the truncation of the 2.75065 in its own selected-output file, and never reads the report, which prints the keyed 2.751.
ModelRaw100kTOCGrepHybridTOC+ReportTOC+Round
Opus 4.611––––
Sonnet 4.6141–––
GPT-5.424––2–
GPT-5.224+11322
GPT-5.124+11322
Gemini 3 Flash11––––
Appendix
Table 8: Full-agent runs per model and condition on the 158 items. Each model has one headline run with Raw100k and one with TOC; the other TOC runs are reruns, and on the 33 items whose report exceeds 100k characters the four TOC runs share one run with a fixed index (Section 2.2 ). +1: the TOC control run of the interventions, with the index built for every report size.
ModelConditionTotalBuildExposureClosureUnsupported
GPT-5.2TOC−- 11.4 [ −- 21.2, −- 1.5]−- 1.5 [ −- 5.6, ++ 2.7]−- 2.5 [ −- 5.3, ++ 0.4]−- 7.7 [ −- 12.6, −- 2.9]++ 0.3 [ −- 6.1, ++ 6.1]
 Grep−- 11.7 [ −- 21.2, −- 2.9]++ 1.8 [ −- 2.7, ++ 6.5]−- 4.7 [ −- 8.7, −- 1.1]−- 7.8 [ −- 13.6, −- 2.7]−- 1.0 [ −- 6.3, ++ 4.0]
 TOC (control)−- 5.1 [ −- 16.1, ++ 5.8]++ 1.3 [ −- 3.1, ++ 6.0]−- 1.7 [ −- 4.7, ++ 0.6]−- 7.4 [ −- 12.7, −- 2.2]++ 2.7 [ −- 3.4, ++ 8.7]
 TOC+Report−- 2.9 [ −- 10.6, ++ 5.5]−- 1.4 [ −- 6.2, ++ 4.3]−- 0.6 [ −- 3.3, ++ 1.6]++ 2.1 [ −- 0.3, ++ 5.0]−- 3.1 [ −- 8.0, ++ 1.6]
 Hybrid−- 1.0 [ −- 8.0, ++ 5.8]−- 1.0 [ −- 5.2, ++ 2.9]++ 0.3 [ −- 1.2, ++ 2.0]−- 0.3 [ −- 2.3, ++ 2.0]++ 0.0 [ −- 4.8, ++ 4.4]
GPT-5.4TOC−- 4.4 [ −- 12.4, ++ 3.4]++ 4.1 [ ++ 0.0, ++ 8.1]−- 2.6 [ −- 4.8, −- 0.5]−- 7.2 [ −- 11.1, −- 4.2]++ 1.3 [ −- 3.2, ++ 5.8]
Appendix
Table 9: Shapley attribution of each condition’s accuracy change relative to the same model’s logged inline run (pp, numeric items), with 95% item-bootstrap intervals (1,000 resamples, items resampled jointly for both conditions). TOC: the three reruns; TOC (control), TOC+Report, and Hybrid: the intervention runs.
RepeatInlineTOCΔ\Delta
192/13181/131−- 8.4
293/13179/131−- 10.7
395/13183/131−- 9.2
All280/393243/393−- 9.4
Mean (%)71.261.8 
Appendix
Table 10: GPT-5.2 replay on 131 frozen checkpoints. Correct answers are shown over continuations per condition; Δ\Delta is TOC minus inline accuracy in pp. The mean uses three repeats of the same items.
Outcome (158 items)TOCRaw100k
Correct4479
Wrong26
Answer only in a chat message3040
Answer file unparseable10
Step budget exhausted, no answer8033
Log incomplete10
Appendix
Table 11: Gemini 3 Flash outcomes on the 158 items in the headline TOC and Raw100k runs. Accuracy when committed : correct among correct and wrong answers.
More in this topic

LLM Tool Use

Large language model tool use studies how models select, call, and coordinate external tools to extend their reasoning and take actions.

See all 289 papers

Explore similar work

CardsList
Jul 1, 2026

PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents

Large language model agents are increasingly connected to scientific software, yet it remains unclear when tool access makes scientific computation more reliable rather than merely more complex. We introduce PHREEQC-MCQ-200, a benchmark for evaluating tool-augmented agents on deterministic aqueous-geochemistry simulations. The benchmark contains 200 multiple-choice questions derived from 21 validated PHREEQC scenarios, requiring agents to construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers. Across multiple frontier and mid-tier model families, simulator access substantially improves aggregate accuracy, confirming that grounded execution is necessary for many scientific-computation tasks. However, the gains are not monotonic: tool-augmented agents also lose items they answered correctly without tools, revealing regressions that average accuracy alone hides. We further show that output-access protocol matters. A table-of-contents interface can reduce token cost while preserving or improving accuracy for stronger models, but it degrades performance for mid-tier models that cannot reliably navigate structured simulator outputs. PHREEQC-MCQ-200 therefore frames scientific tool use as an end-to-end diagnostic problem rather than a simple tool-calling capability. We argue that evaluations of scientific agents should report not only accuracy, but also item-level retention, output-access sensitivity, trajectory failures, and where the computation chain breaks.

Ke Zhang, Sahchit Chundur, Mohammad Javad Qomi +1

University of California, Riverside · University of California, Irvine

Computer-Use Agent BenchmarksAI for ScienceTool-Use EvaluationAI Agent BenchmarksLLM Agent Reliability
AskPDF
Sep 1, 2026

Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents

Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.

Seonghyeon Cho, Chanjun Park

Soongsil University

Tool-Use EvaluationTool-Augmented Language Model AgentsLLM Agent EvaluationLLM Agent Skill RetrievalLLM AgentsLLM Agent Skill Evaluation
AskPDF
May 8, 2026

Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning

Large Language Models (LLMs) have become increasingly capable as tool-using agents, with benchmarks spanning diverse general agentic tasks. Yet rigorous evaluation of scientific tool use remains limited. In chemistry, recent agents can plan syntheses and invoke domain-specific tools, but evaluations often rely on curated demonstrations, expert assessment, or LLM-as-judge scoring rather than exact, judge-free ground truth. We address this gap with chemical procurement cost estimation, a practical task in which an agent must ground chemical identities, retrieve supplier quotes, select valid purchasable packs, normalize quantities, and compute cost from a reaction description. We introduce ChemCost, a benchmark of 1,427 evaluable reactions grounded to a frozen pricing snapshot covering 2,261 chemicals and 230,775 supplier quotes, supporting scalar scoring and stage-level diagnosis of grounding, retrieval, procurement, and arithmetic failures. To evaluate robustness, we further construct controlled noise-injected views that perturb chemical aliases, quantity expressions, missing fields, and input formatting. Experiments with frontier, open-weight, and chemistry-specialized LLM agents show that tool access is necessary but insufficient for solving the task. The strongest agents reach only 50.6% accuracy within 25% relative error on clean inputs and degrade substantially with realistic noise. Stage-level analysis further shows that failures arise from brittle parsing, ineffective evidence integration, invalid pack selection, and non-convergent tool use.

Yuyang Wu, Yue Huang, Shuaike Shen +8

Carnegie Mellon University · University of Notre Dame · University of North Carolina, Chapel Hill +1

AI Agents for Scientific DiscoveryLLM Agent EvaluationAI Agent BenchmarksLLM Tool UseLLM Agent ReliabilityChemical Reasoning in LLMs
AskPDF