cs.AIOct 19, 2026
SaveSame Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
Organizations: University of California, Riverside · University of California, Irvine
Abstract
LLM agents that compute the right answer can still answer wrongly. Agent harnesses increasingly keep long tool outputs out of the context window, saving them to files that the agent searches or reads on demand. We ask where this costs accuracy with a real scientific tool: PHREEQC, a widely used geochemistry simulator from the USGS, whose reports reach 846.8k characters and whose determinism lets every printed value be traced. Agents use it to answer 158 execution-verified multiple-choice items. When agents read the simulator's report through a section index rather than in context, GPT-5.1 and GPT-5.2 lose 11.3 and 7.9 points of accuracy, while Claude Opus 4.6 and Claude Sonnet 4.6 show no detectable cost. Re-running all 11,007 logged simulator calls shows that both GPT models compute the keyed value about as often as before; what changes is the evidence they answer from. Under retrieval they often skip the report, answer from their own higher-precision output files, and misround to a neighboring option, a failure we call evidence substitution. With the simulation held fixed, retrieval still costs GPT-5.2 9.4 points. For GPT-5.4, more frequent correct simulations mask an evidence-use loss of about ten points, leaving a net change indistinguishable from zero. Naming the authoritative output raises accuracy by 6.3-7.9 points over plain retrieval for the three GPT models; for GPT-5.1 and GPT-5.2, inlining outputs that fit works as well, while asking for correct rounding does not. Context-management choices in agent harnesses should be judged by where accuracy is lost, not only by final accuracy.
Figures & tables
Figure 1: Same simulation, different evidence (item 121, GPT-5.1). The agent’s simulation writes the keyed value to the report, at the options’ precision, and to its own output file, with more digits. Under TOC the agent reads only its own file and truncates 2.75065 to the distractor 2.750 (evidence substitution), even when told how to round; told to answer from the report, it answers correctly. Right: for GPT-5.1 and GPT-5.2, report skipping over all 158 items, and wrong answers once the keyed value is seen (100 minus closure) on the 137 numeric items.
| Model | Direct | CoT | Agent-TOC | Agent-Raw100k | TOC inline, pooled (pp) | |
|---|---|---|---|---|---|---|
| Claude Opus 4.6 | 43.0% | 39.9% | 87.3% | 88.0% | 0.6 pp [ 3.8, 2.5] | 1.000 |
| Claude Sonnet 4.6 | 38.0% | 46.2% | 91.8% | 91.1% | 0.5 pp [ 3.8, 3.2] | 1.000 |
| GPT-5.4 | 25.9% | 34.2% | 75.3% | 70.9% | 0.2 pp [ 5.5, 5.7] | 1.000 |
| GPT-5.2 | 27.2% | 32.9% | 57.0% | 65.8% | 7.9 pp [ 14.9, 0.9] | 0.128 |
| GPT-5.1 | 41.1% | 36.1% | 57.0% | 67.7% | 11.3 pp [ 18.0, 4.9] | 0.005 |
| Gemini 3 Flash Preview | 41.1% | 40.5% | 27.8% | 50.0% | – | – |
Table 1: Accuracy on the 158 verified items. Direct, CoT, Agent-TOC, and Agent-Raw100k are the headline runs; unparseable or missing answers count as incorrect. The last two columns report the primary contrast: per-item accuracy under TOC minus inline, with every run of each condition pooled per item (runs in Section 2.2 ), a 95% item-bootstrap interval, and a Holm-corrected sign-flip permutation over the five models.
| Retrieval | Inline | Input tokens per item | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | TOC | Grep | Raw100k | Hybrid | TOC | Raw100k | Hybrid | ||
| Opus 4.6 | 87.3% | – | 88.0% | – | – | – | – | ||
| Sonnet 4.6 | 89.9%–91.8% | 89.2% | 91.1% | – | 100.1k | – | – | ||
| GPT-5.4 | 69.0%–76.6% | – | 70.9%, 76.6% | – | 55.7k | 64.3k | – | ||
| GPT-5.2 | 55.7%–60.1% | 57.6% | 65.8%, 67.7% | 66.5%–69.0% | 61.8k | 65.7k | 59.7k | ||
| GPT-5.1 | 53.2%–59.5% | 48.1% | 67.7%, 65.8% | 60.1%–63.9% | 52.8k | 60.9k | 28.3k † | ||
Table 2: Accuracy over runs on the 158 verified items. TOC ranges cover the headline run and three reruns, which share one run on the 33 long-output items (Section 2.2 ); Raw100k lists the headline and the logged run (GPT-5.1, GPT-5.2, GPT-5.4); Hybrid ranges cover three runs. Tokens are mean input tokens per item from the runs with full logs; full-set logs are unavailable for Opus and for Sonnet inline. † Mostly a change in behavior unrelated to the condition (Appendix B ).
| Stage rates (%) | Change vs. inline (pp) | |||||||||
| Model | Condition | Total | Build | Exposure | Closure | Unsupp. | ||||
| Sonnet 4.6 | TOC | 86.9 | 98.9 | 98.0 | – | – | – | – | – | |
| GPT-5.4 | Raw100k | 73.7 | 96.0 | 99.0 | (reference) | |||||
| TOC | 80.0 | 91.2 | 89.0 | 4.4 | 4.1 | 2.6 | 7.2 | 1.3 | ||
| GPT-5.2 | Raw100k | 65.0 | 96.6 | 95.3 | (reference) | |||||
| TOC | 62.5 | 90.7 | 82.4 | 11.4 | 1.5 | 2.5 | 7.7 | 0.3 | ||
Table 3: Where accuracy is lost, on numeric items. Left: stage rates ( = the agent’s own simulation printed the keyed value; = the value reached the agent, given produced; = correct, given seen). Right: Shapley attribution of each condition’s accuracy change relative to the same model’s logged inline run, in pp (95% item-bootstrap intervals in Table 9 ); unsupported covers answers given without the keyed value in view. TOC averages the three logged reruns. The intervention conditions are in Table 5 .
| Model | Cond. | Skip | Own | Digits | Trap | Corr. P | Corr. D |
|---|---|---|---|---|---|---|---|
| Sonnet 4.6 | TOC | 10.5 | 62.9 | 8.5 | 1/35 | 99.7 | 83.3 |
| GPT-5.4 | TOC | 41.8 | 61.0 | 40.3 | 15/31 | 99.4 | 75.2 |
| GPT-5.2 | TOC | 60.3 | 86.7 | 67.0 | 26/35 | 100.0 | 72.5 |
| GPT-5.1 | TOC | 53.0 | 67.5 | 47.6 | 19/27 | 100.0 | 66.3 |
| GPT-5.4 | Raw | 0.0 | 13.3 | 3.1 | 0/0 | 98.9 | 100.0 |
| GPT-5.2 | Raw | 0.0 | 29.7 | 0.0 | 1/2 | 95.3 | – |
Table 4: Which evidence the agents use (%, except Trap counts). Skip : no part of result.out reached the agent. Own : the agent read one of its own selected-output files. Digits : of trajectories that saw the keyed value, those that saw it only with more digits than the report. Trap : of trajectories that saw a value rounding to the key but truncating to a distractor, those that chose the distractor (each trajectory once). Corr. P / Corr. D : accuracy when the keyed value was seen as the report prints it / only with more digits.
| Accuracy | Evidence use (%) | Stage rates (%) | Closure | |||||||||
| Model | Condition | Runs | % | Skip | Def. | Own | Digits | term | ||||
| GPT-5.2 | Inline | 1 | 67.7% | – | 0.0 | 100.0 | 29.7 | 0.0 | 65.0 | 96.6 | 95.3 | (ref.) |
| TOC control | 1 | 63.3% | – | 57.0 | 98.7 | 87.3 | 63.5 | 67.2 | 92.4 | 83.5 | 7.4 | |
| TOC+Report | 2 | 66.8% | 3.5 | 0.0 | 42.1 | 1.3 | 0.6 | 63.1 | 95.4 | 98.8 | 2.1 | |
| TOC+Round | 2 | 59.5% | 3.8 | 55.1 | 98.4 | 80.7 | 59.9 | 66.4 | 89.0 | 81.5 | 8.5 | |
| Hybrid | 3 | 67.9% | 4.6 | 5.3 | 99.8 | 34.8 | 8.3 | 63.5 | 97.3 | 94.9 | 0.3 | |
Table 5: Intervention runs on the 158 verified items. : per-item accuracy difference from the TOC control run (GPT-5.4: from all four TOC runs; pp). Skip, Own, and Digits as in Table 4 ; Def. : the agent defined a selected-output file. Stage rates and the closure term (Shapley, pp relative to the logged inline run) as in Table 3 . The inline rows are the logged inline runs, shown for reference; GPT-5.4’s TOC row averages its three TOC reruns.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Reason | Items | |
|---|---|---|
| Incorrect key | 1 | 22 |
| Unstable redox-derived value | 17 | 1, 8, 48, 108, 115, 118, 119, 125, 128, 130, 133, 136, 138, 140, 145, 166, 192 |
| Missing question sentence | 3 | 4, 37, 78 |
| Malformed option or species name | 2 | 3, 186 |
| Asks for moles, key is molality | 3 | 54, 64, 66 |
| Wrong process or step in stem | 3 | 56 (precipitated, not dissolved); 5, 6 (key matches step 48) |
Table 6: Excluded items by reason (dataset item IDs).
| Agent-TOC acc. | Agent-Raw100k acc. | TOC Raw100k | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Model | 200 | 158 | 134 | 200 | 158 | 134 | 200 | 158 | 134 |
| Claude Opus 4.6 | 83.5% | 87.3% | 86.6% | 83.0% | 88.0% | 88.1% | 0.5 pp | 0.6 pp | 1.5 pp |
| Claude Sonnet 4.6 | 84.0% | 91.8% | 91.0% | 83.5% | 91.1% | 90.3% | 0.5 pp | 0.6 pp | 0.7 pp |
| GPT-5.4 | 72.5% | 75.3% | 75.4% | 70.0% | 70.9% | 70.9% | 2.5 pp | 4.4 pp | 4.5 pp |
| GPT-5.2 | 54.0% | 57.0% | 59.7% | 61.5% | 65.8% | 69.4% | 7.5 pp | 8.9 pp | 9.7 pp |
| GPT-5.1 | 54.0% | 57.0% | 54.5% | 63.5% | 67.7% | 68.7% | 9.5 pp | 10.8 pp | 14.2 pp |
Table 7: Headline-run results on the full pool (200), the verified items (158), and the strict subset (134).
| Step | Agent call tool result (abridged) |
|---|---|
| 1 | write_file(calc.inp) : SOLUTION 1 Pure water / pH 7 charge / REACTION 1 / H2SO4 1 / 0.001 / SELECTED_OUTPUT -file sel.out -pH true -solution true / END |
| 2 | execute_phreeqc(calc.inp) ok; output_files: phreeqc.log, result.out (6,957 bytes), sel.out (315 bytes); result_out_toc: 19 entries, among them {line 92: Reaction step 1}, {line 112: Solution composition}, {line 118: Description of solution}; hint: Use read_file with start_line/end_line to read specific sections. |
| 3 | read_file(sel.out, 1, 200) sim state soln … step pH pe / 1 i_soln 1 … 6.99738 4 / 1 react 1 … 1 2.75065 14.6924 |
| 4 | write_file(final_answer.txt) : <<< A >>> |
| – | Not read: result.out , line 120 (in “Description of solution” of reaction step 1): pH = 2.751 |
Figure 4: A GPT-5.1 TOC trajectory on item 121 (pH after the first reaction step; options A 2.750, B 2.751, C 2.748, D 2.745; key B), with tool results abridged and “/” marking line breaks. The agent answers A, the truncation of the 2.75065 in its own selected-output file, and never reads the report, which prints the keyed 2.751.
| Model | Raw100k | TOC | Grep | Hybrid | TOC+Report | TOC+Round |
| Opus 4.6 | 1 | 1 | – | – | – | – |
| Sonnet 4.6 | 1 | 4 | 1 | – | – | – |
| GPT-5.4 | 2 | 4 | – | – | 2 | – |
| GPT-5.2 | 2 | 4+1 | 1 | 3 | 2 | 2 |
| GPT-5.1 | 2 | 4+1 | 1 | 3 | 2 | 2 |
| Gemini 3 Flash | 1 | 1 | – | – | – | – |
Table 8: Full-agent runs per model and condition on the 158 items. Each model has one headline run with Raw100k and one with TOC; the other TOC runs are reruns, and on the 33 items whose report exceeds 100k characters the four TOC runs share one run with a fixed index (Section 2.2 ). +1: the TOC control run of the interventions, with the index built for every report size.
| Model | Condition | Total | Build | Exposure | Closure | Unsupported |
|---|---|---|---|---|---|---|
| GPT-5.2 | TOC | 11.4 [ 21.2, 1.5] | 1.5 [ 5.6, 2.7] | 2.5 [ 5.3, 0.4] | 7.7 [ 12.6, 2.9] | 0.3 [ 6.1, 6.1] |
| Grep | 11.7 [ 21.2, 2.9] | 1.8 [ 2.7, 6.5] | 4.7 [ 8.7, 1.1] | 7.8 [ 13.6, 2.7] | 1.0 [ 6.3, 4.0] | |
| TOC (control) | 5.1 [ 16.1, 5.8] | 1.3 [ 3.1, 6.0] | 1.7 [ 4.7, 0.6] | 7.4 [ 12.7, 2.2] | 2.7 [ 3.4, 8.7] | |
| TOC+Report | 2.9 [ 10.6, 5.5] | 1.4 [ 6.2, 4.3] | 0.6 [ 3.3, 1.6] | 2.1 [ 0.3, 5.0] | 3.1 [ 8.0, 1.6] | |
| Hybrid | 1.0 [ 8.0, 5.8] | 1.0 [ 5.2, 2.9] | 0.3 [ 1.2, 2.0] | 0.3 [ 2.3, 2.0] | 0.0 [ 4.8, 4.4] | |
| GPT-5.4 | TOC | 4.4 [ 12.4, 3.4] | 4.1 [ 0.0, 8.1] | 2.6 [ 4.8, 0.5] | 7.2 [ 11.1, 4.2] | 1.3 [ 3.2, 5.8] |
Table 9: Shapley attribution of each condition’s accuracy change relative to the same model’s logged inline run (pp, numeric items), with 95% item-bootstrap intervals (1,000 resamples, items resampled jointly for both conditions). TOC: the three reruns; TOC (control), TOC+Report, and Hybrid: the intervention runs.
| Repeat | Inline | TOC | |
|---|---|---|---|
| 1 | 92/131 | 81/131 | 8.4 |
| 2 | 93/131 | 79/131 | 10.7 |
| 3 | 95/131 | 83/131 | 9.2 |
| All | 280/393 | 243/393 | 9.4 |
| Mean (%) | 71.2 | 61.8 |
Table 10: GPT-5.2 replay on 131 frozen checkpoints. Correct answers are shown over continuations per condition; is TOC minus inline accuracy in pp. The mean uses three repeats of the same items.
| Outcome (158 items) | TOC | Raw100k |
| Correct | 44 | 79 |
| Wrong | 2 | 6 |
| Answer only in a chat message | 30 | 40 |
| Answer file unparseable | 1 | 0 |
| Step budget exhausted, no answer | 80 | 33 |
| Log incomplete | 1 | 0 |
Table 11: Gemini 3 Flash outcomes on the 158 items in the headline TOC and Raw100k runs. Accuracy when committed : correct among correct and wrong answers.
More in this topic
LLM Tool Use
Large language model tool use studies how models select, call, and coordinate external tools to extend their reasoning and take actions.
See all 289 papers