Organizations: Central University, Moscow · Ural Federal University, Yekaterinburg · Skolkovo Institute of Science and Technology, Moscow · AIRI, Moscow
Tool-augmented LLMs must arbitrate between two fallible sources when a tool return conflicts with their parametric memory, yet existing evaluations measure source preference without establishing source correctness. We introduce MemToC, a controlled benchmark for post-tool-return arbitration with executable tools. MemToC comprises 6,504 evaluation episodes constructed from 542 quality-controlled factual questions, independently elicited model-specific closed-book answers, and controlled tool returns of known correctness. These components instantiate four source-correctness cases; tool-error and no-tool conditions are separate controls. Across five open-weight 7-9B models, tool returns strongly dominate elicited closed-book answers. The four instruction-tuned models retain a verified-correct answer against an incorrect tool in only 6.5-17.1% of eligible cases, follow a correct tool in 86.0-93.1%, and repeat the tool return in 78.4-86.0% of cases where both sources are wrong. No cross-model ordering remains stable across three instruction-wording variants with the question and episode content held fixed. We compare prompting with SFT and DPO using chain-level cross-fitting over ToolHop, so questions sharing an underlying fact never straddle training and evaluation. We apply an asymmetric success criterion: correct-answer retention must improve without a detected reduction in correct-tool following. SFT and DPO meet this criterion on the same two of four instruction-tuned backbones. Improvements rarely come cleanly: 19 of 20 tested method-model combinations reduce abstention after tool errors or on unanswerable inputs. Transfer beyond MemToC is positive but partial and depends on the model and presentation frame. Correctness-conditioned arbitration can be improved through fine-tuning, but gains must be evaluated jointly with correct tool use, abstention, and robustness to formulation.
Figures & tables
Figure 1: Schematic illustration of the four source-correctness cases. The displayed combination of answer values is hypothetical and does not represent a single observed MemToC episode; a fixed preference for either source cannot succeed across all four cases.
Figure 2: Overview of MemToC. (a) ToolHop sub-questions are filtered, augmented with type-consistent near and far incorrect returns, and quality-controlled to yield 542 factual items. (b) For each model and prompt formulation, a closed-book pass obtains m , followed by a tool-conditioned pass that produces y under controlled return r . Comparing m and r with verified answer g stratifies responses into four source-correctness cases, separating answer retention from correct- and incorrect-tool following; control conditions are evaluated separately.
Model
Adaptation
Ret. ↑
Tool ↑
Wrong ↓
Err. ↑
Llama-3.1-8B-Instruct
–
17.1
93.1
78.6
84.4
SFT
31.6
92.8
78.6
70.1†
DPO
22.3
92.1
78.0
85.9
gemma-2-9b-it
–
9.2
86.3
79.4
99.5
SFT
26.9
91.8
83.6†
97.9†
DPO
18.6
86.5
77.2†
98.4†
Table 1: Performance before and after fine-tuning on MemToC, pooled across three prompt formulations. Paired changes reported in the text use the corresponding original checkpoint restricted to the same support and need not equal raw differences between displayed rows. Ret., Tool, Wrong, and Err. denote correct-answer retention, correct-tool following, incorrect-tool following, and tool-error abstention, respectively. A dash denotes the original instruction-tuned checkpoint with no additional adaptation, evaluated using the standard benchmark prompt. These rows report full pooled rates, whereas SFT and DPO rows report held-out cross-fitted rates on adaptation-specific case-stable common support. Bold marks a Holm-significant change on a decision metric, † marks an uncorrected p<0.05 on a diagnostic metric, and green or red indicates a desirable or undesirable direction.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: One MemToC episode under four tool-observation conditions. The question and the independently elicited closed-book answer are held fixed while the observed tool return changes. Correct and incorrect factual returns instantiate cells of the correctness matrix; tool error and no tool are evaluated as separate controls. The displayed model outputs are verbatim.
Released file
What is read from it
benchmark/
the evaluated episodes, one file per formulation, plus the repair overlay
results/construction/
the frozen substitution mapping: §3.2 branch counts, answer types, far coverage
results/quality_control/
Step-7 review verdicts, the 542-question analysis set, cross-fitting folds
results/scored_episodes/
per-episode scoring records for the 15 evaluated arms; the untreated rows of Table 1 recompute from these
results/summaries/
every cell of Table 1, the prompting results, pooling across formulations, and all paired comparisons with their intervals
results/finetuning/
the paired fine-tuning changes, the same contrasts as levels, and the 70B scale point
Appendix
Table 2: Which released file carries which reported number. Raw model output text is not released; what the scored-episode files hold is the extracted answer span and the scoring decisions taken on it. Two families cannot be recomputed from the release and are named there rather than omitted: the absolute levels of the fine-tuned rows of Table 1, which need the 48 cross-fitted arms, and the three measurement layers taken on the previous benchmark version.
Where
Pass
Reliability
i
Step 2
Filter review, 54 borderline
19 restored
ii
Step 4
Wording review, 605 candidates
30 removed
iii
Step 5
Distractor authoring, 152
106 differ
iv
Step 7
Semantic review, 575
0.948 agreement
v
Step 5
Generator audits, 2 rounds
12% rejected
vi
§3.4
Target behaviour, 100
κ=0.785
Appendix
Table 3: The eight human annotation passes. Reliability in pass iv is measured on a blinded replication of 58 of the 575 records. In pass iii divergence is expected rather than erroneous — several incorrect values can be equally plausible — so it is resolved by a rule fixed before adjudication, not reported as agreement. Pass vii also found 24 instances with incorrect verified answers, excluded from the corresponding analyses. Pass viii has an empty positive class, so no chance-corrected coefficient is informative.
Figure 4: Correct-answer retention across three instruction-wording variants with fixed question text on untreated models. Markers show the Reference, Paraphrase A, and Paraphrase B estimates; gray segments show their minimum–maximum range and are not confidence intervals. The ranges overlap and model rankings change across formulations. Qwen2.5-7B-Instruct has the highest retention on the Reference formulation and the lowest on Paraphrase A, so we do not include cross-model retention rankings among the core claims.
Model
Prompt
Ret. ↑
Tool ↑
Wrong ↓
Err. ↑
Llama-3.1-8B-Instruct
–
9.3
87.8
71.4
89.3
Warning
21.0
75.1
65.3
16.4
Source-priority
50.6
79.4
60.9
29.7
Abstain-and-flag
12.3
23.5
18.8
89.7
gemma-2-9b-it
–
4.8
80.2
74.1
100.0
Warning
17.7
80.2
65.5
89.7
Appendix
Table 4: Prompting interventions on the four instruction-tuned models, Reference formulation. A dash denotes the same checkpoint under the standard evaluation prompt. Ret., Tool, Wrong and Err. are as in Table 1 of the main paper: correct-answer retention, correct-tool following, incorrect-tool following and tool-error abstention. These are single-formulation levels and therefore differ from the formulation-pooled rows of Table 1; the paired changes with intervals and Holm tests, pooled over the three formulations, are in the released summary layer. Retention, correct-tool following and incorrect-tool following are judge-normalized, tool-error abstention is a deterministic-scorer value.
Arm
Reference
Par. A
Par. B
Pooled
Llama-3.1-8B-I ⋅ SFT
met
fail t
met
met
Llama-3.1-8B-I ⋅ DPO
fail k
fail k
met
met
gemma-2-9b-it ⋅ SFT
met
met
met
met
gemma-2-9b-it ⋅ DPO
fail k
met
met
met
Qwen2.5-7B-I ⋅ SFT
fail
fail
fail
fail
Qwen2.5-7B-I ⋅ DPO
fail
fail
fail
fail
Appendix
Table 5: The asymmetric criterion evaluated per prompt formulation and pooled (judge-normalized layer; Holm correction inside each slice’s own family). k failed because the retention gain does not reach Holm p<0.05 (0.28, 0.051 and 0.44 respectively); t failed on a significant loss in correct-tool following ( − 0.045, p 0.028). The deterministic layer agrees except on Llama-Instruct’s DPO arm under Paraphrase A, which it reads as met. Only gemma’s SFT arm meets the criterion under every formulation; Mistral’s DPO arm, a failure when pooled, meets it under Paraphrase A.
Arm
Value frame
Passage frame
Abstention
Retention
Llama-3.1-8B-Instruct ⋅ SFT
+0.051
+0.044
− 0.068
+0.025
Llama-3.1-8B-Instruct ⋅ DPO
+0.070
0.000
− 0.020
+0.017
gemma-2-9b-it ⋅ DPO
+0.112
+0.047
− 0.076
+0.004
Appendix
Table 6: Out-of-distribution transfer (FaithEval-counterfactual × ARC), adapters trained leakage-free over the full 575-question pool, paired episode bootstrap B=2,000 . Every cell is a change against the corresponding original checkpoint, with its 95% interval; bold marks an interval excluding zero. The two conflict frames present the same counterfactual value either as a bare tool return or inside a passage, and both are read on the common 185-item support so that they are comparable. The 95% intervals are given in the text below. Abstention is measured on the unanswerable split. Retention is the change in closed-book accuracy, and all three arms fall inside the preregistered ±0.03 band.
Model
Guided
Auto
Two-stage
n
Qwen2.5-7B-Instruct
1.000
1.000
0.984
61
Qwen2.5-3B-Instruct
1.000
0.979
0.979
48
Qwen2.5-1.5B-Instruct
1.000
0.970
0.576
33
Llama-3.1-8B-Instruct
0.992
0.328
0.976
125
Llama-3.2-3B-Instruct
1.000
0.291
0.367
79
Mistral-7B-Instruct-v0.3
0.104 †
0.885
0.250 †
96
Appendix
Table 7: Answer agreement with the executed tool return on the memory-right slice, by model and tool-choice protocol. The three protocols are a named schema-guided call, free auto choice, and a two-stage decide-then-call prompt. Bold marks a failing arm. † Mistral’s two schema-guided arms are confounded with a serving defect described below and must not be read as a model property. Hermes-3 is a third-party tool-finetuned checkpoint of the same Llama-3.1-8B base, included as an uncontrolled comparison.
Followed the tool
Kept memory
Control
Model
Median
Top-20
Median
Top-20
Median
Llama-3.1-8B-Instruct, seed 1
1
38/41
1
53/55
421
Llama-3.1-8B-Instruct, seed 2
1
33/37
1
59/63
421
gemma-2-9b-it
2
32/36
1
36/39
538
Qwen2.5-7B-Instruct
26,145
0/25
34,800
0/66
36,598
Appendix
Table 8: Rank of the model’s own elicited closed-book answer in the Jacobian-lens reading at the final input position, on the arbitration case, by what the model then did. Lower is closer to the top of the distribution. The control column applies the same measure to answers the model does not hold for the question at hand, and is a per-model quantity, so the two Llama seeds share it.
Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.
A language model's memory can be worse than no memory at all. Give a model a memory that kept a wrong conclusion but dropped the work behind it, and it re-emits the stale value as a confident answer; give the same model an empty memory, and it abstains. We call this failure brittle memory. The information loss behind it is definitional (an answer cannot be recomputed once its inputs are gone), so the loss is only the setup; the finding is behavioral. Whether a model turns the lost source into a confident error or an abstention is set by disposition, not capability: four of eight models we test emit, and the four that abstain escape only by an interface affordance -- forced through a mandatory structured-output field, as production tool calls are, they commit the inherited wrong value. We measure correctability with reclaim evaluation: induce a known drift, compress the interaction at a fixed budget, deliver a correction that names the error, and score exact recovery of the known answer, judge-free. Correctability is bottlenecked not by capability but by whether the memory kept a re-derivation basis (the source) rather than the answer, so an 8B model and a frontier one wall in the same place. A one-line source-first policy -- keep the recomputable source, drop the re-derivable conclusion -- restores correctability at equal budget wherever the source is compact and identifiable, with a length-matched control that rules out 'more text' and a deployable one-prompt form weaker than the oracle. We map where the fix fails (source size, noise, a silent truncation mode a completeness tag makes loud), show the failure compounds through memory loops, and replicate on three deployed memory systems and on real dialogue (MultiWOZ, where the checkable value is present by construction). We release the harness, the paired memory conditions, and validators built to come out false.
Persistent memory can personalize an LLM agent, but an incorrect durable update can silently distort future behavior. We study the memory-clarification boundary: whether interaction-derived information should be persisted, used only in the current context, re-verified, or clarified with the user. MCB contains 140 primary scenarios, split into 70 development and 70 held-out items, plus a separate 70-item contrast set. It evaluates both action labels and structured tool-call selection. Two non-authors independently label the 70 held-out primary and 70 contrast items (97.1% agreement, Cohen's kappa = 0.962); a blind third resolves four disagreements, replacing eight author labels by non-author majority. Across Claude and Qwen, models verify changing facts more reliably than they ask users to resolve ambiguity. Bare Qwen asks on 0/12 clarification items while verifying 12/18 freshness items. Few-shot prompting raises accuracy from 0.557 to 0.771 (paired delta = +0.214, Holm-adjusted exact McNemar p_H = 0.002), yet clarification recall remains 0.333. The policy prompt reduces erroneous persistence from 0.243 to 0.100 (p_H = 0.038), although its accuracy gain is not significant. Label-tool agreement is 57% for each Claude model and 23% for Qwen; Qwen accuracy falls from 0.557 to 0.343 (p_H = 0.047). Memory evaluation must test both stated decisions and tool-call choices.
Baichuan Li, Junyi Yao, Zihao Zheng
Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, USA · Department of Computer Science & Engineering, Washington University in St. Louis, St. Louis, USA