Organizations: Central University, Moscow · Ural Federal University, Yekaterinburg · Skolkovo Institute of Science and Technology, Moscow · AIRI, Moscow
Tool-augmented LLMs must arbitrate between two fallible sources when a tool return conflicts with their parametric memory, yet existing evaluations measure source preference without establishing source correctness. We introduce MemToC, a controlled benchmark for post-tool-return arbitration with executable tools. MemToC comprises 6,504 evaluation episodes constructed from 542 quality-controlled factual questions, independently elicited model-specific closed-book answers, and controlled tool returns of known correctness. These components instantiate four source-correctness cases; tool-error and no-tool conditions are separate controls. Across five open-weight 7-9B models, tool returns strongly dominate elicited closed-book answers. The four instruction-tuned models retain a verified-correct answer against an incorrect tool in only 6.5-17.1% of eligible cases, follow a correct tool in 86.0-93.1%, and repeat the tool return in 78.4-86.0% of cases where both sources are wrong. No cross-model ordering remains stable across three instruction-wording variants with the question and episode content held fixed. We compare prompting with SFT and DPO using chain-level cross-fitting over ToolHop, so questions sharing an underlying fact never straddle training and evaluation. We apply an asymmetric success criterion: correct-answer retention must improve without a detected reduction in correct-tool following. SFT and DPO meet this criterion on the same two of four instruction-tuned backbones. Improvements rarely come cleanly: 19 of 20 tested method-model combinations reduce abstention after tool errors or on unanswerable inputs. Transfer beyond MemToC is positive but partial and depends on the model and presentation frame. Correctness-conditioned arbitration can be improved through fine-tuning, but gains must be evaluated jointly with correct tool use, abstention, and robustness to formulation.
Figures & tables
Figure 1: Schematic illustration of the four source-correctness cases. The displayed combination of answer values is hypothetical and does not represent a single observed MemToC episode; a fixed preference for either source cannot succeed across all four cases.
Figure 2: Overview of MemToC. (a) ToolHop sub-questions are filtered, augmented with type-consistent near and far incorrect returns, and quality-controlled to yield 542 factual items. (b) For each model and prompt formulation, a closed-book pass obtains m , followed by a tool-conditioned pass that produces y under controlled return r . Comparing m and r with verified answer g stratifies responses into four source-correctness cases, separating answer retention from correct- and incorrect-tool following; control conditions are evaluated separately.
Model
Adaptation
Ret. ↑
Tool ↑
Wrong ↓
Err. ↑
Llama-3.1-8B-Instruct
–
17.1
93.1
78.6
84.4
SFT
31.6
92.8
78.6
70.1†
DPO
22.3
92.1
78.0
85.9
gemma-2-9b-it
–
9.2
86.3
79.4
99.5
SFT
26.9
91.8
83.6†
97.9†
DPO
18.6
86.5
77.2†
98.4†
Table 1: Performance before and after fine-tuning on MemToC, pooled across three prompt formulations. Paired changes reported in the text use the corresponding original checkpoint restricted to the same support and need not equal raw differences between displayed rows. Ret., Tool, Wrong, and Err. denote correct-answer retention, correct-tool following, incorrect-tool following, and tool-error abstention, respectively. A dash denotes the original instruction-tuned checkpoint with no additional adaptation, evaluated using the standard benchmark prompt. These rows report full pooled rates, whereas SFT and DPO rows report held-out cross-fitted rates on adaptation-specific case-stable common support. Bold marks a Holm-significant change on a decision metric, † marks an uncorrected p<0.05 on a diagnostic metric, and green or red indicates a desirable or undesirable direction.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: One MemToC episode under four tool-observation conditions. The question and the independently elicited closed-book answer are held fixed while the observed tool return changes. Correct and incorrect factual returns instantiate cells of the correctness matrix; tool error and no tool are evaluated as separate controls. The displayed model outputs are verbatim.
Released file
What is read from it
benchmark/
the evaluated episodes, one file per formulation, plus the repair overlay
results/construction/
the frozen substitution mapping: §3.2 branch counts, answer types, far coverage
results/quality_control/
Step-7 review verdicts, the 542-question analysis set, cross-fitting folds
results/scored_episodes/
per-episode scoring records for the 15 evaluated arms; the untreated rows of Table 1 recompute from these
results/summaries/
every cell of Table 1, the prompting results, pooling across formulations, and all paired comparisons with their intervals
results/finetuning/
the paired fine-tuning changes, the same contrasts as levels, and the 70B scale point
Appendix
Table 2: Which released file carries which reported number. Raw model output text is not released; what the scored-episode files hold is the extracted answer span and the scoring decisions taken on it. Two families cannot be recomputed from the release and are named there rather than omitted: the absolute levels of the fine-tuned rows of Table 1, which need the 48 cross-fitted arms, and the three measurement layers taken on the previous benchmark version.
Where
Pass
Reliability
i
Step 2
Filter review, 54 borderline
19 restored
ii
Step 4
Wording review, 605 candidates
30 removed
iii
Step 5
Distractor authoring, 152
106 differ
iv
Step 7
Semantic review, 575
0.948 agreement
v
Step 5
Generator audits, 2 rounds
12% rejected
vi
§3.4
Target behaviour, 100
κ=0.785
Appendix
Table 3: The eight human annotation passes. Reliability in pass iv is measured on a blinded replication of 58 of the 575 records. In pass iii divergence is expected rather than erroneous — several incorrect values can be equally plausible — so it is resolved by a rule fixed before adjudication, not reported as agreement. Pass vii also found 24 instances with incorrect verified answers, excluded from the corresponding analyses. Pass viii has an empty positive class, so no chance-corrected coefficient is informative.
Figure 4: Correct-answer retention across three instruction-wording variants with fixed question text on untreated models. Markers show the Reference, Paraphrase A, and Paraphrase B estimates; gray segments show their minimum–maximum range and are not confidence intervals. The ranges overlap and model rankings change across formulations. Qwen2.5-7B-Instruct has the highest retention on the Reference formulation and the lowest on Paraphrase A, so we do not include cross-model retention rankings among the core claims.
Model
Prompt
Ret. ↑
Tool ↑
Wrong ↓
Err. ↑
Llama-3.1-8B-Instruct
–
9.3
87.8
71.4
89.3
Warning
21.0
75.1
65.3
16.4
Source-priority
50.6
79.4
60.9
29.7
Abstain-and-flag
12.3
23.5
18.8
89.7
gemma-2-9b-it
–
4.8
80.2
74.1
100.0
Warning
17.7
80.2
65.5
89.7
Appendix
Table 4: Prompting interventions on the four instruction-tuned models, Reference formulation. A dash denotes the same checkpoint under the standard evaluation prompt. Ret., Tool, Wrong and Err. are as in Table 1 of the main paper: correct-answer retention, correct-tool following, incorrect-tool following and tool-error abstention. These are single-formulation levels and therefore differ from the formulation-pooled rows of Table 1; the paired changes with intervals and Holm tests, pooled over the three formulations, are in the released summary layer. Retention, correct-tool following and incorrect-tool following are judge-normalized, tool-error abstention is a deterministic-scorer value.
Arm
Reference
Par. A
Par. B
Pooled
Llama-3.1-8B-I ⋅ SFT
met
fail t
met
met
Llama-3.1-8B-I ⋅ DPO
fail k
fail k
met
met
gemma-2-9b-it ⋅ SFT
met
met
met
met
gemma-2-9b-it ⋅ DPO
fail k
met
met
met
Qwen2.5-7B-I ⋅ SFT
fail
fail
fail
fail
Qwen2.5-7B-I ⋅ DPO
fail
fail
fail
fail
Appendix
Table 5: The asymmetric criterion evaluated per prompt formulation and pooled (judge-normalized layer; Holm correction inside each slice’s own family). k failed because the retention gain does not reach Holm p<0.05 (0.28, 0.051 and 0.44 respectively); t failed on a significant loss in correct-tool following ( − 0.045, p 0.028). The deterministic layer agrees except on Llama-Instruct’s DPO arm under Paraphrase A, which it reads as met. Only gemma’s SFT arm meets the criterion under every formulation; Mistral’s DPO arm, a failure when pooled, meets it under Paraphrase A.
Arm
Value frame
Passage frame
Abstention
Retention
Llama-3.1-8B-Instruct ⋅ SFT
+0.051
+0.044
− 0.068
+0.025
Llama-3.1-8B-Instruct ⋅ DPO
+0.070
0.000
− 0.020
+0.017
gemma-2-9b-it ⋅ DPO
+0.112
+0.047
− 0.076
+0.004
Appendix
Table 6: Out-of-distribution transfer (FaithEval-counterfactual × ARC), adapters trained leakage-free over the full 575-question pool, paired episode bootstrap B=2,000 . Every cell is a change against the corresponding original checkpoint, with its 95% interval; bold marks an interval excluding zero. The two conflict frames present the same counterfactual value either as a bare tool return or inside a passage, and both are read on the common 185-item support so that they are comparable. The 95% intervals are given in the text below. Abstention is measured on the unanswerable split. Retention is the change in closed-book accuracy, and all three arms fall inside the preregistered ±0.03 band.
Model
Guided
Auto
Two-stage
n
Qwen2.5-7B-Instruct
1.000
1.000
0.984
61
Qwen2.5-3B-Instruct
1.000
0.979
0.979
48
Qwen2.5-1.5B-Instruct
1.000
0.970
0.576
33
Llama-3.1-8B-Instruct
0.992
0.328
0.976
125
Llama-3.2-3B-Instruct
1.000
0.291
0.367
79
Mistral-7B-Instruct-v0.3
0.104 †
0.885
0.250 †
96
Appendix
Table 7: Answer agreement with the executed tool return on the memory-right slice, by model and tool-choice protocol. The three protocols are a named schema-guided call, free auto choice, and a two-stage decide-then-call prompt. Bold marks a failing arm. † Mistral’s two schema-guided arms are confounded with a serving defect described below and must not be read as a model property. Hermes-3 is a third-party tool-finetuned checkpoint of the same Llama-3.1-8B base, included as an uncontrolled comparison.
Followed the tool
Kept memory
Control
Model
Median
Top-20
Median
Top-20
Median
Llama-3.1-8B-Instruct, seed 1
1
38/41
1
53/55
421
Llama-3.1-8B-Instruct, seed 2
1
33/37
1
59/63
421
gemma-2-9b-it
2
32/36
1
36/39
538
Qwen2.5-7B-Instruct
26,145
0/25
34,800
0/66
36,598
Appendix
Table 8: Rank of the model’s own elicited closed-book answer in the Jacobian-lens reading at the final input position, on the arbitration case, by what the model then did. Lower is closer to the top of the distribution. The control column applies the same measure to answers the model does not hold for the question at hand, and is a per-model quantity, so the two Llama seeds share it.
Department of Operations Research and Engineering Management, Southern Methodist University, Dallas, USA · Department of Computer Science & Engineering, Washington University in St. Louis, St. Louis, USA