cs.CLOct 1, 2026

Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models

Authors: Juan S. Santillana

Organizations: Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Keyword-matching benchmarks can credit small models for tool use they never perform. We document such a false positive in a matched-architecture pair of Spanish security language models and propose a ladder of strict, cheap diagnostics. A 661.6M parameter model (approx. 65% code/technical text; no dedicated SFT) and a 1,109M model (web-heavy multi-phase curriculum; 6B-token tool-SFT) share decoder, tokenizer, and special tokens, scoring almost identically on lenient tool-use metrics (B4: 0.660 vs. 0.650). Verbatim-reproduction checks on training examples separate them completely: the 600M emits valid tool calls with generalized arguments on 6/6 examples; the 1B does so on 0/6 across checkpoints. A first-token probe localizes the 1B's failure to a missing prior (prob. 10−410^{-4}--10−510^{-5} on <|tool_call|>), which was erased by its web-heavy training phase. A targeted SFT recipe (diverse corpus, 5x higher learning rate, 2,202 steps, ~3.3 GPU-hours) repairs the 1B using three orders of magnitude fewer tokens than the failed phase. On all 269 corpus rows, valid emission rises from 0.100 to 0.959 (600M: 0.926). On 238 unseen prompts, the repaired 1B passes 0.536 vs. the 600M's 0.428 (p=0.004p = 0.004). Embedding-drift checks show the repair did not move the trigger token's tied embedding (97.7% of the bf16 table remains bit-identical), meaning changes live in the surrounding network. Both models over-trigger, rarely answering negative prompts without a call (0.09 for 600M, 0.17 for repaired 1B). Factorial analyses confirm all repair configurations install the format, though suppression benefits from a diverse corpus remain a hypothesis due to seed sensitivity. This cheap diagnostic ladder costs minutes of CPU time and should gate tool-use claims on small models.

Figures & tables

 VectraYX-600MVectraYX-1B
Parameters661.6M1,109M
Layers1822
dmodeld_{\mathrm{model}}1,7922,048
dffnd_{\mathrm{ffn}}4,8645,504
Heads (Q / KV, GQA)14 / 216 / 4
NoPE period / RoPE θ\theta4 / 10610^{6}4 / 10610^{6}
Table 1. The matched pair. Identical implementation class, tokenizer, and special-token layout; only scale hyperparameters differ.
ComponentConfigured weightRealized fraction
Spanish (Wiki-ES, FineWeb2-ES)0.45≈\approx 0.34
Code and technical (Stack, ICPC)0.23≈\approx 0.65
Cyber (CVE/NVD-derived)0.22≈\approx 0.007
Instruction-formatted SFT0.10≈\approx 0.001
Table 2. The 600M single-phase mixture: configured bin weights vs. realized token fractions ( ≈\approx 10B tokens actually trained). Small bins (cyber, sft) exhaust early; the realized mix reverts to the large code and Spanish bins.
Benchmark600M (step 154K)1B (recorded)
B1 CVE QA (keyword recall)0.3420.342
B2 Classification (acc.)0.2400.205
B3 Commands (tool match)0.170.24
B4 Tool use0.6600.650 †
B5 Conversational0.5890.586
Table 3. Lenient B1–B5 harness scores (single seed). The 1B column is reproduced from project records for its phase-1-era evaluation and is flagged as unreliable for B4 by the strict diagnostic of § 6.2 : reported, not endorsed. Bold marks the per-row leader at face value.
CheckpointPass / trialsGen.PfirstP_{\mathrm{first}}
600M step 154K6 / 6yesgreedy
1B phase-3 step 20K (pre-vision)0 / 4–6—∼10−4\sim 10^{-4} – 10−510^{-5}
1B phase-4b SFT step 400 (post)0 / 4–6—∼10−4\sim 10^{-4} – 10−510^{-5}
1B repaired, step 2202 (§ 6.5 )4 / 4 ‡exact0.996–0.998
historical run, checkpoint lost4 / 4 ‡exact0.995–0.998
Table 4. Verbatim-reproduction diagnostic over real training examples from tool_sft_mini_v1.jsonl , greedy decoding, exact training-time forward pass. “Gen.” marks whether passing emissions generalized (non-identical, contextually sensible arguments) or reproduced training completions exactly. PfirstP_{\mathrm{first}} is the probability of the correct first generated token of the assistant turn ( <|tool_call|> ). The repaired-1B row is the reproduced repair checkpoint of § 6.5 ; the historical run’s checkpoint no longer exists and its numbers are reported beneath it for comparison only (L12).
 Data / configStrict outcome
A: dedicated phase-3 ( ≈\approx 6B tok, 20K steps)30% tool-SFT mixture within broad phase-3 mix0/4–6; Pfirst ⁣∼ ⁣10−4P_{\mathrm{first}}\!\sim\!10^{-4} – 10−510^{-5}
B: narrow retune (486 steps)tool corpus ++ small cyber sample; LR 2×10−62{\times}10^{-6} ; embeddings frozennot validly measured : recorded 0/8 by a harness later found to omit the preamble and drop special tokens; loss fell to 0.05–0.4
C: informed recipe, reproduced (2,202 steps, ≈\approx 3.3 h, one rented GPU)OASST ES/EN ++ tool corpus ++ domain reasoning traces; LR 1×10−51{\times}10^{-5} ; embeddings nominally unfrozen—but numerically inert, § 6.6 ; warm start phase4a_v04/4 verbatim (exact), Pfirst=0.996P_{\mathrm{first}}{=}0.996 – 0.9980.998 ; 8/8 novel-prompt battery (7/7 tool choice, plus correct suppression)
C ∗ : same recipe, historical run (checkpoint and parent lost)identical corpus, LR and step count; warm start v3b_step_0019004/4 verbatim (exact), Pfirst=0.995P_{\mathrm{first}}{=}0.995 – 0.9980.998 ; 7/8 novel-prompt battery
Table 5. Three SFT efforts on the VectraYX-1B lineage. “Strict” is well-formed <|tool_call|> emission under the Level-3 protocol; row B was never measured under it (see text). Volume did not predict the outcome of A vs. C: the successful recipe used ≈\approx 1000 ×\times fewer tokens than the failed dedicated phase. Row C reports the reproduced run , which is the checkpoint every downstream number in this paper was measured on; row C ∗ is the 2026-09 historical run of the same recipe, whose output checkpoint and warm-start parent were both lost before they could be re-measured (L12). The two agree within reproduction noise, but only C is an artifact we still hold.
StepPˉskip\bar{P}_{\mathrm{skip}}Pˉlit\bar{P}_{\mathrm{lit}}no-callEmitGen.Suppr.
1K0.8583.6×10−73.6{\times}10^{-7}0.6735/54no
5K0.9731.6×10−61.6{\times}10^{-6}0.9975/53no
10K0.9932.9×10−72.9{\times}10^{-7}0.9885/53no
20K0.9964.5×10−94.5{\times}10^{-9}0.999915/53no
30K0.99983.4×10−113.4{\times}10^{-11}0.99985/53no
60K0.99867.8×10−127.8{\times}10^{-12}0.99775/53no
Table 6. VectraYX-600M trajectory probe, 8 of 15 sampled checkpoints (the omitted seven interleave without changing the shape). Pˉ\bar{P} is averaged over the five call-expecting examples; “no-call” is the single suppression example, on the PskipP_{\mathrm{skip}} series. “Emit” counts well-formed calls on the five; “Gen.” counts how many of those used non-identical arguments; “Suppr.” is whether the no-call example was correctly answered without a call.
Phase 2 (web-heavy pretraining)Phase 3 (dedicated tool-SFT)
StepPˉskip\bar{P}_{\mathrm{skip}}Pˉlit\bar{P}_{\mathrm{lit}}no-callEmitGen.StepPˉskip\bar{P}_{\mathrm{skip}}Pˉlit\bar{P}_{\mathrm{lit}}no-callEmitB4
36K0.4323.2×10−143.2{\times}10^{-14}2.5×10−22.5{\times}10^{-2}3/504.0K3.7×10−33.7{\times}10^{-3}3.7×10−33.7{\times}10^{-3}5.2×10−35.2{\times}10^{-3}0/5—
38K3.0×10−33.0{\times}10^{-3}1.0×10−91.0{\times}10^{-9}1.3×10−41.3{\times}10^{-4}0/5—5.7K1.0×10−21.0{\times}10^{-2}1.0×10−21.0{\times}10^{-2}8.5×10−38.5{\times}10^{-3}0/50.020
40K4.0×10−44.0{\times}10^{-4}3.8×10−63.8{\times}10^{-6}4.0×10−54.0{\times}10^{-5}0/5—6.0K1.4×10−2\mathbf{1.4{\times}10^{-2}}1.4×10−21.4{\times}10^{-2}9.1×10−39.1{\times}10^{-3}0/5—
44K6.2×10−76.2{\times}10^{-7}2.1×10−62.1{\times}10^{-6}2.4×10−72.4{\times}10^{-7}0/5—8.0K1.2×10−21.2{\times}10^{-2}1.2×10−21.2{\times}10^{-2}1.4×10−21.4{\times}10^{-2}0/50.075
48K5.9×10−75.9{\times}10^{-7}3.3×10−53.3{\times}10^{-5}1.9×10−71.9{\times}10^{-7}0/5—10.0K §4.8×10−44.8{\times}10^{-4}4.8×10−44.8{\times}10^{-4}2.5×10−82.5{\times}10^{-8}0/50.020
Table 7. VectraYX-1B trajectory probe. Left: phase-2 pretraining, 9 of 13 sampled checkpoints, with the 36K–48K window densified to test whether the 36K value is a singleton (the four omitted points, 76K/116K/134K/174K, sit on the same floor). Right: all ten archived phase-3 (tool-SFT) checkpoints, with the project’s contemporaneous B4 harness score where one was recorded for that checkpoint. Columns as in Table 6 ; “Emit” is 0/5 at every 1B checkpoint after 36K. § marks the first checkpoint after the tool_sft weight cut (0.20 →\to 0.16, applied to the mixture between steps 8K and 10K); the weight was restored to 0.20 for the continuation running from step 10K onward.
AxisVectraYX-600M1B pre-rep.1B repaired
Decision1.0000.0061.000
Well-formed0.9940.0061.000
Tool correct0.776—0.717
Arg. shape0.855—0.819
Entity match0.560—0.667
Headline pass0.4280.0000.536
Table 8. Generalization battery, 166 call-expecting prompts over entities absent from the training corpus. Axes are nested: each is conditioned on the previous one being available, so entity is scored only on the 134–135 fixed-schema items and soft miss only on the 30–31 bash_exec -family items carrying a command regex. Brackets are 95% Wilson intervals.
FamilynnVectraYX-600M1B repaired
nvd_get_cve280.8931.000
anchor70.7140.857
cisa_kev_check280.6070.571
otx_check_ioc270.4440.519
bash_exec280.2140.500
terse_control200.2000.500
Table 9. Headline pass by template family. terse_control re-asks the four fixed-schema tools in the corpus’s own clipped register (“ detalle CVE-2025-3187 ”); anchor is the reconstructed n=8n{=}8 battery of § 6.5 . The pre-remediation 1B scores 0.000 in every family and is omitted.
ArmHead.ToolEntitySoftSupp.Over
 passcorr.matchmisspasstrig.
narrow / high LR0.6080.7830.7480.4190.4580.542
narrow / low LR0.5420.6850.7020.3870.3750.625
diverse / low LR0.5840.7590.7190.4520.5830.403
diverse / high (s42)0.6020.7950.7410.4520.5140.486
diverse / high (s43)0.6020.7710.7260.3870.5000.500
Table 10. F3 factorial: 238-prompt generalization battery, scored on the same axes as Table 8 . Denominators follow the scorer’s own conditioning: headline over the 166 call items, tool over the well-formed call items (165–166), entity over the 134–135 fixed-schema items, soft miss over the 31 command-regex items, suppression over the 72 no-call items. The two right-hand columns are complements of each other up to a single degenerate answer and are not independent evidence. Best in column in bold ; for soft miss and over-triggering the better value is the lower one, as in Table 8 .
Arm @ step 880Head.ToolEntitySoftSupp.Over
 passcorr.matchmisspasstrig.
narrow / low (s42)0.5360.6910.7010.4190.3890.611
narrow / low (s43)0.5480.6850.6870.3550.5690.431
diverse / low (s42)0.5780.7590.7110.4520.5560.431
Table 11. Matched-step check at step 880 of 2,202 (40% of training): the two low-LR arms of Table 10 and an interrupted second seed of the narrow/low-LR cell. Same battery, scorer and denominators as Table 10 . These values are comparable with each other, not with the final-step table.
 600MSFTSFT ++Discordant
 (no SFT)baselinedistilled( pp )
Headline pass (166)0.4280.5480.5485 / 5 (1.00)
Well-formed0.9940.9340.9227 / 5 (0.77)
Tool correct0.7760.7940.8171 / 6 (0.13)
Entity match0.5600.7140.7122 / 4 (0.69)
Suppression pass (72)—0.0140.0560 / 3 (0.25)
Table 12. Tool SFT of the 600M, one seed per arm, on the 238-prompt battery. The first column is the un-fine-tuned 600M, copied from Table 8 and § 6.8 ; it is not paired-tested against the SFT arms here. Tool is conditioned on a well-formed call and entity on the fixed-schema items, as in Table 8 ; paired tests use the items scored in both arms. “Discordant” counts items passing only in the baseline arm vs. only in the distilled arm, with the exact McNemar pp .

Explore similar work

CardsList
Jul 6, 2026cs.CL

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Harsh Soni

UC Berkeley

Large Language Model AgentsLarge Language Model Tool Use
AskPDF
Apr 22, 2026cs.CL

Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models

Can small language models achieve strong tool-use performance without complex adaptation mechanisms? This paper investigates this question through Meta-Tool, a controlled empirical study comparing hypernetwork-based LoRA adaptation against carefully designed few-shot prompting. Using a Llama-3.2-3B-Instruct backbone, we evaluate four adaptation mechanisms--few-shot prompting, documentation encoding, hypernetwork-generated LoRA weights, and value-guided beam search--across four diverse benchmarks: Gorilla APIBench, Spider 2.0, WebArena, and InterCode. Our central finding is a well-supported negative result: despite generating non-trivial weight matrices, the 227.8M-parameter hypernetwork provides no measurable improvement over few-shot prompting alone. Comprehensive ablation studies reveal that few-shot examples contribute +21.5% to performance and documentation contributes +5.0%, while the hypernetwork adds 0%. A 3B model with well-designed prompts achieves 79.7% of GPT-5's average performance at 10×10 \times lower latency. Error analysis across 722 failure cases spanning all shot counts (0--5) shows that at the 5-shot configuration (106 failures), failure modes are task-dependent: schema-heavy tasks (Spider 2.0, WebArena) show near-zero format errors with remaining failures semantic, while format errors dominate on Gorilla (100%) and InterCode (70%). These findings redirect practitioners toward prompt engineering and example curation rather than complex adaptation architectures.

Sachin Kumar

LexisNexis, USA

Large Language Model AdaptationSmall Large Language ModelsFew-Shot LearningPassage Encoding
AskPDF
Sep 13, 2026cs.SE

Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return

Tool-augmented language models are evaluated on whether they reach the right answer, not on whether they report honestly when a tool fails to supply one. We isolate this post-failure decision with a benchmark of 1,024 items spanning 16 internal-system domains and eight tool-failure types, in which a tool call is enforced and the returned payload is guaranteed to be unusable. Under a deployment-style system prompt, 14.10% of responses are dishonest: the model either asserts a value the payload cannot support or declines while citing a fabricated policy or capability limit. The rate is governed almost entirely by whether the failure is signalled. When the tool returns status:error, dishonesty is absent (0.0%); when it returns status:ok with a redacted, corrupted, stale, malformed, empty or truncated value, dishonesty reaches 45.3%. The behaviour is not an artefact of our prompts: it appears under a neutral prompt (10.17%) and under the shipped prompt of every production agent framework we evaluate, reaching 24.67% under CrewAI's, and none of the nine frameworks we audit specifies what the model should do when a tool fails. Comparing prompt-level defences, we find that the operative variable is not deference to tool output but the absence of a named failure state. Appending a single sentence that requires the model to emit retrieval_status: OK or FAILED before answering reduces dishonesty from 14.10% to 0.87%, with one item of 688 worsening against 92 improving, and transfers unchanged into three foreign agent scaffolds. The emitted flag is faithful in 99.7-99.9% of declarations, giving a runtime detector that needs only a regular expression.

Arham Sethi, Arsen Kenzhebayev, Saanvi Paturi +3

The Shishukunj International School · Haileybury Astana · UWCSEA East Campus +2

HonestySystemverilog Assertions
AskPDF