cs.AIOct 2, 2026

Toward SLM-based agentic task-tool intent matching

Authors: Chiara Troiani, Arash Salarian, Majed El Helou, Benjamin Ryder, Jean Diaconu, Hervé Muyal, Marcelo Yannuzzi

Organizations: Cisco Systems, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Tool-equipped AI agents use tool calls to access data and act on external systems. Horizontal growth of agentic systems increases the number of these interactions, and further motivates the need for automated, per-call oversight that can operate at low latency and/or on-prem. Conventional authorization schemes can determine whether an agent is allowed to invoke a tool, but cannot assess the agent's underlying cognition, specifically, whether the tool selection represents a logical, relevant step toward satisfying the intent of the task or not. Consequently, an allowed call may still deviate from the task's intent: a rogue agent might deviate the calls or nudge other agents to make a combination of calls that would not align with the intent of the task. Therefore, every call needs to be verified. In this study we investigate the applicability of Small Language Models (SLMs) to this purpose: an SLM functions as a task-tool relevance classifier that evaluates every selected tool independently against the assigned task and returns a relevance signal for downstream enforcement. Equipped with a novel dataset with multi-tool tasks whose required tools span distinct Model Context Protocol (MCP) servers, we used prompt-optimization, supervised fine-tuning, and reinforcement learning through GRPO to optimize and specialize SLMs.

Figures & tables

ModelMetricBaseGEPASFTGRPO
Gemma 3 1BFailed parses ↓\downarrow103—1751
E2E Acc. ↑\uparrow60.94 [59.97, 61.91]—78.63 [77.80, 79.42]89.60 [88.99, 90.18]
Accuracy ↑\uparrow61.59 [60.64, 62.59]—80.06 [79.26, 80.87]89.60 [88.99, 90.20]
Precision ↑\uparrow72.53 [71.30, 73.69]—94.54 [93.83, 95.16]94.61 [94.01, 95.19]
Recall ↑\uparrow61.80 [60.56, 63.04]—72.13 [70.99, 73.25]88.39 [87.55, 89.16]
F1F_{1} ↑\uparrow66.74 [65.72, 67.75]—81.82 [81.01, 82.63]91.39 [90.85, 91.91]
Table 1: Task-tool relevance classification and end-to-end performance on the held-out test set for Gemma 3 4B and Gemma 3 1B. Scores are percentages; brackets give 9595 % confidence intervals. Bold indicates the best result for each metric.
Figure 1: False-positive rate (FPR) and false-negative rate (FNR) on the held-out test set ( 97369736 samples) for Gemma 3 1B and Gemma 3 4B across successive stages of the specialization pipeline: Base, GEPA, SFT, and GRPO. GRPO achieves the lowest rates for Gemma 3 4B with FPR of 5.425.42 % and FNR of 2.942.94 %. Because the stages were applied sequentially, the results represent cumulative improvements rather than independent ablations. For Gemma 3 1B, an em dash indicates that GEPA produced no validation improvement and therefore no separate configuration was selected for test evaluation.

Explore similar work

CardsList

  1. Measuring the Microtask Eligibility Gap: When Is an Off-the-Shelf SLM Enough for an Agent Harness?

    Sep 1, 2026Jundong Hu, Shekar RamachandranSmall Language ModelsLLM Agent Harnesses

    PayPal AI

  2. Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents

    Apr 6, 2026Yizhou Liu, Qi Sun, Yulin Chen +2Agentic SearchSmall Language Models

    University of Illinois Urbana-Champaign · New York University · Nanyang Technological University

  3. How Do Agentic LLMs Decide to Call Tools? A Tool-Call Vector Shaped by Suppression

    Oct 7, 2026Xijie Gong, Tingxu Han, Jiahao Zhang +5LLM Tool UseLLM Decision-Making

    Mohamed bin Zayed University of Artificial Intelligence · University of Electronic Science and Technology of China · Nanjing University +3