On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation

Moghis Fereidouni; Anthony Arnold; Sumit Gulwani; Mark Marron; A. B. Siddique

cs.AISep 27, 2026

On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation

Authors: Moghis Fereidouni, Anthony Arnold, Sumit Gulwani, Mark Marron, A. B. Siddique

Organizations: University of Kentucky · Microsoft

Abstract

Agentic AI is increasingly being embedded in software applications to provide natural language interfaces to features and functionality. In most cases these agents are powered by enterprise (100+ billion parameter) or frontier class large language models that require substantial computational resources run and depend on cloud hosted inference to handle the task of transforming natural language inputs into actionable software operations. This reliance on cloud-hosted inference introduces substantial network latency on top of LLM inference times, creates data privacy concerns, and, given the costs of running these models, can rapidly escalate expenses associated with supporting agentic features. This paper introduces a novel means of converting the NL-to-Action problem from a generative one into a classification-centric formulation via on-device operation caches. These caches allow an agentic system to handle frequently occurring classes of actions completely on-device -- reducing latency, enhancing privacy, and lowering operational costs. We show that for a classic NL-to-Formula task, generating Excel Formula in response to user requests, this approach reduces total inference cost by 56% when compared to cloud-only model-routing based inference and, on cache hits, reduces the latency to response latency by 5x.

Figures & tables

Figure 1. Interaction between on-device operation caches and agentic natural language integration. A spreadsheet sends a natural-language query and table context to an on-device semantic cache. A high-confidence cache hit returns a specialized formula locally; a miss routes the request to a general-purpose cloud agent.

Figure 2. On-device operation cache architecture. Natural-language input and table context enter an on-device cache containing parameterized operation idioms and a semantic matcher. The matcher binds formula parameters and applies a confidence threshold, returning either a specialized formula or a miss that triggers cloud fallback.

Query	Context	Operation
"Sum the values in column A"	{"headers": ["A", "B"], …]}	"SUM(A1:A10)"
"Total of Sales"	{"name": "Table1", "headers": ["Items", "Quantity", "Sales"], …]}	"SUM(Table1[Sales])"
"how mnay wins"	{"name": "Games", "headers": ["Opponent", "Win", "Loss"], …]}	"SUM(Games[Win])"
…	…	…
"Which item sold best in June"	{"name": "Sales", "headers": ["Item", "Jan", …], …]}	"INDEX(Sales[Item], MATCH(MAX(Sales[July]), Sales[July], 0))"

Figure 3. Example of the input data format for cache construction. Each entry consists of a natural language query, the relevant spreadsheet context, and the corresponding spreadsheet formula.

Cluster	$\alpha$ -Operation
$\begin{aligned} \left\{\begin{array}[]{ll}\text{"Sum the values in column A"}&\{"headers":["A","B"],\dots]\}\\ \text{"Total of Sales"}&\{"name":"Table1","headers":["Items","Quantity","Sales"],\dots]\}\\ \text{"how mnay wins"}&\{"name":"Games","headers":["Opponent","Win","Loss"],\dots]\}\\ \ldots&\ldots\\ \end{array}\right\}\end{aligned}$	"SUM( $\alpha$ )"
…	…
$\begin{aligned} \left\{\begin{array}[]{ll}\text{"Which item sold best in June"}&\{"name":"Sales","headers":["Item","Jan",\ldots],\dots]\}\\ \ldots&\ldots\\ \end{array}\right\}\end{aligned}$	"INDEX( $\alpha$ , MATCH(MAX( $\beta$ ), $\beta$ , 0))"

Figure 4. Idiom abstracted and clustered dataset corresponding to the input data format shown in Figure 3 .

	Cached + Cloud	Cloud Only
Hit Rate	55%	-
Cost (USD)	$0.67	$1.54
Correctness	97%(hit)/83%(miss)	85%
Avg. Latency (s)	0.31(hit)/1.6(miss)	1.5

Table 1. Cache performance comparison between using a local cache with cloud fall back and using cloud only. Hit rate indicates the fraction of queries served by the local cache. Cost is the total inference cost (in USD at API costs) to run all queries in the experiment. Avg. Latency indicates the average time taken to serve a query, with separate values for cache hits and misses – cloud only has a single value as all queries are served from the cloud.

Model	Correct Percentage
Gemma 3–1B	$1\%$
Gemma 3–4B	$6\%$
Gemma 3–12B	$17\%$
Gemma 3–27B	OOM

Table 2. Performance of local Gemma 3 instruction-tuned model variants on the NL2Formula benchmark. The correctness percentage is over all queries (full recall).

Figure 5. Impact of confidence gating on cache performance. Hit rate and correctness curves are shown for varying values of the confidence threshold.

Figure 6. Impact of noise on cache performance. Hit rate and correctness curves are shown for varying levels of noise in the natural language queries.

LLM Inference Efficiency

Large language model inference efficiency reduces the time, memory, energy, and cost of generating useful outputs.

See all 352 papers

TabAgent: Distilling LLM Agent Components into Cheap Classifiers
Feb 18, 2026Ido Levy, Segev Shlomov, Avi Yaeli +5LLM Agent Workflow Optimization AI Agent Architecture
IBM Research, Haifa, Israel
Agent-X: Full Pipeline Acceleration of On-device AI Agents
May 11, 2026Jinha Chung, Byeongjun Shin, Jiin Kim +1LLM Inference Efficiency On-Device Language Model Inference
KAIST Republic of Korea, Daejeon
Koa-decision: Fast and Consistent Structured Decision Making with Generative LLMs
Sep 28, 2026Shenghong Dai, Shiva Kumar Pentyala, Yingchi Liu +6LLM Decision-Making LLM Inference Acceleration
Salesforce AI · University of Wisconsin–Madison

Feb 18, 2026J/K move · Enter open · S save

TabAgent: Distilling LLM Agent Components into Cheap Classifiers

Ido Levy, Segev Shlomov, Avi Yaeli, Nir Mashkif+4

IBM Research, Haifa, Israel

LLM Agent Workflow Optimization AI Agent Architecture LLM Decision-Making Classification

Open paper AskPDF arXiv

Agentic systems, AI architectures that autonomously execute multi-step workflows to achieve complex goals, are often built using repeated large language model (LLM) calls for closed-set decision tasks such as routing, shortlisting, gating, and verification. While convenient, this design makes deployments slow and expensive due to cumulative latency and token usage. We propose TabAgent, a framework for replacing generative decision components in closed-set selection tasks with a compact textual-tabular classifier trained on execution traces. TabAgent (i) extracts structured schema, state, and dependency features from trajectories (TabSchema), (ii) augments coverage with schema-aligned synthetic supervision (TabSynth), and (iii) scores candidates with a lightweight classifier (TabHead). On the long-horizon AppWorld benchmark, TabAgent maintains task-level success while eliminating shortlist-time LLM calls, reducing latency by approximately 95% and inference cost by 85-91%. Beyond tool shortlisting, TabAgent generalizes to other agentic decision heads, establishing a paradigm for learned discriminative replacements of generative bottlenecks in production agent architectures.

Abstract

LLM Inference Efficiency

Explore similar work

TabAgent: Distilling LLM Agent Components into Cheap Classifiers

Agent-X: Full Pipeline Acceleration of On-device AI Agents

Koa-decision: Fast and Consistent Structured Decision Making with Generative LLMs