Agentic AI is increasingly being embedded in software applications to provide natural language interfaces to features and functionality. In most cases these agents are powered by enterprise (100+ billion parameter) or frontier class large language models that require substantial computational resources run and depend on cloud hosted inference to handle the task of transforming natural language inputs into actionable software operations. This reliance on cloud-hosted inference introduces substantial network latency on top of LLM inference times, creates data privacy concerns, and, given the costs of running these models, can rapidly escalate expenses associated with supporting agentic features. This paper introduces a novel means of converting the NL-to-Action problem from a generative one into a classification-centric formulation via on-device operation caches. These caches allow an agentic system to handle frequently occurring classes of actions completely on-device -- reducing latency, enhancing privacy, and lowering operational costs. We show that for a classic NL-to-Formula task, generating Excel Formula in response to user requests, this approach reduces total inference cost by 56% when compared to cloud-only model-routing based inference and, on cache hits, reduces the latency to response latency by 5x.
Figures & tables
Figure 1. Interaction between on-device operation caches and agentic natural language integration. A spreadsheet sends a natural-language query and table context to an on-device semantic cache. A high-confidence cache hit returns a specialized formula locally; a miss routes the request to a general-purpose cloud agent.
Figure 2. On-device operation cache architecture. Natural-language input and table context enter an on-device cache containing parameterized operation idioms and a semantic matcher. The matcher binds formula parameters and applies a confidence threshold, returning either a specialized formula or a miss that triggers cloud fallback.
Figure 3. Example of the input data format for cache construction. Each entry consists of a natural language query, the relevant spreadsheet context, and the corresponding spreadsheet formula.
Cluster
α -Operation
\begin{aligned} \left\{\begin{array}[]{ll}\text{"Sum the values in column A"}&\{"headers":["A","B"],\dots]\}\\ \text{"Total of Sales"}&\{"name":"Table1","headers":["Items","Quantity","Sales"],\dots]\}\\ \text{"how mnay wins"}&\{"name":"Games","headers":["Opponent","Win","Loss"],\dots]\}\\ \ldots&\ldots\\ \end{array}\right\}\end{aligned}
"SUM( α )"
…
…
\begin{aligned} \left\{\begin{array}[]{ll}\text{"Which item sold best in June"}&\{"name":"Sales","headers":["Item","Jan",\ldots],\dots]\}\\ \ldots&\ldots\\ \end{array}\right\}\end{aligned}
"INDEX( α , MATCH(MAX( β ), β , 0))"
Figure 4. Idiom abstracted and clustered dataset corresponding to the input data format shown in Figure 3 .
Cached + Cloud
Cloud Only
Hit Rate
55%
-
Cost (USD)
$0.67
$1.54
Correctness
97%(hit)/83%(miss)
85%
Avg. Latency (s)
0.31(hit)/1.6(miss)
1.5
Table 1. Cache performance comparison between using a local cache with cloud fall back and using cloud only. Hit rate indicates the fraction of queries served by the local cache. Cost is the total inference cost (in USD at API costs) to run all queries in the experiment. Avg. Latency indicates the average time taken to serve a query, with separate values for cache hits and misses – cloud only has a single value as all queries are served from the cloud.
Model
Correct Percentage
Gemma 3–1B
1%
Gemma 3–4B
6%
Gemma 3–12B
17%
Gemma 3–27B
OOM
Table 2. Performance of local Gemma 3 instruction-tuned model variants on the NL2Formula benchmark. The correctness percentage is over all queries (full recall).
Figure 5. Impact of confidence gating on cache performance. Hit rate and correctness curves are shown for varying values of the confidence threshold.
Figure 6. Impact of noise on cache performance. Hit rate and correctness curves are shown for varying levels of noise in the natural language queries.
Agentic systems, AI architectures that autonomously execute multi-step workflows to achieve complex goals, are often built using repeated large language model (LLM) calls for closed-set decision tasks such as routing, shortlisting, gating, and verification. While convenient, this design makes deployments slow and expensive due to cumulative latency and token usage. We propose TabAgent, a framework for replacing generative decision components in closed-set selection tasks with a compact textual-tabular classifier trained on execution traces. TabAgent (i) extracts structured schema, state, and dependency features from trajectories (TabSchema), (ii) augments coverage with schema-aligned synthetic supervision (TabSynth), and (iii) scores candidates with a lightweight classifier (TabHead). On the long-horizon AppWorld benchmark, TabAgent maintains task-level success while eliminating shortlist-time LLM calls, reducing latency by approximately 95% and inference cost by 85-91%. Beyond tool shortlisting, TabAgent generalizes to other agentic decision heads, establishing a paradigm for learned discriminative replacements of generative bottlenecks in production agent architectures.
LLM-based agents deliver state-of-the-art performance across tasks but incur high end-to-end latency on edge devices. We introduce Agent-X, a software-only, accuracy-preserving framework that accelerates both the prefill and decode stages of on-device agent workloads. Agent-X's two key components rewrite prompts to leverage prefix caching tailored to agent-specific input-token patterns and enable LLM-free speculative decoding for fast token generation with minimal overhead. On representative agentic workloads, Agent-X achieves a 1.61x end-to-end speedup in real systems with no accuracy loss and can be seamlessly integrated into existing on-device AI agents. To the best of our knowledge, ours is the first to systematically characterize and eliminate latency bottlenecks in on-device agents.
Industry applications often demand low-latency classification, yet current large language model (LLM) approaches remain poorly suited for latency-critical applications. Existing prompting and constrained decoding produce verbose, multi-token outputs that require expensive token-by-token generation, while encoder-based models achieve faster inference but sacrifice task flexibility. We propose Koa-decision, a framework for low-latency atomic actions -- fast, single-step decisions such as classification, semantic endpointing, Boolean checks, and scoring -- formulated as constrained generation with single-token outputs. By introducing atomic label tokens and applying supervised fine-tuning, our method reduces classification to a deterministic one-step decoding problem. Across standard benchmarks, Koa-decision delivers competitive accuracy with consistently low and stable latency. On a production intent-routing benchmark, Koa-decision reaches 85.5% accuracy -- competitive with the strongest frontier models (Claude-4.8-Opus, Gemini-Pro-3.1) and ahead of GPT-5 and Gemini-2.5-Pro -- while answering in about half a second, several-fold faster than every frontier model (up to ~7.5x at the median) under identical serving conditions. Against the dedicated single-token system Jev/TypeSafe, Koa-decision is competitive on accuracy and faster at the median, while also handling multimodal inputs and multi-label outputs that single-label text systems do not.
Shenghong Dai, Shiva Kumar Pentyala, Yingchi Liu +6