Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge. Each model may require coordinated changes across conversion, graph construction, operators, and hardware backends. Existing coding agents often miss these cross-layer dependencies and struggle to diagnose failures across host--device execution paths. We present EngineAgent, an agentic system for automating on-device LLM model enablement. Its key insight is to separate reusable engine and hardware knowledge from model-specific implementation. Before enablement, EngineAgent constructs a hierarchy-aware engine index that captures engine abstraction layers and cross-layer dependencies, and pre-validates backend-specific build, execution, and debugging workflows using already supported models. For each new model, a builder uses this knowledge to coordinate cross-layer changes, while an auditor checks implementation completeness and drives iterative debugging through device-side validation. Across 12 tasks spanning three mobile LLMs and four engine--backend pairs on CPUs, GPUs, and NPUs, EngineAgent achieves 75% end-to-end success with either of two underlying LLMs, versus 33.3% for the strongest coding-agent baseline. Success requires compilation, on-device execution, and output-logit agreement with reference implementations. EngineAgent also reduces average per-task time and model-API cost relative to baselines using the same underlying LLMs.
Figures & tables
Figure 1. Position and workflow of EngineAgent: agentically mapping rapidly evolving LLM architectures onto heterogeneous mobile hardware for inference engines.
Model
Release
Engine
First Enablement
Backend
Qwen3.5-2B
2026-03-02
ExecuTorch
04-01 / 30 days
CPU (text-only)
MNN
03-02 / 0 day
CPU / GPU / NPU
llama.cpp
03-02 / 0 day
CPU
MLC-LLM
03-08 / 6 days
GPU
Gemma4-E2B-it
2026-04-02
ExecuTorch
N.A.
unsupported
MNN
04-09 / 7 days
CPU
Table 1. Model enablement 2 2 2 The enablement date is defined as the date first verified releases or pull requests become public available. backend coverage and delay across on-device LLM inference engines ( Foundation, 2025 ; Lv et al., 2022 ; Gerganov, 2026 ; MLC team, 2023 ) .
Figure 2. On-device LLM engines comprise export and execution stages, which share three abstraction layers.
Figure 3. Model enablement workloads: mapping new model features to engine abstraction layers and then perform host-device build-verify-debug loop.
Engine Backend
Agent
Failed Stage
Failure Category
Symptom
MNN CPU
CodePlan
Planning
Missing implementation
Localized config changes but missed dependent vision and runtime implementations.
SWE-Agent
Implementation
Missing implementation
Modified model components but missed export–runtime propagation.
Codex
Verification
Premature acceptance
Stopped after incomplete build and skipped device validation.
llama.cpp Vulkan
OpenHands
Backend integration
Backend failure
Failed to resolve Vulkan compilation and deployment.
OpenCode
Backend integration
Backend failure
Host adaptation completed but Vulkan execution path remained incomplete.
ExecuTorch QNN
CodePlan
Planning
Missing implementation
Followed local dependencies but missed QNN export.
Table 2. Representative failure cases of existing coding agents during on-device LLM model enablement.
Figure 4. Two-stage workflow of EngineAgent, from engine-specific knowledge construction to model-dependent enablement and verification.
Figure 5. CodeGraph ( McHenry, 2026 ) only captures syntactic structures rather than hierarchical understandin gs. Extracted relations in MNN CodeGraph as an example are illustrated on the left.
Figure 6. Hierarchy-aware engine modeling for MNN. The original CodeGraph (center) is organized into abstraction-layer subgraphs for export (top, L1–L8) and execution (bottom, L1–L9), with layer definitions shown on the left. Colored subgraphs group code entities by engine responsibility, while gray edges indicate cross-layer dependencies.
Figure 7. Instantiation subgraphs within MNN’s execution-stage hierarchy, exposing modality- and backend-specific code structures within each layer.
Figure 8. Examples of generated unit tests and debug skills for MNN-CPU execution stage by Deepseek-V4-Flash.
Agent Model API
Agentic Systems
E2E Success Rate
Average
Time
Cost ($)
DeepSeek V4 Flash
SWE-Agent
0/12
4.85h
2.11
OpenCode
1/12
4.78h
2.28
CodeGraph + OpenCode + AgenTracer
1/12
4.91h
2.44
EngineAgent
9/12
3.82h
1.53
GPT-5.6 Sol
Codex
4/12
4.82h
14.90
Table 3. Overall model enablement performance across the 12 benchmark cases.
Agent Model API
Agentic Systems
Stage-wise Success Rate
Export
Host
Device
T
MM
T
MM
DeepSeek V4 Flash
SWE-Agent
4/12
3/12
0/12
1/12
0/12
OpenCode
4/12
4/12
2/12
1/12
1/12
CodeGraph + OpenCode + AgenTracer
4/12
4/12
4/12
1/12
1/12
EngineAgent
10/12
10/12
10/12
9/12
9/12
Table 4. Stage-wise model enablement success across the 12 benchmark cases. (T: Text-only backbone success rate on host/device, MM: Multi-Modal success rate on host/device)
Figure 9. Failure breakdown across the 12 benchmark cases.
Figure 10. Progress Comparison of Agents enabling Qwen3.5-2B in llama.cpp-Vulkan.
Figure 11. Progress Comparison of Agents enabling MiniCPM-V 4.6 in ExecuTorch-QNN.
Engine
Snapshot
Index Cost ($)
Skills Cost ($)
MNN
f41d4638
1.04
1.83
llama.cpp
e06088da
0.66
2.27
MLC-LLM
84050697
0.77
2.40
ExecuTorch
b7781552
1.25
2.68
Table 5. Initial construction cost of pre-enablement artifacts on four engine snapshots.
Commit Update
Index Update Cost ($)
Skills Update Cost ($)
f41d4638 → c5903082
0.0041
0.0000
c5903082 → 0bff03cb
0.0195
0.1262
0bff03cb → 17c00ba6
0.0051
0.0000
17c00ba6 → 4419e33b
0.0047
0.0507
4419e33b → 30dfb9bb
0.0049
0.1459
Average
0.0077
0.0646
Table 6. Update cost of pre-constructed artifacts across MNN versions.
Figure 12. Ablation study of the hierarchy-aware index and hardware-aware skills.
LLM-based agents deliver state-of-the-art performance across tasks but incur high end-to-end latency on edge devices. We introduce Agent-X, a software-only, accuracy-preserving framework that accelerates both the prefill and decode stages of on-device agent workloads. Agent-X's two key components rewrite prompts to leverage prefix caching tailored to agent-specific input-token patterns and enable LLM-free speculative decoding for fast token generation with minimal overhead. On representative agentic workloads, Agent-X achieves a 1.61x end-to-end speedup in real systems with no accuracy loss and can be seamlessly integrated into existing on-device AI agents. To the best of our knowledge, ours is the first to systematically characterize and eliminate latency bottlenecks in on-device agents.
Large language models (LLMs) on mobile devices can act as agents that call external software tools to complete complex tasks, but the major challenge of supporting on-device mobile agentic AI with LLMs is the long latency of LLM tool calling on mobile devices. In particular, the decode phase is the main contributor of such long latency, but is hard to speed up on mobile processors as it is bound by memory bandwidth rather than compute power. To address this limitation, in this paper we present SchemaWeaver, a new on-device mobile agentic AI technique that enforces schema-guided parallel decoding in the decode phase of LLM tool calling, to reduce its latency by producing multiple tokens in each decode pass. SchemaWeaver can be generically applied to the LLM inference engine on mobile devices without involving any training efforts or changes on LLMs, and our experiment results on major mobile SoCs and LLM families show that SchemaWeaver can reduce the LLM tool calling latency by up to 3.9 times, compared to the competitive baselines.
Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce near-real-time responses and exhibit broad hardware compatibility, maximizing user reach. We present a methodology for designing such models using hardware-in-the-loop architecture search under mobile latency constraints. This system is amenable to industry-scale deployment: it generates models deployable without custom kernels and compatible with standard mobile runtimes like Executorch. Our methodology avoids specialized attention mechanisms and instead uses attention skipping for long-context acceleration. Our approach jointly optimizes model architecture (layers, dimensions) and attention pattern. To efficiently evaluate candidates, we treat each as a pruned version of a pretrained backbone with inherited weights, thereby achieving high accuracy with minimal continued pretraining. We leverage the low cost of latency evaluation in a staged process: learning an accurate latency model first, then searching for the Pareto-frontier across latency and quality. This yields MobileLLM-Flash, a family of foundation models (350M, 650M, 1.4B) for efficient on-device use with strong capabilities, supporting up to 8k context length. MobileLLM-Flash delivers up to 1.8x and 1.6x faster prefill and decode on mobile CPUs with comparable or superior quality. Our analysis of Pareto-frontier design choices offers actionable principles for OD-LLM design.