Rapidly evolving LLM architectures and heterogeneous mobile hardware make supporting new models in on-device inference engines a recurring and costly system challenge. Each model may require coordinated changes across conversion, graph construction, operators, and hardware backends. Existing coding agents often miss these cross-layer dependencies and struggle to diagnose failures across host--device execution paths. We present EngineAgent, an agentic system for automating on-device LLM model enablement. Its key insight is to separate reusable engine and hardware knowledge from model-specific implementation. Before enablement, EngineAgent constructs a hierarchy-aware engine index that captures engine abstraction layers and cross-layer dependencies, and pre-validates backend-specific build, execution, and debugging workflows using already supported models. For each new model, a builder uses this knowledge to coordinate cross-layer changes, while an auditor checks implementation completeness and drives iterative debugging through device-side validation. Across 12 tasks spanning three mobile LLMs and four engine--backend pairs on CPUs, GPUs, and NPUs, EngineAgent achieves 75% end-to-end success with either of two underlying LLMs, versus 33.3% for the strongest coding-agent baseline. Success requires compilation, on-device execution, and output-logit agreement with reference implementations. EngineAgent also reduces average per-task time and model-API cost relative to baselines using the same underlying LLMs.
Figures & tables
Figure 1. Position and workflow of EngineAgent: agentically mapping rapidly evolving LLM architectures onto heterogeneous mobile hardware for inference engines.
Model
Release
Engine
First Enablement
Backend
Qwen3.5-2B
2026-03-02
ExecuTorch
04-01 / 30 days
CPU (text-only)
MNN
03-02 / 0 day
CPU / GPU / NPU
llama.cpp
03-02 / 0 day
CPU
MLC-LLM
03-08 / 6 days
GPU
Gemma4-E2B-it
2026-04-02
ExecuTorch
N.A.
unsupported
MNN
04-09 / 7 days
CPU
Table 1. Model enablement 2 2 2 The enablement date is defined as the date first verified releases or pull requests become public available. backend coverage and delay across on-device LLM inference engines ( Foundation, 2025 ; Lv et al., 2022 ; Gerganov, 2026 ; MLC team, 2023 ) .
Figure 2. On-device LLM engines comprise export and execution stages, which share three abstraction layers.
Figure 3. Model enablement workloads: mapping new model features to engine abstraction layers and then perform host-device build-verify-debug loop.
Engine Backend
Agent
Failed Stage
Failure Category
Symptom
MNN CPU
CodePlan
Planning
Missing implementation
Localized config changes but missed dependent vision and runtime implementations.
SWE-Agent
Implementation
Missing implementation
Modified model components but missed export–runtime propagation.
Codex
Verification
Premature acceptance
Stopped after incomplete build and skipped device validation.
llama.cpp Vulkan
OpenHands
Backend integration
Backend failure
Failed to resolve Vulkan compilation and deployment.
OpenCode
Backend integration
Backend failure
Host adaptation completed but Vulkan execution path remained incomplete.
ExecuTorch QNN
CodePlan
Planning
Missing implementation
Followed local dependencies but missed QNN export.
Table 2. Representative failure cases of existing coding agents during on-device LLM model enablement.
Figure 4. Two-stage workflow of EngineAgent, from engine-specific knowledge construction to model-dependent enablement and verification.
Figure 5. CodeGraph ( McHenry, 2026 ) only captures syntactic structures rather than hierarchical understandin gs. Extracted relations in MNN CodeGraph as an example are illustrated on the left.
Figure 6. Hierarchy-aware engine modeling for MNN. The original CodeGraph (center) is organized into abstraction-layer subgraphs for export (top, L1–L8) and execution (bottom, L1–L9), with layer definitions shown on the left. Colored subgraphs group code entities by engine responsibility, while gray edges indicate cross-layer dependencies.
Figure 7. Instantiation subgraphs within MNN’s execution-stage hierarchy, exposing modality- and backend-specific code structures within each layer.
Figure 8. Examples of generated unit tests and debug skills for MNN-CPU execution stage by Deepseek-V4-Flash.
Agent Model API
Agentic Systems
E2E Success Rate
Average
Time
Cost ($)
DeepSeek V4 Flash
SWE-Agent
0/12
4.85h
2.11
OpenCode
1/12
4.78h
2.28
CodeGraph + OpenCode + AgenTracer
1/12
4.91h
2.44
EngineAgent
9/12
3.82h
1.53
GPT-5.6 Sol
Codex
4/12
4.82h
14.90
Table 3. Overall model enablement performance across the 12 benchmark cases.
Agent Model API
Agentic Systems
Stage-wise Success Rate
Export
Host
Device
T
MM
T
MM
DeepSeek V4 Flash
SWE-Agent
4/12
3/12
0/12
1/12
0/12
OpenCode
4/12
4/12
2/12
1/12
1/12
CodeGraph + OpenCode + AgenTracer
4/12
4/12
4/12
1/12
1/12
EngineAgent
10/12
10/12
10/12
9/12
9/12
Table 4. Stage-wise model enablement success across the 12 benchmark cases. (T: Text-only backbone success rate on host/device, MM: Multi-Modal success rate on host/device)
Figure 9. Failure breakdown across the 12 benchmark cases.
Figure 10. Progress Comparison of Agents enabling Qwen3.5-2B in llama.cpp-Vulkan.
Figure 11. Progress Comparison of Agents enabling MiniCPM-V 4.6 in ExecuTorch-QNN.
Engine
Snapshot
Index Cost ($)
Skills Cost ($)
MNN
f41d4638
1.04
1.83
llama.cpp
e06088da
0.66
2.27
MLC-LLM
84050697
0.77
2.40
ExecuTorch
b7781552
1.25
2.68
Table 5. Initial construction cost of pre-enablement artifacts on four engine snapshots.
Commit Update
Index Update Cost ($)
Skills Update Cost ($)
f41d4638 → c5903082
0.0041
0.0000
c5903082 → 0bff03cb
0.0195
0.1262
0bff03cb → 17c00ba6
0.0051
0.0000
17c00ba6 → 4419e33b
0.0047
0.0507
4419e33b → 30dfb9bb
0.0049
0.1459
Average
0.0077
0.0646
Table 6. Update cost of pre-constructed artifacts across MNN versions.
Figure 12. Ablation study of the hierarchy-aware index and hardware-aware skills.