Large language models (LLMs) have demonstrated remarkable capabilities in code generation, yet their potential for generating kernels specifically for mobile devices remains largely unexplored. In this work, we extend the scope of automated kernel generation to the mobile domain to investigate the central question: Can LLMs write efficient kernels for mobile devices? To enable systematic investigation, we introduce MobileKernelBench, a comprehensive evaluation framework comprising a benchmark prioritizing operator diversity and cross-framework interoperability, coupled with an automated pipeline that bridges the host-device gap for on-device verification. Leveraging this framework, we conduct extensive evaluation on the CPU backend of Mobile Neural Network (MNN), revealing that current LLMs struggle with the engineering complexity and data scarcity inherent to mobile frameworks; standard models and even fine-tuned variants exhibit high compilation failure rates (over 54%) and negligible performance gains due to hallucinations and a lack of domain-specific grounding. To overcome these limitations, we propose the Mobile Kernel Agent (MoKA), a multi-agent system equipped with repository-aware reasoning and a plan-and-execute paradigm. Validated on MobileKernelBench, MoKA achieves state-of-the-art performance, boosting compilation success to 93.7% and enabling 27.4% of generated kernels to deliver measurable speedups over native libraries.
Figures & tables
Figure 1: Performance evaluation on MobileKernelBench across three metrics: compilation success rate (CSR), functional correctness rate (FCR), and performance speedup (fast p ) [ 1 ] . (a) Baseline LLM performance: We benchmark prevalent open- and closed-source LLMs, revealing significant shortcomings in their ability to generate functional and efficient mobile kernels. (b) Method comparison : We compare our proposed MoKA against common training methods, including LoRA and GRPO. The red circle (marked at 50%) corresponds to the outer limit of plot (a), highlighting that MoKA achieves substantial improvements, surpassing the performance ceiling of both baseline models and naive fine-tuning approaches.
Dimension
Server Side
Mobile Side
Hardware
CPU, GPU, TPU, NPU ( ASIC )
CPU, GPU, NPU ( SoC )
Backend
CUDA, TensorRT, ROCm
CPU, OpenCL, Vulkan, CoreML, Metal
Language
CUDA, Triton, PTX
C++, Assembly
Feature
Homogeneous
Heterogeneous & Fragmented
Frameworks
PyTorch, TensorFlow, JAX
MNN, NCNN, TFLite, CANN
Workload
Training & Inference
Inference
Table 1: Comparison between server-side and mobile-side computing. Unlike the server side, which utilizes unified CUDA-based backends on high-performance GPUs, the mobile side faces a fragmented landscape with diverse compute backends primarily optimized for inference tasks.
Feature
KernelBench
MobileKernelBench (Ours)
Diversity
43 (Level 1)
95
Tasks
250
190
Format
PyTorch Models
(PyTorch, ONNX) Pairs
Domain
GPUs
Mobile SoCs (CPU, Adreno, etc.)
Target
Optimization ONLY
Framework Adaptation & Optimization.
Table 2: Comparison between KernelBench and MobileKernelBench. Compared to KernelBench, MobileKernelBench emphasizes operator diversity, covering the majority of common operator types in ONNX Opset 20. The (PyTorch, ONNX) pairs format facilitates kernel evaluation across different frameworks, targeting both framework adaptation and optimization on mobile devices.
Figure 2: Overview of our proposed framework. The system consists of two core components: (a) MobileKernelBench , which establishes the evaluation environment by integrating a target-driven data curation process with an automated, hardware-in-the-loop evaluation pipeline; and (b) MoKA , a multi-role agentic system where Coder, Debugger, and Accelerator agents collaborate to iteratively generate and refine kernels based on feedback from the benchmark.
Categories
Representatives
#Operators
#Tasks
Unary
exp, sign, ceil, floor
11
11
Binary
add, div, mod
7
8
Trigonometry
sin, acos, atanh, tan
12
12
Activation
hardsigmoid, softmax, celu, relu
6
12
Normalization
batchnorm, layernorm, instancenorm
3
12
Pooling
maxpool, averagepool
4
15
Table 3: Overview of MobileKernelBench. This benchmark comprises 190 tasks derived from 95 primitive operators. These operators are classified into 12 categories, encompassing common operators found in the ONNX ecosystem. A primitive operator may yield multiple distinct tasks based on differences in input shapes or parameter settings.
Figure 6
Figure 4: Fine-grained performance of LLMs across different operator categories. We visualize the evaluation metrics for five representative operator types. The results highlight significant disparities in model capabilities when handling operators with varying levels of algorithmic complexity.
Method
CSR
FCR
fast p
(%)
(%)
0.5
1.0
1.5
Supervised Fine-Tuning (SFT)
Qwen3-32B
16.7
13.9
8.3
2.8
0.0
Qwen3-32B-LoRA
25 (+8.3)
18.5 (+5.6)
11.1 (+2.8)
5.6 (+2.8)
0.0 (–)
Reinforcement Learning (RL)
Qwen3-4B-Instruct-2507
10.0
5.0
5.0
5.0
0.0
Table 5: Comprehensive performance evaluation on MobileKernelBench. We compare our proposed MoKA against standard prompting, SFT, and RL methods. CSR and FCR represent compilation success rate and functional correctness rate, respectively. Bold values indicate the best performance, and values in parentheses denote the absolute gains over the corresponding baselines.
Figure 5: Fine-grained performance of MoKA across different operator categories. The agent demonstrates robust generalization across diverse operator types, achieving high correctness on structurally simpler operations (e.g., activation, normalization) while maintaining competitive performance on complex tasks like convolution and matrix operations.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: MoKA optimize LayerNorm2D kernel process. We conduct the optimization process for 10 iterations. MoKA shows that LLMs can provide diverse optimization methods like SIMD and cache blocking, achieve remarkable speedups.
Epoch
Stage
Performance / ms
Baseline / ms
Speedup
1
compilation
–
0.409
–
2
optimization
0..466
0.409
0.88
3
optimization
0.199
0.409
2.06
4
optimization
0.193
0.409
2.12
5
optimization
0.09
0.409
4.54
6
optimization
0.37
0.409
1.11
Appendix
Table 6: Case study: LayerNorm2D. We demonstrate MoKA’s capability to automatically optimize kernel implementations through 10 iterative epochs. MoKA achieves a peak speedup of 6.82× at epoch 8, perform an average speedup of 2.82x.