Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce near-real-time responses and exhibit broad hardware compatibility, maximizing user reach. We present a methodology for designing such models using hardware-in-the-loop architecture search under mobile latency constraints. This system is amenable to industry-scale deployment: it generates models deployable without custom kernels and compatible with standard mobile runtimes like Executorch. Our methodology avoids specialized attention mechanisms and instead uses attention skipping for long-context acceleration. Our approach jointly optimizes model architecture (layers, dimensions) and attention pattern. To efficiently evaluate candidates, we treat each as a pruned version of a pretrained backbone with inherited weights, thereby achieving high accuracy with minimal continued pretraining. We leverage the low cost of latency evaluation in a staged process: learning an accurate latency model first, then searching for the Pareto-frontier across latency and quality. This yields MobileLLM-Flash, a family of foundation models (350M, 650M, 1.4B) for efficient on-device use with strong capabilities, supporting up to 8k context length. MobileLLM-Flash delivers up to 1.8x and 1.6x faster prefill and decode on mobile CPUs with comparable or superior quality. Our analysis of Pareto-frontier design choices offers actionable principles for OD-LLM design.
Figures & tables
Figure 1: Comparison between MobileLLM-Flash and state-of-the-art OD-LLMs. MobileLLM-Flash achieves up to 1.8×/1.6× faster prefill/decode on mobile CPUs with superior accuracy than LFM2 Amini et al. (2025) . Evaluation details are in Sec. 4.2 .
Feature
MobileLLM-Flash
LFM2 Amini et al. (2025)
Jet-Nemotron Gu et al. (2025) , Nemotron-Flash Fu et al. (2025)
Directly optimizes mobile CPU latency
✓
✓
✗
Unified architecture (shape + attention) search
✓
✗
✗
Efficient pruning-based search
✓
✗
✗
Compatible with any hardware / runtime
✓
✓
✗
Table 1: Comparison of methodologies.
Figure 2: Overview of our two-stage OD-LLM design. We jointly search the architecture and attention pattern by pruning a pretrained model a0 (Sec. 3.3 ) using Bayesian Optimization (BO) with Ax. In Stage 1, we sample pruned architectures and measure their latency on phone to cheaply learn a latency model. In Stage 2, leveraging the latency model, we efficiently search the space to generate the Accuracy-Latency Pareto-frontier (Sec. 3.4 ).
Figure 3: A latency-loss Pareto-frontier
Figure 4: Pruned model loss evolution.
Parameters
Parameter Choices
dL
{10,11,12,13,14,15,16}
dffn
{2048,2304,2560,…8192}
dmodel
{1024,1152,1280…2048}
pi
{full_attn, SWA, skip_attn}
Table 3: Search space S .
Figure 5: A Pareto curve with different model depths.
TQA
NQ
WinoG
With >3 consecutive skip attention
8.8%
2.5%
59.7%
Without >3 consecutive skip attention
33.2%
10.0%
61.2%
Table 4: Candidates with identical latency / identical architecture and skip attention counts can differ in performance due to the consecutive attention patterns.
Model (Parameter Count)
HellaSwag
BoolQ
PIQA
SocialIQA
TQA
NQ
ARC-c
ARC-e
WinoG
Avg ↑
Prefill TTFT(s) ↓
Decode rate (tok/s) ↑
Zellers et al. (2019)
Clark et al. (2019)
Bisk et al. (2019)
Sap et al. (2019)
Joshi et al. (2017)
Kwiatkowski et al. (2019)
Clark et al. (2018)
Clark et al. (2018)
Sakaguchi et al. (2021)
1k
2k
4k
1k
2k
4k
Gemma3 270M Team et al. (2025)
41.38
58.17
68.34
39.71
15.4
4.04
29.0
57.32
53.59
40.77
0.99
3.13
5.64
123.71
82.85
52.40
LFM2 350M Amini et al. (2025)
49.00
64.37
69.48
35.01
14.97
4.96
44.54
66.04
55.96
44.92
0.84
2.18
4.52
147.54
96.90
63.60
MobileLLM-Flash 350M
49.16
62.39
70.08
43.50
19.84
5.90
38.89
63.26
56.09
45.46
0.91
2.78
5.33
165.56
112.58
95.55
Qwen3 0.6B Yang et al. (2025a)
53.80
69.39
69.86
43.14
2.41
4.32
38.57
58.08
58.88
44.27
4.63
11.59
25.40
44.56
28.67
18.48
Table 6: Comparison of model quality and efficiency performance across various on-device scale models
MMLU
MBPP
HumanEval
Open Rewrite
TLDR9+
Gemma3-1B
29.90
35.20
41.50
–
–
Llama3.2-1B
49.30
39.60
37.80
41.60
16.80
MobileLLM-Flash 650M
35.37
33.00
45.12
46.84
14.93
MobileLLM-Flash 1.4B
47.89
35.60
46.34
40.10
16.89
Table 7: Comparison of downstream task results
TTFT 1k (s) ↓
S25
iPhone 17
LFM2 350M
0.84
1.47
MobileLLM-Flash 350M
0.91
1.55
LFM2 700M
2.05
3.40
MobileLLM-Flash 650M
1.62
2.81
LFM2 1.2B
3.46
5.51
MobileLLM-Flash 1.4B
3.40
4.96
Table 8: MobileLLM-Flash maintains competitive or superior TTFT across devices.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Kendall Tau correlation between TTFT at 2k sequence length vs. model parameter count and FLOPs.
Figure 7: Kendall Tau correlation between TTFT at 1k sequence length vs. model parameter count and FLOPs.
Figure 8: Kendall Tau correlation between decode latency at 2k sequence length vs. model parameter count and FLOPs.
Figure 9: Kendall Tau correlation between decode latency at 1k sequence length vs. model parameter count and FLOPs.
Large language models (LLMs) are increasingly needed for interactive mobile applications, but high-quality models exceed the limited DRAM available on smartphones. Flash storage can hold larger models, yet flash-backed inference is slow because autoregressive decoding repeatedly invokes the target model and incurs costly I/O. We observe that speculative decoding is a natural fit for this setting: a small draft model can remain in DRAM, while a larger flash-resident target model verifies multiple candidate tokens per invocation. However, existing methods assume server-class accelerators and fail to account for prolonged I/O latency, limited computation parallelism, and irregular speculation execution. We present Lever, an end-to-end system for efficient flash-backed LLM inference on smartphones. Lever jointly optimizes the three stages of speculative decoding under mobile constraints. For drafting, it builds token trees using an I/O- and compute-aware gain-cost objective. For verification, it prunes low-value branches through early-exit prediction to reduce target-model computation. For execution, it maps speculation efficiently across mobile CPU-NPU hardware to improve utilization. Comprehensive evaluations show that Lever reduces inference latency by an average of 2.93x over baseline flash-offloaded inference and 1.50x over conventional speculative decoding, narrowing the latency gap between flash-backed and memory-resident LLM inference.
Tuowei Wang, Fengzu Li, Yanfan Sun +2
Tsinghua University · Beijing, China · Beihang University +1
Mixture-of-Experts (MoE) has become the de facto architecture for hundred-billion-parameter language models, yet its advantages at sub-billion scales for on-device deployment remain largely unexplored. To close this gap, we present MobileMoE, a family of on-device MoE language models with sub-billion active parameters (0.3-0.9B active and 1.3-5.3B total) that establish a new Pareto frontier for on-device LLMs. We first formulate an on-device MoE scaling law that jointly optimizes MoE architecture under mobile memory and compute constraints, identifying an on-device sweet spot - moderate sparsity with fine-grained and shared experts - that is simultaneously memory and compute-optimal. Building on the derived architectures, we train MobileMoE with a four-stage recipe covering pre-training, mid-training, instruction fine-tuning, and quantization-aware training, all on open-source datasets. Across 14 benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs with 2-4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters. To bridge the last mile to mobile deployment, we provide the first efficient MoE inference on commodity smartphones with comprehensive on-device profiling. At comparable INT4 weight memory, MobileMoE-S delivers 1.8-3.8× faster prefill and 2.2-3.4× faster decode than the dense baseline MobileLLM-Pro.
We present IronLLM-0.6B, a 654M-parameter language model designed for efficient on-device inference. IronLLM-0.6B combines a hybrid attention architecture with X-MTP, a lightweight shared-KV multi-token prediction design that eliminates per-depth KV-cache replay and employs a lightweight verification head for rollback-free drafting, achieving a 1.48x decoding speedup. The model is pretrained on approximately 6.2 trillion tokens using a quality-oriented data pipeline and is further post-trained with Multi-Domain On-Policy Distillation to integrate capabilities from domain-specialized teachers. To better meet the low-latency requirements of on-device scenarios, IronLLM-0.6B adopts an Instruct-Only design. Evaluations show that IronLLM-0.6B achieves competitive performance relative to larger models such as Qwen3.5-0.8B and MiniCPM5-1B, while producing more concise responses on many tasks. We further present IronLLM-0.6B-Light, which replaces RMSNorm with Dynamic Tanh and simplifies several computationally expensive components to improve inference and quantization efficiency. Together, the IronLLM models provide an effective performance-efficiency trade-off for resource-constrained deployment.