MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment

Hanxian Huang; Igor Fedorov; Andrey Gromov; Bernard Beckerman; Naveen Suda; David Eriksson; Maximilian Balandat; Rylan Conway

cs.LGMar 16, 2026

MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment

Authors: Hanxian Huang, Igor Fedorov, Andrey Gromov, Bernard Beckerman, Naveen Suda, David Eriksson, Maximilian Balandat, Rylan Conway,

Organizations: Meta AI

Abstract

Real-time AI experiences call for on-device large language models (OD-LLMs) optimized for efficient deployment on resource-constrained hardware. The most useful OD-LLMs produce near-real-time responses and exhibit broad hardware compatibility, maximizing user reach. We present a methodology for designing such models using hardware-in-the-loop architecture search under mobile latency constraints. This system is amenable to industry-scale deployment: it generates models deployable without custom kernels and compatible with standard mobile runtimes like Executorch. Our methodology avoids specialized attention mechanisms and instead uses attention skipping for long-context acceleration. Our approach jointly optimizes model architecture (layers, dimensions) and attention pattern. To efficiently evaluate candidates, we treat each as a pruned version of a pretrained backbone with inherited weights, thereby achieving high accuracy with minimal continued pretraining. We leverage the low cost of latency evaluation in a staged process: learning an accurate latency model first, then searching for the Pareto-frontier across latency and quality. This yields MobileLLM-Flash, a family of foundation models (350M, 650M, 1.4B) for efficient on-device use with strong capabilities, supporting up to 8k context length. MobileLLM-Flash delivers up to 1.8x and 1.6x faster prefill and decode on mobile CPUs with comparable or superior quality. Our analysis of Pareto-frontier design choices offers actionable principles for OD-LLM design.

Figures & tables

Figure 1: Comparison between MobileLLM-Flash and state-of-the-art OD-LLMs. MobileLLM-Flash achieves up to $1.8\times/1.6\times$ faster prefill/decode on mobile CPUs with superior accuracy than LFM2 Amini et al. (2025) . Evaluation details are in Sec. 4.2 .

Feature	MobileLLM-Flash	LFM2 Amini et al. (2025)	Jet-Nemotron Gu et al. (2025) , Nemotron-Flash Fu et al. (2025)
Directly optimizes mobile CPU latency	✓	✓	✗
Unified architecture (shape + attention) search	✓	✗	✗
Efficient pruning-based search	✓	✗	✗
Compatible with any hardware / runtime	✓	✓	✗

Table 1: Comparison of methodologies.

Figure 2: Overview of our two-stage OD-LLM design. We jointly search the architecture and attention pattern by pruning a pretrained model $a_{0}$ (Sec. 3.3 ) using Bayesian Optimization (BO) with Ax. In Stage 1, we sample pruned architectures and measure their latency on phone to cheaply learn a latency model. In Stage 2, leveraging the latency model, we efficiently search the space to generate the Accuracy-Latency Pareto-frontier (Sec. 3.4 ).

Figure 3: A latency-loss Pareto-frontier

Parameters	Parameter Choices
$d_{\textrm{L}}$	$\{10,11,12,13,14,15,16\}$
$d_{\textrm{ffn}}$	$\{2048,2304,2560,\ldots 8192\}$
$d_{\textrm{model}}$	$\{1024,1152,1280\ldots 2048\}$
$p_{i}$	{full_attn, SWA, skip_attn}

Table 3: Search space

\mathcal{S}

Figure 5: A Pareto curve with different model depths.

	TQA	NQ	WinoG
With >3 consecutive skip attention	8.8%	2.5%	59.7%
Without >3 consecutive skip attention	33.2%	10.0%	61.2%

Table 4: Candidates with identical latency / identical architecture and skip attention counts can differ in performance due to the consecutive attention patterns.

Model (Parameter Count)	HellaSwag	BoolQ	PIQA	SocialIQA	TQA	NQ	ARC-c	ARC-e	WinoG	Avg $\uparrow$	Prefill TTFT(s) $\downarrow$			Decode rate (tok/s) $\uparrow$
	Zellers et al. (2019)	Clark et al. (2019)	Bisk et al. (2019)	Sap et al. (2019)	Joshi et al. (2017)	Kwiatkowski et al. (2019)	Clark et al. (2018)	Clark et al. (2018)	Sakaguchi et al. (2021)
											1k	2k	4k	1k	2k	4k
Gemma3 270M Team et al. (2025)	41.38	58.17	68.34	39.71	15.4	4.04	29.0	57.32	53.59	40.77	0.99	3.13	5.64	123.71	82.85	52.40
LFM2 350M Amini et al. (2025)	49.00	64.37	69.48	35.01	14.97	4.96	44.54	66.04	55.96	44.92	0.84	2.18	4.52	147.54	96.90	63.60
MobileLLM-Flash 350M	49.16	62.39	70.08	43.50	19.84	5.90	38.89	63.26	56.09	45.46	0.91	2.78	5.33	165.56	112.58	95.55
Qwen3 0.6B Yang et al. (2025a)	53.80	69.39	69.86	43.14	2.41	4.32	38.57	58.08	58.88	44.27	4.63	11.59	25.40	44.56	28.67	18.48

Table 6: Comparison of model quality and efficiency performance across various on-device scale models

	MMLU	MBPP	HumanEval	Open Rewrite	TLDR9+
Gemma3-1B	29.90	35.20	41.50	–	–
Llama3.2-1B	49.30	39.60	37.80	41.60	16.80
MobileLLM-Flash 650M	35.37	33.00	45.12	46.84	14.93
MobileLLM-Flash 1.4B	47.89	35.60	46.34	40.10	16.89

Table 7: Comparison of downstream task results

TTFT 1k (s) $\downarrow$	S25	iPhone 17
LFM2 350M	0.84	1.47
MobileLLM-Flash 350M	0.91	1.55
LFM2 700M	2.05	3.40
MobileLLM-Flash 650M	1.62	2.81
LFM2 1.2B	3.46	5.51
MobileLLM-Flash 1.4B	3.40	4.96

Table 8: MobileLLM-Flash maintains competitive or superior TTFT across devices.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Kendall Tau correlation between TTFT at 2k sequence length vs. model parameter count and FLOPs.

Figure 7: Kendall Tau correlation between TTFT at 1k sequence length vs. model parameter count and FLOPs.

Figure 8: Kendall Tau correlation between decode latency at 2k sequence length vs. model parameter count and FLOPs.

Figure 9: Kendall Tau correlation between decode latency at 1k sequence length vs. model parameter count and FLOPs.

On-Device Language Model Inference

On-device language model inference runs language models locally on phones, computers, and edge devices under tight memory, power, and latency limits.

See all 164 papers

Lever: Speculative LLM Inference on Smartphones
May 16, 2026Tuowei Wang, Fengzu Li, Yanfan Sun +2On-Device Language Model Inference LLM Inference
Tsinghua University · Beijing, China · Beihang University +1
MobileMoE: Scaling On-Device Mixture of Experts
May 26, 2026Yanbei Chen, Hanxian Huang, Ernie Chang +5Mixture-of-Experts Language Models On-Device Language Model Inference
Meta AI
IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligence
Sep 29, 2026Changdi Yang, Fengquan Jiao, Haochih Lin +14Multi-Token Prediction On-Device Language Model Inference
Robotics Foundation Model Team, Xpeng Inc.

May 16, 2026J/K move · Enter open · S save

Lever: Speculative LLM Inference on Smartphones

Tuowei Wang, Fengzu Li, Yanfan Sun, Wei Gao+1

Tsinghua University · Beijing, China · Beihang University+1

On-Device Language Model Inference LLM Inference Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly needed for interactive mobile applications, but high-quality models exceed the limited DRAM available on smartphones. Flash storage can hold larger models, yet flash-backed inference is slow because autoregressive decoding repeatedly invokes the target model and incurs costly I/O. We observe that speculative decoding is a natural fit for this setting: a small draft model can remain in DRAM, while a larger flash-resident target model verifies multiple candidate tokens per invocation. However, existing methods assume server-class accelerators and fail to account for prolonged I/O latency, limited computation parallelism, and irregular speculation execution. We present Lever, an end-to-end system for efficient flash-backed LLM inference on smartphones. Lever jointly optimizes the three stages of speculative decoding under mobile constraints. For drafting, it builds token trees using an I/O- and compute-aware gain-cost objective. For verification, it prunes low-value branches through early-exit prediction to reduce target-model computation. For execution, it maps speculation efficiently across mobile CPU-NPU hardware to improve utilization. Comprehensive evaluations show that Lever reduces inference latency by an average of 2.93x over baseline flash-offloaded inference and 1.50x over conventional speculative decoding, narrowing the latency gap between flash-backed and memory-resident LLM inference.

Abstract

On-Device Language Model Inference

Explore similar work

Lever: Speculative LLM Inference on Smartphones

MobileMoE: Scaling On-Device Mixture of Experts

IronLLM: Forging Compact Edge-Native Language Models for Real-Time Embodied Intelligence