Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19× less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25% of easy and moderate cases and in 60% of hard ones. Code is available at: https://github.com/evanly-gh/SLM_Factory.
Figures & tables
Figure 1. Mobile AI Factory. It takes a task specification and a budget, and builds a specialized model for a phone.
Figure 2. Qwen3-0.6B at 4 bits on four phones, two inference engines and two backends (CPU/GPU). The star marks the fastest option on each phone.
Figure 3. Each point is one model; the dashed line marks a tie. (a) GPU prefill throughput relative to the CPU on the same phone. (b) MNN decode throughput relative to llama.cpp on the CPU, at 4 bits. (c) 8-bit decode throughput relative to 4-bit on the CPU. Whether a rule holds depends on the phone.
Weights
llama.cpp
MNN
MNN
Q4_K_M
default
HQQ-32
SmolLM2-360M, fine-tuned
0.767
0.659
0.747
Qwen3.5-0.8B, fine-tuned
0.864
0.850
0.859
Qwen3-1.7B, fine-tuned
0.886
0.851
0.875
Qwen3-4B, fine-tuned
0.915
0.897
0.918
SmolLM2-360M, zero-shot
0.114
0.104
0.119
Table 1. Macro-F1 on CLINC150 for the same weights under three 4-bit recipes. All three are labelled 4-bit. Run-to-run noise is about ± 0.015.
Phone
Model
Engine, backend
F1
Budget: 1 s
OnePlus 15
Qwen3.5-0.8B, 4-bit
MNN, GPU
0.835
Galaxy S23
SmolLM2-360M, 4-bit
MNN, CPU
0.744
OnePlus 8 Pro
SmolLM2-135M, 4-bit
llama.cpp, CPU
0.719
Redmi 9A
none (fastest option takes 10 s)
Budget: 4 s
Table 2. The most accurate deployment of the entity-extraction model for each phone. When several engines and backends reach the same accuracy, we list the fastest.
Figure 4. Mobile AI Factory. A setup stage reads the phone’s measured profile and builds a frozen evaluation set; a loop then curates data, trains, scores the quantized artifact and decides what to do next. Dark orange is the orchestrator LLM, and light orange nodes call it. Blue nodes, including rollback and the decision to escalate, are deterministic code.
Claude
DeepSeek
Task
Metric
0-shot / 5-shot
0-shot / 5-shot
Untrained
First FT
Factory
Factory model
MB
Output-format bound
BC5CDR
span F1
0.165 / 0.606
0.128 / 0.508
0.000
0.705
0.830
Qwen3.5-0.8B Q4
517
Calendar JSON
arg. match
0.514 / 0.634
0.505 / 0.669
0.036
0.535
0.811
Qwen3.5-0.8B Q4
517
TOPv2
exact match
0.241 / 0.242
0.145 / 0.085
0.003
0.534
0.599
Qwen3-4B Q8
4,082
MultiCoNER
micro F1
0.491 / 0.587
0.428 / 0.521
0.284
0.604
0.653
Qwen3-4B Q8
4,082
Table 3. Factory models against frontier models, on each task’s own scorer and the same frozen held-out set. Frontier models are given no examples (0-shot) or five worked examples (5-shot). The three student columns describe the factory’s best model: untrained, after its first fine-tune, and after search. Q4 and Q8 are 4- and 8-bit weights; MB is the model’s file size. Bold marks the best score in each row; on the last two rows, the factory and Claude are within our 0.015 noise floor. GoEmotions is scored as macro F1 over its Ekman emotion groups.
Figure 5. What search adds over each tier’s first fine-tune. Each dot is one tier of one task’s run. The grey band is the 0.015 improvement threshold.
Figure 6. Search gains split across interventions, (a) per task and (b) per tier. Each bar sums the gains of the kept steps of each kind; the number on the right is the total.
Figure 7. (a) Best score against GPU-hours for three search strategies on biomedical entity extraction, with orchestrator spend. (b) One-shot pick compared against the best model from the tiered search on each task, labelled with [picked model vs tiered winner]. The grey band is the noise floor.
Figure 8. Every decision to raise the goal or stop, by how many iterations the tier had taken to meet the goal and by how much it beat it. The grey band is the noise floor, set at 0.015.
Figure 9. Phone-aware factory deployment. (a, b) Two representative end-to-end runs: biomedical entity extraction on a OnePlus 8 Pro (moderate requirement) and call summaries on a Galaxy S23 (hard requirement); labels give each tier’s model, precision, engine and backend. (c–e) Across 60 requirements: (c) how often the factory’s most accurate model for the task meets the requirement on the phone; (d) how often a configuration picked without measurements meets the requirement. The measured-selection bar is a reference: it meets every requirement by construction; (e) the share of the phone’s configurations that measurements rule out before any training.
Task
CPU
GPU
Task
CPU
GPU
BC5CDR
− .0004
− .0017
CLINC150
− .0004
− .0004
Calendar
.0000
− .0019
SMS Spam
− .0160
− .0119
TOPv2
.0000
+ .0030
GoEmo.
− .0062
− .0079
MultiCoN.
+ .0005
− .0005
BEA-19
− .0007
+ .0015
BFCL
+ .0010
+ .0010
DialogSum
− .0014
− .0022
Table 4. Accuracy of each factory model on the phone’s CPU and GPU, minus its accuracy on the server.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Five-shot
Task
Rows
Base
Tuned
Claude
DeepSeek
BC5CDR
4,000
0.000
0.828
0.652
0.560
Calendar
867
0.036
0.818
0.621
0.644
TOPv2
5,000
0.002
0.562
0.221
0.112
MultiCoNER
5,000
0.292
0.619
0.594
0.508
DialogSum
459
0.241
0.451
0.387
0.347
Appendix
Table 5. Every model on the unseen test set, which the factory never used for training or for choosing a model. Base is the untrained model, and Tuned is the factory’s fine-tuned model. Frontier models get the same five examples as in Table 3 . Bold marks the best score and any score within the 0.015 noise floor of it. CLINC is intent classification; (0.99) marks the run with the goal pinned at 0.99. DialogSum’s rows have one reference summary instead of three, so all its scores are lower than in Table 3 .
Figure 10. Best score per tier, starting from (a) 100 and (b) 151 real rows, with and without synthesis, against full real data. The x-axis gives each tier’s model.
Deployment kind
Memory
First token
Answer
Keyboard
300 MB
0.3 s
1 s
Voice assistant
1,536 MB
0.5 s
2 s
Feature in an app
1,024 MB
0.5 s
3 s
Background service
512 MB
1 s
5 s
Dedicated app
40% of RAM
1 s
5 s
Overnight job
50% of RAM
6 h ÷ requests
Appendix
Table 6. Memory budget and response time by kind of deployment. An overnight job must finish all of a night’s requests in 6 hours, at most 60 s each.
ID
Task
Phone
Deployment
Memory (MB)
Prefill
Decode
Energy (J)
E01
GoEmotions
S23
Keyboard mood insights
≤ 3,500
≥ 1.9
≥ 1
≤ 135
E02
SMS Spam
OP15
Scam-check app
≤ 6,000
≥ 42
≥ 1
≤ 101
E03
BEA-19
OP8P
English journal corrections
≤ 5,800
≥ 0.8
≥ 1
≤ 261
E04
DialogSum
S23
Overnight call summaries
≤ 3,500
≥ 5.83
≥ 2
≤ 1,350
E05
Calendar JSON
OP15
Planner app
≤ 6,000
≥ 40
≥ 22
≤ 101
E06
BFCL
OP15
Shortcuts app
≤ 6,000
≥ 35
≥ 9
≤ 50.6
Appendix
Table 7. Easy requirements: at least 60% of the phone’s configurations meet all four limits. Prefill and decode are minimum speeds in tokens/s. OP15 = OnePlus 15, S23 = Galaxy S23, OP8P = OnePlus 8 Pro, R9A = Redmi 9A.
ID
Task
Phone
Deployment
Memory (MB)
Prefill
Decode
Energy (J)
M01
CLINC150
OP15
Offline assistant app
≤ 6,000
≥ 515
≥ 1
≤ 33.7
M02
GoEmotions
OP8P
Comment tone check
≤ 1,024
≥ 114
≥ 2
≤ 31.3
M03
SMS Spam
S23
Scam-check app
≤ 2,800
≥ 42
≥ 1
≤ 54
M04
BEA-19
R9A
English journal corrections
≤ 1,900
≥ 0.8
≥ 1
≤ 290
M05
BFCL
OP15
Voice-assistant actions
≤ 1,536
≥ 70
≥ 24
≤ 50.6
M06
Calendar JSON
S23
Planner app
≤ 2,800
≥ 40
≥ 22
≤ 54
Appendix
Table 8. Moderate requirements: 30–60% of the phone’s configurations meet all four limits. Prefill and decode are minimum speeds in tokens/s.
ID
Task
Phone
Deployment
Memory (MB)
Prefill
Decode
Energy (J)
H01
CLINC150
S23
Voice-assistant intent router
≤ 1,536
≥ 1,030
≥ 3
≤ 27
H02
SMS Spam
OP15
Messaging app spam check
≤ 512
≥ 42
≥ 1
≤ 10.1
H03
Calendar JSON
OP8P
Voice “add to my calendar”
≤ 1,536
≥ 80
≥ 57
≤ 104
H04
BFCL
R9A
Smartwatch actions relay
≤ 512
≥ 35
≥ 9
≤ 17.4
H05
GoEmotions
S23
Keyboard tone check
≤ 300
≥ 190
≥ 8
≤ 5.4
H06
BEA-19
OP15
English-learner keyboard
≤ 300
≥ 80
≥ 29
≤ 5.06
Appendix
Table 9. Hard requirements: under 30% of the phone’s configurations meet all four limits. Prefill and decode are minimum speeds in tokens/s.