This technical report presents MNN-Pocket-0.3B, a compact 300M-parameter dense language model designed for efficient on-device deployment and supporting context lengths of up to 32K tokens. With an INT4 deployment footprint of only 173 MB and decoding throughput of up to 466 tokens/s on iPad Pro M5 (Metal), it can be seamlessly integrated into applications on edge devices such as smartphones and smart glasses. MNN-Pocket-0.3B is developed through the co-design of its architecture, data, training pipeline, and inference system. Hardware-aware architecture selection and dependency-aware vocabulary pruning balance model capability with on-device memory and compute constraints. A four-stage pre-training pipeline covering approximately 4.1 trillion high-quality Chinese and English tokens equips the model with broad knowledge and reasoning capabilities. This is followed by a three-stage post-training pipeline comprising instruction fine-tuning, multi-teacher on-policy distillation, and quantization-aware training. Finally, hardware-specific optimizations in the MNN inference engine enable fast, memory-efficient inference and long-context processing. Extensive evaluations demonstrate that MNN-Pocket-0.3B achieves frontier-level performance among models of comparable size, with particularly strong results in Chinese and English knowledge and reasoning, mathematics, coding, and instruction following. It consistently outperforms similarly sized open-source models and even surpasses several 0.5B-parameter models on instruction-following benchmarks. We release MNN-Pocket-0.3B with open weights and ready-to-use MNN deployment packages. We hope it will provide a practical foundation for capable, responsive, and memory-efficient edge applications while advancing research and development in on-device language models.
Figures & tables
Figure 1: MNN-Pocket-0.3B Model Overview.
Figure 2: Hardware-aware architecture selection. (a) Model-scale selection on iPad Pro M5 using the MNN Metal backend. The approximately 300M-parameter model is the largest tested candidate with peak memory below 300 MB and decoding throughput above 200 tokens/s; shading marks the feasible tested candidates. (b) Architecture search at a comparable 300M-parameter budget. The 28-layer model achieves the same evaluation average as the 36-layer model, with lower training loss, lower peak memory, and higher prefill and decoding throughput. (c) Final architecture hyperparameters of MNN-Pocket-0.3B.
Figure 3: Training pipeline overview. Four-stage pre-training produces MNN-Pocket-0.3B-Base, followed by supervised fine-tuning (SFT) on 3 million high-quality question–answer pairs to obtain MNN-Pocket-0.3B-SFT. Four domain-specialized reinforcement learning (RL) teachers covering knowledge, instruction following, mathematics, and code guide MOPD of the SFT student to obtain MNN-Pocket-0.3B. Quantization-aware training (QAT) further converts the consolidated model into MNN-Pocket-0.3B-INT4 for on-device deployment. The consolidated model also initializes MNN-Pocket-0.3B-Dec for non-autoregressive decision modeling.
Figure 4: Pre-training data composition across training stages. From outer to inner, the concentric rings show domain, style, knowledge density, difficulty, and quality distributions. Knowledge density (D1–D3), difficulty (H1–H3), and quality (Q1–Q3) are each grouped into three levels, from low to high. Percentages are normalized independently within each dimension using recorded token weights. Center labels indicate the training token budget for each stage, in trillions (T) or billions (B).
Hyperparameter
Stable Stage 1
Stable Stage 2
Short Decay
Long Decay
Context length
4,096
4,096
4,096
32,768
Total tokens
∼ 3T
∼ 800B
∼ 200B
∼ 100B
Global batch size
1024
1024
1024
128
Tokens per step
4.2M
4.2M
4.2M
4.2M
Peak learning rate
3×10−4
1×10−4
4×10−5
4×10−5
LR schedule
Constant
Constant
Cosine
Cosine
Table 1: Training hyperparameters for each pre-training stage of MNN-Pocket-0.3B-Base.
Figure 5: SFT Loop Engine. An agent invokes the training/evaluation (1) and data refinement (2) skills. Subagents categorize failures (A), aggregate results and allocate quotas (B), and synthesize, deduplicate, and screen data for leakage (C). The refined data feeds the next SFT iteration.
Category
Samples
Tokens
Percent
Family
Multilingual Creation & Open-Ended Generation
607K
289M
15.81%
Mathematics & Quantitative Reasoning
722K
393M
21.54%
Programming & Algorithmic Coding
801K
433M
23.70%
Application Development & General QA
247K
148M
8.08%
General Chinese-Language Tasks
450K
337M
18.43%
Table 2: Token-based composition of the final filtered SFT data recipe across taxonomy families and capabilities.
Figure 6: Two-step consolidation and low-precision recovery. Both steps run the same on-policy distillation loop. Step 1 (left): Multi-Teacher On-Policy Distillation (MOPD), in which four same-origin RL teachers—code, mathematics, knowledge, and instruction following—are routed by domain label and the selected teacher prefills the MNN-Pocket-0.3B student’s own rollout, so that domain-routed Top- K reverse-KL distillation supervises the student in BF16. Step 2 (right): Self-OPD with quantization-aware training (QAT), which instead uses a single frozen teacher: MNN-Pocket-0.3B in BF16 serves as the self-teacher, and MNN-Pocket-0.3B-INT4 in W4A16 form, initialized by post-training quantization, is aligned to it on its own rollout.
Figure 7: MNN-Pocket-0.3B-Dec architecture. Left: abridged Typed Decisions [ 33 ] customer-service input. Right: the shared model architecture with dataset reference probabilities shown for illustration. Markers are semantic aliases of reserved tokens.
Metric
MNN-Pocket 0.3B
Gemma-3 270M
ERNIE-4.5 0.3B
LFM2.5 350M
SmolLM2 360M
MiniCPM4 0.5B
Hunyuan 0.5B
Qwen3 0.6B
MNN inference file size (MB)
173
162
207
202
206
247
307
340
Peak Memory (MB)
228
272
287
258
257
305
449
427
Table 3: Memory and deployment footprint of on-device LLMs on iPad Pro (M5) at 512 token context.
Device
Backend
Stage
MNN-Pocket 0.3B
Hunyuan 0.5B
Qwen3 0.6B
MiniCPM4 0.5B
SmolLM2 360M
ERNIE-4.5 0.3B
LFM2.5 350M
Gemma-3 270M
iPad Pro (M5)
Metal
Prefill
8721
5330
6027
6973
7067
7218
8026
9519
Decode
466
262
286
357
397
395
504
455
CPU
Prefill
2228
1474
1396
1818
1824
2185
2234
3793
Decode
231
149
135
197
187
224
262
290
Android (8 Gen 5)
OpenCL
Prefill
3403
1757
1627
2368
2291
2532
3070
4209
Decode
138
88
93
117
127
119
175
118
Table 4: Inference speed (tok/s) of on-device LLMs across different platforms at 512 token context.
Draft Precision
Training Source
Verification Model
Math
Code
Chat
Avg.
MATH-500
AIME25
MBPP
HumanEval
MT-Bench
A-Hard
BF16
MNN
MNN
5.34
4.59
5.55
5.83
3.55
3.18
4.67
BF16
MNN
MNN-INT4
5.30
4.49
5.57
5.75
3.62
3.16
4.65
BF16
MNN-INT4
MNN-INT4
5.34
4.55
5.64
5.82
3.70
3.18
4.70
INT4
MNN
MNN-INT4
5.16
4.37
5.48
5.65
3.50
3.06
4.54
INT4
MNN-INT4
MNN-INT4
5.22
4.47
5.55
5.76
3.65
3.10
4.63
Table 5: Mean accepted length across draft–target configurations for MNN-Pocket-0.3B.
Device
Workload
AR tok/s
Mean AL
Spec. tok/s
Speedup
iPad Pro (M5)
Math
435
4.49
709
1.63 ×
Code
5.73
895
2.06 ×
Chat
3.51
547
1.26 ×
Avg.
4.57
717
1.65 ×
Android (8 Gen 5)
Math
135
4.48
159
1.17 ×
Code
5.72
186
1.38 ×
Table 6: Performance with MNN-Pocket-0.3B MNN-INT4 target and the INT4 MNN-trained draft.
∼ 300M Models
∼ 500M Models
Benchmark
MNN-Pocket 0.3B
MNN-Pocket 0.3B-INT4
Gemma-3 270M
ERNIE-4.5 0.3B
LFM2.5 350M
SmolLM2 360M
Hunyuan 0.5B
MiniCPM4 0.5B
Qwen3 0.6B
Average
50.00
49.59
6.42
31.99
31.28
18.43
31.22
46.81
44.28
Knowledge
C-Eval
38.86
40.27
6.84
32.17
32.10
23.77
35.29
67.98
44.06
CMMLU
40.11
39.65
4.94
33.91
31.65
24.87
35.69
66.04
44.50
MMLU-Redux
44.00
43.40
13.58
32.68
41.01
24.17
36.29
52.85
51.63
Table 7: Performance of post-trained MNN-Pocket-0.3B model and other baselines across 11 benchmarks.
∼ 300M Models
∼ 500M Models
Benchmark
MNN-Pocket 0.3B-Base
Gemma-3 270M Base
ERNIE-4.5 0.3B Base
LFM2.5 350M Base
SmolLM2 360M Base
Hunyuan 0.5B-Pretrain
Qwen3 0.6B Base
Average
33.58
16.94
34.00
29.86
22.25
42.60
44.86
General Tasks
C-Eval
36.55
23.25
40.56
35.66
22.07
57.95
54.68
CMMLU
38.13
25.26
39.54
36.78
25.30
54.54
53.29
MMLU-Redux
47.07
24.47
46.59
48.44
30.19
59.34
57.45
Table 8: Performance of pre-trained MNN-Pocket-0.3B-Base model and other baselines across 11 benchmarks.
Figure 8: Benchmark evolution from the SFT initialization to the final MOPD model. Gray-blue bars denote SFT scores, and blue segments denote gains after MOPD. Final scores match Table 7 .
Table 9: Prompt and example for E-Commerce Review Opinion Mining.
Language backbone
Params
F1 (%) ↑
MNN-Pocket-0.3B
0.3B
80.15
Qwen3-1.7B
1.7B
78.59
Qwen3-0.6B
0.6B
77.47
Hunyuan-0.5B
0.5B
77.27
ERNIE-4.5-0.3B
0.3B
76.73
LFM2.5-350M
0.35B
76.19
Table 10: E-Commerce Review Opinion Mining results. All backbones are LoRA fine-tuned with the identical recipe of Section 7.1.1 and scored on the official test split; a quadruple counts as correct only under exact match on all four fields. Parameter counts are the nominal sizes given by the model names. Rows are ordered by F1.
Figure 9: Fully on-device speech-to-speech dialogue pipeline. Processed microphone audio feeds Silero VAD and INT8 X-ASR for speech activity detection and streaming transcription. Transcript-stability checks and Smart Turn determine when to commit the user utterance to INT4 MNN-Pocket-0.3B. Generated text is segmented and normalized for incremental synthesis by MOSS-TTS-Nano, with audio chunks queued for playback. Generation, synthesis, and playback run asynchronously; new user speech interrupts all three, while the playback signal provides a reference for acoustic echo cancellation. All neural modules run locally with MNN.
Metric
iPhone
iPad
Android
Device / SoC
13 Pro / A15
Pro / M5
Canoe / 8Gen5
Model storage (MB) ↓
2,018
2,018
2,018
Peak memory (MB) ↓
1,591
1,858
1,704
ASR RTF ↓
0.073
0.047
0.045
LLM decoding speed (token/s) ↑
130.2
324.9
231.7
TTS RTF ↓
0.653
0.366
0.651
Table 11: On-device performance of the MNN-Pocket-0.3B Voice Assistant.
Language backbone
Common
WildVoice
SD-QA
OBQA
MMSU
BBH
IFEval
AdvBench
AlpacaEval
Overall
MNN-Pocket-0.3B
40.23
36.72
5.75
32.75
27.91
58.90
64.11
97.69
44.91
45.44
Gemma3-270M
37.17
31.54
6.43
25.93
25.02
49.40
18.32
93.27
38.56
36.18
ERNIE-4.5-0.3B
37.33
32.50
4.87
28.57
25.63
50.60
33.98
39.42
38.46
32.37
LFM2.5-350M
47.17
39.96
11.06
22.86
25.76
46.30
57.17
88.65
50.47
43.27
SmolLM2-360M
37.93
32.26
15.78
21.98
24.59
46.40
32.59
23.46
41.20
30.69
Hunyuan-0.5B
22.67
22.19
2.02
29.67
28.24
48.70
18.06
23.85
23.73
24.35
Table 12: VoiceBench results of the MNN-Pocket-0.3B Voice Assistant and other models.
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.