Running an AI call agent on-device makes it private, network-independent, and free of per-query cloud cost, but shipping one as a real-world service imposes constraints a cloud deployment never faces: ~1GiB memory budget, in-call latency and thermal limits, static-integer NPU execution, Korean STT input far from sLM pretraining, and cloud-level quality expectations. We bring ixi-sLM, a 1.2B Korean sLM, to mobile NPUs through ON-DEVICE MODELING: domain-adaptive modeling trims the vocabulary by call frequency to shrink the embedding and LM head that take nearly half the model under INT4 transformer weights and refits the tokenizer to call speech (18% fewer tokens on call traffic), while task-aware quantization calibrates on real calls and assigns per-module mixed precision by activation sensitivity. A single domain-adapted model runs ixi-O's two production services, AI Call Summary and AI Call Answering, at 94% of a state-of-the-art cloud LLM's task-weighted quality; our on-device modeling carries that quality onto the NPU with only a marginal drop in overall score, in an 803MiB model (28% of FP16) at 23% of the CPU's per-call energy. The same recipe deploys across Qualcomm and Apple NPUs and serves several million ixi-O subscribers in production.
Figures & tables
Figure 1: ixi-sLM (blue) compared with a cloud LLM, its FP16 counterpart, and the CPU: (a) 94% of cloud quality, (b) 803 MiB, 28% of FP16, (c) 22% of CPU latency, (d) 23% of CPU energy.
Figure 2: Overview of ixi-O, our on-device AI call agent, and its core functions: AI call summary, real-time call script, AI call answering, fraud prevention, and smart call search.
Figure 3: Overview of the on-device sLM converting pipeline. Opt.Studio applies task-aware quantization and then exports the model to Qualcomm and Apple NPUs from a single recipe.
Config.
Vocab.
Tok/call
Δ
Size (MiB)
Original
102,400
1,324
–
954
Merge only
102,400
1,087
− 17.9%
954
Trim only
65,536
1,326
+ 0.2%
803
Both
65,536
1,090
− 17.7%
803
Table 1: Merging accounts for the entire tokenization gain; trimming acts purely on footprint. Sizes in MiB.
Figure 4: (a) Embedding (FP16) vs. transformer (INT4) share across 1B-class sLMs. (b) shows the same FP16-embedding accounting as (a), isolating the effect of trimming; the deployed model quantizes the embedding further (4.2).
Quant.
Size (MiB)
Bits
ROUGE-L (%)
Call
WikiText
W4A16
726
4.10
80.63
79.42
MW4A16
803
5.09
82.23
79.47
W8A16
1,111
8.10
84.29
81.67
Table 2: Real-call vs. WikiText calibration per quantization setting. Sizes are after vocabulary trimming; MW4A16 is the mixed-precision setting initialized from W8A16.
Summary
Answering
Cloud LLM
99.60
99.00
Cloud sLM
94.38
95.70
ixi-sLM (Ours)
94.33
95.40
Table 3: Task accuracy via LLM judge (weighted rubric, 0–100). Cloud sLM: our domain-adapted FP16 model; ixi-sLM: the same model quantized ( < 1 GiB).
Latency (s)
Energy (mAh)
CPU (ExecuTorch)
45
7.79
NPU (ixi-sLM)
10
1.79
Table 4: Per-call latency and energy, CPU vs. NPU.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hardware
Framework
OS
Modeling
CPU
GPU
NPU
ExecuTorch
AOS/iOS
△
◯
◯
△
llama.cpp
AOS/iOS
×
◯
◯
△
ANEMLL
iOS
×
◯
×
◯
Appendix
Table 5: Comparison of generic on-device LLM frameworks. “Modeling” denotes whether the framework exposes tokenizer/quantization adaptation as first-class levers ( ◯ : supported, △ : partial, × : unsupported).
Figure 5: Module-wise SeqMSE on EXAONE4.0 1.2B (W8A16). Modules above the threshold (blue region) are kept at INT8, while those below it (gray region) are lowered to INT4.
Figure 6: Alignment score comparison across quantized model sizes. (a) shows the final-layer alignment score as a function of model size, where the 803 MiB model preserves the final-layer alignment of the 1,111 MiB W8A16 model at a smaller size. (b) shows the layer-wise alignment score in the late layers, indicating how each quantized model follows the original model before the final prediction.
Cloud LLM
Cloud sLM
ixi-sLM
Sub-task
Wt.
Raw
Rel.
Fin.
Raw
Rel.
Fin.
Raw
Rel.
Fin.
Service 1: AI Call Summarization
One-line summary
25%
4.77
100.00
25.00
4.61
96.65
24.16
4.66
97.69
24.42
Detailed summary
50%
22.00
100.00
50.00
21.42
97.36
48.68
21.48
97.64
48.82
Keyword extraction
10%
4.81
100.00
10.00
4.67
97.09
9.71
4.73
98.34
9.83
Task suggestion (schedule)
6%
4.22
95.91
5.75
4.38
99.55
5.97
3.83
87.05
5.22
Appendix
Table 6: Detailed task-accuracy rubric for both services. For each sub-task we report the raw judge score (Raw), its relative score against a per-sub-task reference, and the weighted contribution to the total (Fin. = Wt × Rel.). The reference is the best score observed across systems and judges, so it can exceed the highest value shown. Raw scales differ across sub-tasks. Scores are averaged over a three-LLM judge panel under cross-evaluation.
LLaMA3.2 1B
EXAONE4.0 1.2B
OS
Hardware
Framework
Prefill
Decode
Size
Prefill
Decode
Size
Test Device 1: Samsung Galaxy S25 Ultra (Snapdragon 8 Elite)
Android
CPU
llama.cpp
195.0
40.0
728 MiB
49.4
39.4
786 MiB
ExecuTorch
526.4
66.3
821 MiB
145.7
54.2
853 MiB
[3.5pt/2.5pt]
NPU
llama.cpp
132.4
25.5
851 MiB
97.2
17.1
887 MiB
ExecuTorch
2639.5
42.6
1,037 MiB
2327.1
32.5
1,026 MiB
Appendix
Table 7: Prefill and decode throughput (token/sec) and model size (MiB) across OS, Hardware, and Frameworks. Dashes mark unsupported configurations. All models were benchmarked with Max 4K Tokens (3K prompt tokens and 0.5 ∼ 1K generated tokens)
Figure 7: Battery current traces over one representative call, CPU vs. NPU, segmented into prefill and decode phases.
Stage
Prefill
Decode
Total
CPU
4.42
3.37
7.79
NPU
0.26
1.53
1.79
Appendix
Table 8: Per-call Energy (mAh) by phase, CPU vs. NPU.
0
30
60
90
120
150
180
210
240
270
300
Average
CPU
29.9
36.5
43.0
43.5
44.0
44.5
45.0
44.5
44.0
44.6
45.3
42.3
NPU (ixi-sLM)
29.9
30.4
31.6
32.2
33.0
33.1
33.3
33.5
33.5
34.0
34.5
32.6
Appendix
Table 9: Temperature (°C) at 30 s intervals over a 300 s sustained run, CPU against NPU. The final column is the average.
Figure 8: Device surface temperature over a 300 s sustained inference run, CPU vs. NPU. The dashed line marks the 36 °C throttling threshold.
Figure 9: (a) AI Call Summary: After a call ends, the agent produces an on-device summary capturing the conversation’s intent and key points, classifies it with semantic tags, and surfaces actionable suggestions (e.g., adding an event to the calendar). (b) AI Call Answering: The agent autonomously carries out a voice conversation on behalf of the user, handling task-oriented dialogue with an external party to fulfill the user’s request.