On-Device Language Model Inference

The field studies how to run useful language models locally, where limited memory, battery life, heat, and compute can constrain both model size and response speed. Common approaches include quantization and pruning, compact or mixture-of-experts models, KV-cache compression, speculative decoding, and hardware-aware runtimes for mobile CPUs, GPUs, and NPUs. Other systems split work between a device and the cloud or route tasks among local models to balance privacy, cost, and capability. A central tension is that gains in speed or memory do not always preserve accuracy, reliability, or energy efficiency, so evaluation increasingly measures full workloads on real devices rather than relying on model size or peak throughput alone.