On-Device Inference

On-device inference studies how to run models locally on phones, wearables, robots, and tiny microcontrollers when connectivity, privacy, latency, memory, or battery life is limited. Common approaches include quantization, pruning and distillation, compact hardware-aware architectures, and runtimes that manage memory, accelerators, and computation across devices. Applications range from keyword spotting and wearable activity recognition to mobile language models, image generation, medical screening, and robot control, evaluated on target hardware and task benchmarks. A central challenge is balancing accuracy and capability against real latency, energy, and memory costs, which vary across hardware; deciding when to compute locally, adapt, or offload remains an open systems trade-off.