Overview

On-device inference studies how to run models locally on phones, wearables, robots, and tiny microcontrollers when connectivity, privacy, latency, memory, or battery life is limited. Common approaches include quantization, pruning and distillation, compact hardware-aware architectures, and runtimes that manage memory, accelerators, and computation across devices. Applications range from keyword spotting and wearable activity recognition to mobile language models, image generation, medical screening, and robot control, evaluated on target hardware and task benchmarks. A central challenge is balancing accuracy and capability against real latency, energy, and memory costs, which vary across hardware; deciding when to compute locally, adapt, or offload remains an open systems trade-off.

Latest papers

CardsList

204 papers23 papers in the last four weeks, up 21% on the four weeks before. 0.2% of all new papers.

  1. LEDGE: An Efficient Simulator for Distributed LLM Serving on Heterogeneous Edge Devices

    Nov 18, 2026Zihao Wang, Ye Tian, Tajana Rosing +1LLM ServingParallel LLM Inference

  2. Does synthetic pre-training pay off on a microcontroller? Accuracy, latency and energy across two MCUs

    Nov 15, 2026Achintya YadavTiny MLSynthetic Data Pretraining

  3. dit.xpu : Taming DiT Sparse Inference on Mobile SoCs

    Nov 14, 2026Jiayu Chen, Jinye Hu, Maoliang Li +8Diffusion Model Inference AccelerationOn-Device Inference

  4. SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding

    Nov 13, 2026Jai Jain, Chihyo Ahn, Yong Liu +10Speculative DecodingEdge-Cloud Collaborative Inference

  5. A3^3: Adaptive Audio Agent Framework for Collaborative Edge Computing

    Nov 13, 2026Zexin Li, Xueyi Zhang, Tao Ren +1Edge-Cloud Collaborative InferenceEdge Inference

  6. On-Edge Acoustic Continual Anomaly Detection: A Cross-Platform Comparative Study

    Nov 11, 2026Rami Zewail, Yehia Ehab, Karim Ali +3Anomaly DetectionContinual Anomaly Detection

  7. GigaEdge Spench: An SDK for Practical Streaming Speech Enhancement

    Nov 9, 2026Artem Sokolov, Artem Khaidarov, Yana Lysova +2Speech EnhancementStreaming Inference

  8. MCUFlowNet: Hardware-Aware Neural Architecture Search for Optical Flow on Microcontrollers

    Nov 5, 2026Enmin Lin, Jona BeysensOptical FlowEfficient Neural Network Inference

  9. Streaming Speech Recognition on a Microcontroller: a 65.7M-Parameter Transducer in 2 and 4 Bits on the ESP32-P4

    Oct 28, 2026Billal BoumedineAutomatic Speech RecognitionOn-Device Inference

  10. MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices

    Oct 27, 2026Kailai Feng, Haoxiang Rao, Yuxiang Wei +5Image Editing EvaluationImage Generation Evaluation

  11. AIPC: Evidence-Guided Agent Workflows for PyTorch Project Deployment

    Oct 26, 2026Zhanwei Wu, Jianhao Su, Brian Zhang +11AI Accelerator InferenceLLM Agent Workflow Optimization

  12. VLA-HWBench: Where the Time Goes When VLA Models Run on Edge Hardware

    Oct 26, 2026Borui Xu, Hongshi Tan, Wenqi Jiang +4Efficient VLA InferenceOn-Device Inference

  13. Energy and Latency First: How Deployment Constraints Reverse Time-Series Model Selection

    Oct 22, 2026Shruti Bothe, Illyyne Saffar, Ayush K. VarshneyModel SelectionEnergy-Efficient ML

  14. AgentDP: Towards Automated Model Deployment for Heterogeneous Embedded AI Applications

    Oct 22, 2026Leming Shen, Qiang Yang, Xinyu Huang +2MLOpsOn-Device Inference

  15. GANet: A Lightweight Group Attention Network for Plant Disease Identification on Embedded Systems

    Oct 21, 2026Anubhab Tripathi, Bibek Das, Asfak Ali +1Plant Disease ClassificationLightweight CNNs

  16. Fault Detection and Bit-Error Rate Diagnostics in Optical Wireless Links Using Edge-AI

    Oct 15, 2026Sarthak Bhattarai, Sheshadri Shikhar Bandyopadhyay, Suprabh Dhakal +1Fault DiagnosisWireless Communications

  17. EdgeAgent: Orchestrating On-Device LLM inference for End-User Multi-Agent Systems on CPU-GPU Unified Memory Architectures

    Oct 2, 2026Yuhai Long, Yuanxin Wei, Kai Wu +3On-Device InferenceLLM Inference Scheduling

  18. LiteEMG-FM: An Efficient and Deployable Foundation Model for Robust EMG Sensing

    Oct 1, 2026Tianhao Wu, Xu Wu, Amirmohammad Radmehr +4EMG-Based Gesture RecognitionTime-Series Foundation Models