Machine learning models are widely integrated into modern mobile apps to analyze user behaviors and deliver personalized services. Ensuring low-latency on-device model execution is critical for maintaining high-quality user experiences. While prior research has primarily focused on accelerating model inference with given input features, we identify an overlooked bottleneck in real-world on-device model execution pipelines: extracting input features from raw application logs. In this work, we explore a new direction of feature extraction optimization by analyzing and eliminating redundant extraction operations across different model features and consecutive model inferences. We then introduce AutoFeature, an automated feature extraction engine designed to accelerate on-device feature extraction process without compromising model inference accuracy. AutoFeature comprises three core designs: (1) graph abstraction to formulate the extraction workflows of different input features as one directed acyclic graph, (2) graph optimization to identify and fuse redundant operation nodes across different features within the graph; (3) efficient caching to minimize operations on overlapping raw data between consecutive model inferences. We implement a system prototype of AutoFeature and integrate it into five industrial mobile services spanning search, video and e-commerce domains. Online evaluations show that AutoFeature reduces end-to-end on-device model execution latency by 1.33x-3.93x during daytime and 1.43x-4.53x at night.
Figures & tables
Figure 1. Workflow of real-world mobile application services. Workflow of real-world mobile application services.
Figure 2. A complete on-device model execution pipeline in industrial mobile apps. A complete on-device model execution pipeline in industrial mobile apps.
Figure 3
Figure 5. Proportion of user features (left) and numbers of various features (right) across on-device ML models.
Figure 6. Analysis on real-world on-device ML models to reveal redundancy across features and inferences. Analysis on real-world on-device ML models to reveal redundancy across features and inferences.
Figure 7. AutoFeature overview and workflow. AutoFeature overview and workflow.
Figure 8. Four atomic operations of feature extraction. Four atomic operations of feature extraction.
Figure 9. Additional cost introduced by feature fusion. Additional cost introduced by feature fusion.
Figure 10. Latency breakdown of extracting user features from behavior events with different attributes. Latency breakdown of extracting user features from behavior events with different attributes.
Figure 11. Comparison between original design and hierarchical filtering. Each line denotes a comparison. Comparison between original design and hierarchical filtering. Each line denotes a comparison.
Figure 12. User feature definitions and online execution frequencies of on-device ML models in our evaluation. User feature definitions and online execution frequencies of on-device ML models in our evaluation.
Figure 13. Model structure of common on-device ML models. Model structure of common on-device ML models.
Figure 14. Statistically similar usage patterns between testing users and thousands of real-world users. Statistically similar usage patterns between testing users and thousands of real-world users.
Figure 15. Specific behavior traces of testing users. Specific behavior traces of testing users.
Figure 15
System
Offloaded Operations
Storage Structure
Introduced Storage
AutoFeature
✘
Each Row: One Behavior Event
✘
Each Column: Compressed Attributes
Decoded Log
Decode
Each Row: One Behavior Event
Massive Columns
Each Column: One Unique Attribute
Feature Store
Decode
Each Row: One Behavior Event required by One Feature
Redundant Rows
Retrieve
Each Column: One Unique Attribute
Massive Columns
Table 1. Detailed introduction to cloud-side feature extraction baselines. Detailed introduction to cloud-side feature extraction baselines.
Figure 18. Performance comparison between AutoFeature and cloud-side feature extraction baselines.
Figure 18
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 1. Behavior traces of testing users across three time periods (noon, evening and night). Each subfigure focuses on one video-related behavior and we plot the frequencies within each 10-minute segment of users with different activity levels (P90, P80, P70, P60, P50, and P30 traces).
On-device inference runs machine-learning models locally on phones, wearables, microcontrollers, and edge hardware to reduce latency, energy use, and data sharing.
In on-device NLP tasks, limited resources of embedded hardware, such as the Raspberry Pi 5, require efficient inference strategies. This paper introduces FrugalSOT (Frugal Search Over The Models), a resource-aware model selection architecture for on-device NLP inference. FrugalSOT estimates each request's complexity by extracting features such as prompt length, named entity density, and syntactic complexity. The request is first made to the least complex model that is likely to pass a relevance threshold. If the output of that model falls short of the threshold, the request is made to a more complex model. It is important to note that the relevance threshold undergoes continuous updates in the background. using past validation outcomes in an adaptation process using a low-pass filtering mechanism, thus imparting adaptation to changing input patterns. Experimental results achieved on a Raspberry Pi 5 show that FrugalSOT reduces average inference time and overall computational resource use to a significant extent compared to a single-model baseline approach, without compromising output relevance to the same extent as the most sophisticated model. These results confirm that adaptive model selection can enable efficient, high-quality natural language processing inference on limited devices.
Pradheep P, Yuvanesh S, Harish KB +4
Computer Science and Engineering Vellore Institute of Technology Vellore, India · Electronics and Communication Engineering Vellore Institute of Technology Vellore, India · Department of Analytics Vellore Institute of Technology Vellore, India +2
On-device model adaptation is essential to enable lifelong personalization on resource-constrained hardware, but compute, power, and memory limitations of such devices make end-to-end backpropagation impractical for modern deep neural networks. This work proposes a heterogeneous adaptation pipeline that repurposes a commercial edge AI inference accelerator, Hailo-8L, for frozen-backbone feature extraction during on-device training. The computational graph is partitioned so that the pre-trained backbone is quantized to INT8 and run on the accelerator, while only a lightweight FP32 classification head is fine-tuned on the host CPU, enabling frequent, energy-efficient in-field updates with most weights remaining fixed. Across multiple architectures and datasets, this pipeline achieves up to 15.4x faster wall-clock training time compared to a Raspberry Pi 5 CPU baseline, offers competitive throughput in favorable settings, and consistently reduces energy per sample. Post-training quantization restoration is shown to be crucial for preserving the quality of accelerator-generated features and mitigating accuracy loss in quantization-sensitive architectures. Overall, the results demonstrate a practical approach to efficient on-device adaptation using inference-oriented edge accelerators. The implementation is available at https://github.com/MatPiech/accelerator-training.
Mateusz Piechocki, Alessandro Capotondi, Marek Kraft
Poznan University of Technology, Institute of Robotics and Machine Intelligence, Poznan, Poland · University of Modena and Reggio Emilia, The High-Performance Real-Time Laboratory, Modena, Italy
Mobile graphical user interface (GUI) agents enable AI models to autonomously operate smartphones on behalf of users. However, most existing systems focus primarily on optimizing task accuracy and rely on cloud-hosted models for inference, which introduces privacy concerns and network-dependent latency. As a result, fully on-device deployment of mobile GUI agents remains underexplored. We propose MobileExplorer, a new framework that accelerates on-device inference for vision-based mobile GUI agents via online exploration. The key idea is to exploit the long per-step reasoning time of vision-language models (VLMs) by performing lightweight, parallel exploration of UI elements. During model inference, the agent proactively probes semantically relevant UI elements and records these exploration traces as structured memory. To ensure reliable execution in live mobile environments, we design a two-level rollback mechanism that robustly restores the initial UI state when a fast but naive backtracking strategy fails. The collected exploration traces are then summarized into concise contextual hints and injected into the prompt to enhance the subsequent reasoning step. We evaluate MobileExplorer on multiple off-the-shelf devices using the AndroidWorld benchmark, as well as newly designed, more complex tasks and dynamic on-device environments. MobileExplorer reduces the average number of reasoning steps and end-to-end latency by 23%, while maintaining or improving task success rates by up to 5%. A video demonstration of MobileExplorer performance in the real world is available at https://youtu.be/thK7MJmdlvM .
Runxi Huang, Liyu Zhang, Shengzhong Liu +1
Hong Kong University of Science and Technology · Shanghai Jiao Tong University