Optimizing Feature Extraction for On-device Model Inference with User Behavior Sequences

Chen Gong; Zhenzhe Zheng; Yiliu Chen; Sheng Wang; Fan Wu; Guihai Chen

cs.LGMar 23, 2026

Optimizing Feature Extraction for On-device Model Inference with User Behavior Sequences

Authors: Chen Gong, Zhenzhe Zheng, Yiliu Chen, Sheng Wang, Fan Wu, Guihai Chen

Organizations: Shanghai Jiao Tong University, Shanghai, China · ByteDance, Hangzhou, China

Abstract

Machine learning models are widely integrated into modern mobile apps to analyze user behaviors and deliver personalized services. Ensuring low-latency on-device model execution is critical for maintaining high-quality user experiences. While prior research has primarily focused on accelerating model inference with given input features, we identify an overlooked bottleneck in real-world on-device model execution pipelines: extracting input features from raw application logs. In this work, we explore a new direction of feature extraction optimization by analyzing and eliminating redundant extraction operations across different model features and consecutive model inferences. We then introduce AutoFeature, an automated feature extraction engine designed to accelerate on-device feature extraction process without compromising model inference accuracy. AutoFeature comprises three core designs: (1) graph abstraction to formulate the extraction workflows of different input features as one directed acyclic graph, (2) graph optimization to identify and fuse redundant operation nodes across different features within the graph; (3) efficient caching to minimize operations on overlapping raw data between consecutive model inferences. We implement a system prototype of AutoFeature and integrate it into five industrial mobile services spanning search, video and e-commerce domains. Online evaluations show that AutoFeature reduces end-to-end on-device model execution latency by 1.33x-3.93x during daytime and 1.43x-4.53x at night.

Figures & tables

Figure 1. Workflow of real-world mobile application services. Workflow of real-world mobile application services.

Figure 2. A complete on-device model execution pipeline in industrial mobile apps. A complete on-device model execution pipeline in industrial mobile apps.

Figure 5. Proportion of user features (left) and numbers of various features (right) across on-device ML models.

Panel of Figure 6 — Figure 6. Analysis on real-world on-device ML models to reveal redundancy across features and inferences. Analysis on real-world on-device ML models to reveal redundancy across features and inferences.

Figure 7. AutoFeature overview and workflow. AutoFeature overview and workflow.

Figure 8. Four atomic operations of feature extraction. Four atomic operations of feature extraction.

Figure 9. Additional cost introduced by feature fusion. Additional cost introduced by feature fusion.

Figure 10. Latency breakdown of extracting user features from behavior events with different attributes. Latency breakdown of extracting user features from behavior events with different attributes.

Figure 11. Comparison between original design and hierarchical filtering. Each line denotes a comparison. Comparison between original design and hierarchical filtering. Each line denotes a comparison.

Panel of Figure 12 — Figure 12. User feature definitions and online execution frequencies of on-device ML models in our evaluation. User feature definitions and online execution frequencies of on-device ML models in our evaluation.

Figure 13. Model structure of common on-device ML models. Model structure of common on-device ML models.

Figure 14. Statistically similar usage patterns between testing users and thousands of real-world users. Statistically similar usage patterns between testing users and thousands of real-world users.

Figure 15. Specific behavior traces of testing users. Specific behavior traces of testing users.

System	Offloaded Operations	Storage Structure	Introduced Storage
AutoFeature	✘	Each Row: One Behavior Event	✘
		Each Column: Compressed Attributes
Decoded Log	Decode	Each Row: One Behavior Event	Massive Columns
		Each Column: One Unique Attribute
Feature Store	Decode	Each Row: One Behavior Event required by One Feature	Redundant Rows
	Retrieve	Each Column: One Unique Attribute	Massive Columns

Table 1. Detailed introduction to cloud-side feature extraction baselines. Detailed introduction to cloud-side feature extraction baselines.

Panel of Figure 18 — Figure 18. Performance comparison between AutoFeature and cloud-side feature extraction baselines.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1. Behavior traces of testing users across three time periods (noon, evening and night). Each subfigure focuses on one video-related behavior and we plot the frequencies within each 10-minute segment of users with different activity levels (P90, P80, P70, P60, P50, and P30 traces).

On-Device Inference

On-device inference runs machine-learning models locally on phones, wearables, microcontrollers, and edge hardware to reduce latency, energy use, and data sharing.

See all 188 papers

FrugalSOT - Frugal Search Over the Models
Aug 21, 2026Pradheep P, Yuvanesh S, Harish KB +4Language Model Cascading On-Device Language Model Inference
Computer Science and Engineering Vellore Institute of Technology Vellore, India · Electronics and Communication Engineering Vellore Institute of Technology Vellore, India · Department of Analytics Vellore Institute of Technology Vellore, India +2
Empowering On-Device Model Adaptation with an Edge AI Inference Accelerator
Jul 20, 2026Mateusz Piechocki, Alessandro Capotondi, Marek KraftOn-Device Training Edge ML
Poznan University of Technology, Institute of Robotics and Machine Intelligence, Poznan, Poland · University of Modena and Reggio Emilia, The High-Performance Real-Time Laboratory, Modena, Italy
MobileExplorer: Accelerating On-Device Inference for Mobile GUI Agents via Online Exploration
May 26, 2026Runxi Huang, Liyu Zhang, Shengzhong Liu +1Mobile GUI Automation Efficient VLM Inference
Hong Kong University of Science and Technology · Shanghai Jiao Tong University

Aug 21, 2026J/K move · Enter open · S save

FrugalSOT - Frugal Search Over the Models

Pradheep P, Yuvanesh S, Harish KB, Keerthan Saai Reddy S+3

Computer Science and Engineering Vellore Institute of Technology Vellore, India · Electronics and Communication Engineering Vellore Institute of Technology Vellore, India · Department of Analytics Vellore Institute of Technology Vellore, India+2

Language Model Cascading On-Device Language Model Inference Efficient Language Model Inference Cost-Aware Inference LLM Routing

Open paper AskPDF arXiv

In on-device NLP tasks, limited resources of embedded hardware, such as the Raspberry Pi 5, require efficient inference strategies. This paper introduces FrugalSOT (Frugal Search Over The Models), a resource-aware model selection architecture for on-device NLP inference. FrugalSOT estimates each request's complexity by extracting features such as prompt length, named entity density, and syntactic complexity. The request is first made to the least complex model that is likely to pass a relevance threshold. If the output of that model falls short of the threshold, the request is made to a more complex model. It is important to note that the relevance threshold undergoes continuous updates in the background. using past validation outcomes in an adaptation process using a low-pass filtering mechanism, thus imparting adaptation to changing input patterns. Experimental results achieved on a Raspberry Pi 5 show that FrugalSOT reduces average inference time and overall computational resource use to a significant extent compared to a single-model baseline approach, without compromising output relevance to the same extent as the most sophisticated model. These results confirm that adaptive model selection can enable efficient, high-quality natural language processing inference on limited devices.

Abstract

On-Device Inference

Explore similar work

FrugalSOT - Frugal Search Over the Models

Empowering On-Device Model Adaptation with an Edge AI Inference Accelerator

MobileExplorer: Accelerating On-Device Inference for Mobile GUI Agents via Online Exploration