Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge

Jiesong Chen; Jun You; Zhidan Liu; Zhenjiang Li

cs.ARApr 11, 2026

Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge

Authors: Jiesong Chen, Jun You, Zhidan Liu, Zhenjiang Li

Organizations: Department of Computer Science, City University of Hong Kong, Hong Kong, China · INTR Thrust, System Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

Abstract

Precise estimation of model inference latency is crucial for time-critical mobile edge applications, enabling devices to calculate latency margins against deadlines and trade them for enhanced model performance or resource savings. However, the ubiquity of Dynamic Voltage and Frequency Scaling (DVFS) renders traditional static profiling invalid in real-world deployments, as inference latency fluctuates with varying processor (CPU and GPU) frequencies. While extensive profiling across frequency combinations is theoretically possible, it is prohibitively expensive, particularly for emerging Small Language Models (SLMs), where variable context lengths explode the profiling up to days. We observe that simple analytic scaling fails to predict these fluctuations due to the complex asynchronous coupling between CPU (kernel launching) and GPU (execution). In this paper, we introduce FLAME to accurately estimate inference latency across frequency combinations. It features a novel layer-wise modeling that quantifies the overlapping parallelism and then aggregates dynamic pipeline bubbles caused by asynchronous processor interactions when extending to the full model. This bottom-up approach ensures generalizability across diverse models from DNNs to SLMs, and its precise modeling allows for profiling a sparse subset of samples, cutting DNN profiling from hours to minutes and SLM profiling from days to mere minutes, while maintaining small estimation errors across frequencies. We further showcase FLAME's utility in a deadline-aware DVFS, outperforming the state-of-the-art approach in both power efficiency and latency guarantees.

Figures & tables

Fig. 1: Dynamic timing factor

\Delta_{\ell}(f_{c},f_{g})

, denoted as

\Delta_{\ell}

in the above figure for clear illustration, exists when a mobile edge device processes any model layer

\ell

due to asynchronous interaction between its CPU and GPU. This asynchrony leads to (a) overlapping and (b) idle waiting between the CPU and GPU execution times.

Model	Time	Model	Time
ResNet50	43 min	GPT2-large	113 h
VGG16	54 min	Qwen2-1.5B	151 h
DenseNet121	102 min	Qwen2-7B	304 h

TABLE I: Overhead of exhaustive profiling for DNN and SLM models on NVIDIA Jetson AGX Orin.

Fig. 2: CDF of the ratio between the duration of the timing factor

\Delta_{\ell}(f_{c},f_{g})

and the execution latency of the corresponding entire layer.

Fig. 3: Latency estimation error of existing methods for (a) DNN models and (b) SLM models.

Fig. 4: System architecture of FLAME.

Fig. 5: CDF of estimating independent (a) CPU and (b) GPU times for constituent layers.

Fig. 6: The selected top-10 HPCs most relevant to the inference latency of the convolution layer.

Fig. 7: Estimation error of (a) each layer in ResNet50 and (b) one layer in GPT2-large across context lengths.

Fig. 8: (a) Ideal serialization and (b) our constructed CPU-GPU processing timeline.

Fig. 9: Estimated (Est) model-wise latency of FLAME compared to the ground truth (GT) across all frequency combinations for (a) ResNet50 and (b) GPT2-large.

Fig. 10: Deadline-aware DVFS vs. other strategies.

Fig. 20: Varying deadlines with (a) ResNet50 and (b) GPT2-large. The rate achieved (blue) higher than the required rate (red) means deadline requirement is met.

Fig. 21: Online adaptation for (a) ResNet50 and (b) GPT2-large with concurrent workloads.

Model	Time	Model	Time
ResNet50	2 min (43 min)	GPT2-large	2 min (113 h)
VGG16	3 min (54 min)	Qwen2-1.5B	2 min (151 h)
DenseNet	6 min (102 min)	Qwen2-7B	4 min (304 h)

TABLE II: Profiling overhead of FLAME. The numbers in parentheses are the time needed for full profiling.

Edge Inference

Edge inference runs trained AI models on nearby devices such as phones, cameras, robots, and embedded computers under tight resource and latency limits.

See all 228 papers

PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling
Sep 9, 2026Weisi Yang, Stephen XiaLLM Inference Efficiency On-Device Language Model Inference
Northwestern University Evanston, Illinois, USA
Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors
Aug 7, 2026Saeid Shokoufa, Mohammad Erfan Sadeghi, Mehdi Kamal +1LLM Inference Energy-Aware LLM Inference
University of Southern California Los Angeles, California, USA
EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems
Jul 10, 2026Shuo Huai, Hao Kong, Shiqing Li +5Efficient Neural Network Inference Edge Computing
School of Computer Science and Engineering, Nanyang Technological University, Singapore · HP-NTU Digital Manufacturing Corporate Lab, Nanyang Technological University, Singapore · HP Inc., Palo Alto, California, USA

Sep 9, 2026J/K move · Enter open · S save

PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

Weisi Yang, Stephen Xia

Northwestern University Evanston, Illinois, USA

LLM Inference Efficiency On-Device Language Model Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Deploying Large Language Models (LLMs) directly on mobile platforms at the edge is gaining traction due to a myriad of benefits, such as increased privacy, personalization, and reduced latency. However, LLMs have heavy computational requirements, which are difficult for resource-constrained mobile and edge platforms to fulfill. In addition to limited compute resources, mobile and edge systems often have a compact form factor and lack physical mechanisms to dissipate heat generated from high processor usage rates (e.g., fans) to prevent throttling and reduced processing power, which LLMs can easily cause. To mitigate these effects, prior works have proposed various power governing strategies, such as dynamic voltage and frequency scaling (DVFS), for reducing power and heat generation for heavy computational tasks on mobile platforms. Recently, DVFS methods tailored for mobile LLMs have also been proposed. However, these methods mostly focus on optimizing hardware parameters and processor frequencies, and they fall short under some thermally constrained scenarios. Drawing from recent advances in machine learning, we identify and take advantage of the key insight that not all tokens require full-depth inference to maintain high-quality generation. Motivated by this, we present PELM, a solution that augments traditional DVFS processor frequency tuning with two additional workload-specific knobs: 1) speculative decoding and 2) variable verification depth to expand the optimization space to multiple dimensions for more power efficient on-device LLM inference. In extensive evaluations across hardware platforms and datasets, PELM demonstrates superior performance compared to state-of-the-art power governing methods, with up to 23.1% speedup and 52.4% reduction in energy consumption, while maintaining comparable task performance. The source code is available at https://github.com/imec-nu/PELM.

Abstract

Edge Inference

Explore similar work

PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling

Multi-Level Modeling of Large Language Model Inference Latency and Energy via Hybrid Analytical--Machine-Learning Predictors

EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems