Deploying deep neural networks (DNNs) on power-sensitive edge devices presents a formidable challenge. While Dynamic Voltage and Frequency Scaling (DVFS) is widely employed for energy optimization, traditional model-level scaling is often too coarse to capture intra-inference variations, whereas fine-grained operator-level scaling suffers from prohibitive performance degradation due to significant hardware switching latency. This paper presents SparseDVFS, a fine-grained, sparse-aware DVFS framework designed for energy-efficient edge inference. Our key insight is that operator sparsity is a primary metric for hardware frequency modulation. By distinguishing between compute-bound dense operators and memory-bound sparse operators, the system can apply specialized frequency triplets to maximize energy efficiency. To overcome switching overheads and component interference, SparseDVFS incorporates three key innovations: (1) an offline modeler that established a deterministic mapping between operator sparsity and optimal frequency triplets (CPU/GPU/EMC) via white-box timeline analysis; (2) a runtime graph partitioner that utilizes a greedy merging heuristic to aggregate operators into super-blocks, balancing scaling granularity and DVFS switching latency through a latency amortization constraint; and (3) a unified co-governor that employs a frequency unified scaling engine (FUSE) and a look-ahead instruction queue to eliminate antagonistic effects between independent controllers and hide hardware transition latencies. Extensive evaluations show that SparseDVFS achieves an average 78.17% energy efficiency gain over state-of-the-art solutions while maintaining a superior 14% cost-gain ratio.
Figures & tables
Figure 1. Comparison of DVFS governor granularities for edge DNN inference. (a) Model-level DVFS governor employs a single V/f setting. (b) Operator-level fine-grained DVFS governor. (c) Our sparsity-aware block-level DVFS governor. Comparison of DVFS governor granularities for edge DNN inference. (a) Model-level DVFS governor employs a single V/f setting. (b) Operator-level fine-grained DVFS governor. (c) Our sparsity-aware block-level DVFS governor.
Figure 2. Roofline model across diverse DNN architectures. For CNN-based models (i.e., ResNet-18/101), dense operators (e.g., Conv2d, Linear) reside on the horizontal plateau (i.e., Compute-Bound), while activation and normalization layers fall into the sloped region (i.e., Memory-Bound). For Transformer-based models (i.e., ViT-B16/L16), dense operators (e.g., Conv2d) reside on the horizontal plateau (i.e., Compute-Bound), while linear and normalization layers fall into the sloped region (i.e., Memory-Bound). Roofline model across diverse DNN architectures. For CNN-based models (i.e., ResNet-18/101), dense operators (e.g., Conv2d, Linear) reside on the horizontal plateau (i.e., Compute-Bound), while activation and normalization layers fall into the sloped region (i.e., Memory-Bound). For Transformer-based models (i.e., ViT-B16/L16), dense operators (e.g., Conv2d) reside on the horizontal plateau (i.e., Compute-Bound), while linear and normalization layers fall into the sloped region (i.e., Memory-Bound).
Figure 3. Dynamic sparsity distribution (CDF) for CNN and Transformer models on ImageNet-2012 validation dataset (about 50k images). A substantial portion of operators in ResNet and ViT exhibit high sparsity (>50%). Dynamic sparsity distribution (CDF) for CNN and Transformer models on ImageNet-2012 validation dataset (about 50k images). A substantial portion of operators in ResNet and ViT exhibit high sparsity (>50%).
Figure 4. Runtime traces of CPU and GPU frequencies under default DVFS governors for ResNet-18. Independent and often inverse fluctuations between frequencies create an antagonistic effect. Runtime traces of CPU and GPU frequencies under default DVFS governors for ResNet-18. Independent and often inverse fluctuations between frequencies create an antagonistic effect.
Figure 5. Comparison between DVFS switching overhead and end-to-end inference latency for ResNet-18. Comparison between DVFS switching overhead and end-to-end inference latency for ResNet-18.
Figure 6. The impact of CPU and GPU clock frequencies on system performance for ResNet-18. The heatmaps illustrate the non-linear sensitivity of (a) inference latency and (b) power consumption to frequency scaling. The impact of CPU and GPU clock frequencies on system performance for ResNet-18. The heatmaps illustrate the non-linear sensitivity of (a) inference latency and (b) power consumption to frequency scaling.
Figure 7. Overview of the SparseDVFS framework. The system is architected in three synergistic stages: (1) the offline modeler establishes a deterministic mapping between operator characteristics and optimal frequency triplets through timeline-based performance analysis and thermal-aware power modeling. (2) the runtime graph partitioner performs greedy operator aggregation to construct super-blocks, ensuring that fine-grained DVFS scaling satisfies the latency amortization constraint. (3) the unified co-governor executes coordinated frequency scaling across heterogeneous components via the frequency unified scaling engine (FUSE) strategy, while masking DVFS switching latency through a pipelined look-ahead mechanism. Overview of the SparseDVFS framework. The system is architected in three synergistic stages: (1) the offline modeler establishes a deterministic mapping between operator characteristics and optimal frequency triplets through timeline-based performance analysis and thermal-aware power modeling. (2) the runtime graph partitioner performs greedy operator aggregation to construct super-blocks, ensuring that fine-grained DVFS scaling satisfies the latency amortization constraint. (3) the unified co-governor executes coordinated frequency scaling across heterogeneous components via the frequency unified scaling engine (FUSE) strategy, while masking DVFS switching latency through a pipelined look-ahead mechanism.
Predictor
Model
±10% Accuracy
Model Size
FLOPs-based ( Kim et al., 2019 )
ResNet-18
18.4%
8B
ResNet-101
12.8%
ViT-B16
11.5%
ViT-L16
8.2%
nn-Meter ( Zhang et al., 2021 )
ResNet-18
91.5%
~856MB
ResNet-101
88.1%
Table 1. ±10% latency prediction accuracy and model size of different predictors.
Figure 8. An example of offline modeler for ResNet-18. (a) latency prediction under different CPU/GPU frequency combinations and (b) thermal-aware power prediction. An example of offline modeler for ResNet-18. (a) latency prediction under different CPU/GPU frequency combinations and (b) thermal-aware power prediction.
Figure 9. Impact of runtime graph partitioner on operator aggregation granularity and DVFS switching latency. We set aggregation factor N=5 . Impact of runtime graph partitioner on operator aggregation granularity and DVFS switching latency. We set aggregation factor N=5.
Table 2. Comparison of DVFS Switching Latency between Traditional DVFS (serial execution) and Pipeline-based Look-Ahead. We set aggregation factor N=5 .
Model
Op. Count
FLOPs (G)
Size (M)
ResNet-18
21
1.82
11.69
ResNet-101
105
7.87
44.55
ViT-B16
38
11.29
58.07
ViT-L16
74
39.86
203.36
Table 3. Detailed configuration of four DNN models.
Figure 11. Comparison of end-to-end inference latency across different DNN models. Comparison of end-to-end inference latency across different DNN models.
Figure 12. Energy and power profile comparison across different DNN models. Energy and power profile comparison across different DNN models.
Figure 13. Normalized energy efficiency gain relative to the Default DVFS baseline. Normalized energy efficiency gain relative to the Default DVFS baseline.
Figure 14. Comparison of Cost-Gain Ratio across different DVFS strategies. Comparison of Cost-Gain Ratio across different DVFS strategies.
Figure 15. Ablation studies of each component of the unified co-governor on (a) DOTA-v1.0 and (b) VisDrone 2019 datasets, respectively. We use ViT-B16 as test load. Ablation studies of each component of the unified co-governor on (a) DOTA-v1.0 and (b) VisDrone 2019 datasets, respectively. We use ViT-B16 as test load.
Figure 16. Thermal stability and frame rate jitter analysis of NVIDIA Jetson Orin Nano under sustained load (ViT-B16). Thermal stability and frame rate jitter analysis under sustained load (ViT-B16).
Figure 17. Impact of the amortization factor N on super-block aggregation granularity. Increasing N dictates the trade-off between operator aggregation granularity and DVFS switching latency. Impact of the amortization factor N on super-block aggregation granularity. Increasing N dictates the trade-off between operator aggregation granularity and DVFS switching latency.
Figure 18. Trade-off between inference latency and switching overhead relative to N . Trade-off between inference latency and switching overhead relative to N.
Figure 19. Impact of the aggregation factor N on system power and energy consumption. Impact of the aggregation factor N on system power and energy consumption.
Passive cooling eliminates the energy overhead and mechanical failure modes of fans, making it attractive for edge deployment, yet sustained Deep Neural Network (DNN) inference on passively cooled edge Systems-on-Chip (SoCs) is bottlenecked by thermal throttling. To address this, we propose an empirically calibrated, state-aware Dynamic Voltage and Frequency Scaling (DVFS) scheduler. Unlike heuristic-driven controllers, our methodology utilizes time-domain guards and absolute temperature bounds, with derivative triggers acting as safeguards against sharp thermal spikes. Evaluated on a passively cooled Raspberry Pi 5 running YOLOv8n, our scheduler eliminates all observed thermal throttling events during sustained 30-minute workloads. It outperforms a temperature-only reactive baseline by achieving a 6.8% higher frame rate (Cohen's d = 8.73) while consuming 1.9% less energy per frame. Furthermore, our optimized passive scheduling surpasses an actively cooled reference system in energy efficiency (Joules/frame), though active cooling remains superior for raw throughput. Through isolated ablations, we show that the dwell guard is necessary for run-to-run reproducibility. Finally, exploratory boundary probes indicate that the passive operating envelope closes at ambient temperatures (≥27∘C) where nonlinear leakage defeats DVFS-based control. These results indicate that, within the mapped envelope, correct scheduling can make mechanical cooling unnecessary for sustained edge inference on this platform.
Aayush Marasini, Zhaoxian Zhou
School of Computing Sciences and Computer Engineering University of Southern Mississippi, Hattiesburg, MS, USA
Deep neural network (DNN) inference on mobile devices often incurs high latency and energy consumption due to limited computing and memory resources. To enable energy-efficient DNN inference, most existing studies focus on dynamic voltage and frequency scaling (DVFS) for adjusting the computing frequency, while the impact of memory frequency on the inference performance has been greatly overlooked. In this paper, we consider the impact of memory frequency and computing frequency on DNN inference time, and jointly optimize these two frequencies together with communication resources for energy-efficient DNN inference. Based on a realistic inference time model, we formulate an optimization problem to minimize the energy consumption of all mobile devices under the deadline constraint. For local inference, we derive a near-optimal closed-form solution via convex optimization, while an optimal closed-form solution for transmission power is obtained for edge inference with the given bandwidth. Furthermore, we propose a low-complexity heuristic algorithm to effectively solve the overall problem with polynomial time complexity. Simulation results based on measured data show that the proposed near-optimal solution for local inference can achieve optimal performance under strict deadline constraints, with a performance gap of up to 2.5% compared with the optimal solution. Meanwhile, our proposed algorithm significantly reduces the energy consumption of devices by up to 10.4% compared to other methods.
Yunchu Han, Zhaojun Nan, Sheng Zhou +1
Beijing National Research Center for Information Science and Technology Department of Electronic Engineering, Tsinghua University, Beijing 100084, China · School of Electronics and Internet of Things Chongqing Polytechnic University of Electronic Technology, Chongqing 401331, China
Dynamic Voltage Frequency Scaling (DVFS) on resource-constrained embedded GPU platforms is essential for energy-efficient small language model (SLM) fine-tuning, as privacy- and personalization-driven adaptation increasingly requires local execution and involves repeated forward-backward optimization over many mini-batches, making it substantially more time- and energy-intensive than single-pass inference. To this end, 1) we first characterize the fine-tuning behavior of representative encoder-only SLMs of BERT variants, and autoregressive decoder-only SLMs of Pythia variants on GLUE benchmarks. In addition to the characterizations, 2) we propose a simple yet effective ML-based model selection that selects energy-optimal GPU DVFS settings on resource-constrained embedded platforms. Our results on NVIDIA Jetson AGX Orin demonstrate average 13.11% energy savings (up to 26.73%) over MAXN Mode 0, which has no explicit power cap.
Jurn-Gyu Park, Sanzhar Zholdybayev, Aidar Amangeldi +1
School of Engineering and Digital Sciences, Nazarbayev University, 010000 Astana, Kazakhstan