Partition-Aware Scheduling for Mobile Heterogeneous Inference Co-Execution

Zhuojin Li; Marco Paolieri; Leana Golubchik

cs.DCSep 13, 2026

Partition-Aware Scheduling for Mobile Heterogeneous Inference Co-Execution

Authors: Zhuojin Li, Marco Paolieri, Leana Golubchik

Organizations: University of Southern California, Los Angeles, CA, 90089, USA

Abstract

Modern mobile inference runs on heterogeneous platforms combining mobile GPUs with multiple CPU core clusters. Existing optimizations typically exploit either inter-operator parallelism, by assigning entire operators to CPU cores or to the GPU, or intra-operator parallelism, by partitioning each operator for CPU-GPU co-execution. We consider these two forms of parallelism together, to improve inference latency of tasks that can be represented by a static DAG of operators with predefined input/output tensor shapes (e.g., CNNs or vision transformers). We define the problem of partition-aware DAG scheduling for mobile heterogeneous inference, illustrating that the best strategy depends on the structure of the inference DAG, thus motivating a joint formulation capturing operator partition choices, device assignment, and execution order. We propose an online iterative search framework, which decomposes large DAGs into stages, focuses search on critical operators, and uses latency predictors to estimate partitioned execution without exhaustive profiling. Across representative mobile inference workloads, our approach achieves latency close to an offline solution while keeping scheduling overhead to a fraction of the model initialization cost, allowing platform-specific scheduling at deployment time.

Figures & tables

Platform	Processor	GPU	CPU (L)	CPU (M)
OnePlus 11	Snapdragon 8 Gen 2	Adreno 740	$1{\times}$ 3.2 GHz Cortex-X3	$2{\times}$ 2.8 GHz Cortex-A715
Motorola Edge Plus 2022	Snapdragon 8 Gen 1	Adreno 730	$1{\times}$ 3.0 GHz Cortex-X2	$2{\times}$ 2.5 GHz Cortex-A710
Pixel 5	Snapdragon 765G	Adreno 620	$1{\times}$ 2.4 GHz Kryo 475	$1{\times}$ 2.2 GHz Kryo 475
Pixel 4	Snapdragon 855	Adreno 640	$1{\times}$ 2.84 GHz Kryo 485	$2{\times}$ 2.42 GHz Kryo 485

Table 1: Mobile platforms used in the evaluation. CPU (L) and CPU (M) denote the large-core and medium-core CPU clusters exposed as logical scheduler devices in our experiments.

Panel of Figure 1 — Figure 1: Chain-dominated ResNet16 on OnePlus 11 phone. Schedule-only execution leaves CPU clusters mostly idle because few operators are ready at the same time. Partitioning operators creates intra-operator parallelism.

Panel of Figure 2 — Figure 2: Branch-structured models on OnePlus 11 phone. Inception-v3 has limited branch-level parallelism, so partitioning can use idle devices. HRNet exposes more ready operators, so partition-only execution can block useful inter-operator parallelism. Joint partitioning and scheduling balance these effects.

Symbol	Description
$G=(V,E)$	Inference DAG with operators $V$ and dependencies $E$
$\mathrm{pred}(i),\mathrm{succ}(i)$	Predecessor and successor sets of operator $i$
$\mathcal{D}$	Set of devices, e.g., GPU and CPU core clusters
$V_{p}$	Set of partitionable operators
$\mathcal{A}_{i}$	Candidate partition strategies for operator $i$
$a_{i}$	Selected partition strategy of operator $i$ , e.g., none, output-channel, input-channel, or spatial

Table 2: Notation used in the optimization model and baseline definitions.

Family	# Models	# Convolution Range	# Operators Range
Inception / Inception-ResNet	4	94–244	170–429
SqueezeNet / SqueezeResNet	4	26	46–51
PeleeNet	1	113	144
HRNet	9	91–325	149–535

Table 3: Workload summary. The full per-model list is reported in the Appendix ( Table 8 ).

Approach	Variant	OnePlus 11		Motorola 2022		Pixel 4		Pixel 5
Approach	Variant	Avg	Worst	Avg	Worst	Avg	Worst	Avg	Worst
Single-device	GPU	1.64	2.35	1.61	2.23	2.10	2.94	2.13	2.44
	CPU (L)	7.57	10.05	7.71	9.46	6.31	7.27	4.87	6.30
	CPU (M)	6.83	8.89	6.39	7.45	3.81	4.33	5.16	6.51
Partition-only	Min-local-latency	1.20	1.39	1.22	1.59	1.16	1.42	1.11	1.28
Schedule-only	Best	1.28	1.44	1.30	1.47	1.47	1.78	1.51	1.84

Table 4: Main simulation results. Latency is normalized to Joint-Gurobi . Best denotes the best result among evaluated list-scheduling heuristics within each baseline method, while Gurobi denotes the offline solver-based result for the corresponding restricted search space. Avg ( Worst ) reports the average (worst) normalized latency across 18 models. Full results are reported in the Appendix ( Table 9 ).

Figure 3: Distribution of normalized simulation latency across 18 models on OnePlus 11 (annotated values report the median). Latency is normalized to Joint-Gurobi . Additional results on other devices are reported in the Appendix ( Fig. 7 ).

Approach	Variant	OnePlus 11		Motorola 2022		Pixel 4		Pixel 5
Approach	Variant	Avg	Worst	Avg	Worst	Avg	Worst	Avg	Worst
Single-device	GPU	1.62	1.95	1.60	2.01	1.94	2.46	1.93	2.10
	CPU (L)	7.68	11.72	7.45	10.29	6.60	9.11	4.45	5.54
	CPU (M)	5.49	8.64	5.21	7.32	3.21	3.75	4.71	5.76
Partition-only	Min-local-latency	1.37	2.92	1.25	1.61	1.20	1.93	1.12	1.52
Schedule-only	Best	1.15	1.33	1.08	1.26	1.25	1.52	1.33	1.66

Table 5: Main measurement results. Latency is normalized to Joint-Gurobi . Full results are reported in the Appendix ( Table 10 ).

Figure 4: Distribution of measured end-to-end latency across 18 models on OnePlus 11, normalized to Joint-Gurobi (annotated values report the median). Additional results on other devices are reported in the Appendix ( Fig. 8 ).

Figure 5: Scheduling overhead compared to TFLite model initialization time, measured on OnePlus 11. Each bar is stacked by prediction overhead and iterative scheduling overhead. Full measurements are reported in the Appendix ( Table 11 ).

Panel of Figure 6 — Figure 6: Sensitivity of scheduling quality to iterative-search cost. Latency is normalized to Joint-Gurobi . Each point corresponds to one configuration, where lower normalized latency and lower search cost are preferred. Full data are reported in the Appendix ( Table 12 ).

Panel of Table 6 — Table 6: Cross-platform scheduling sensitivity. Each entry reports normalized latency when the schedule is generated using the row device and executed on the column device.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

1	$\boldsymbol{\pi}\leftarrow\text{initial partition }\{\pi_{i}\}_{i\in V}$
2	$S\leftarrow\textnormal{{Schedule}}(G,\boldsymbol{\pi},\hat{\tau},\mathcal{D})$
3	for $t=1,\ldots,B$
4	$V_{\mathrm{samp}}\leftarrow\textnormal{{SampleOperators}}(G,V_{p},\boldsymbol{\pi},S)$
5	$\boldsymbol{\pi}^{\star}\leftarrow\boldsymbol{\pi}$ , $S^{\star}\leftarrow S$ , $\Delta^{\star}\leftarrow 0$
6	for each $i\in V_{\mathrm{samp}}$
7	$\mathcal{M}_{i}\leftarrow\textnormal{{GenerateUpdates}}(i,\Pi_{i},\pi_{i})$

Appendix

Algorithm 1 Iterative Search for Partition-Aware Scheduling

Operator	Device	Features
Convolution	GPU	Input/output height, input/output width, input/output channels, filter shape, stride, group count, input/output sizes, filter size, FLOPs, grid size ( $x,y,z$ -dims), workgroup size ( $x,y,z$ -dims), workgroup count ( $x,y,z$ -dims), total workgroup count
	CPU	Input/output height, input/output width, input/output channels, filter shape, stride, group count, input/output sizes, filter size, FLOPs
Linear	GPU	Input/output height, input/output width, input/output channels, input/output sizes, weight size, FLOPs, grid size ( $x,y,z$ -dims), workgroup size ( $x,y,z$ -dims), workgroup count ( $x,y,z$ -dims), total workgroup count
	CPU	Input/output height, input/output width, input/output channels, input/output sizes, weight size, FLOPs
Pooling	CPU / GPU	Input/output height, input/output width, input/output channels, input/output sizes, kernel shape, stride
Others	CPU / GPU	Input/output height, input/output width, input/output channels, input/output sizes

Appendix

Table 7: Features used for training GBDT predictors across different operators and devices.

Model	# Conv	# Ops
inceptionv3	94	170
inceptionv4	149	271
inceptionresnetv1	132	231
inceptionresnetv2	244	429
squeezenet_v1_0	26	47
squeezenet_v1_1	26	46

Appendix

Table 8: Neural-network workloads. All models are exported to TFLite and evaluated with input resolution

224{\times}224

Approach	Variant	OnePlus 11		Motorola 2022		Pixel 4		Pixel 5
Approach	Variant	Avg	Worst	Avg	Worst	Avg	Worst	Avg	Worst
Joint-search	Gurobi	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Single-device	GPU	1.64	2.35	1.61	2.23	2.10	2.94	2.13	2.44
	CPU (L)	7.57	10.05	7.71	9.46	6.31	7.27	4.87	6.30
	CPU (M)	6.83	8.89	6.39	7.45	3.81	4.33	5.16	6.51
Partition-only	Min-local-latency	1.20	1.39	1.22	1.59	1.16	1.42	1.11	1.28

Appendix

Table 9: Simulation results across four phones. Latency is normalized to Joint-Gurobi ; lower is better.

Panel of Figure 7 — Figure 7: Distribution of simulation latency across 18 models, normalized to Joint-Gurobi (annotated values report the median)

Approach	Variant	OnePlus 11		Motorola 2022		Pixel 4		Pixel 5
Approach	Variant	Avg	Worst	Avg	Worst	Avg	Worst	Avg	Worst
Joint-search	Gurobi	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Single-device	GPU	1.62	1.95	1.60	2.01	1.94	2.46	1.93	2.10
	CPU (L)	7.68	11.72	7.45	10.29	6.60	9.11	4.45	5.54
	CPU (M)	5.49	8.64	5.21	7.32	3.21	3.75	4.71	5.76
Partition-only	Min-local-latency	1.37	2.92	1.25	1.61	1.20	1.93	1.12	1.52

Appendix

Table 10: Full measurement results across four phones. Latency is normalized to Joint-Gurobi ; lower is better

Panel of Figure 8 — Figure 8: Distribution of measured end-to-end latency across 18 models, normalized to Joint-Gurobi (annotated values report the median)

Model	Prediction overhead (ms)	Scheduling overhead (ms)	Initialization time (ms)
inceptionv3	135	303	1048
inceptionv4	130	420	1534
inceptionresnetv1	88	249	1100
inceptionresnetv2	123	627	1783
squeezenet_v1_0	62	25	479
squeezenet_v1_1	51	24	482

Appendix

Table 11: Overhead compared with TFLite model initialization time

	Configuration	Normalized latency	Search cost (s)
Default	HEFT, split=3, staging	1.03	10.8
Heuristics	PEFT	1.04	12.3
	PSLS	1.04	16.7
	Lookahead-HEFT	1.03	28.3
	Min-local-latency Init	1.05	17.7
Staging	no staging	1.14	24.7

Appendix

Table 12: Ablation and split-sensitivity results for iterative search. Lower normalized latency (on OnePlus 11) and lower search cost are preferred.

Timeout (s)	Normalized latency	Search cost (s)
Unlimited	1.03	10.78
0.75	1.03	9.30
0.5	1.04	7.96
0.4	1.04	7.12
0.3	1.04	6.08
0.2	1.05	4.54

Appendix

Table 13: Timeout-sensitivity results for iterative search. Lower timeout values reduce search cost but can degrade normalized latency (measured on OnePlus 11).

On-Device Inference

On-device inference runs machine-learning models locally on phones, wearables, microcontrollers, and edge hardware to reduce latency, energy use, and data sharing.

See all 189 papers

Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds
May 20, 2026Lakshani Manamperi, Disumi Pathirana, Thiwanka Pathirana +2Efficient Neural Network Inference Edge Computing
Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka.
Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inference
May 31, 2026Zhiyao Xu, Aoxue Liu, Zhanjie Ding +3Communication-Efficient Mixture-of-Experts Mixture-of-Experts Inference
Tsinghua Shenzhen International Graduate School, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China
Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge
Apr 11, 2026Jiesong Chen, Jun You, Zhidan Liu +1Edge Inference Efficient Neural Network Inference
Department of Computer Science, City University of Hong Kong, Hong Kong, China · INTR Thrust, System Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

May 20, 2026J/K move · Enter open · S save

Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds

Lakshani Manamperi, Disumi Pathirana, Thiwanka Pathirana, Nipun Premarathna+1

Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka.

Efficient Neural Network Inference Edge Computing Memory-Efficient Inference On-Device Inference

Open paper AskPDF arXiv

Deploying large deep neural networks on memory-constrained mobile devices is a central challenge in edge ML. While compression, pruning, and quantization reduce per-parameter cost, transformer-based models remain too large for the 3.3-7.4 GB RAM envelope of commodity Android handsets. We present the DNN pipeline scheduling subsystem of CROWDio, which achieves practical ONNX inference across resource-constrained Android workers without model modification, by distributing memory pressure across devices via five mechanisms: JIT deferred partition loading, a single-partition-resident constraint, a 4-tier affinity scheduler, a zlib-compressed tensor transport, and a streaming 1:1 dependency model. Evaluated on DistilBERT (Sanh et al., 2019) (approximately 67 M parameters, SST-2) across five Android handsets over ten runs, our system holds peak per-device RSS to 43+-2 MB and limits battery draw to 50+-3 mAh per run, while streaming concurrency cuts batch latency 34% below barrier synchronisation.

Abstract

On-Device Inference

Explore similar work

Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds

Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inference

Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge