PhaseGate: Phase-Aware CPU Retrieval Scheduling for On-Device LLMs on Unified Memory

Seoyoon Yum; Sehoon Kim

cs.LGOct 3, 2026

PhaseGate: Phase-Aware CPU Retrieval Scheduling for On-Device LLMs on Unified Memory

Authors: Seoyoon Yum, Sehoon Kim

Organizations: KAIST

Abstract

On-device assistants run GPU-based LLM inference alongside CPU retrieval on unified-memory systems. Under a saturated local-retrieval workload, four concurrent retrieval workers raise 95th-percentile (p95) decode latency by 60-61% on two M4 systems, whereas prefill latency rises by only 5.7-6.9%. We study LLM phase as an admission signal for independent CPU retrieval under controlled LLM workloads. PHASEGATE calibrates separate concurrency limits for prefill and decode, selecting four and one on our base-M4 configuration. Under a backlogged queue, it achieves 2.0 times the aggregate retrieval throughput of the best tested feasible fixed policy, with both p95 LLM latency metrics within 1.25 times their no-retrieval baselines in all seven held-out runs. A phase-blind control, TimeGate, uses the same two limits on a calibration-derived schedule without observing LLM phase. It achieves similar retrieval throughput but violates the output-token latency limit in every run. M2 and M2 Pro Mac minis reproduce the policy ordering, while output-length sweeps show that the advantage narrows as decode occupies more of each request.

Figures & tables

Figure 1: (a) A discrete-GPU server’s CPU search and GPU LLM use separate memory pools, whereas Apple Silicon shares unified memory. (b) On the measured base-M4 configuration, explicit PASS/FAIL labels show the tradeoff: Fixed-1 meets both latency limits but restricts retrieval, while Fixed-2 retrieves more but violates the output-token limit. PhaseGate raises its admission cap during prefill and lowers it during decode; in-flight work may cross the boundary. Schematic bars denote admission caps; PASS means both latency limits are met at

B=1.25

Figure 2: (a) Four concurrent HNSW workers show the same p95 asymmetry on a fanless M4 Air and fan-cooled base-M4 Mini; each Mini bar is the median of three runs with 300 LLM requests per run. (b) On the base-M4 Mini, PhaseGate /Fixed throughput falls with output length (five pairs per length); the right axis shows the median prefill fraction measured under PhaseGate

4{\rightarrow}1

Fixed cap	Retrieval throughput (kQPS) $\uparrow$	Worst p95 TPOT $\downarrow$ ( $\times$ baseline)	PhaseGate caps	Retrieval throughput (kQPS) $\uparrow$	Worst p95 TPOT $\downarrow$ ( $\times$ baseline)
0	0	1.021	2 $\rightarrow$ 0	0.79	1.265 $\dagger$
1	0.78	1.247	2 $\rightarrow$ 1	1.1	1.225
2	1.4	1.393 $\dagger$	4 $\rightarrow$ 0	1.2	1.271 $\dagger$
4	2.1	1.690 $\dagger$	4 $\rightarrow$ 1	1.6	1.177
			4 $\rightarrow$ 2	1.8	1.384 $\dagger$

Table 1: Base-M4 calibration sweep at

B=1.25

. Throughput is the median of three runs; TPOT is the maximum per-run p95 ratio. Unrounded ratios determine eligibility;

\dagger

exceeds

B

. All maximum TTFT ratios are

<1.1

; bold policies are selected.

Metric	Fixed -1	PhaseGate $4{\rightarrow}1$	TimeGate 4/1
Runs meeting both limits	7/7	7/7	0/7
p95 TPOT increase (%)	17	18	60
p95 TTFT increase (%)	0.61	4.6	4.3
Retrieval throughput (kQPS)	0.78	1.6	1.5
p95 transition gap (ms)	18	23	24

Table 2: Base-M4 evaluation at

B=1.25

. Latency, throughput, and transition rows report medians across seven paired randomized repetitions with 250 requests per policy; “7/7” means both latency limits pass in all seven.

Device	$B$	vs. Fixed throughput ( $\times$ )	vs. TimeGate throughput ( $\times$ )	Fixed passing runs	PhaseGate passing runs	TimeGate passing runs
Base-M4 Mini	1.25	2.0	1.0	7/7	7/7	0/7
Base-M2 Mini	1.25	1.8	1.0	5/5	5/5	0/5
M2 Pro Mini	1.10	2.5	–	7/7	7/7	–

Table 3: Mac mini evaluations, each comparing Fixed -1 and PhaseGate

4{\rightarrow}1

. Throughput entries are median paired run ratios of PhaseGate to each control (two significant digits); passing runs meet both device-specific latency criteria (Appendix A.1 ), and “–” denotes an unevaluated control.

Measure	Median of per-run p95s (ms)	Observed maximum (ms)
16-query call service time	27.7	34.3
Drain time to at most one active call	26.6	31.5
Completion of all calls active at decode entry	29.0	32.2

Table 4: CPU service and drain times from seven held-out PhaseGate

4{\rightarrow}1

traces (250 requests per run). The p95 column reports the median of seven per-run p95s; maxima span all seven traces.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

M4 primary	Mac mini (Mac16,10); 4P+6E CPU, 10-core GPU, 16 GB UMA
M2 replication	Mac mini (Mac14,3); 4P+4E CPU, 10-core GPU, 16 GB UMA
M2 Pro earlier	Mac mini (Mac14,12); 6P+4E CPU, 16-core GPU, 16 GB UMA
M4 runtime	macOS 26.5/26.6.1 (continuous study/duty sweep); MLX 0.31.2; MLX-LM 0.31.3; FAISS 1.14.3; MLX memory limit 5.5 GB
M4 model	mlx-community/Qwen2.5-1.5B-Instruct-4bit , revision 8b403126fc14f14cfc99bb4cfa72ecbc129ea677
Workload	2,048 input tokens/128 decode steps; 100k 384-D vectors; queue kept nonempty

Appendix

Table A1: Fan-cooled scheduler devices and shared setup (P/E CPU cores; UMA unified memory).

Policy	Median	p95 TPOT / baseline			Max TTFT	Eligible
	kQPS	Run 0	Run 1	Run 2	/ baseline	at $B=1.25$
Fixed -0	0.000	1.009	0.996	1.021	0.980	Yes
Fixed -1	0.784	1.213	1.235	1.247	1.008	Yes
Fixed -2	1.379	1.325	1.294	1.393	1.016	No
Fixed -4	2.136	1.667	1.690	1.667	1.074	No
PhaseGate $2{\rightarrow}0$	0.793	1.261	1.265	1.258	1.002	No

Appendix

Table A2: Per-repetition calibration results. Ratios are rounded to three decimals for display; eligibility uses unrounded values. Run indices are repetition identifiers, not a simultaneous paired comparison.

Figure A1: M2 Mini output-length replication with unchanged PhaseGate

4{\rightarrow}1

and Fixed -1 caps. The 128-token point summarizes five pairs and the 512-token point summarizes three; M2 is analyzed separately from M4.

LLM Inference Scheduling

Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.

See all 166 papers

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators
Jun 14, 2026Shun Usami, Venkatram Vishwanath, E. Wes BethelAI Accelerator Inference LLM Inference
Department of Computer Science San Francisco State University San Francisco, CA, 94132 · Argonne National Laboratory Lemont, IL, 60439 · Lawrence Berkeley National Laboratory Berkeley, CA, 94720
Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)
Mar 29, 2026Rajveer Bachkaniwala, Chengqi Luo, Richard So +2LLM Inference LLM Inference Serving
Georgia Tech, Atlanta, USA
Threshold-Based Exclusive Batching for LLM Inference
May 30, 2026Weifang Zhang, Yuzhou Nie, Bowen Pang +2GPU Acceleration LLM Inference
The Hong Kong Polytechnic University · UC Santa Barbara · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences +1

Jun 14, 2026J/K move · Enter open · S save

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

Shun Usami, Venkatram Vishwanath, E. Wes Bethel

Department of Computer Science San Francisco State University San Francisco, CA, 94132 · Argonne National Laboratory Lemont, IL, 60439 · Lawrence Berkeley National Laboratory Berkeley, CA, 94720

AI Accelerator Inference LLM Inference Prefill-Decode Disaggregation LLM Inference Serving

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge. While GPUs dominate current deployments, a growing number of AI accelerators claim advantages for LLM inference, yet it remains unclear under which conditions such accelerators outperform GPUs in practice. Recent inference systems decompose execution into Prefill and Decode phases, which exhibit distinct computational characteristics and latency metrics, commonly captured by time to first token (TTFT) and time per output token (TPOT). This paper presents a phase-aware evaluation of LLM inference performance across GPUs and emerging AI accelerators using a common model, Llama2-7B. By separately measuring Prefill and Decode performance, we reveal that accelerator advantages differ by phase and metric. Our results show that GPUs consistently excel in the compute-intensive Prefill phase, while GroqRack achieves significantly lower TPOT during Decode (batching not currently supported). However, GPUs regain an advantage in Decode throughput as batch size increases. These findings demonstrate that each platform exhibits distinct phase-dependent strengths. We further analyze heterogeneous Prefill/Decode disaggregation across different accelerator platforms, identifying performance gains and the workload and network conditions under which such gains are realized.

Abstract

LLM Inference Scheduling

Explore similar work

Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT)

Threshold-Based Exclusive Batching for LLM Inference