HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding

Hanbin Bae; Heejin Choi; Joohyung Lee; Byoung Jin Choi; Sangjun Park; Hoon-Young Cho

cs.CLOct 15, 2026

HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding

Authors: Hanbin Bae, Heejin Choi, Joohyung Lee, Byoung Jin Choi, Sangjun Park, Hoon-Young Cho

Organizations: AI Solution Team, Samsung Research, Seoul, Republic of Korea

Abstract

Device-cloud speculative decoding reduces server computation, but growing a deep draft tree with a small language model (SLM) requires repeated passes on a mobile neural processing unit (NPU). We present HyCo, which runs the server model's native multi-token prediction (MTP) head on the device to build a deep chain and uses one SLM pass to add alternative branches. The server returns the hidden state, input embedding, and key-value entries needed by the head instead of executing its sequential steps. Quantization, a restricted vocabulary, and a 1K-token window fit both drafters to the NPU. An ε-threshold rule additionally relaxes exact preservation of the verifier's sampling distribution. In single-client runs with a Galaxy S26 Ultra over LTE/5G, HyCo reduces server GPU time per output token relative to server-side MTP self-speculation by 38.6% on Spec-Bench and 29.2% on 32K-token summarization. With the same acceptance rule, three runs of each system on Spec-Bench show 9.0 percentage points more saving and 1.9x delivery speed than an SLM-only drafter. HyCo differs from Cloud+MTP by +0.2 judge-score points (paired 95% CI: -0.9 to +1.3); exploratory translation favors SpecEdge-ε. Without relaxation, HyCo exact saves 27.7% and 11.9% on the two datasets.

Figures & tables

Figure 1: HyCo on Spec-Bench: server GPU time relative to exact Cloud+MTP and delivery rate relative to the standalone SLM. HyCo and HyCo -2P use

\epsilon{=}0.005

; HyCo -2P uses two SLM passes. Italic labels denote illustrative, not evaluated, deployment priorities. KV: key–value. HyCo shows a three-run mean; other results use one run.

	D	H	S	V
SpecEdge	✓	–	–	✓
SLED	✓	–	–	✓
HAT	✓	–	–	✓
Li et al. (2026a)	✓	–	✓	✓
Synera	✓	–	–	–
Cloud+MTP	–	✓	✓	✓

Table 1: Comparison of draft placement and verification. D : device-side drafting; H : verifier-native draft head; S : server-computed hidden states or KV; V : token verification. For HyCo , D+H+S denotes running the native head on the device using server states. † Verification with

\epsilon

-threshold acceptance does not preserve the sampling distribution.

Figure 2: Hybrid draft tree in one HyCo round. The MTP chain

y_{1:4}

and siblings

y^{\prime}_{k}

are augmented by a single SLM pass, then pruned to

B{=}32

draft nodes plus the root. Twelve nodes are shown for clarity; the green path illustrates exact verification (default

\epsilon

rule: § 3.5 ).

Figure 3: MAT against judge score of each acceptance rule on HyCo ’s tree on UC-val. The shaded band is the observed score range of exact-verification reference runs used for screening, not a quality-equivalence region. Repeat run:

\epsilon{=}0.02

at every node, run again with node logging. Over the other runs,

g

falls as MAT rises.

Configuration	Rule	$\Delta g$ (%)	MAT	Tok/s	$\Delta g$ (%)	MAT	Tok/s
		SB-480			$\infty$ Bench Sum 32K
Cloud+MTP	exact	0.0	3.05	–	0.0	2.37	–
SpecEdge-style	exact	$-20.6$	3.53	16.1	$-2.9$	2.52	10.0
SpecEdge-style + PD	exact	$-23.9$	3.69	12.1	$-2.9$	2.52	8.2
SpecEdge- $\epsilon$	$\epsilon$	$-29.6\pm 0.2$	4.13	$18.9\pm 0.1$	$-15.7$	3.08	12.4
HyCo	$\epsilon$	$-38.6\pm 0.2$	4.71	$35.0\pm 0.7$	$-29.2$	3.73	21.0

Table 2: Server cost, acceptance, and delivery speed.

\Delta g

is the paired change from Cloud+MTP; its reference

g

is 7.81/23.91 ms per token on SB/32K. Lower

\Delta g

is better. HyCo and SpecEdge-

\epsilon

SB entries average three runs (

\pm

: run SD); all other entries use one run. MAT run SDs are both 0.012 for the repeated systems. –: Cloud+MTP was measured without the phone, so no mobile delivery rate is available. PD uses 558 SB turns and 49 summaries; recording details are in Appendix G . Matching rules does not enforce equal quality.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	SLM Drafter	MTP Drafter
Weight bit map	the checkpoint’s 2/4/8 bits	W4 (W8: post-training)
Activations	16-bit	16-bit
KV / inputs	8-bit KV cache	16-bit (8-bit on the wire)
Embeddings	2/4-bit LUT	8-bit LUT, 242 MB ∗
Size	754 MB NPU graph	123 MB NPU graph at W4
Our training	none (graph surgery only)	QAT, 100 steps (W4)

Appendix

Table 3: Integer formats of the two drafters on the NPU. W4, W8: 4- and 8-bit weights. LUT: lookup table on the CPU. ∗ For the MTP Drafter’s own drafted tokens.

Initialization	BF16 release
Teacher	original head (BF16)
Trained	linear weights only (norms frozen)
Weight bits	4, all linear layers incl. output layer
Weight clipping	per-channel mean-squared-error search, ratio 1.0–0.6
Loss	KL on response tokens
Optimizer	AdamW, $\beta{=}(0.9,0.95)$ , no weight decay, gradient-norm clip 1.0

Appendix

Table 4: Final UC-tr QAT of the MTP head’s 4-bit weights.

	MTP head
Configuration	Place	Weights	Vocabulary	Window	$K$	Siblings	SLM passes	Nodes / rows	Acceptance
Cloud+MTP	server	BF16	262K	whole prefix	4	–	–	4 / 5	exact
SpecEdge-style	–	–	–	–	0	–	4	32 / 33	exact
+ PD	–	–	–	–	0	–	4 + PD	32 / 33	exact
SpecEdge- $\epsilon$	–	–	–	–	0	–	4	32 / 33	$\epsilon{=}0.005$
HyCo	mobile	W4	86K	1,024	4	depths 1–2	1	32 / 33	$\epsilon{=}0.005$

Appendix

Table 5: Settings of every configuration in Tables 2 and 7 . Every tree drafted on the mobile device uses the SLM Drafter with the 86K keep set and a 1,024-token context.

K

: MTP chain depth; Siblings: depths of the runner-up siblings; Nodes / rows: draft nodes after the trim and verifier rows per round, including the root.

\epsilon

\epsilon

-threshold acceptance at every node (§ 3.5 ). PD continues drafting while a verification reply is in flight. All SLM-based device configurations use the final UC-tr activation calibration.

Keep set
Setting	MAT	$g$	Device
21K	2.98	7.00	45.7
45K	3.25	6.44	47.8
86K ✓	3.36	6.23	52.9
262K, full vocabulary	2.85	7.57	77.9
SLM context

Appendix

Table 6: Selection runs on UC-val for the keep set, the SLM Drafter’s context, and the MTP head’s window. Blocks ran in different sessions and setups, so compare rows only within a block. ✓: chosen;

g

: server ms per output token; Device: on-device ms per round.

Configuration	Generations	Judge mean	Paired $\Delta$ vs. Cloud+MTP	Paired $n$
Cloud+MTP	1	90.15	–	–
SpecEdge-style	1	90.21	$+0.1$ $[-1.3,+1.4]$	398
SpecEdge-style + PD	1	90.05	$-0.1$ $[-1.6,+1.2]$	398
SpecEdge- $\epsilon$	3	$90.60\pm 0.61$	$+0.5$ $[-0.7,+1.8]$	396
HyCo	3	$90.34\pm 0.50$	$+0.2$ $[-0.9,+1.3]$	398
HyCo -2P	1	90.38	$+0.2$ $[-1.0,+1.5]$	398

Appendix

Table 7: SB judge results. For the repeated systems, the judge mean is the mean

\pm

sample SD of the three run-level means, each using its available eligible answers. Paired differences instead use per-question three-run averages on the common complete-case set. Thus, a paired difference need not equal the difference of the displayed marginal means. Other rows use one generation and one judge score per output. All intervals are paired question-bootstrap 95% CIs.

Comparison	All	MT-bench	Translation	Summarization	Math	RAG
HyCo $-$ SpecEdge- $\epsilon$	$-0.2$ $[-1.2,+0.8]$	$+0.2$ $[-1.5,+1.8]$	$-2.6$ $[-5.2,-0.2]$	$+0.3$ $[-0.1,+0.7]$	$+0.5$ $[-1.3,+2.7]$	$+0.6$ $[-2.7,+3.7]$
HyCo $-$ Cloud+MTP	$+0.2$ $[-0.9,+1.3]$	$-0.2$ $[-1.9,+1.4]$	$+0.3$ $[-2.6,+3.3]$	$+0.0$ $[-0.3,+0.4]$	$-0.1$ $[-1.6,+1.9]$	$+0.9$ $[-2.9,+4.9]$
SpecEdge- $\epsilon$ $-$ Cloud+MTP	$+0.5$ $[-0.7,+1.8]$	$-0.4$ $[-2.1,+1.5]$	$+2.8$ $[+0.1,+6.2]$	$-0.3$ $[-0.8,+0.2]$	$-0.2$ $[-2.5,+2.1]$	$+0.3$ $[-4.5,+5.2]$

Appendix

Table 8: Repeated SB quality contrasts, in judge points: mean difference and paired 95% CI. All: non-QA first turns,

n=398

for HyCo –Cloud+MTP and

n=396

otherwise. Task sizes are 80 except summarization (78) and math in comparisons involving SpecEdge-

\epsilon

(78). These 15 task-wise contrasts are exploratory and not multiplicity-adjusted. A CI spanning zero does not establish equivalence.

Configuration	F1	$\Delta$ F1	$\Delta$ recall	$\Delta$ precision
Cloud+MTP	29.6	–	–	–
SpecEdge-style	31.3	$+1.6$ $[-1.2,+4.4]$	$+1.3$ $[-0.7,+3.3]$	$+5.8$ $[+1.3,+10.2]$
SpecEdge-style + PD	29.1	$-0.9$ $[-4.1,+2.1]$	$+1.1$ $[-1.0,+3.0]$	$-1.7$ $[-7.1,+4.1]$
SpecEdge- $\epsilon$	28.5	$-1.2$ $[-3.6,+1.4]$	$-2.2$ $[-4.3,0.0]$	$+6.2$ $[+2.0,+10.6]$
HyCo	30.0	$+0.4$ $[-2.3,+3.1]$	$-0.1$ $[-2.3,+2.1]$	$+1.9$ $[-2.6,+6.2]$
HyCo -2P	28.0	$-1.7$ $[-4.3,+1.0]$	$-1.9$ $[-4.1,+0.3]$	$+4.3$ $[-0.3,+8.9]$

Appendix

Table 9: HELMET model-based evaluation on 50

\infty

Bench Sum 32K summaries per configuration (49 for PD), one generated answer and one metric evaluation per answer. Values are percentage points; differences are against Cloud+MTP with paired 95% CIs. F1 includes the fluency factor. PD differences use the 49 common summaries, so they need not equal differences of displayed marginal means. Recall measures key-point coverage; precision measures support by the reference expert summary. These are not human factuality judgments against full novels.

Our classification	Decreases	Increases
Substantive HyCo error	6	–
Minor difference	2	1
Score/content discrepancy	5	9
Cloud+MTP defect	–	2
Selected cases	13	12

Appendix

Table 10: Manual inspection of large score changes. No scored item is removed from the quantitative analysis.

SB-480
Configuration	$g$ (ms/token)	Tok/s	Draft ms/token	Tokens/GPU-busy s	GPU-h/1M tokens
Cloud+MTP	7.81	–	–	128.1	2.17
SpecEdge-style	6.20	16.1	42.8	161.3	1.72
SpecEdge-style + PD	5.94	12.1	40.1	168.4	1.65
SpecEdge- $\epsilon$	5.50	18.9	37.0	181.9	1.53
HyCo	4.80	35.0	12.1	208.5	1.33

Appendix

Table 11: Metrics recomputed from the finalized per-request records. Repeated SB configurations average each run’s metric; other entries use one run. Tokens per GPU-busy second equal

1000/g

per run, and GPU-hours per million tokens equal

g/3.6

when

g

is in milliseconds. These are execution-efficiency measures for one client, not multi-client serving throughput or priced service costs. –: no applicable server or device measurement.

Artifact	Use	License
Models
Gemma 4 26B-A4B-it ( Gemma Team, 2026 ) , 25.2B parameters (3.8B active), AWQ 4-bit ( Lin et al., 2024 ) : cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit	verifier	Apache 2.0
Its MTP head, 420M parameters: google/gemma-4-26B-A4B-it-assistant	MTP Drafter	Apache 2.0
Gemma 4 E2B-it, Google’s QAT checkpoint, 2.3B effective parameters: google/gemma-4-E2B-it-qat-mobile-transformers ( Google DeepMind, 2026 )	SLM Drafter	Apache 2.0
Muse-Glimmer-30B ( Meta Superintelligence Lab, 2026 ) , 29.6B parameters, NVFP4: nvidia/Muse-Glimmer-30B-NVFP4 ( NVIDIA, 2026 )	cross-family verifier (App. H )	Apache 2.0
Its DFlash2 head ( Chen et al., 2026 ) , 2.8B parameters: incoai/Muse-Glimmer-30B-DFlash2 ( Inco AI, 2026 )	its server-side head	Apache 2.0

Appendix

Table 12: Artifacts we used, with the licenses that their distributors declare on Hugging Face, on GitHub or in the license file. Model checkpoints are Hugging Face repositories. a Research, evaluation and education; commercial deployment needs a separate license. b UC-tr: keep-set counts and 60 disjoint prompts for final head-QAT and activation calibration (Appendix A ).

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
Oct 11, 2026Hongliang Cao, Xiuyu Li, Siqi Wang +3LLM Inference Acceleration Speculative Decoding
Beihang University · Monash University · The Hong Kong University of Science and Technology
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Jul 14, 2026Jincheng Xie, Runheng Liu, Heyan Huang +4Mixture-of-Experts Inference Speculative Decoding
Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra +2
AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices
Apr 28, 2026Ma Zirui, Fan Zhihua, Li Wenxing +4Compute-in-Memory On-Device Language Model Inference
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Oct 11, 2026J/K move · Enter open · S save

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

Hongliang Cao, Xiuyu Li, Siqi Wang, Ziming Wang+2

Beihang University · Monash University · The Hong Kong University of Science and Technology

LLM Inference Acceleration Speculative Decoding Mixture-of-Experts Inference Expert Offloading

Open paper AskPDF arXiv

Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3. Code is available at https://github.com/Chlience/SpecFit.

Abstract

Speculative Decoding

Explore similar work

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices