Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LM

Krzysztof Fonal

cs.CLMar 22, 2026

Cross-Family Speculative Decoding for Polish Language Models on AppleSilicon: An Empirical Evaluation of Bielik11B with UAG-Extended MLX-LM

Authors: Krzysztof Fonal

Organizations: Wrocław University of Science and Technology, Poland

Abstract

Speculative decoding accelerates LLM inference by using a small draft model to propose k candidate tokens for a target model to verify. While effective for same-tokenizer pairs on high-bandwidth GPUs, its applicability to cross-family pairs with mismatched tokenizers and consumer-grade unified memory remains underexplored. We extend the MLX-LM framework with Universal Assisted Generation (UAG) to enable cross-tokenizer speculative decoding on Apple Silicon. We evaluate Bielik 11B-Instruct (Mistral-based) as the target model, paired with three draft models: Bielik 1.5B (Qwen-based with custom tokenizer), Qwen2.5-1.5B, and Llama 3.2-1B. Experiments on three Polish-language datasets (Wikipedia, pl_alpaca, synthetic) use draft lengths k in {2, 4, 6} to compare naive and context-aware token translation. Results show: (1) context-aware translation consistently improves acceptance rates across all configurations; (2) the Polish-specialized Bielik 1.5B achieves lower acceptance than general-purpose Qwen2.5 and Llama 3.2 drafters; (3) throughput on Apple Silicon is content-dependent, reaching 1.7x speedup for structured text but failing for varied instructions; and (4) verification cost on unified memory does not amortize as theory predicts because both models are memory-bandwidth bound, making sequential drafting expensive relative to batched verification. We propose a hardware-aware speedup formula and characterize conditions for cross-family speculative decoding on Apple Silicon. This is the first systematic evaluation of cross-family speculative decoding for Polish LLMs and the first empirical study of UAG-based decoding on unified memory architectures.

Figures & tables

Model	Role	Base Arch.	Tokenizer
Bielik 11B v3 Instruct	Target	Mistral	Mistral ( $\sim$ 32K)
Bielik 1.5B v3 Instruct	Draft	Qwen2.5	APT4 (Polish)
Qwen2.5 1.5B Instruct	Draft	Qwen2.5	Qwen ( $\sim$ 152K)
Llama 3.2 1B Instruct	Draft	Llama 3.2	Llama ( $\sim$ 128K)

Table 1: Models used in experiments. All models are cross-family relative to the Bielik 11B target.

Drafter	Condition	Wikipedia	pl_alpaca	Synthetic
Bielik-1.5B	No translation	28.5	12.5	11.8
	Naive translation	19.1	11.9	16.7
	Context-aware	31.1	23.9	22.2
Llama-3.2-1B	No translation	26.7	12.1	10.6
	Naive translation	8.9	4.2	3.1
	Context-aware	42.0	36.0	36.5

Table 2: Mean token acceptance rate (%) for

k=2

across datasets and translation conditions.

		Wikipedia		pl_alpaca		Synthetic
Drafter	Condition	TPS	Speedup	TPS	Speedup	TPS	Speedup
—	Baseline	14.83	1.00 $\times$	14.70	1.00 $\times$	14.72	1.00 $\times$
Bielik-1.5B	No translation	13.72	0.93 $\times$	10.80	0.74 $\times$	10.68	0.73 $\times$
	Naive	10.32	0.70 $\times$	9.69	0.66 $\times$	9.72	0.66 $\times$
	Context-aware	12.85	0.87 $\times$	11.92	0.81 $\times$	11.02	0.75 $\times$
Llama-3.2-1B	No translation	14.03	0.95 $\times$	11.24	0.77 $\times$	11.02	0.75 $\times$

Table 3: Mean generation throughput (TPS) and speedup for

k=2

. Speedup is relative to the per-dataset autoregressive baseline.

		Wikipedia		pl_alpaca		Synthetic
Drafter	$k$	$\alpha$ (%)	Speedup	$\alpha$ (%)	Speedup	$\alpha$ (%)	Speedup
Bielik-1.5B	2	31.1	0.87 $\times$	23.9	0.81 $\times$	22.2	0.75 $\times$
	4	38.5	0.59 $\times$	29.5	0.51 $\times$	29.1	0.46 $\times$
Llama-3.2-1B	2	42.0	1.06 $\times$	36.0	0.97 $\times$	36.5	0.98 $\times$
	4	49.7	0.66 $\times$	43.7	0.59 $\times$	44.5	0.61 $\times$
Qwen2.5-1.5B	2	44.6	1.06 $\times$	41.0	0.98 $\times$	42.7	1.00 $\times$

Table 4: Effect of draft length

k

on throughput and acceptance rate (context-aware translation, all three datasets).

Dataset	Drafter	$k=2$	$k=4$
Wikipedia	Bielik-1.5B	44.0 $\pm$ 7.1%	90.1 $\pm$ 19.8%
	Llama-3.2-1B	38.3 $\pm$ 5.5%	81.2 $\pm$ 13.3%
	Qwen2.5-1.5B	40.8 $\pm$ 5.3%	77.7 $\pm$ 13.1%
pl_alpaca	Bielik-1.5B	40.2 $\pm$ 6.8%	89.9 $\pm$ 18.5%
	Llama-3.2-1B	38.1 $\pm$ 4.8%	82.0 $\pm$ 14.4%
	Qwen2.5-1.5B	42.2 $\pm$ 6.4%	81.3 $\pm$ 14.1%

Table 5: Break-even token acceptance rate (%) required to match autoregressive baseline TPS, with 95% confidence intervals. Context-aware translation only.

$\alpha$ bucket	Bielik-1.5B	Llama-3.2-1B	Qwen2.5-1.5B
$<\!30\%$	0.78 $\times$	0.79 $\times$	0.73 $\times$
30–40%	0.88 $\times$	0.92 $\times$	0.89 $\times$
40–50%	1.03 $\times$	1.04 $\times$	1.03 $\times$
50–60%	1.12 $\times$	1.23 $\times$	1.20 $\times$
60–70%	—	1.59 $\times$	1.43 $\times$
$\geq\!70\%$	—	—	—

Table 6: Mean speedup by acceptance rate bucket, Wikipedia dataset,

k=2

, context-aware translation. Bold values

\geq 1.00\times

. Dashes indicate no samples in that bucket.

Hardware	BW (GB/s)	$k=2$	$k=4$	$k=6$	$k=8$
M2 Pro (empirical)	200	$\sim$ 40%	$\sim$ 77%	—	—
M2 Pro (formula)	200	44%	81%	$>$ 100%	$>$ 100%

Table 7: Break-even acceptance rate

\alpha_{\mathrm{be}}

for the M2 Pro at

r=0.071

and

\beta=0.19

. Empirical values are OLS-derived means across all drafter–dataset pairs; formula values use Equation 6 .

	$k=2$			$k=4$
$\alpha$ bucket	Bielik	Llama	Qwen	Bielik	Llama	Qwen
$<\!30\%$	0.78 $\times$	0.79 $\times$	0.73 $\times$	0.48 $\times$	0.43 $\times$	0.34 $\times$
30–40%	0.88 $\times$	0.92 $\times$	0.89 $\times$	0.54 $\times$	0.50 $\times$	0.48 $\times$
40–50%	1.03 $\times$	1.04 $\times$	1.03 $\times$	0.61 $\times$	0.57 $\times$	0.59 $\times$
50–60%	1.12 $\times$	1.23 $\times$	1.20 $\times$	0.71 $\times$	0.67 $\times$	0.66 $\times$
60–70%	—	1.59 $\times$	1.43 $\times$	0.93 $\times$	0.82 $\times$	0.81 $\times$

Table 8: Mean speedup by token acceptance rate bucket, Wikipedia dataset, context-aware translation. Bold values exceed baseline (

\geq 1.00\times

). Dashes indicate no samples in that bucket.

	$k=2$			$k=4$
$\alpha$ bucket	Bielik	Llama	Qwen	Bielik	Llama	Qwen
$<\!30\%$	0.74 $\times$	0.83 $\times$	0.80 $\times$	0.43 $\times$	0.46 $\times$	0.46 $\times$
30–40%	0.85 $\times$	0.94 $\times$	0.90 $\times$	0.47 $\times$	0.51 $\times$	0.51 $\times$
40–50%	—	1.08 $\times$	1.01 $\times$	0.51 $\times$	0.61 $\times$	0.54 $\times$
50–60%	1.21 $\times$	1.32 $\times$	1.18 $\times$	—	0.69 $\times$	0.64 $\times$
60–70%	—	—	—	0.81 $\times$	0.82 $\times$	0.72 $\times$

Table 9: Mean speedup by token acceptance rate bucket, Synthetic (short questions) dataset, context-aware translation. Bold values exceed baseline (

\geq 1.00\times

). Dashes indicate no samples in that bucket.

	$k=2$			$k=4$
$\alpha$ bucket	Bielik	Llama	Qwen	Bielik	Llama	Qwen
$<\!30\%$	0.77 $\times$	0.83 $\times$	0.79 $\times$	0.47 $\times$	0.44 $\times$	0.43 $\times$
30–40%	0.88 $\times$	0.93 $\times$	0.91 $\times$	0.51 $\times$	0.51 $\times$	0.45 $\times$
40–50%	1.01 $\times$	1.04 $\times$	1.01 $\times$	0.57 $\times$	0.58 $\times$	0.51 $\times$
50–60%	1.29 $\times$	1.34 $\times$	1.11 $\times$	0.73 $\times$	0.66 $\times$	0.59 $\times$
60–70%	—	—	1.46 $\times$	1.10 $\times$	0.92 $\times$	0.81 $\times$

Table 10: Mean speedup by token acceptance rate bucket, pl_alpaca dataset, context-aware translation. Bold values exceed baseline (

\geq 1.00\times

). Dashes indicate no samples in that bucket.

Drafter	Hit rate (%)	TPS	Speedup	Break-even $\alpha$
Bielik-1.5B	20.6	3.98	0.28 $\times$	$>$ 100% (297%, $R^{2}\!=\!0.07$ )
Llama-3.2-1B	41.7	5.01	0.35 $\times$	$>$ 100% (147%, $R^{2}\!=\!0.93$ )
Qwen2.5-1.5B	45.0	5.81	0.40 $\times$	$>$ 100% (120%, $R^{2}\!=\!0.37$ )
Baseline (no drafter)		14.38	1.00 $\times$	—

Table 11: Throughput at

k=6

, pl_alpaca dataset, context-aware translation only (

n=31

–

33

). All conditions are well below baseline. OLS break-even is extrapolated beyond the physically achievable range (

>100\%

) for all three drafters.

Hardware	Class	TDP (W)	BW (GB/s)	TPS	J/token
Apple M2 Pro (chip)	Consumer	30	200	15 0 (meas.)	2.0
Apple M2 Pro (system)	Consumer	$\sim$ 40	200	15 0 (meas.)	$\sim$ 2.7
NVIDIA L4	Datacenter	72	300	$\sim$ 22 (est.)	$\sim$ 3.3
NVIDIA A100 PCIe 40GB	Datacenter	250	1555	$\sim$ 110 (est.)	$\sim$ 2.3
NVIDIA RTX 4090	Consumer	450	1008	$\sim$ 80 (est.)	$\sim$ 5.6
NVIDIA RTX 4090 (sys.)	Consumer	$\sim$ 600	1008	$\sim$ 80 (est.)	$\sim$ 7.5

Table 12: Indicative energy efficiency comparison for 11B INT8 model inference across hardware platforms. NVIDIA TPS values are estimates from published benchmarks; M2 Pro TPS is measured in this study.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

Jul 19, 2026

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

Single-stream autoregressive decoding of large language models is bound by memory bandwidth: each generated token requires one full forward pass through the target model, and successive passes cannot be parallelized. Speculative decoding restructures this computation: a small draft model proposes

K

tokens autoregressively, the target model scores all of them in one batched pass, and a rejection-sampling rule provably preserves the target model's output distribution. We present a from-scratch, device-agnostic (CUDA/MPS/CPU) implementation and an empirical study across five draft/target backend configurations on a consumer Apple-silicon laptop. Distribution equivalence is verified at three levels, culminating in a two-sample test over roughly 9,200 real-model tokens per method (

χ^2 = 162.5

, dof

= 200

p = 0.976

) and exact greedy-sequence agreement. The best configuration reaches a measured

1.61\times

wall-clock speedup at

K=6

, on an acceptance profile declining from 69.7% at

K=1

to 37.8% at the optimum, while three of five configurations decelerate, either because the draft fails to out-speed a small target or because the quantized Metal backend executes "parallel" verification serially, an effect we isolate and quantify. The failures are as instructive as the successes: speculative decoding pays off only when verification is genuinely batch-parallel and the draft/target latency gap is real.

Param Chordiya

University of California, San Diego

On-Device Language Model Inference Speculative Decoding LLM Inference Acceleration Parallel Decoding Parallel Speculative Decoding

Ask PDF

May 28, 2026

Speculative Decoding Across Languages

Speculative decoding has become a crucial component of large language model (LLM) inference, enabling faster generation by drafting multiple tokens and verifying them in parallel. However, small draft models tend to suffer from disproportionately poor multilingual capabilities. Thus, when generating text in a non-English language, speculative decoding is far less effective. We compare three strategies to improve speculative decoding efficiency for eleven languages: finetuning the draft model on task-specific data (translation); finetuning the draft model on unlabeled monolingual corpora; and training simple n-gram draft models on the same monolingual corpora. We evaluate efficiency on translation (from English into the target language) and the held-out task of story generation. We find that while task-specific distillation can significantly improve efficiency, distilled models generalize poorly to a new task. Meanwhile, n-gram draft models, despite lower acceptance rates, consistently provide large speed-ups due to much faster draft generation.

Nirajan Paudel, Michael Ginn, Luc De Nardi +1

University of Colorado

Multilingual Language Models Speculative Decoding LLM Inference Acceleration Language Model Distillation Language Modeling Parallel LLM Inference

Ask PDF

May 31, 2026

Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding

Large Language Model (LLM) generation remains expensive because autoregressive decoding calls the model once for each new token. Speculative decoding reduces this cost by drafting multiple tokens and verifying them with the target model in one step, but its speedup depends on how many drafted tokens are accepted. Parameter-free draft sources can propose long continuations at low cost in structured and agentic workloads, yet a cache match that looks promising at one generation step may have low payoff at the next. We propose Hybrid Verified Decoding, which predicts the accepted length of a cache draft before verification and uses this payoff estimate to choose between cache verification and a model-based drafter. Across three LLMs and sixteen datasets, Hybrid Verified Decoding is especially effective on agentic workflows, where it outperforms EAGLE3 in every setting with a 2.73x average speedup. Our analysis shows how prompt structure creates cache opportunities, how high-payoff cache drafts concentrate in a small part of the draft space, and how payoff-guided selection reduces sequential decoding work, pointing to runtime draft selection as a promising direction for speculative decoding.

Xin Su, Dawid Majchrowski, Fangyuan Yu +5

Thoughtworks · Nvidia

Speculative Decoding LLM Inference Acceleration

Ask PDF

Abstract

Speculative Decoding

Explore similar work

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

Speculative Decoding Across Languages

Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding