Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LM

Krzysztof Fonal

cs.CLMar 22, 2026

Cross-Family Speculative Decoding for Polish Language Models on AppleSilicon: An Empirical Evaluation of Bielik11B with UAG-Extended MLX-LM

Authors: Krzysztof Fonal

Organizations: Wrocław University of Science and Technology, Poland

Abstract

Speculative decoding accelerates LLM inference by using a small draft model to propose k candidate tokens for a target model to verify. While effective for same-tokenizer pairs on high-bandwidth GPUs, its applicability to cross-family pairs with mismatched tokenizers and consumer-grade unified memory remains underexplored. We extend the MLX-LM framework with Universal Assisted Generation (UAG) to enable cross-tokenizer speculative decoding on Apple Silicon. We evaluate Bielik 11B-Instruct (Mistral-based) as the target model, paired with three draft models: Bielik 1.5B (Qwen-based with custom tokenizer), Qwen2.5-1.5B, and Llama 3.2-1B. Experiments on three Polish-language datasets (Wikipedia, pl_alpaca, synthetic) use draft lengths k in {2, 4, 6} to compare naive and context-aware token translation. Results show: (1) context-aware translation consistently improves acceptance rates across all configurations; (2) the Polish-specialized Bielik 1.5B achieves lower acceptance than general-purpose Qwen2.5 and Llama 3.2 drafters; (3) throughput on Apple Silicon is content-dependent, reaching 1.7x speedup for structured text but failing for varied instructions; and (4) verification cost on unified memory does not amortize as theory predicts because both models are memory-bandwidth bound, making sequential drafting expensive relative to batched verification. We propose a hardware-aware speedup formula and characterize conditions for cross-family speculative decoding on Apple Silicon. This is the first systematic evaluation of cross-family speculative decoding for Polish LLMs and the first empirical study of UAG-based decoding on unified memory architectures.

Figures & tables

Model	Role	Base Arch.	Tokenizer
Bielik 11B v3 Instruct	Target	Mistral	Mistral ( $\sim$ 32K)
Bielik 1.5B v3 Instruct	Draft	Qwen2.5	APT4 (Polish)
Qwen2.5 1.5B Instruct	Draft	Qwen2.5	Qwen ( $\sim$ 152K)
Llama 3.2 1B Instruct	Draft	Llama 3.2	Llama ( $\sim$ 128K)

Table 1: Models used in experiments. All models are cross-family relative to the Bielik 11B target.

Drafter	Condition	Wikipedia	pl_alpaca	Synthetic
Bielik-1.5B	No translation	28.5	12.5	11.8
	Naive translation	19.1	11.9	16.7
	Context-aware	31.1	23.9	22.2
Llama-3.2-1B	No translation	26.7	12.1	10.6
	Naive translation	8.9	4.2	3.1
	Context-aware	42.0	36.0	36.5

Table 2: Mean token acceptance rate (%) for

k=2

across datasets and translation conditions.

		Wikipedia		pl_alpaca		Synthetic
Drafter	Condition	TPS	Speedup	TPS	Speedup	TPS	Speedup
—	Baseline	14.83	1.00 $\times$	14.70	1.00 $\times$	14.72	1.00 $\times$
Bielik-1.5B	No translation	13.72	0.93 $\times$	10.80	0.74 $\times$	10.68	0.73 $\times$
	Naive	10.32	0.70 $\times$	9.69	0.66 $\times$	9.72	0.66 $\times$
	Context-aware	12.85	0.87 $\times$	11.92	0.81 $\times$	11.02	0.75 $\times$
Llama-3.2-1B	No translation	14.03	0.95 $\times$	11.24	0.77 $\times$	11.02	0.75 $\times$

Table 3: Mean generation throughput (TPS) and speedup for

k=2

. Speedup is relative to the per-dataset autoregressive baseline.

		Wikipedia		pl_alpaca		Synthetic
Drafter	$k$	$\alpha$ (%)	Speedup	$\alpha$ (%)	Speedup	$\alpha$ (%)	Speedup
Bielik-1.5B	2	31.1	0.87 $\times$	23.9	0.81 $\times$	22.2	0.75 $\times$
	4	38.5	0.59 $\times$	29.5	0.51 $\times$	29.1	0.46 $\times$
Llama-3.2-1B	2	42.0	1.06 $\times$	36.0	0.97 $\times$	36.5	0.98 $\times$
	4	49.7	0.66 $\times$	43.7	0.59 $\times$	44.5	0.61 $\times$
Qwen2.5-1.5B	2	44.6	1.06 $\times$	41.0	0.98 $\times$	42.7	1.00 $\times$

Table 4: Effect of draft length

k

on throughput and acceptance rate (context-aware translation, all three datasets).

Dataset	Drafter	$k=2$	$k=4$
Wikipedia	Bielik-1.5B	44.0 $\pm$ 7.1%	90.1 $\pm$ 19.8%
	Llama-3.2-1B	38.3 $\pm$ 5.5%	81.2 $\pm$ 13.3%
	Qwen2.5-1.5B	40.8 $\pm$ 5.3%	77.7 $\pm$ 13.1%
pl_alpaca	Bielik-1.5B	40.2 $\pm$ 6.8%	89.9 $\pm$ 18.5%
	Llama-3.2-1B	38.1 $\pm$ 4.8%	82.0 $\pm$ 14.4%
	Qwen2.5-1.5B	42.2 $\pm$ 6.4%	81.3 $\pm$ 14.1%

Table 5: Break-even token acceptance rate (%) required to match autoregressive baseline TPS, with 95% confidence intervals. Context-aware translation only.

$\alpha$ bucket	Bielik-1.5B	Llama-3.2-1B	Qwen2.5-1.5B
$<\!30\%$	0.78 $\times$	0.79 $\times$	0.73 $\times$
30–40%	0.88 $\times$	0.92 $\times$	0.89 $\times$
40–50%	1.03 $\times$	1.04 $\times$	1.03 $\times$
50–60%	1.12 $\times$	1.23 $\times$	1.20 $\times$
60–70%	—	1.59 $\times$	1.43 $\times$
$\geq\!70\%$	—	—	—

Table 6: Mean speedup by acceptance rate bucket, Wikipedia dataset,

k=2

, context-aware translation. Bold values

\geq 1.00\times

. Dashes indicate no samples in that bucket.

Hardware	BW (GB/s)	$k=2$	$k=4$	$k=6$	$k=8$
M2 Pro (empirical)	200	$\sim$ 40%	$\sim$ 77%	—	—
M2 Pro (formula)	200	44%	81%	$>$ 100%	$>$ 100%

Table 7: Break-even acceptance rate

\alpha_{\mathrm{be}}

for the M2 Pro at

r=0.071

and

\beta=0.19

. Empirical values are OLS-derived means across all drafter–dataset pairs; formula values use Equation 6 .

	$k=2$			$k=4$
$\alpha$ bucket	Bielik	Llama	Qwen	Bielik	Llama	Qwen
$<\!30\%$	0.78 $\times$	0.79 $\times$	0.73 $\times$	0.48 $\times$	0.43 $\times$	0.34 $\times$
30–40%	0.88 $\times$	0.92 $\times$	0.89 $\times$	0.54 $\times$	0.50 $\times$	0.48 $\times$
40–50%	1.03 $\times$	1.04 $\times$	1.03 $\times$	0.61 $\times$	0.57 $\times$	0.59 $\times$
50–60%	1.12 $\times$	1.23 $\times$	1.20 $\times$	0.71 $\times$	0.67 $\times$	0.66 $\times$
60–70%	—	1.59 $\times$	1.43 $\times$	0.93 $\times$	0.82 $\times$	0.81 $\times$

Table 8: Mean speedup by token acceptance rate bucket, Wikipedia dataset, context-aware translation. Bold values exceed baseline (

\geq 1.00\times

). Dashes indicate no samples in that bucket.

	$k=2$			$k=4$
$\alpha$ bucket	Bielik	Llama	Qwen	Bielik	Llama	Qwen
$<\!30\%$	0.74 $\times$	0.83 $\times$	0.80 $\times$	0.43 $\times$	0.46 $\times$	0.46 $\times$
30–40%	0.85 $\times$	0.94 $\times$	0.90 $\times$	0.47 $\times$	0.51 $\times$	0.51 $\times$
40–50%	—	1.08 $\times$	1.01 $\times$	0.51 $\times$	0.61 $\times$	0.54 $\times$
50–60%	1.21 $\times$	1.32 $\times$	1.18 $\times$	—	0.69 $\times$	0.64 $\times$
60–70%	—	—	—	0.81 $\times$	0.82 $\times$	0.72 $\times$

Table 9: Mean speedup by token acceptance rate bucket, Synthetic (short questions) dataset, context-aware translation. Bold values exceed baseline (

\geq 1.00\times

). Dashes indicate no samples in that bucket.

	$k=2$			$k=4$
$\alpha$ bucket	Bielik	Llama	Qwen	Bielik	Llama	Qwen
$<\!30\%$	0.77 $\times$	0.83 $\times$	0.79 $\times$	0.47 $\times$	0.44 $\times$	0.43 $\times$
30–40%	0.88 $\times$	0.93 $\times$	0.91 $\times$	0.51 $\times$	0.51 $\times$	0.45 $\times$
40–50%	1.01 $\times$	1.04 $\times$	1.01 $\times$	0.57 $\times$	0.58 $\times$	0.51 $\times$
50–60%	1.29 $\times$	1.34 $\times$	1.11 $\times$	0.73 $\times$	0.66 $\times$	0.59 $\times$
60–70%	—	—	1.46 $\times$	1.10 $\times$	0.92 $\times$	0.81 $\times$

Table 10: Mean speedup by token acceptance rate bucket, pl_alpaca dataset, context-aware translation. Bold values exceed baseline (

\geq 1.00\times

). Dashes indicate no samples in that bucket.

Drafter	Hit rate (%)	TPS	Speedup	Break-even $\alpha$
Bielik-1.5B	20.6	3.98	0.28 $\times$	$>$ 100% (297%, $R^{2}\!=\!0.07$ )
Llama-3.2-1B	41.7	5.01	0.35 $\times$	$>$ 100% (147%, $R^{2}\!=\!0.93$ )
Qwen2.5-1.5B	45.0	5.81	0.40 $\times$	$>$ 100% (120%, $R^{2}\!=\!0.37$ )
Baseline (no drafter)		14.38	1.00 $\times$	—

Table 11: Throughput at

k=6

, pl_alpaca dataset, context-aware translation only (

n=31

–

33

). All conditions are well below baseline. OLS break-even is extrapolated beyond the physically achievable range (

>100\%

) for all three drafters.

Hardware	Class	TDP (W)	BW (GB/s)	TPS	J/token
Apple M2 Pro (chip)	Consumer	30	200	15 0 (meas.)	2.0
Apple M2 Pro (system)	Consumer	$\sim$ 40	200	15 0 (meas.)	$\sim$ 2.7
NVIDIA L4	Datacenter	72	300	$\sim$ 22 (est.)	$\sim$ 3.3
NVIDIA A100 PCIe 40GB	Datacenter	250	1555	$\sim$ 110 (est.)	$\sim$ 2.3
NVIDIA RTX 4090	Consumer	450	1008	$\sim$ 80 (est.)	$\sim$ 5.6
NVIDIA RTX 4090 (sys.)	Consumer	$\sim$ 600	1008	$\sim$ 80 (est.)	$\sim$ 7.5

Table 12: Indicative energy efficiency comparison for 11B INT8 model inference across hardware platforms. NVIDIA TPS values are estimates from published benchmarks; M2 Pro TPS is measured in this study.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
Jul 19, 2026Param ChordiyaOn-Device Language Model Inference Speculative Decoding
University of California, San Diego
Speculative Decoding Across Languages
May 28, 2026Nirajan Paudel, Michael Ginn, Luc De Nardi +1Multilingual Language Models Speculative Decoding
University of Colorado
Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
May 31, 2026Xin Su, Dawid Majchrowski, Fangyuan Yu +5Speculative Decoding LLM Inference Acceleration
Thoughtworks · Nvidia

Jul 19, 2026J/K move · Enter open · S save

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

Param Chordiya

University of California, San Diego

On-Device Language Model Inference Speculative Decoding LLM Inference Acceleration Parallel Decoding Parallel Speculative Decoding

Open paper AskPDF arXiv

Single-stream autoregressive decoding of large language models is bound by memory bandwidth: each generated token requires one full forward pass through the target model, and successive passes cannot be parallelized. Speculative decoding restructures this computation: a small draft model proposes

K

tokens autoregressively, the target model scores all of them in one batched pass, and a rejection-sampling rule provably preserves the target model's output distribution. We present a from-scratch, device-agnostic (CUDA/MPS/CPU) implementation and an empirical study across five draft/target backend configurations on a consumer Apple-silicon laptop. Distribution equivalence is verified at three levels, culminating in a two-sample test over roughly 9,200 real-model tokens per method (

χ^2 = 162.5

, dof

= 200

p = 0.976

) and exact greedy-sequence agreement. The best configuration reaches a measured

1.61\times

wall-clock speedup at

K=6

, on an acceptance profile declining from 69.7% at

K=1

to 37.8% at the optimum, while three of five configurations decelerate, either because the draft fails to out-speed a small target or because the quantized Metal backend executes "parallel" verification serially, an effect we isolate and quantify. The failures are as instructive as the successes: speculative decoding pays off only when verification is genuinely batch-parallel and the draft/target latency gap is real.

Abstract

Speculative Decoding

Explore similar work

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

Speculative Decoding Across Languages

Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding