LLM inference on edge devices is constrained by computational and memory resources, making efficient autoregressive decoding challenging. Speculative decoding alleviates this bottleneck by generating tokens with a smaller draft model and verifying multiple tokens in parallel with a batched target model pass. However, verification introduces a runtime-dependent intermediate regime between memory-bound general matrix-vector (GEMV) operations in decoding and compute-bound general matrix-matrix (GEMM) operations in prefill, as its arithmetic intensity varies with speculation length and acceptance rate. We present SPECTRA, a runtime-reconfigurable tiled architecture that sustains high utilization across the full speculative decoding pipeline. Within each tile, the compute engine switches between systolic execution for GEMMs and vector-lane execution for GEMVs. Across tiles, SPECTRA dynamically adapts computation parallelism by selecting tile count, kernel partitioning, and communication pattern. Both tile-level and system-level reconfiguration operate on a per-kernel basis, enabling efficient execution across these diverse regimes. Evaluated on a 20-tile FPGA prototype across the Pythia, SmolLM2, and GPT-2 families, SPECTRA achieves up to 2.09× speedup from tile-level reconfiguration and a further 1.25× gain from system-level adaptability over fixed designs.
Figure 4. Block diagram of the fused post-processing unit.
Figure 5. SPECTRA intra- and inter-tile mapping strategy.
Figure 6. Execution latency with architecture flexibility. V, S, and R represent vector-only, systolic-only, and reconfigurable architecture of SPECTRA , respectively.
Pythia
SmolLM2
GPT-2
out
R/V
R/S
TR/SDR
R/V
R/S
TR/SDR
R/V
R/S
TR/SDR
128
3.38
1.47
1.15
5.14
2.16
1.70
4.65
1.32
1.82
256
3.20
1.50
1.37
4.86
2.20
1.93
4.41
1.35
2.12
512
2.67
1.42
1.44
3.44
1.86
1.83
3.21
1.25
1.91
Table 1. Scalability at γ=8 . All entries report speedup.
Configuration
LUT
FF
BRAM
URAM
DSP
Vector-only
52,149
69,791
68
12
276
Systolic-only
48,426
57,863
104
18
279
Reconfigurable
88,206
103,564
112
18
308
Table 2. Resource utilization across datapath configurations.
Figure 7. Latency with sharding flexibility. p=32,γ=8 .
Figure 8. Performance with p=32 and varying γ , out for Pythia-160M → 410M. T and SD mean target-only and speculative decoding, respectively.
Jetson Orin NX
Jetson TX2
SPECTRA
Compute Unit
Ampere GPU 8 SMs / 1024 CUDA cores
Pascal GPU 2 SMs / 256 CUDA cores
14 accels 100 MHz
On-chip Memory
11.8–15.8 MB cache/local
∼ 5.4 MB cache/local
18.84 MB BRAM+URAM
Off-chip Bandwidth
102 GB/s LPDDR5
59.7 GB/s LPDDR4
45 GB/s DDR
Power (W)
25
15
19.9
Thr. (TPS)
0.19
0.19
0.69
Pwr Eff. (TPS/W)
0.008
0.012
0.035
Table 3. Platform comparison on the GPT-2 speculative decoding workload ( p=128 ). Jetson figures are derived from per-token latencies reported by ( Xu et al., 2025 ) ; SPECTRA results are measured on the FPGA prototype at 100 MHz.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to speculate multiple tokens, reducing expensive target model decoding steps. Its effectiveness depends heavily on draft selection, motivating adaptive methods that exploit variation across inputs and generation stages. On memory-constrained edge devices, however, these methods often fail to improve end-to-end throughput due to the overhead of switching between draft models. We identify a key limitation in this setting: the mismatch between draft selection and draft availability under tight memory budgets. To address this challenge, we present MemSpec, a prediction-guided, memory-aware runtime for adaptive speculative decoding on edge devices. MemSpec decouples draft selection from execution through proactive resident working-set management. A lightweight predictor estimates draft effectiveness from prompt and generation context, while a memory-aware scheduler reduces reactive model loading overhead. Experiments on a Jetson Orin Nano show that MemSpec improves steady-state generation throughput by 40.7% on average over state-of-the-art bandit-based adaptive methods while closely approaching the oracle upper bound.
Eunjeong Kim, Yeong Jun Jeon, Myeonggyun Han
Kyungpook National University Daegu, Republic of Korea
Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.
Tanishq Kumar, Tri Dao, Avner May
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI
Auto-regressive decoding in Large Language Models (LLMs) is inherently memory-bound: every generation step requires loading the model weights and intermediate results from memory (e.g., High-Bandwidth Memory (HBM) for GPU servers), making throughput bottlenecked by memory bandwidth rather than compute. Speculative decoding addresses this by enabling parallel verification of multiple draft tokens, effectively amortizing the cost of each target-model call. However, existing speculative decoding methods are designed under the assumption that HBM is sufficiently large to hold both the target model and an auxiliary draft model simultaneously -- an assumption that breaks down on memory-constrained devices such as edge platforms with limited DRAM. We analyze the inference bottleneck in this memory-limited regime and propose CATS, a self-speculative decoding framework that conducts cascaded verification and correction based on the memory budget and parameter offloading patterns on memory-limited devices. This design maximizes token acceptance rate and end-to-end speedup while keeping the peak memory footprint on the device equal to that of the target model alone. We evaluate CATS on different models across five benchmarks on real edge devices. CATS can achieve a wall-clock speedup of up to 5.08x with no degradation in generation quality, outperforming the SOTA method by up to 1.45x under edge memory constraints.