Speculative decoding accelerates vision-language model (VLM) inference without altering the output distribution, yet two factors limit its gains. Existing drafters either lack the image evidence required for visually grounded tokens or pay to condition on every visual token, although a short draft block typically depends on only a few image regions. Moreover, the value of verifying additional candidates varies with GPU load, so no fixed verification structure is efficient across serving conditions. We present Uni-LaSpec, a block-parallel speculative decoding framework that addresses both factors. To improve draft quality, Uni-LaSpec reuses the verifier's own attention: the query retained from the preceding verification ranks the cached, int8-quantized visual keys, and a cross-layer vote selects a few visual tokens that the drafter fuses through gated cross-attention. Acceptance-aware training further weights each draft position by its contribution to the accepted prefix. To allocate verification, a marginal-cost criterion selects the structure per request and per round: draft trees exploit idle verifier capacity under light load, whereas a dynamic head truncates linear drafts under heavy load. Across six VLMs from three model families and 17 text and multimodal benchmarks, selecting eight visual tokens retains at least 98.6% of the acceptance obtained from the full image at sub-millisecond selection cost. With greedy decoding at concurrency one, Uni-LaSpec achieves up to 5.11x text and 3.24x multimodal throughput over autoregressive decoding. Across six load levels and three GPU types, it outperforms autoregressive decoding in every setting and matches or exceeds the best fixed verification strategy in 123 of 144 settings; under bursty arrivals, it reduces mean end-to-end latency by 47.6% relative to linear verification.
Figures & tables
Fig. 2: Comparison of visual evidence acquisition mechanisms in MSD [ 17 ] , DREAM [ 18 ] , ViSpec [ 21 ] , and Uni-LaSpec. In each panel, the verifier is on the left and the drafter is on the right. Uni-LaSpec scores the visual tokens with the quantized visual keys cached by the verifier, fuses only the selected tokens through gated cross-attention, and drafts a masked block in one pass.
Fig. 3: Overview of Uni-LaSpec. Left: the frozen verifier caches the auxiliary hidden states Haux and quantized visual keys. Its last retained query, at position j−1 for the current root xj , scores the visual tokens in every selector layer, and a cross-layer vote selects the positions Sj . Right: the drafter fuses the projected Haux with the selected visual embeddings through gated cross-attention and decodes a masked block in one pass. The LM head emits draft tokens, and the dynamic head emits survival probabilities. Bottom: the candidates (a dynamically truncated linear draft or a draft tree) are verified in one verifier pass, and the accepted tokens start the next round.
Fig. 4: Attention-guided visual evidence selection for schema extraction. (a) Colored regions link output fields to supporting image patches; diagonal splits indicate patches shared by multiple fields. The insets show verifier attention scoring, cross-layer voting, and fusion of the selected visual embeddings into the drafter. (b) Top- k patches selected for each field, with k=3 shown for illustration. Selection is refreshed in each drafting round to follow the field being generated.
Fig. 5: Visual input availability across prefill and decoding. In vLLM, step-local gathering restricts visual inputs to the current scheduled window. Image embeddings reach the drafter during prefill, whereas subsequent decoding windows contain no visual tokens, leaving the visual branch inactive. Uni-LaSpec caches image embeddings for each request and refreshes the top- k selection using the verifier’s last retained query in every drafting round, maintaining access to visual evidence throughout generation.
Fig. 6: Parallel verification and scheduling of draft trees. (a) Attention mask for a single request. The root is the previous round’s bonus token; each node attends to the cached prefix x1:j−1 , its ancestors, and itself. (b) Best-first tree construction with nodes indexed in topological order. Rotary positions follow tree depth, pos(u)=j+∣u∣ , independently of the packed KV-cache indices. (c) Batched execution of two draft trees and a prefill chunk. The logical attention mask is block diagonal across requests, preserving each tree’s ancestor dependencies and the prefill chunk’s causal order. (d) Scheduling under a shared token budget. The numbered events illustrate deferral of a tree that exceeds the remaining budget (1), allocation of residual capacity to prefill (2), verification of the root alone near the context limit (3), discarding drafts during continued prefill (4), preemption under KV-cache pressure (5), and suspension of new admissions (6). (e) Compaction of the accepted path. Retained entries are gathered into contiguous slots within each request’s storage region, and entries from rejected branches are discarded.
Fig. 7: Dynamic truncation of linear drafts using predicted survival probabilities. (a) The dynamic head estimates ρ^i for each draft position. The longest prefix satisfying ρ^i≥δ defines the raw width γ^ . (b) Width relaxation maps γ^ to an admissible draft width γ . Slack Δ is applied only above the first bucket, followed by upward bucket quantization. (c) Comparison with fixed-width truncation across a batch. Adapting widths to individual requests reduces verification of rejected candidates and retains longer acceptable prefixes.
Fig. 8: Hybrid verification under variable serving load. (a) At verifier admission, the scheduler selects a draft tree or a linear prefix from stored candidates according to current batch load and available capacity, subject to the shared token budget Q . KV-cache and context constraints may reduce a request to its root or a shorter prefix. (b) A single verifier pass processes both structures. The logical attention mask isolates requests: tree queries attend to their own cached history, ancestors, and themselves, while linear queries follow causal order within their request. (c) Acceptance and state retention are resolved independently for each request. The accepted tree path or linear prefix is retained with its corresponding KV cache and recurrent state. Accepted tokens are committed with a bonus token. Structure selection is repeated in the next round as load changes.
Text Benchmarks
Multimodal Benchmarks
Samp.
Model
Method
Math
Code
Chat
Document & OCR
General Vision
Capt. & Chat
GSM8K
MATH
AIME
HEval
MBPP
LCB
MT-B
ODoc
OCR2
DVQA
olmO
CXiv
TVQA
MMMU
GQA
COCO
MMMT
T=0
Q3.5-4B
EAGLE-3
3.53
4.25
3.91
4.49
4.25
3.34
2.84
2.07
2.38
2.61
2.49
2.74
2.32
2.30
2.18
2.01
2.37
MTP
6.11
6.29
5.66
6.13
5.98
4.73
4.58
4.14
4.02
5.25
4.48
4.65
4.18
3.98
3.72
3.54
4.13
DFlash (T)
4.92
6.41
6.52
5.27
4.99
4.48
3.48
2.81
3.24
3.40
4.29
4.66
3.09
4.01
3.00
2.66
3.74
DFlash (M)
5.07
6.53
6.57
5.36
5.01
4.55
3.60
3.01
3.51
3.83
4.71
4.92
3.47
4.11
3.34
3.07
4.00
TABLE I: Mean accepted length τ per benchmark at concurrency R=1 , reported by verifier, drafter, and sampling temperature. DFlash (T) and DFlash (M) denote training on text-only prompts and the full multimodal mixture, respectively. Benchmark abbreviations: MATH (MATH-500), AIME (AIME25), HEval (HumanEval), LCB (LiveCodeBench), MT-B (MT-Bench), ODoc (OmniDocBench), OCR2 (OCRBench-v2), DVQA (DocVQA), olmO (olmOCR), CXiv (CharXiv), TVQA (TextVQA), MMMU (MMMU-Pro), and MMMT (MM-MT-Bench).
Text Benchmarks
Multimodal Benchmarks
Samp.
Model
Method
Math
Code
Chat
Document & OCR
General Vision
Capt. & Chat
GSM8K
MATH
AIME
HEval
MBPP
LCB
MT-B
ODoc
OCR2
DVQA
olmO
CXiv
TVQA
MMMU
GQA
COCO
MMMT
T=0
Q3.5-4B
EAGLE-3
1.33 ×
1.58 ×
1.45 ×
1.64 ×
1.59 ×
1.26 ×
1.06 ×
0.79 ×
0.93 ×
0.99 ×
0.94 ×
1.05 ×
0.86 ×
0.88 ×
0.82 ×
0.75 ×
0.92 ×
MTP
1.68 ×
1.71 ×
1.58 ×
1.68 ×
1.64 ×
1.30 ×
1.26 ×
1.14 ×
1.11 ×
1.37 ×
1.23 ×
1.28 ×
1.11 ×
1.09 ×
1.01 ×
0.97 ×
1.13 ×
DFlash (T)
3.29 ×
4.25 ×
4.31 ×
3.51 ×
3.26 ×
2.98 ×
2.28 ×
1.81 ×
2.06 ×
1.96 ×
2.68 ×
2.98 ×
1.91 ×
2.55 ×
1.92 ×
1.69 ×
2.36 ×
DFlash (M)
3.35 ×
4.06 ×
4.34 ×
3.53 ×
3.22 ×
2.98 ×
2.29 ×
1.95 ×
2.29 ×
2.24 ×
2.99 ×
3.08 ×
2.17 ×
2.62 ×
2.11 ×
1.94 ×
2.53 ×
TABLE II: Speedup relative to autoregressive decoding at concurrency R=1 , reported by benchmark, verifier, drafter, and sampling temperature. Speedup is defined as the ratio of speculative to autoregressive output throughput for the same verifier, temperature, and prompt set. Benchmark abbreviations follow Table I .
Samp.
Model
Method
Text Benchmarks
Multimodal Benchmarks
R=1
R=4
R=8
R=16
R=32
R=64
Avg. τ
R=1
R=4
R=8
R=16
R=32
R=64
Avg. τ
T=0
Q3.5-4B
EAGLE-3
1.41 ×
1.54 ×
1.53 ×
1.33 ×
0.95 ×
0.76 ×
3.82
0.89 ×
0.95 ×
0.98 ×
0.84 ×
0.63 ×
0.51 ×
2.34
MTP
1.55 ×
1.59 ×
1.55 ×
1.34 ×
1.08 ×
0.90 ×
5.61
1.14 ×
1.18 ×
1.12 ×
1.00 ×
0.83 ×
0.71 ×
4.20
DFlash (T)
3.41 ×
3.25 ×
2.86 ×
1.97 ×
1.32 ×
1.00 ×
5.16
2.19 ×
2.15 ×
1.88 ×
1.31 ×
0.94 ×
0.72 ×
3.49
DFlash (M)
3.40 ×
3.28 ×
2.92 ×
2.04 ×
1.38 ×
1.03 ×
5.23
2.39 ×
2.32 ×
2.03 ×
1.44 ×
1.03 ×
0.78 ×
3.79
Uni-LaSpec
3.60 ×
3.42 ×
2.95 ×
2.11 ×
1.41 ×
1.06 ×
5.38
2.60 ×
2.48 ×
2.12 ×
1.49 ×
1.08 ×
0.82 ×
4.07
TABLE III: Output throughput ratios relative to autoregressive decoding across client concurrency R and sampling temperature. Each ratio is the quotient of unweighted mean output throughputs over the same benchmark group, evaluated for the same verifier, R , and temperature. Avg. τ denotes the unweighted mean accepted length across benchmarks within the group and all six concurrency levels. A ratio below unity indicates lower throughput than autoregressive decoding.
Fig. 9: Visual grounding in general VQA and schema-guided OCR. (a) Scene entities and attributes are grounded in image regions, while language priors and context support the surrounding description. (b) The schema constrains field names and output structure, while exact field values depend on local document regions. The relevant regions change as generation proceeds across fields. Colors associate output spans with supporting visual regions; gray spans indicate language context or schema structure.
Fig. 10: Out-of-vocabulary (OOV) rates of generated outputs, defined as the fraction of tokens outside the draft vocabulary. (a) Uni-LaSpec rates by verifier and benchmark; Overall columns use token-weighted aggregation. (b) Ratio of the Uni-LaSpec OOV rate to that of EAGLE-3, aggregated using the geometric mean over paired verifiers. Ratios below 1× indicate lower OOV rates for Uni-LaSpec. Rates are approximate estimates from saved outputs re-tokenized with each verifier’s tokenizer. Benchmark abbreviations follow Table I .
Model
Method
R=1
R=4
R=8
R=16
R=32
R=64
Avg. τ
Q3.5-2B -SFT
MTP
1.49 ×
1.18 ×
1.02 ×
1.06 ×
0.69 ×
0.54 ×
5.553
DFlash (M)
5.45 ×
3.88 ×
2.70 ×
2.73 ×
1.83 ×
1.47 ×
9.328
Uni-LaSpec
5.59 ×
4.11 ×
3.15 ×
3.02 ×
1.85 ×
1.80 ×
9.923
Q3.5-4B -SFT
MTP
1.55 ×
1.50 ×
1.41 ×
1.39 ×
1.28 ×
1.18 ×
5.664
DFlash (M)
5.57 ×
3.83 ×
3.16 ×
2.54 ×
2.12 ×
1.85 ×
9.369
Uni-LaSpec
5.70 ×
3.86 ×
3.18 ×
2.60 ×
2.18 ×
1.90 ×
9.920
TABLE IV: Throughput scaling of business models fine-tuned on in-house data for medical-document extraction at sampling temperature T=0 . Concurrency columns report the ratio of speculative to autoregressive output throughput for the same model and R . Avg. τ denotes the unweighted mean accepted length across the six concurrency levels, with all requests included. Q3.5-2B-SFT at R=4 pools 5 repetitions of 100 prompts each; throughput is total output tokens divided by total benchmark duration, and acceptance pools the draft and accepted-token counters.
Verifier
Modality
None
Top-8
Top-32
Top-64
All
Quant.
Softmax
κ
Quant.
Softmax
κ
Quant.
Softmax
κ
τ
Csel
τ
Csel
τ
Csel
τ
Csel
τ
Csel
τ
Csel
Q3.5-4B
Text
5.34
5.34
5.34
5.38
5.35
MM
3.86
4.06
0.595
4.06
11.411
19.17 ×
4.07
0.846
4.08
18.104
21.40 ×
4.07
1.156
4.07
20.973
18.14 ×
4.04
Q3.5-9B
Text
5.26
5.26
5.26
5.21
5.25
MM
3.79
3.95
0.722
3.93
18.291
25.35 ×
3.93
1.229
3.93
25.362
20.63 ×
3.95
1.549
3.94
30.479
19.67 ×
3.94
TABLE V: Comparison of visual input policies at concurrency R=32 and sampling temperature T=0 . None disables the visual branch; All provides the complete set of visual features to the drafter. Quant. and Softmax correspond to int8 ranking and full softmax scoring, respectively. Text and MM report unweighted benchmark averages of accepted length τ for text and multimodal tasks. Csel (ms) is the mean wall-clock time per selector invocation on multimodal inputs; κ is the ratio of Softmax to Quant. selector time.
Verifier
Scoring
Kernels
τ
R=1
R=4
R=16
R=32
R=64
Csel
ηsel
Csel
ηsel
Csel
ηsel
Csel
ηsel
Csel
ηsel
Q3.5-4B
Softmax
Unfused
4.07
0.99
1.00 ×
2.12
1.00 ×
4.16
1.00 ×
8.61
1.00 ×
8.95
1.00 ×
Quant.
Unfused
4.08
0.74
1.34 ×
0.75
2.83 ×
0.98
4.25 ×
1.22
7.07 ×
1.41
6.36 ×
Quant.
Fused
4.07
0.38
2.58 ×
0.44
4.83 ×
0.62
6.73 ×
0.76
11.40 ×
0.94
9.54 ×
Q3.5-9B
Softmax
Unfused
3.92
1.03
1.00 ×
2.07
1.00 ×
5.17
1.00 ×
13.98
1.00 ×
15.71
1.00 ×
Quant.
Unfused
3.92
0.70
1.48 ×
0.75
2.76 ×
1.15
4.48 ×
1.47
9.49 ×
1.55
10.15 ×
TABLE VI: Latency of visual selection across scoring rules and kernel implementations at concurrency R . Csel (ms) is the mean wall-clock time per selector invocation. The time ratio ηsel is defined as the unfused Softmax latency divided by the latency of the evaluated configuration, for the same verifier and R . Reported τ values are unweighted averages over multimodal benchmarks and concurrency levels.
Samp.
Method
Text
Multimodal
∣A∣
τ
TPS
S
τ
TPS
S
T=0
AR
–
–
321.1
1.00 ×
–
306.3
1.00 ×
Uni-LaSpec
3
5.35
1047.5
3.26 ×
4.00
728.8
2.38 ×
Uni-LaSpec
5
5.33
1037.2
3.23 ×
3.98
730.9
2.39 ×
Uni-LaSpec
8
5.38
1090.2
3.40 ×
4.08
756.3
2.47 ×
T=1
AR
–
–
317.2
1.00 ×
–
308.0
1.00 ×
TABLE VII: Comparison of auxiliary-feature configurations under Top-8 visual selection at concurrency R=4 . ∣A∣ denotes the number of verifier layers providing auxiliary features. τ and TPS report unweighted benchmark means of accepted length and output throughput (tokens/s), respectively. S is the ratio of TPS to the corresponding autoregressive throughput at the same sampling temperature. Results are aggregated separately for text and multimodal benchmarks. Acceptance length applies only to speculative methods.
Fig. 11: Scheduling decisions and verification workload of Uni-LaSpec. From top to bottom: verification structure, decoding request count, tokens verified for decoding, mean draft tokens verified per request, and the survival threshold δ for linear truncation. Green and blue indicate tree and linear verification, respectively. Gray bars denote requests undergoing prefill; the gray trace shows the scheduler’s token reservation before draft trimming. Red dashed lines mark the request and token limits for tree verification under Rt(1+B)≤W⋆ .
Text Benchmarks
Multimodal Benchmarks
Model
Strategy
Math
Code
Chat
Document & OCR
General Vision
Capt. & Chat
GSM8K
MATH
AIME
HEval
MBPP
LCB
MT-B
ODoc
OCR2
DVQA
olmO
CXiv
TVQA
MMMU
GQA
COCO
MMMT
Q3.5-4B
Linear
5.31
6.65
6.55
5.46
4.99
4.64
3.81
3.03
3.81
4.12
4.92
5.25
3.66
4.26
3.70
3.56
4.30
Trunc. N=5
4.02
4.53
4.54
4.09
3.87
3.78
3.27
2.78
3.26
3.51
3.93
3.98
3.28
3.56
3.28
3.22
3.56
Trunc. N=7
4.58
5.36
5.41
4.61
4.31
4.19
3.47
2.90
3.52
3.78
4.38
4.41
3.41
3.92
3.48
3.42
3.86
Dyn. trunc.
4.93
6.24
6.16
5.10
4.69
4.25
3.45
2.87
3.45
3.75
4.57
4.64
3.29
3.88
3.25
3.07
3.84
TABLE VIII: Mean accepted length τ of verification strategies at concurrency R=1 and sampling temperature T=0 . All strategies for a given verifier share the same drafter. Linear verifies the full draft; Trunc. retains a fixed prefix of N draft tokens; Dyn. trunc. selects prefix length from predicted survival probabilities; Tree verifies B candidate nodes; Uni-LaSpec selects the verification structure according to load. Benchmark abbreviations follow Table I .
Text Benchmarks
Multimodal Benchmarks
Model
Strategy
Math
Code
Chat
Document & OCR
General Vision
Capt. & Chat
GSM8K
MATH
AIME
HEval
MBPP
LCB
MT-B
ODoc
OCR2
DVQA
olmO
CXiv
TVQA
MMMU
GQA
COCO
MMMT
Q3.5-4B
Linear
3.71 ×
4.62 ×
4.55 ×
3.80 ×
3.48 ×
3.22 ×
2.67 ×
2.09 ×
2.61 ×
2.59 ×
3.33 ×
3.57 ×
2.47 ×
2.93 ×
2.55 ×
2.48 ×
2.94 ×
Trunc. N=5
2.86 ×
3.21 ×
3.20 ×
2.91 ×
2.76 ×
2.68 ×
2.33 ×
1.96 ×
2.29 ×
2.30 ×
2.74 ×
2.79 ×
2.27 ×
2.52 ×
2.31 ×
2.28 ×
2.49 ×
Trunc. N=7
3.24 ×
3.78 ×
3.74 ×
3.25 ×
3.04 ×
2.95 ×
2.46 ×
2.02 ×
2.44 ×
2.43 ×
3.00 ×
3.05 ×
2.33 ×
2.73 ×
2.43 ×
2.41 ×
2.67 ×
Dyn. trunc.
3.43 ×
4.32 ×
4.24 ×
3.55 ×
3.27 ×
2.97 ×
2.37 ×
1.98 ×
2.36 ×
2.39 ×
3.05 ×
3.15 ×
2.21 ×
2.67 ×
2.24 ×
2.13 ×
2.62 ×
TABLE IX: Speedup of verification strategies relative to autoregressive decoding at concurrency R=1 and sampling temperature T=0 . Speedup is defined as the ratio of each strategy’s output throughput to the corresponding autoregressive throughput for the same verifier and prompt set. Verification strategies are defined in Table VIII .
Model
Strategy
Text Benchmarks
Multimodal Benchmarks
R=1
R=4
R=8
R=16
R=32
R=64
Avg. τ
R=1
R=4
R=8
R=16
R=32
R=64
Avg. τ
Q3.5-4B
Linear
3.72 ×
3.30 ×
2.70 ×
2.26 ×
1.34 ×
1.01 ×
5.36
2.76 ×
2.34 ×
1.90 ×
1.37 ×
1.08 ×
0.78 ×
4.05
Trunc. N=5
2.85 ×
2.74 ×
2.59 ×
2.53 ×
1.67 ×
1.36 ×
4.02
2.40 ×
2.22 ×
2.04 ×
1.65 ×
1.41 ×
1.10 ×
3.42
Trunc. N=7
3.21 ×
3.06 ×
2.80 ×
2.71 ×
1.70 ×
1.34 ×
4.56
2.55 ×
2.35 ×
2.09 ×
1.67 ×
1.38 ×
1.06 ×
3.71
Dyn. trunc.
3.45 ×
3.19 ×
2.85 ×
2.73 ×
1.78 ×
1.40 ×
4.95
2.48 ×
2.30 ×
2.05 ×
1.68 ×
1.45 ×
1.11 ×
3.66
Tree B=15
3.94 ×
3.18 ×
2.38 ×
1.90 ×
1.10 ×
0.83 ×
6.15
2.96 ×
2.30 ×
1.73 ×
1.19 ×
0.93 ×
0.67 ×
4.75
TABLE X: Output throughput ratios of verification strategies relative to autoregressive decoding across client concurrency R at sampling temperature T=0 . Each ratio is the quotient of unweighted mean output throughputs over the same benchmark group, evaluated for the same verifier and R . Avg. τ denotes the unweighted mean accepted length across benchmarks within the group and all six concurrency levels. A ratio below unity indicates lower throughput than autoregressive decoding. Verification strategies are defined in Table VIII .
Device
Model
Strategy
Text Benchmarks
Multimodal Benchmarks
R=1
R=4
R=8
R=16
R=32
R=64
Avg. τ
R=1
R=4
R=8
R=16
R=32
R=64
Avg. τ
RTX 5090
Q3.5-4B
Linear
3.05 ×
3.24 ×
2.90 ×
2.08 ×
1.35 ×
0.90 ×
5.35
2.32 ×
2.35 ×
2.13 ×
1.51 ×
1.03 ×
0.77 ×
4.06
Trunc. N=5
2.41 ×
2.61 ×
2.65 ×
2.23 ×
1.68 ×
1.21 ×
4.01
2.05 ×
2.13 ×
2.17 ×
1.74 ×
1.34 ×
1.06 ×
3.42
Trunc. N=7
2.67 ×
2.90 ×
2.95 ×
2.41 ×
1.67 ×
1.16 ×
4.55
2.17 ×
2.24 ×
2.28 ×
1.81 ×
1.30 ×
1.02 ×
3.72
Dyn. trunc.
2.87 ×
3.01 ×
2.98 ×
2.36 ×
1.70 ×
1.16 ×
4.96
2.11 ×
2.15 ×
2.14 ×
1.74 ×
1.32 ×
1.02 ×
3.66
Tree B=15
3.23 ×
3.25 ×
2.23 ×
1.82 ×
1.18 ×
0.81 ×
6.16
2.43 ×
2.43 ×
1.63 ×
1.38 ×
0.93 ×
0.70 ×
4.75
TABLE XI: Output throughput ratios of verification strategies relative to autoregressive decoding on an RTX 5090 (32 GB), an A800 (80 GB), and an H100 (80 GB), across client concurrency R at sampling temperature T=0 . H100 results are shown in the continuation.
Fig. 12: Temporal behavior of verification strategies under bursty arrivals. All strategies serve the same arrival trace. From top to bottom: arrival rate, outstanding requests, Uni-LaSpec’s verification structure, tokens verified per step, mean draft tokens verified per decoding request, and accepted length τ . Arrival ticks mark individual requests. Pale blue and orange shading distinguish requests in flight from those queued at the client; the dashed line marks the concurrency cap. The structure strip uses green for tree verification and blue for linear verification.
Department of Mathematics, National University of Singapore, Singapore · Yale University, Center for Algorithms, Data, and Market Design, USA · Department of Electrical and Computer Engineering, National University of Singapore, Singapore