ChainFlow: Few-Step Latent Flow Matching for Speculative Decoding
Authors: Selim Aktas, Yusuf Erdem Erdem, Ahmed Zeer, Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali
Organizations: Department of Electronics Engineering, Koç University, Istanbul, Türkiye · Department of Computer Engineering, Yıldız Technical University, Istanbul, Türkiye
Speculative decoding has become a standard way to accelerate LLM inference: a lightweight drafter proposes tokens that the target verifies in one parallel pass. Existing drafters face trade-offs that limits their speedups: autoregressive drafters generate tokens one at a time, so drafting cost grows with draft length, while diffusion drafters generate tokens in parallel but predict each position independently, ignoring inter-token dependencies. We introduce ChainFlow, a new speculative decoding paradigm and the first flow-matching drafter conditioned on the target's hidden states. ChainFlow treats drafting as a short transport over hidden states, needing one or two flow steps instead of the many denoising steps of generation from noise. A variational autoencoder compresses these states so drafting cost barely grows with target size, lightweight feedback heads restore inter-token dependencies, and tree drafting raises acceptance. On Qwen3.5 models, at matched drafter size and training data, ChainFlow outperforms EAGLE-3 and DFlash in vLLM by 39% and 11%, respectively. These gains further widen as drafters are adapted to new languages sequentially. Our ablations cover drafter design and concurrent serving, and we open-source our code and drafter models.
Figures & tables
Figure 1: ChainFlow against EAGLE-3 and DFlash. Hatched: gain from 8 × 4 tree verification. (a) Served speedup over autoregressive decoding at batch one on English domains, ran on one NVIDIA B300 with vLLM 0.31.0 and greedy decoding. The tree beats both in every domain. (b) Acceptance length τ (offline) after Turkish adaptation; on average even the chain beats both, showing adaptability.
Figure 2: One draft–verify cycle of ChainFlow on a letter example. The drafter starts each position from the extrapolation z0i=zC+i(zC−zC−1) of the context latents. The illustration uses N=2 Euler steps; served experiments use N=1 . Here E,F,G are accepted and H is replaced by the target’s own token H∗ .
Figure 3: Tree drafting and attention mask (illustrative). Each draft token attends only to itself and its ancestors, so all branches are verified in one target pass.
Method
Code
Math
QA
MT
RAG
Summ.
Tool
Transl.
Mean
τ
V
τ
V
τ
V
τ
V
τ
V
τ
V
τ
V
τ
V
τ
V
EAGLE-3
3.09
1.64 ×
3.80
2.00 ×
2.09
1.14 ×
2.12
1.17 ×
2.48
1.36 ×
2.10
1.17 ×
2.41
1.31 ×
2.30
1.11 ×
2.49
1.35 ×
DFlash
3.02
2.11 ×
3.68
2.55 ×
1.89
1.38 ×
1.98
1.44 ×
2.34
1.69 ×
2.03
1.49 ×
2.29
1.64 ×
2.24
1.38 ×
2.35
1.69 ×
ChainFlow C
2.43
1.45 ×
2.91
1.73 ×
1.72
1.06 ×
1.76
1.08 ×
1.99
1.23 ×
1.72
1.07 ×
1.98
1.21 ×
1.85
1.06 ×
2.01
1.23 ×
ChainFlow T1
3.31
2.19 ×
4.07
2.67 ×
2.43
1.66 ×
2.44
1.68 ×
2.73
1.85 ×
2.32
1.61 ×
2.74
1.84 ×
2.58
1.39 ×
2.78
1.87 ×
ChainFlow T2
3.41
2.14 ×
4.51
2.78 ×
2.31
1.51 ×
2.33
1.53 ×
2.65
1.72 ×
2.23
1.48 ×
2.64
1.69 ×
2.46
1.32 ×
2.73
1.75 ×
Table 1: English results at Qwen3.5-4B, batch one: acceptance length τ and speedup V over autoregressive decoding by domain and pooled mean. C denotes chain drafting ( K=3 ); T1 and T2 denote 8×4 and 5×7 trees (nodes per depth × depth), respectively, with T1 as our headline configuration. Each drafter has ∼ 0.52B parameters. EAGLE-3 and DFlash draft K=4 tokens per round. Evaluated on NVIDIA B300 with vLLM 0.31.0 and greedy decoding.
Target
Hidden D
Latent Z
Drafter params
Qwen3.5-4B
2560
640
529M
Qwen3.5-9B
4096
1024
1.16B
Qwen3.5-27B
5120
1024
1.64B
Table 2: Drafters. Parameter counts exclude the target’s shared, frozen embedding and lm_head .
Target
Flow ms
AR step ms
τ
Chain
Tree
4B
0.30
3.1
2.78
1.23 ×
1.87 ×
9B
0.38
4.2
2.66
1.30 ×
1.78 ×
27B
0.38
11.2
2.73
1.61 ×
2.12 ×
Table 3: Scaling with target size. Flow and AR latencies are per draft and decoding step (ms), respectively. τ is measured on a fixed 8×4 tree; chain uses K=3 .
Figure 4: Conditional acceptance at each draft position, given that all earlier positions were accepted (offline, 4B). Left: English drafters on English. Right: after Turkish adaptation, on Turkish. Chain curves end at position 7; the tree is 8×4 .
Target
Code
Math
QA
MT
RAG
Summ.
Tool
Transl.
4B
2.19
2.67
1.66
1.68
1.85
1.61
1.84
1.39
9B
2.13
2.67
1.52
1.54
1.73
1.52
1.83
1.42
27B
2.51
3.23
1.82
1.87
2.09
1.79
2.09
1.74
Table 4: Per-domain tree speedup ( 8×4 ) by target size in English.
Figure 5: Effect of language adaptation on offline decoding speedup ( z ) for ChainFlow (tree mode), DFlash, and EAGLE-3. Adapting to new languages improves their decoding speedups, with ChainFlow achieving the largest gains, while English performance declines slightly across all three methods.
Drafter
acc@1
acc@4
τ
Speedup
Flow, context only
0.96
0.44
2.68
0.96 ×
+ deep-position loss
0.96
0.44
2.89
0.99 ×
+ Markov head
0.95
0.46
2.93
1.01 ×
Autoregressive
0.99
0.57
3.21
1.00 ×
Table 5: Token feedback. acc@ i : acceptance at draft position i ; speedup measured live.
Experts × positions
Plateau
Epochs to 90%
2×4 (ChainFlow)
1.00
≈ 85
4×2
≈ 0.99
≈ 80
Table 6: Splitting the draft block across experts. Plateau: validation expected acceptance relative to 2×4 . Values are approximate.
Dev. sweep
4B, τ
Source z0
epochs to 90%
N=1
N=2
N=4
Noise
≈ 57
1.97
2.55
2.47
Repeat last
≈ 38
2.58
2.66
2.42
Extrapolation
≈ 20
2.49
2.53
2.50
Table 7: Source distribution. Left: training epochs to 90% of the validation plateau. Right: inference-time Euler steps for 4B drafters trained at N=2 (offline, English chain).
Figure 6: Training loss (top) and validation expected acceptance (bottom) across epochs for three initializations: noise, last-latent repetition, and trajectory extrapolation (Delta).
Figure 7: Capacity and context sweeps for flow with MLP experts.
Tree Shape
Positions
Tokens/s
τ
Speedup
8×4
33
610.9
2.82
1.87 ×
8×5
41
606.2
2.84
1.85 ×
8×6
49
596.4
2.87
1.83 ×
12×4
49
593.6
2.88
1.82 ×
6×5
31
593.4
2.74
1.82 ×
16×4
65
589.0
2.96
1.80 ×
Table 8: Tree shape ( Wb×D : nodes kept per depth × depth) at Qwen3.5-4B, batch one, eight English domains. Positions: tokens verified per round, including the committed token. Speedup is relative to plain decoding (327 tokens/s). The headline 8×4 tree is best.
Concurrent requests
Method
1
4
8
16
64
128
EAGLE-3
1.33
1.26
1.23
1.20
1.20
1.06
DFlash
1.66
1.48
1.39
1.33
1.32
1.14
ChainFlow
1.85
1.12
0.94
0.81
0.63
0.45
Table 9: Speedup under concurrency at Qwen3.5-4B: output tokens per second relative to autoregressive decoding at the same number of concurrent requests.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Language
Source
Domain
Rows
English
Code Alpaca ( Chaudhary, 2023 )
Code
5,000
Dolly ( Conover et al., 2023 )
Instruction
5,000
Nemotron v1, math ( Nathawani et al., 2025 )
Math
5,000
Nemotron v1, STEM ( Nathawani et al., 2025 )
Science
5,000
GSM8K ( Cobbe et al., 2021 )
Math
5,000
UltraChat ( Ding et al., 2023 )
Dialogue
5,000
Appendix
Table 10: Training sources by language and domain. Drafters are trained on English first, then adapted in sequence to Turkish, German and French without English replay. English totals 45,887 rows (16.6M tokens), while Turkish, German and French 8,000 each. Replay stages add 2,088 Turkish rows (20.7%). Rows are prompts, and training responses are the target’s own greedy continuations.
Block diffusion speculative decoding improves LLM inference efficiency by proposing a block of future tokens in parallel and verifying them with a single forward pass through the target model. However, existing methods retain only the accepted prefix and discard the rejected suffix, preventing the computation spent on these positions from benefiting subsequent drafting rounds and forcing the drafter to repeatedly reconstruct representations for future tokens from scratch. We observe that rejection only determines whether a proposed token can be committed, while the verifier representations at rejected positions can still provide useful information for subsequent predictions. Based on this observation, we propose DFlow, a simple yet effective framework that enables verifier information to flow across drafting rounds. DFlow reuses the hidden states produced by the target verifier for the rejected suffix to guide subsequent drafting without additional target computation. To effectively learn this information flow across drafting rounds, we introduce a self-condition train strategy that feeds verifier representations from earlier predictions back into subsequent predictions. Experiments on Qwen3 models across diverse benchmarks demonstrate that DFlow consistently improves draft quality and acceptance length over DFlash.
Yaojie Zhang, Linfeng Zhang, Bin Cui +1
School of Computer Science & Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems, Peking University · University of Electronic Science and Technology of China · Shanghai Jiao Tong University +1
Speculative decoding accelerates large language model (LLM) inference by using a lightweight drafter to propose a block of tokens that the target model verifies in one parallel pass. Autoregressive drafters generate this block one token at a time, whereas block diffusion drafters such as DFlash generate it in one pass, making drafting faster. However, these diffusion drafters predict each token independently, ignoring dependencies between tokens in the block. We introduce FlashMap, a continuous flow drafter that conditions a flow map language model on the target model's hidden states and generates the whole block in one forward pass while capturing inter-token dependencies. On Qwen3-8B, FlashMap is faster than a matched DFlash baseline on every benchmark, with 7% higher average speedup and up to 6.1x lossless acceleration.
Jeff Shen, Zhejian Zhou, Ryan Lee +2
Polymathic AI · Princeton University · USC Information Sciences Institute +2
Speculative decoding accelerates LLM inference by drafting a tree of candidate continuations and verifying it in one target forward. Existing drafters fall into two camps with opposite weaknesses. Autoregressive drafters such as EAGLE-3 preserve dependence along each draft path but call the drafter once per tree depth, making drafting a non-trivial share of per-iteration latency. Parallel drafters cut drafter calls by predicting multiple future positions in one forward, but each position is predicted without seeing the others, producing paths the verifier rejects. In this paper, we propose SpecBlock, a block-iterative drafter that combines path dependence with cheap drafting. Each drafter forward produces K dependent positions and we call this a block. The draft tree grows through repeated block expansions. Two mechanisms explicitly carry path dependence to keep later draft positions accurate. Within each block, a layer-wise shift carries the previous position's hidden state into every decoder layer. Across blocks, each new block can start from any position of the previous block, inheriting its hidden state to extend the path. To spend verifier budget where acceptance is likely, a co-trained rank head replaces the fixed top-k tree by allocating per-position branching during drafting. To avoid training the drafter on prefixes it never produces at inference, a valid-prefix mask drops the loss at later positions once an earlier one is wrong. Beyond static drafting, a cost-aware bandit at deployment uses free verifier feedback to update the drafter selectively, only when the expected throughput gain exceeds the update cost. Experiments show that SpecBlock improves mean speedup by 8-13% over EAGLE-3 at 44-52% of its drafting cost, and cost-aware adaptation extends this lead to 11-19%.
Weijie Shi, Qiang Xu, Fan Deng +9
Hong Kong University of Science and Technology · MetaX · Zhejiang Normal University +1