ChainFlow: Few-Step Latent Flow Matching for Speculative Decoding
Authors: Selim Aktas, Yusuf Erdem Erdem, Ahmed Zeer, Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali
Organizations: Department of Electronics Engineering, Koç University, Istanbul, Türkiye · Department of Computer Engineering, Yıldız Technical University, Istanbul, Türkiye
Speculative decoding has become a standard way to accelerate LLM inference: a lightweight drafter proposes tokens that the target verifies in one parallel pass. Existing drafters face trade-offs that limits their speedups: autoregressive drafters generate tokens one at a time, so drafting cost grows with draft length, while diffusion drafters generate tokens in parallel but predict each position independently, ignoring inter-token dependencies. We introduce ChainFlow, a new speculative decoding paradigm and the first flow-matching drafter conditioned on the target's hidden states. ChainFlow treats drafting as a short transport over hidden states, needing one or two flow steps instead of the many denoising steps of generation from noise. A variational autoencoder compresses these states so drafting cost barely grows with target size, lightweight feedback heads restore inter-token dependencies, and tree drafting raises acceptance. On Qwen3.5 models, at matched drafter size and training data, ChainFlow outperforms EAGLE-3 and DFlash in vLLM by 39% and 11%, respectively. These gains further widen as drafters are adapted to new languages sequentially. Our ablations cover drafter design and concurrent serving, and we open-source our code and drafter models.
Figures & tables
Figure 1: ChainFlow against EAGLE-3 and DFlash. Hatched: gain from 8 × 4 tree verification. (a) Served speedup over autoregressive decoding at batch one on English domains, ran on one NVIDIA B300 with vLLM 0.31.0 and greedy decoding. The tree beats both in every domain. (b) Acceptance length τ (offline) after Turkish adaptation; on average even the chain beats both, showing adaptability.
Figure 2: One draft–verify cycle of ChainFlow on a letter example. The drafter starts each position from the extrapolation z0i=zC+i(zC−zC−1) of the context latents. The illustration uses N=2 Euler steps; served experiments use N=1 . Here E,F,G are accepted and H is replaced by the target’s own token H∗ .
Figure 3: Tree drafting and attention mask (illustrative). Each draft token attends only to itself and its ancestors, so all branches are verified in one target pass.
Method
Code
Math
QA
MT
RAG
Summ.
Tool
Transl.
Mean
τ
V
τ
V
τ
V
τ
V
τ
V
τ
V
τ
V
τ
V
τ
V
EAGLE-3
3.09
1.64 ×
3.80
2.00 ×
2.09
1.14 ×
2.12
1.17 ×
2.48
1.36 ×
2.10
1.17 ×
2.41
1.31 ×
2.30
1.11 ×
2.49
1.35 ×
DFlash
3.02
2.11 ×
3.68
2.55 ×
1.89
1.38 ×
1.98
1.44 ×
2.34
1.69 ×
2.03
1.49 ×
2.29
1.64 ×
2.24
1.38 ×
2.35
1.69 ×
ChainFlow C
2.43
1.45 ×
2.91
1.73 ×
1.72
1.06 ×
1.76
1.08 ×
1.99
1.23 ×
1.72
1.07 ×
1.98
1.21 ×
1.85
1.06 ×
2.01
1.23 ×
ChainFlow T1
3.31
2.19 ×
4.07
2.67 ×
2.43
1.66 ×
2.44
1.68 ×
2.73
1.85 ×
2.32
1.61 ×
2.74
1.84 ×
2.58
1.39 ×
2.78
1.87 ×
ChainFlow T2
3.41
2.14 ×
4.51
2.78 ×
2.31
1.51 ×
2.33
1.53 ×
2.65
1.72 ×
2.23
1.48 ×
2.64
1.69 ×
2.46
1.32 ×
2.73
1.75 ×
Table 1: English results at Qwen3.5-4B, batch one: acceptance length τ and speedup V over autoregressive decoding by domain and pooled mean. C denotes chain drafting ( K=3 ); T1 and T2 denote 8×4 and 5×7 trees (nodes per depth × depth), respectively, with T1 as our headline configuration. Each drafter has ∼ 0.52B parameters. EAGLE-3 and DFlash draft K=4 tokens per round. Evaluated on NVIDIA B300 with vLLM 0.31.0 and greedy decoding.
Target
Hidden D
Latent Z
Drafter params
Qwen3.5-4B
2560
640
529M
Qwen3.5-9B
4096
1024
1.16B
Qwen3.5-27B
5120
1024
1.64B
Table 2: Drafters. Parameter counts exclude the target’s shared, frozen embedding and lm_head .
Target
Flow ms
AR step ms
τ
Chain
Tree
4B
0.30
3.1
2.78
1.23 ×
1.87 ×
9B
0.38
4.2
2.66
1.30 ×
1.78 ×
27B
0.38
11.2
2.73
1.61 ×
2.12 ×
Table 3: Scaling with target size. Flow and AR latencies are per draft and decoding step (ms), respectively. τ is measured on a fixed 8×4 tree; chain uses K=3 .
Figure 4: Conditional acceptance at each draft position, given that all earlier positions were accepted (offline, 4B). Left: English drafters on English. Right: after Turkish adaptation, on Turkish. Chain curves end at position 7; the tree is 8×4 .
Target
Code
Math
QA
MT
RAG
Summ.
Tool
Transl.
4B
2.19
2.67
1.66
1.68
1.85
1.61
1.84
1.39
9B
2.13
2.67
1.52
1.54
1.73
1.52
1.83
1.42
27B
2.51
3.23
1.82
1.87
2.09
1.79
2.09
1.74
Table 4: Per-domain tree speedup ( 8×4 ) by target size in English.
Figure 5: Effect of language adaptation on offline decoding speedup ( z ) for ChainFlow (tree mode), DFlash, and EAGLE-3. Adapting to new languages improves their decoding speedups, with ChainFlow achieving the largest gains, while English performance declines slightly across all three methods.
Drafter
acc@1
acc@4
τ
Speedup
Flow, context only
0.96
0.44
2.68
0.96 ×
+ deep-position loss
0.96
0.44
2.89
0.99 ×
+ Markov head
0.95
0.46
2.93
1.01 ×
Autoregressive
0.99
0.57
3.21
1.00 ×
Table 5: Token feedback. acc@ i : acceptance at draft position i ; speedup measured live.
Experts × positions
Plateau
Epochs to 90%
2×4 (ChainFlow)
1.00
≈ 85
4×2
≈ 0.99
≈ 80
Table 6: Splitting the draft block across experts. Plateau: validation expected acceptance relative to 2×4 . Values are approximate.
Dev. sweep
4B, τ
Source z0
epochs to 90%
N=1
N=2
N=4
Noise
≈ 57
1.97
2.55
2.47
Repeat last
≈ 38
2.58
2.66
2.42
Extrapolation
≈ 20
2.49
2.53
2.50
Table 7: Source distribution. Left: training epochs to 90% of the validation plateau. Right: inference-time Euler steps for 4B drafters trained at N=2 (offline, English chain).
Figure 6: Training loss (top) and validation expected acceptance (bottom) across epochs for three initializations: noise, last-latent repetition, and trajectory extrapolation (Delta).
Figure 7: Capacity and context sweeps for flow with MLP experts.
Tree Shape
Positions
Tokens/s
τ
Speedup
8×4
33
610.9
2.82
1.87 ×
8×5
41
606.2
2.84
1.85 ×
8×6
49
596.4
2.87
1.83 ×
12×4
49
593.6
2.88
1.82 ×
6×5
31
593.4
2.74
1.82 ×
16×4
65
589.0
2.96
1.80 ×
Table 8: Tree shape ( Wb×D : nodes kept per depth × depth) at Qwen3.5-4B, batch one, eight English domains. Positions: tokens verified per round, including the committed token. Speedup is relative to plain decoding (327 tokens/s). The headline 8×4 tree is best.
Concurrent requests
Method
1
4
8
16
64
128
EAGLE-3
1.33
1.26
1.23
1.20
1.20
1.06
DFlash
1.66
1.48
1.39
1.33
1.32
1.14
ChainFlow
1.85
1.12
0.94
0.81
0.63
0.45
Table 9: Speedup under concurrency at Qwen3.5-4B: output tokens per second relative to autoregressive decoding at the same number of concurrent requests.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Language
Source
Domain
Rows
English
Code Alpaca ( Chaudhary, 2023 )
Code
5,000
Dolly ( Conover et al., 2023 )
Instruction
5,000
Nemotron v1, math ( Nathawani et al., 2025 )
Math
5,000
Nemotron v1, STEM ( Nathawani et al., 2025 )
Science
5,000
GSM8K ( Cobbe et al., 2021 )
Math
5,000
UltraChat ( Ding et al., 2023 )
Dialogue
5,000
Appendix
Table 10: Training sources by language and domain. Drafters are trained on English first, then adapted in sequence to Turkish, German and French without English replay. English totals 45,887 rows (16.6M tokens), while Turkish, German and French 8,000 each. Replay stages add 2,088 Turkish rows (20.7%). Rows are prompts, and training responses are the target’s own greedy continuations.
School of Computer Science & Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems, Peking University · University of Electronic Science and Technology of China · Shanghai Jiao Tong University +1