Flide: Accelerating Decision Inference with Exact State Forking

Bobo Li; Jiajia Song; Hao Fei; Wynne Hsu; Mong Li Lee

cs.CLOct 30, 2026

Flide: Accelerating Decision Inference with Exact State Forking

Authors: Bobo Li, Jiajia Song, Hao Fei, Wynne Hsu, Mong Li Lee

Organizations: National University of Singapore · University of Oxford

Abstract

Decision models such as Jev have emerged as a highly promising paradigm for LLM agent systems, providing lightweight and reliable categorical reasoning for rapid System One execution. However, existing generation-oriented inference engines fail to support these workloads effectively, suffering from severe prefix recomputation overheads or distorted output probabilities. To address this challenge, we propose Flide, an efficient and exact inference engine that decouples the logical tree structure of a decision request from its physical execution within a single flattened token stream. Through ForkAttention, Flide navigates request dependencies using GPU-resident indices across both standard attention and recurrent Gated DeltaNet layers, guaranteeing in exact arithmetic that every query branch evaluates identically to an isolated forward pass. By expressing branch topologies directly in GPU-resident metadata, Flide accommodates diverse request mixtures with a compact set of pre-recorded execution graphs while enabling expanding agent histories to resume from checkpoints saved in earlier passes. Extensive evaluations on public hybrid and decision models demonstrate that Flide answers queries up to 9.5 times faster than vLLM and SGLang while staying closest to a float32 reference in total variation distance. By unifying physical execution throughput with mathematical exactness, Flide establishes a principled foundation for scalable, latency-critical decision intelligence in agent systems.

Figures & tables

Panel of Figure 1 — Figure 1: One request with three questions, run three ways. Forks show the logical structure and strips the physical sequence, where red bars are delimiters and a tilde, as in $\tilde{p}_{1}$ , marks an answer that may differ.

Engine	One pass	Same input	Hybrid fork	Graph replay	Bitwise
vLLM ( vLLM Project, 2026d )	✗	✓	✗	✓	✗
SGLang ( SGLang Project, 2026c )	✗	✓	✗	✓	✓
SGLang MIS ( SGLang Project, 2026a )	✓	✗	✓	✗	✓
vllm-jev ( mode-io, 2026 )	✗	✓	✗	✓	✗
Flide	✓	✓	✓	✓	✓

Table 1: Decision serving in current engines. Each mark follows the public documentation and code of the engine, and bitwise marks an optional mode that keeps answers bit-identical under any load on hybrid models.

Figure 2: Overview of Flide. (a) Requests, each a tree of a shared state and its questions. (b) ForkAttention over one flat stream. (c) Flat graphs over GPU-resident round metadata. (d) Incremental forks of a growing state.

Workload	Source	Requests	Questions	Per request	State	Stream	Sequences	Copies	Per level
RACE	RACE-M test	256	256	1.0	250	325	325	4	1,024
QuALITY	QuALITY val., cut	256	256	1.0	3,355	3,443	3,443	4	1,024
DI	Decision Index	679	3,635	5.4	323	1,349	3,387	2	1,358
RACE, all	RACE-M test	256	1,019	4.0	255	546	1,283	4	1,024
QuALITY, all	QuALITY val.	115	2,086	18.1	5,835	7,454	107,085	2	230
MultiRC	MultiRC val.	83	3,948	47.6	327	2,663	17,808	4	332

Table 2: Workloads. Mean tokens of the state, of the stream with the state once and every question once, and of one sequence per question, in the prompts of Flide and vLLM.

		One question per state						All questions of a state
		RACE			QuALITY			DI			RACE			QuALITY			MultiRC
Model	Engine	Lat. $\downarrow$	C8 $\uparrow$	C64 $\uparrow$	Lat. $\downarrow$	C8 $\uparrow$	C64 $\uparrow$	Lat. $\downarrow$	C8 $\uparrow$	C64 $\uparrow$	Lat. $\downarrow$	C8 $\uparrow$	C64 $\uparrow$	Lat. $\downarrow$	C8 $\uparrow$	C64 $\uparrow$	Lat. $\downarrow$	C8 $\uparrow$	C64 $\uparrow$
Qwen3.5
0.8B	vLLM	19.9	154	527	35.1	84.1	89.4	23.2	69.1	107	47.3	88.2	208	362	4.5	4.6	154	14.1	14.4
	SGLang	27.0	153	418	35.3	76.5	78.8	28.6	61.5	98.2	31.0	95.7	174	359	4.6	4.4	130	16.6	15.1
	Flide	6.2	539	672	17.9	93.6	96.5	7.1	196	236	7.2	410	540	37.7	39.1	39.1	16.8	115	124

Table 3: Flide leads most where a state carries many questions, and shading gives its gain over the best baseline.

		One question per state						All questions of a state
		RACE			QuALITY			DI		RACE			QuALITY			MultiRC
Qwen3.5	Engine	Agr. $\uparrow$	Acc. $\uparrow$	Stab. $\uparrow$	Agr. $\uparrow$	Acc. $\uparrow$	Stab. $\uparrow$	Agr. $\uparrow$	Stab. $\uparrow$	Agr. $\uparrow$	Acc. $\uparrow$	Stab. $\uparrow$	Agr. $\uparrow$	Acc. $\uparrow$	Stab. $\uparrow$	Agr. $\uparrow$	Acc. $\uparrow$	Stab. $\uparrow$
0.8B	vLLM	98.0	66.4	97.8	98.0	39.5	96.8	97.8	99.0	97.6	71.7	97.9	98.0	46.1	99.1	99.7	44.4	100.0
	SGLang	98.0	67.2	98.3	98.0	39.5	99.2	96.9	99.7	97.1	71.6	99.3	97.7	46.2	99.5	98.9	47.1	99.9
	Flide	99.6	67.2	100.0	98.4	38.3	100.0	98.8	99.9	99.0	71.7	100.0	98.1	45.6	100.0	99.8	44.5	100.0
4B	vLLM	100.0	87.9	99.6	99.2	63.3	99.5	99.1	99.7	99.7	89.1	100.0	99.2	77.0	99.7	99.2	79.9	100.0

Table 4: On Qwen3.5, Flide’s answers stay closest to a float32 reference, with all values in percent.

Figure 3: Flide leads on every workload where a state carries several questions. Speed of each engine over the fastest one, geometric mean over the four sizes. RC, QL and MRC stand for RACE, QuALITY and MultiRC, and -1 marks one question per state.

	DI		QuALITY, all		RACE
Variant	Lat. $\downarrow$	Req/s $\uparrow$	Lat. $\downarrow$	Req/s $\uparrow$	Req/s $\uparrow$
Flide	8.8	48.9	135.6	8.6	210.3
without forks	8.7	20.7	1868.5	0.6	211.1
without flat graphs	8.5	43.5	135.7	8.5	134.9
without CUDA graphs	76.1	48.9	137.2	8.5	191.6
without selective readout	8.9	47.8	140.1	8.3	205.2

Table 5: Ablation on Qwen3.5-4B, timed inside the engine without its HTTP server. Each row removes one component of Flide, and Req/s counts the requests per second of a round.

		RACE					QuALITY
Model	Engine	Lat. $\downarrow$	C8 $\uparrow$	C32 $\uparrow$	C64 $\uparrow$	Acc. $\uparrow$	Lat. $\downarrow$	C8 $\uparrow$	C32 $\uparrow$	C64 $\uparrow$	Acc. $\uparrow$
Open-Jev-2B	vllm-jev	41.8	96.5	134.8	133.9	75.4	61.6	33.4	33.2	32.5	50.8
	+ RadixFlow	34.3	95.1	115.8	114.8	74.6	72.8	29.3	29.2	29.0	50.8
	Flide	8.9	344.5	401.4	412.1	74.2	37.3	47.1	47.9	47.9	50.8
Open-Jev-9B	vllm-jev	57.8	31.8	32.9	32.9	84.4	141.9	8.5	8.5	8.5	68.4
	+ RadixFlow	47.5	30.4	31.7	31.7	84.4	153.9	8.2	8.1	8.2	68.4

Table 6: Flide against the engine written for these models. Shading gives Flide’s gain over the better vllm-jev setting.

Figure 4: Multi-item scoring shifts the whole distribution. Total variation of each answer from the float32 reference on Qwen3.5-4B, every question of a state in one request.

Figure 5: Flide adds little latency per question and per step. Latency on Qwen3.5-4B as the questions on one state and the history of an agent grow.

	Qwen3.5-0.8B					Qwen3.5-4B					Qwen3.5-9B
	One agent, ms			Steps/s		One agent, ms			Steps/s		One agent, ms			Steps/s
Engine	${<}$ 4k $\downarrow$	${>}$ 12k $\downarrow$	Growth $\downarrow$	1 $\uparrow$	32 $\uparrow$	${<}$ 4k $\downarrow$	${>}$ 12k $\downarrow$	Growth $\downarrow$	1 $\uparrow$	32 $\uparrow$	${<}$ 4k $\downarrow$	${>}$ 12k $\downarrow$	Growth $\downarrow$	1 $\uparrow$	32 $\uparrow$
vLLM	173.8	333.7	13.5	4.6	7.8	218.7	384.6	13.8	3.8	6.0	218.5	382.7	13.6	3.8	5.9
SGLang	64.2	197.5	11.1	9.7	14.0	116.2	279.8	13.2	6.0	12.4	167.6	355.5	14.5	4.5	8.2
MIS	52.4	88.3	2.9	16.4	28.9	81.8	294.9	17.2	7.5	8.9	102.2	453.8	28.5	5.2	5.7
Flide	11.7	18.7	0.6	72.4	208.2	22.4	36.1	0.9	38.5	57.9	32.7	48.2	1.0	27.4	36.1

Table 7: Agents whose state grows by one step at a time. Latency of a step in ms with a history under 4k and over 12k tokens, its growth per thousand tokens of history, and steps per second at 1 and 32 agents.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Engine	Version	Endpoint and settings
vLLM	0.32.0rc1 of 14 October, 0.32.1rc1	decision endpoint, --enable-prefix-caching --logprobs-mode raw_logprobs , the later build also --prefix-match-unit 32 ; Gemma 4 also through the completion endpoint of 0.32.0rc1
SGLang	0.5.21	decision endpoint, default settings
SGLang MIS	0.5.21	/v1/score , --enable-mis --attention-backend flashinfer --linear-attn-prefill-backend triton , which turns CUDA graphs, the radix cache and chunked prefill off
vllm-jev	0.3.7 on vLLM 0.29.0	vllm-jev serve with its defaults, and with --radixflow true
Flide	release	serve with two front-end processes and default batching

Appendix

Table 8: Engine versions and launch settings.

		1 client			8 clients			32 clients			64 clients
Workload	Engine	Med. $\downarrow$	P99 $\downarrow$	Thr. $\uparrow$	Med. $\downarrow$	P99 $\downarrow$	Thr. $\uparrow$	Med. $\downarrow$	P99 $\downarrow$	Thr. $\uparrow$	Med. $\downarrow$	P99 $\downarrow$	Thr. $\uparrow$
RACE, one	vLLM	24.6	37.6	38.5	59.9	80.6	130	164	194	192	315	428	196
	SGLang	31.6	46.8	30.1	83.4	99.2	96.5	182	216	173	351	440	179
	Flide	10.9	20.1	88.8	45.2	58.3	175	155	191	201	308	393	204
QuALITY, one	vLLM	68.6	79.1	14.7	395	699	19.3	1.7k	1.8k	19.3	3.3k	3.4k	19.3
	SGLang	75.3	85.6	13.6	386	523	18.5	1.7k	1.8k	18.5	3.4k	3.6k	18.5

Appendix

Table 9: Full results on Qwen3.5 4B. Median and 99th percentile latency in ms (k for thousands) and requests per second at every number of clients.

		1 client			8 clients			32 clients			64 clients
Workload	Engine	Med. $\downarrow$	P99 $\downarrow$	Thr. $\uparrow$	Med. $\downarrow$	P99 $\downarrow$	Thr. $\uparrow$	Med. $\downarrow$	P99 $\downarrow$	Thr. $\uparrow$	Med. $\downarrow$	P99 $\downarrow$	Thr. $\uparrow$
RACE, one	vLLM	14.5	23.9	67.1	51.1	66.5	154	167	212	187	325	379	192
	SGLang	36.6	45.7	27.1	76.0	89.4	105	203	278	155	361	474	173
	Flide	11.3	17.2	87.5	45.9	56.0	172	169	193	188	327	362	193
QuALITY, one	vLLM	65.7	71.4	15.6	396	532	18.0	1.7k	1.8k	18.0	3.5k	3.7k	18.0
	SGLang	95.3	103	10.9	543	727	13.2	2.3k	2.5k	13.2	4.9k	5.0k	13.2

Appendix

Table 10: Full results on Gemma 4 E4B. Median and 99th percentile latency in ms (k for thousands) and requests per second at every number of clients.

Figure 7: Flide serves more requests at a lower latency. Requests per second against median latency for 1, 8, 32 and 64 clients on Qwen3.5-4B.

Figure 8: Flide serves more requests at a lower latency on Gemma 4. Requests per second against median latency for 1, 8, 32 and 64 clients on Gemma 4 E4B.

		DI			RACE			QuALITY			MultiRC
Qwen3.5	Engine	TV $\downarrow$	P99 $\downarrow$	Flip $\downarrow$	TV $\downarrow$	P99 $\downarrow$	Flip $\downarrow$	TV $\downarrow$	P99 $\downarrow$	Flip $\downarrow$	TV $\downarrow$	P99 $\downarrow$	Flip $\downarrow$
0.8B	vLLM	1.05	3.3	2.3	1.08	4.3	1.4	1.34	4.3	3.2	0.58	2.7	0.3
	SGLang	1.09	3.3	3.2	1.07	4.2	1.4	1.41	4.5	2.6	0.83	3.1	1.1
	MIS	7.47	40.2	12.1	9.16	32.3	14.1	14.80	42.3	28.9	3.10	16.1	1.7
	Flide	0.67	2.2	1.0	0.91	3.7	2.3	1.11	3.9	2.1	0.37	1.9	0.2
4B	vLLM	0.73	3.2	1.0	0.28	2.9	0.3	0.56	4.0	0.9	0.59	3.0	0.8

Appendix

Table 11: Multi-item scoring moves the answers it shares, with every question of a state in one request.

		One question per state						All questions of a state
		RACE			QuALITY			DI			RACE			QuALITY			MultiRC
Model	Mode	Lat. $\downarrow$	C64 $\uparrow$	Bits $\uparrow$	Lat. $\downarrow$	C64 $\uparrow$	Bits $\uparrow$	Lat. $\downarrow$	C64 $\uparrow$	Bits $\uparrow$	Lat. $\downarrow$	C64 $\uparrow$	Bits $\uparrow$	Lat. $\downarrow$	C64 $\uparrow$	Bits $\uparrow$	Lat. $\downarrow$	C64 $\uparrow$	Bits $\uparrow$
Qwen3.5
0.8B	Default	6.1	654	88.8	17.9	96.7	99.5	6.8	237	85.5	7.4	544	73.4	37.3	38.6	98.2	17.2	121	94.1
0.8B	Invariant	7.7	754	100	18.3	94.4	100	8.0	229	100	8.9	505	100	38.8	36.9	100	17.8	115	100
4B	Default	10.8	204	26.7	62.4	19.9	86.4	11.7	50.4	67.1	14.3	118	33.2	142	8.6	92.6	50.7	25.6	31.4

Appendix

Table 12: Identical bits at any load. Bits gives the percent of answers whose bits do not change with the load.

	DI		QuALITY, all		MultiRC
	Lat. $\downarrow$	Tok. $\downarrow$	Lat. $\downarrow$	Tok. $\downarrow$	Lat. $\downarrow$	Tok. $\downarrow$
vLLM, endpoint	26.7	2098	537	12448	154	17314
vLLM, token ids	20.1	2049	350	12448	97.3	17314
SGLang, endpoint	28.6	3268	359	30826	130	13358
SGLang, token ids	26.4	2574	155	30826	86.2	13625
MIS	53.1	1364	66.2	7479	55.2	2717

Appendix

Table 13: Where the time goes. One request alone on 0.8B, ms and tokens computed.

Engine	Peak $\downarrow$	Graphs $\downarrow$	Busy	Tok. $\uparrow$	Busy	Tok. $\uparrow$	Busy	Tok. $\uparrow$	Busy	Tok. $\uparrow$	Busy	Tok. $\uparrow$	Busy	Tok. $\uparrow$
	Memory, GiB		DI				RACE				QuALITY, all
			1 client		64 clients		1 client		64 clients		1 client		64 clients
vLLM	135.3	0.6	50	17.9	77	33.6	35	12.5	60	64.9	39	11.3	51	15.5
SGLang	131.7	1.9	59	19.9	91	37.1	28	10.4	60	60.5	67	10.5	84	16.5
MIS	129.0	–	32	16.7	71	46.4	17	5.0	53	46.6	86	47.7	84	51.0
vLLM, 10%	17.2	0.1	46	15.9	76	31.5	27	9.8	60	65.0	34	9.8	44	12.9

Appendix

Table 14: Memory and GPU use on Qwen3.5-4B. Busy is the percent of time a kernel runs, Tok. the useful tokens per second in thousands, and the 10% rows cap the engine at a tenth of the card.

Figure 9: The lead under load. Flide over the best baseline in throughput on Qwen3.5, mean over 0.8B, 4B and 9B.

Table 23

Start with	Ready (s) $\downarrow$	Compiled $\downarrow$	Tuned $\downarrow$
Bundle	36.3	0	0
Bundle, other install	33.3	0	0
Copied cache	181.0	231	200
Empty cache	180.6	231	200

Appendix

Table 17: Kernels shipped with the package. Start of a 0.8B server, kernels compiled and tuned.

			Speedup $\uparrow$
Model	Backbone	Release	DI	RACE	MultiRC	QuALITY	GQA
Fork
Intern-Decision 4B	Qwen3.5 4B	transformers	4.40	4.35	1.15	1.46	1.47
Senova 4B	Qwen3.5 4B	SGLang	3.13	3.65	28.10	1.18
Quyet Medium	Qwen3.5 4B	transformers	5.95	5.90	7.02	1.37
cerno 4B	Qwen3.5 4B	vLLM	4.72	2.75	9.06	1.13

Appendix

Table 18: Decision models served by Flide. Median latency of each release engine over that of Flide for one request at a time, where custom marks an engine that the release ships itself.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,109 papers

SparseEngine: Sparse-First Inference Engine
Sep 30, 2026Jitai Hao, Quansheng Gu, Qiang Huang +1LLM Inference Long-Context Language Model Inference
Stateful Inference for Low-Latency Multi-Agent Tool Calling
May 25, 2026Victor NorgrenLLM Inference Efficiency Multi-Agent LLM Systems
LayerScale, Inc.
Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agents
Apr 9, 2026Khushal SethiInference-Time Compute Allocation LLM Inference Efficiency
Stanford University

Sep 30, 2026J/K move · Enter open · S save

SparseEngine: Sparse-First Inference Engine

Jitai Hao, Quansheng Gu, Qiang Huang, Jun Yu

LLM Inference Long-Context Language Model Inference LLM Inference Acceleration Sparse Attention KV-Cache Compression Long-Horizon LLM Agents

Open paper AskPDF arXiv

Long-context LLM agents accumulate interaction histories that strain KV-cache memory and attention computation. Although sparse attention reduces these costs, heterogeneous cache representations and workflows hinder integration with existing inference engines, while prior sparse-serving abstractions support only specific layouts or workflows. We present SparseEngine, a ground-up, sparse-first inference engine whose shared lifecycle contract lets each method control its KV representation and computation while coordinating state transitions with common serving infrastructure. SparseEngine supports 15 methods across four categories and enables cross-request state management through Chain Cache, which resumes KV-eviction methods from retained history, and controllable Prefix-Cache Pruning, which removes KV from selected history regions while preserving logical-prefix matching. While maintaining method quality, SparseEngine delivers over 10x higher throughput with KV eviction, over 2.5x faster decoding at matched concurrency than vLLM, and over 2x end-to-end speedup on agent benchmarks. The code is available at https://github.com/CURRENTF/SparseEngine.

Abstract

LLM Inference Acceleration

Explore similar work

SparseEngine: Sparse-First Inference Engine

Stateful Inference for Low-Latency Multi-Agent Tool Calling

Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agents