Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability

Xingyu Xie; Zhaochen Yu; Yue Liao; Tao Wang; Kim-Chuan Toh; Shuicheng Yan

cs.LGMar 12, 2026

Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability

Authors: Xingyu Xie, Zhaochen Yu, Yue Liao, Tao Wang, Kim-Chuan Toh, Shuicheng Yan

Organizations: National University of Singapore · ByteDance

Abstract

Long-context autoregressive decoding remains expensive because each decoding step must repeatedly process a growing history. We observe a consistent pattern during decoding: within a sentence, and more generally within a short semantically coherent span, the dominant attention support often remains largely stable. Motivated by this observation, we propose Slow-Fast Inference (SFI), a training-free decoding framework that decouples generation into frequent low-cost fast steps and occasional dense-attention slow steps. Fast steps reuse a compact sparse memory for efficient decoding. Slow steps are triggered near semantic boundaries. At slow steps, the model revisits the broader context and uses the Selector to refresh the selected memory for subsequent fast steps. Across the evaluated context lengths, SFI delivers approximately $1.6\times$ -- $14.4\times$ higher decoding throughput while generally maintaining quality on par with the full-KV baseline across long-context and long-CoT settings. Because SFI is training-free and applies directly to existing checkpoints, it offers a practical path to reducing inference cost for contemporary autoregressive reasoning models in long-context, long-horizon, and agentic workloads.

Figures & tables

Figure 1: The SFI Framework. (A) Motivation: Attention maps from Qwen3-0.6B illustrate a common pattern of within-sentence support stability : across consecutive decoding steps within a sentence, and more generally within a short semantically coherent span, the dominant attended positions remain largely stable rather than changing abruptly at every step. (B) Slow-Fast paradigm and speedup: SFI decouples decoding into many low-cost Fast Steps and occasional dense Slow Steps . The speedup plot reports the average end-to-end throughput gain across the Qwen series, from 0.6B to 235B, and shows that the advantage of this slow-fast schedule grows with context length.

Figure 2: The Slow-Fast Inference framework. Top: Across multiple model scales, attention mass often remains concentrated on a largely stable set of positions within a semantic unit, illustrating within-sentence support stability . Bottom: SFI exploits this pattern by alternating frequent low-cost fast steps , which attend to a managed sparse state ( sink + selected + recent ), with occasional slow steps . A slow step is triggered when a boundary token is generated (Eq. ( 5 )) or when a fixed refresh interval is reached; it then performs dense attention, collects masked attention logits over the allowed candidate set, and invokes the Selector to refresh the selected memory for the next segment.

Figure 3: System infrastructure for efficient Slow-Fast Inference. (A) Asynchronous pipeline: SFI overlaps the main attention computation with slow-step maintenance across two execution streams. While the Main Stream computes attention for layer $i+1$ , the Aux Stream concurrently runs the Selector and cache reorganization for layer $i$ , so that most maintenance overhead is hidden behind ongoing layer execution. (B) Memory-coalesced sparse kernel: Native sparse attention suffers from scattered KV reads and poor bandwidth utilization. We therefore use a two-segment layout in which sink and selected tokens are packed into a contiguous compact buffer, enabling high-bandwidth sequential access over most of the sparse context, while recent tokens are read in place from paged KV.

Figure 4: End-to-end decoding throughput across model scales and context lengths. Throughput (tok/s) is measured for the full-KV baseline (full kv cache) and SFI with up to 2048 generated tokens per request. SFI consistently improves decoding throughput, and the advantage grows with context length.

Retention Ratio (%)	1.6	6.3	12.5	25.0	37.5	50.0	75.0	98.4	100
Speedup	10.67 $\times$	9.56 $\times$	7.15 $\times$	3.96 $\times$	2.75 $\times$	2.10 $\times$	1.43 $\times$	1.10 $\times$	1.00 $\times$

Table 1: Kernel-level sparse-attention speedup at KV length 16K. This measurement isolates the attention kernel itself rather than end-to-end decoding throughput. Speedup is computed as dense-attention latency divided by sparse-attention latency. Batch size = 16, bf16.

		Qwen3-4B		Qwen3-30B-A3B		Qwen3-235B-A22B
Category	Task	Slow	SFI (Ours)	Slow	SFI (Ours)	Slow	SFI (Ours)
Single-Doc QA	Qasper	40.60	44.20 $\uparrow$ 3.6	38.96	42.14 $\uparrow$ 3.2	45.77	44.67
	MultiFieldQA-en	46.56	49.31 $\uparrow$ 2.8	50.32	53.22 $\uparrow$ 2.9	51.21	50.92
	MultiFieldQA-zh	61.21	63.81 $\uparrow$ 2.6	63.42	66.00 $\uparrow$ 2.6	67.08	66.37
Multi-Doc QA	HotpotQA	55.34	59.00 $\uparrow$ 3.7	61.68	63.37 $\uparrow$ 1.7	67.13	67.65 $\uparrow$ 0.5
	2WikiMQA	42.02	44.50 $\uparrow$ 2.5	54.68	55.98 $\uparrow$ 1.3	64.14	65.31 $\uparrow$ 1.2

Table 2: LongBench-V1 results across Qwen3 scales. SFI matches or improves full-KV decoding (Slow) on most subsets, with the clearest average gains at 4B and 30B-A3B. Bold marks the better result within each backbone; green values with

\uparrow

denote gains over Slow.

Backbone	Method	Overall	Easy	Hard	Short	Medium	Long
Qwen3-4B	Slow	34.2	37.5	32.2	35.0	33.5	34.3
Qwen3-4B	SFI (Ours)	34.8 $\uparrow$ 0.6	38.5 $\uparrow$ 1.0	32.5	35.0	34.9 $\uparrow$ 1.4	34.3
Qwen3-30B-A3B	Slow	35.1	36.0	34.6	38.5	32.8	28.6
Qwen3-30B-A3B	SFI (Ours)	35.1	36.0	34.6	38.5	32.8	28.6
Qwen3-235B-A22B	Slow	46.0	50.0	43.7	48.0	44.1	47.6
Qwen3-235B-A22B	SFI (Ours)	46.0	50.0	43.7	47.5	44.1	52.4 $\uparrow$ 4.8

Table 3: LongBench-V2 results across Qwen3 scales. SFI remains stable across model sizes and shows the strongest gain on the longest-context subset of Qwen3-235B-A22B. Bold marks the better result within each backbone; green values with

\uparrow

denote gains over Slow.

	Difficulty		Context Length
Method	Easy	Hard	Short	Medium	Long	Avg.
Slow	37.50	32.20	35.00	33.50	34.30	34.20
StreamingLLM ( Xiao et al., 2024b )	39.58	26.69	36.67	28.84	28.70	31.61
TOVA ( Oren et al., 2024 )	38.54	27.65	36.11	31.63	25.00	31.81
Think ( Xu et al., 2025b )	36.98	27.65	31.67	31.63	29.63	31.21
SnapKV ( Li et al., 2024 )	36.98	29.26	36.11	32.56	25.00	32.21

Table 4: LongBench-V2 comparison with representative training-free KV-cache compression baselines on Qwen3-4B-Instruct-2507. All baselines use 50% compression, whereas SFI retains only 15–20% of tokens on average and still achieves the best overall score. Bold indicates the best result and underline indicates the second-best result among compression methods in each column.

Model	GPQA		MMLU
	Slow	SFI (Ours)	Slow	SFI (Ours)
Qwen3-4B-Thinking-2507	64.14	63.70	63.00	63.00
Qwen3-30B-A3B-Thinking-2507	69.70	71.21	70.90	70.60
Qwen3-235B-A22B-Thinking-2507	80.80	80.80	90.09	90.30

Table 5: Long-CoT reasoning results on GPQA and MMLU. SFI matches the full-KV baseline at medium and large scales, with only minor variation at 4B. Bold marks the better result within each model.

$\lambda_{\text{clip}}$	S-QA	M-QA	Summ.	Few-shot	Synthetic	Code	Avg.
0.005	43.05	50.32	21.64	64.24	72.33	24.74	46.05
0.02	43.14	50.29	21.73	64.21	72.40	24.83	46.10
0.05	43.02	50.27	21.60	64.21	72.33	24.81	46.04
0.15	42.95	50.20	21.53	64.00	72.33	24.89	45.98
0.25	42.60	49.95	21.66	63.85	72.17	24.90	45.86
0.40	42.23	50.28	21.58	63.48	72.50	24.69	45.79

Table 6: Effect of the clip threshold

\lambda_{\text{clip}}

on LongBench-V1 category averages. Performance is stable in the small-

\lambda_{\text{clip}}

regime and degrades as

\lambda_{\text{clip}}

becomes larger, indicating that mild prior clipping is sufficient while stronger prior injection becomes over-regularized. Bold denotes the best result per column; the shaded row marks the chosen configuration.

$\alpha_{\text{cross}}$	$\alpha_{\text{soft}}$	S-QA	M-QA	Summ.	Few-shot	Synthetic	Code	Avg.
(a) Varying $\alpha_{\text{cross}}$ with $\alpha_{\text{soft}}=0.0$ fixed
0.00	0.0	43.21	50.02	21.80	64.39	72.35	25.22	46.52
0.15	0.0	42.98	50.55	21.62	64.44	72.48	24.90	46.55
0.25	0.0	43.44	50.63	21.66	64.26	72.33	24.82	46.61
0.35	0.0	43.45	50.29	21.84	64.44	72.33	25.06	46.70
0.50	0.0	43.19	50.11	21.62	64.29	72.19	24.82	46.45

Table 7: Effect of score-refinement hyperparameters on LongBench-V1. Moderate cross-head competition and moderate Soft-NMS produce the best overall result, supporting refinement that improves support diversity without suppressing useful tokens too aggressively. Bold denotes the best result per column; the shaded rows mark the chosen configuration.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,084 papers

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Long-Context Language Model Inference Speculative Decoding
University of Science and Technology of China
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
Sep 16, 2026Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2Long-Context Language Model Inference LLM Inference Scheduling
University of Southern California, Los Angeles, CA, USA
RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
Apr 16, 2026Zihong Zhang, Zuchao Li, Lefei Zhang +2Speculative Decoding LLM Inference Acceleration
School of Artiﬁcial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China +1

Jul 30, 2026J/K move · Enter open · S save

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

Yuesong Liu, Yuan Zeng, Min Lyu, Ruilin Liu+2

University of Science and Technology of China

Long-Context Language Model Inference Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.

Abstract

LLM Inference Acceleration

Explore similar work

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding