Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining

João Eduardo Batista; Emil Vatai; Mohamed Wahib

cs.CLNov 4, 2026

Accelerating Long-Context Decoding in Large-Scale Hybrid LLMs Without Retraining

Authors: João Eduardo Batista, Emil Vatai, Mohamed Wahib

Organizations: RIKEN Center for Computational Science, Kobe, Hyogo, Japan.

Abstract

The full-attention layers of large language models (LLMs) make every generated token more expensive than the previous one, slowing models down on long-context tasks. Hybrid models reduce this cost by replacing most full-attention layers with linear-attention layers, but the first still dominate decoding at long contexts. We accelerate these remaining layers without any training or weight changes using combined sparse attention: each layer attends to a sliding window of recent tokens, a pinned prefix, and a fixed budget of most relevant tokens in the context. The window size and budget are found on a per-layer basis using a multi-objective search algorithm. Across six hybrid models from three families, from 4B to 397B parameters, the selected configurations preserve their baseline model's accuracy on 21 long-context datasets. Decoding 64k/256k tokens becomes up to 1.26x/2.22x faster on Qwen3.5-4B, 1.20x/2.23x on Qwen3.5-122B-A10B, and 1.16x/2.06x on Ornith-1.0-397B.

Figures & tables

Figure 1: Time spent decoding one token by Qwen3.5-4B in each kernel (averaging 256 tokens), at different context lengths.

Method	Approach	Training required	Long context	Largest model reported
Pretraining – architecture decided before weights are released
Mamba, Mamba-3	SSM	full pretraining	synthetic only (1M), 2k on text	1.5–2.8B
Hyena	long convolution	full pretraining	synthetic recall (131k), 2k on text	1.3B
RWKV	linear recurrence	full pretraining	16k	14B
Qwen3.5, Gemma 4, gpt-oss, kimi-k3	hybrid full + linear/SWA	full pretraining	yes (1M)	2.8T
Post-training linearization – continued training on top of a released model

Table 1: Methods for reducing attention costs in LLMs. The first block covers pretraining methods, the second covers post-training, and the third is training-free. Our pipeline is training-free and tested in hybrid models across three model families (Qwen3.5, Gemma 4, gpt-oss). Green : no training, long-context capability, or more than 300B parameters; orange : post-training, limited long-context capability, or 10–300B parameters; red : full pretraining or over 1B token training, no long-context evaluation, or fewer than 10B parameters.

Figure 2: Overview of the pipeline used to obtain configurations using NSGA-II. (1) We fill an table with measured times with different window/budget values to be used by the surrogate cost function; (2) we generate the initial solutions; (3) the solutions are evaluated in the evaluation samples and using the cost function; (4) the best solutions are combined and mutated, and (5) when the search is over, we extract 3 configurations from the Pareto front.

Figure 3: Combined sparse attention: one static sink (pinned prefix, 256 tokens, as

4\times\hbox to6.43pt{\vbox to6.43pt{\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-3.21388pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {\lx@inpgf@ignorespaces}{{\lx@inpgf@ignorespaces}{}}{{}}{} {{\lx@inpgf@ignorespaces}{}}{{}}{}{}{}{}{{}}{}\lxSVG@begingroup@{_scopebegin=1} \color[rgb]{0.9219,0.7,0.2}\lxSVG@begingroup@{stroke=#A17000} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{}\lxSVG@fillstroke\lxSVG@drawpath@unclipped{M 0 -4.17 M 0 -3.62 L 0 3.62 C 0 3.92 0.25 4.17 0.55 4.17 L 7.79 4.17 C 8.09 4.17 8.34 3.92 8.34 3.62 L 8.34 -3.62 C 8.34 -3.92 8.09 -4.17 7.79 -4.17 L 0.55 -4.17 C 0.25 -4.17 0 -3.92 0 -3.62 Z M 8.34 4.17}{} \lx@inpgf@ignorespaces \lxSVG@closescope \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}

), sliding window (

w=6\times\hbox to6.43pt{\vbox to6.43pt{\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-3.21388pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {\lx@inpgf@ignorespaces}{{\lx@inpgf@ignorespaces}{}}{{}}{} {{\lx@inpgf@ignorespaces}{}}{{}}{}{}{}{}{{}}{}\lxSVG@begingroup@{_scopebegin=1} \color[rgb]{0.2,0.5562,0.7594}\lxSVG@begingroup@{stroke=#004F7D} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{}\lxSVG@fillstroke\lxSVG@drawpath@unclipped{M 0 -4.17 M 0 -3.62 L 0 3.62 C 0 3.92 0.25 4.17 0.55 4.17 L 7.79 4.17 C 8.09 4.17 8.34 3.92 8.34 3.62 L 8.34 -3.62 C 8.34 -3.92 8.09 -4.17 7.79 -4.17 L 0.55 -4.17 C 0.25 -4.17 0 -3.92 0 -3.62 Z M 8.34 4.17}{} \lx@inpgf@ignorespaces \lxSVG@closescope \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}

, and dynamic (Quest) sinks (

q=4\times\hbox to6.43pt{\vbox to6.43pt{\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-3.21388pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {\lx@inpgf@ignorespaces}{{\lx@inpgf@ignorespaces}{}}{{}}{} {{\lx@inpgf@ignorespaces}{}}{{}}{}{}{}{}{{}}{}\lxSVG@begingroup@{_scopebegin=1} \color[rgb]{0.2,0.6969,0.5594}\lxSVG@begingroup@{stroke=#006F50} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{}\lxSVG@fillstroke\lxSVG@drawpath@unclipped{M 0 -4.17 M 0 -3.62 L 0 3.62 C 0 3.92 0.25 4.17 0.55 4.17 L 7.79 4.17 C 8.09 4.17 8.34 3.92 8.34 3.62 L 8.34 -3.62 C 8.34 -3.92 8.09 -4.17 7.79 -4.17 L 0.55 -4.17 C 0.25 -4.17 0 -3.92 0 -3.62 Z M 8.34 4.17}{} \lx@inpgf@ignorespaces \lxSVG@closescope \lxSVG@closescope {\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}

), refreshed every

R

tokens (R=3).

Model	Thro.	NSGA-II	Accuracy
Qwen3.5-4B	10.5	8.6	97.2
Qwen3.5-27B	22.9	34.2	465.4
Gemma-4-31B	38.4	27.5	693.4
gpt-oss-120b	13.5	16.2	220.2
Total (A100)	85.3	86.5	1476.2
Qwen3.5-122B-A10B	52.3	31.3	116.2

Table 2: Total number of GPU-hours spent on the throughput (Thro.), NSGA-II, and accuracy runs.

		Parameters
Model	Type	Total	Active	Layers	Full attn.	Other layers	Context	Hardware
Qwen3.5-4B	dense	4B	4B	32	8	GDN	262k	1 $\times$ A100
Qwen3.5-27B	dense	27B	27B	64	16	GDN	262k	1 $\times$ A100
Qwen3.5-122B-A10B-FP8	MoE	122B	10B	48	12	GDN	262k	4 $\times$ GB200
Ornith-1.0-397B-FP8	MoE	397B	17B	60	15	GDN	262k	4 $\times$ GB200
Gemma-4-31B	dense	30.7B	30.7B	60	10	SWA-1024	262k	2 $\times$ A100

Table 3: Models studied in this work and hardware used. For multi-GPU runs, we use tensor parallelism of 2 and 4.

Panel of Figure 4 — Figure 4: Token/s throughput using different configurations and batch sizes. The baseline models are faster for short sequences due to the computation overhead from Quest. For longer sequences starting from 64k tokens, we see reasonable speedups in Qwen-4B, Qwen-122B, and Ornith. These speedups range 1.16–1.26x in context lengths of 64k tokens and 2.06–2.23x for 256k tokens

Model /	Generated tokens
Configuration	16k	64k	256k
Qwen3.5-4B
Baseline	472.3	382.4	217.6
Best Accuracy	0.99 $\times$ (466.8)	1.19 $\times$ (453.4)	1.87 $\times$ (407.3)
Best Speed	1.02 $\times$ (483.5)	1.26 $\times$ (483.5)	2.22 $\times$ (483.0)
R32	1.01 $\times$ (478.3)	1.25 $\times$ (478.2)	2.20 $\times$ (477.7)

Table 4: Decode speedup over the baseline at the same context length and batch size 4, followed by the throughput in brackets (tokens/s, all requests). For each model, we show the results of the baseline, Best Accuracy , the fastest between Balance and Best Speed configurations, and R32 (applying a uniform 1024-token budget to all layers, updating quest every 32 steps). The GPT-OSS run returned a single configuration. Ornith’s Best Speed and R32 configurations are the same. ‘—’ marks a point that was not measured either because the context is beyond the model’s limit (GPT-OSS) or the KV cache would not fit into memory at batch size 4 (Qwen-27B and Gemma runs).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Throughput of the baseline Qwen3.5-4B model, best-accuracy configuration, and two uniform configurations with SWA and Quest budgets or 2048 and 4096 tokens.

Figure 6: Speedup obtained in 4 polybench kernels (gemm, correlation, heat-3d and lu) as the conversation with each configuration moved towards 256k tokens. The lack of lines implies there was no correct code in that interval.

Figure 7: Average time spent in each kernel category when decoding 256 tokens, starting at a context length of 64k tokens.

Panel of Figure 8 — Figure 8: Tokens-per-second throughput using different configurations and batch sizes. We see that the baseline models are faster for short sequences due to the computation overhead from Quest. For longer sequences starting from 64k tokens, we see reasonable speedups in Qwen-4B, Qwen-122B, and Ornith. These speedups range 1.16–1.26x in context lengths of 64k tokens and 2.06–2.23x for 256k tokens

Qwen3.5-4B		3	7	11	15	19	23	27	31
Best Accuracy	$W$	256	4096	1024	4096	full	2048	256	1024
	$B$	512	256	512	2048	–	2048	2048	2048
Balanced	$W$	256	512	256	4096	256	256	256	256
	$B$	1024	256	512	2048	1024	512	512	2048
Best Speed	$W$	256	256	256	256	256	256	256	256
	$B$	512	512	512	512	512	512	512	512

Appendix

Table 6: Configurations evaluated for each model: the window

W

and Quest budget

B

(tokens) of every full-attention layer, indexed by its position in the model. ‘full’ marks a layer left at full attention, whose budget is irrelevant (‘–’). On gpt-oss-120b, the search front contained a single configuration; on Ornith, Best Speed is the Uniform 1024 configuration.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,171 papers

Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
Apr 8, 2026Quantong Qiu, Zhiyi Hong, Yi Yang +5Self-Attention Long-Context Language Model Inference
School of Computer Science and Technology, Soochow University · Baidu Inc, China
SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
Jun 22, 2026Huzama Ahmad, Se-Young YunSparse Attention Block-Sparse Attention
KAIST
Hybrid Latent Attention for Looped Language Models
Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model Inference LLM Inference Acceleration
Virginia Tech · Independent Researcher

Apr 8, 2026J/K move · Enter open · S save

Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference

Quantong Qiu, Zhiyi Hong, Yi Yang, Haitian Wang+4

School of Computer Science and Technology, Soochow University · Baidu Inc, China

Self-Attention Long-Context Language Model Inference LLM Inference Acceleration Sparse Attention Attention Mechanisms Hybrid Attention

Open paper AskPDF arXiv

The quadratic computational complexity of standard attention mechanisms presents a severe scalability bottleneck for LLMs in long-context scenarios. While hybrid attention mechanisms combining Full Attention (FA) and Sparse Attention (SA) offer a potential solution, existing methods typically rely on static allocation ratios that fail to accommodate the variable retrieval demands of different tasks. Furthermore, head-level dynamic sparsity often introduces severe computational load imbalance and synchronization long-tails, which hinder hardware acceleration during autoregressive decoding. To bridge this gap, we introduce Flux Attention, a context-aware framework that dynamically optimizes attention computation at the layer level. By integrating a lightweight Layer Router into frozen pretrained LLMs, the proposed method adaptively routes each layer to FA or SA based on the input context. This layer-wise routing preserves high-fidelity information retrieval while ensuring contiguous memory access, translating theoretical computational reductions into practical wall-clock speedups. As a parameter-efficient approach, our framework requires only 12 hours of training on 8

\times

A800 GPUs. Extensive experiments across multiple long-context and mathematical reasoning benchmarks demonstrate that Flux Attention achieves a superior trade-off between performance and inference speed compared with baseline models, with speed improvements of up to

2.8\times

and

2.0\times

in the prefill and decode stages.

Abstract

LLM Inference Acceleration

Explore similar work

Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference

SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers

Hybrid Latent Attention for Looped Language Models