Efficient attention mechanisms, such as linear and sliding window attention, aim to reduce the computational complexity of Transformers from quadratic to linear. However, their degraded in-context recall often necessitates interleaving full attention layers, leaving the quadratic bottleneck unresolved in long-context processing. To address this, we introduce Orthros attention, which shifts the efficiency paradigm from structural allocation to phase-aware decoupling. As a unified module, it dynamically switches its computation pattern across different phases of sequence processing, enabling linear complexity prefilling and full attention decoding. Our design grants Orthros attention a lower asymptotic complexity compared to vanilla Transformers in prevalent long-context scenarios characterized by heavy prefilling and light decoding. Beyond theoretical formulation and complexity analysis, extensive empirical evaluations demonstrate Orthros's superiority across multiple dimensions. First, we validate its fundamental language modeling capabilities, demonstrating scalability across varying model sizes and strong competitiveness in comprehensive architectural comparisons. Second, Orthros exhibits exceptional conversion feasibility via a data-efficient Transformer-to-Orthros conversion, yielding performance that even surpasses the original Transformer. Finally, Orthros achieves a 21.9× prefilling speedup over Transformers with end-to-end inference gains, while preserving long-context capabilities, as demonstrated by near-perfect 128K needle-in-a-haystack results and Transformer-level performance on real-world in-context recall tasks. Overall, these advantages in high performance, flexibility, and efficiency position Orthros as a promising candidate to succeed the Transformer in industrial applications.
Figures & tables
Figure 1: Overview of Orthros attention. The top two sub-figures describe the Orthros architecture and the bottom four sub-figures show the comparison of four sequence-processing regimes. SWA stands for sliding window attention, and CA stands for full cross-attention.
Figure 2: The ablation and scaling study of Orthros. The transformer serves as a baseline. Orthros w/o SO removes different scalers α and offsets β .
Model
ARC-C
ARC-E
HS
OBQA
PIQA
SIQA
Wino
Avg.
Norm
Norm
Norm
Transformer
37.80
70.58
56.21
37.80
72.91
41.10
57.46
53.41
Orthros
39.08
71.09
57.19
40.20
71.82
42.12
56.35
53.98
GDN
38.82
70.66
56.28
40.20
71.44
40.63
55.01
53.29
GLA
36.86
67.97
53.29
38.20
70.73
40.07
56.20
51.90
RetNet
35.84
69.82
54.33
37.00
71.93
39.76
55.01
51.96
Table 1: Common Sense Reasoning evaluation of different architectures at 1.3B scale. Higher is better for all metrics.
Base
Model
ARC-C
ARC-E
HS
OBQA
PIQA
SIQA
Wino
MMLU
Avg.
Norm
Norm
Norm
Llama3.1
Base
53.50
81.57
78.99
45.20
79.76
46.67
73.09
65.00
65.47
Orthros Conversion
57.08
84.13
79.10
45.60
80.25
48.93
74.35
68.65
67.26
Base CPT
57.51
83.80
78.93
45.40
80.25
47.80
73.40
68.89
67.00
Qwen3
Base
57.34
81.73
78.49
42.20
79.27
54.61
72.53
76.82
67.87
Orthros Conversion
59.39
84.81
78.06
44.80
79.27
50.51
72.38
77.76
68.37
Table 2: Evaluation results of converting Llama3.1 and Qwen3 to Orthros. Higher is better for all metrics.
Model
DP
NQ
SW
TQ
FD
SQ
Avg.
Trans.
23.2
22.4
36.7
62.3
16.4
42.6
33.9
Orthros
23.3
20.8
39.9
59.8
24.1
38.7
34.4
GDN
22.8
20.6
21.8
58.2
12.0
35.1
28.4
GLA
21.4
16.7
21.4
58.6
10.7
33.6
27.1
RetNet
21.0
15.8
17.6
57.7
8.3
33.7
25.7
SWA
22.8
14.1
17.3
61.0
16.3
41.1
28.8
Table 3: In-context recall-intensive task evaluation at 1.3B scale. Higher is better for all metrics.
Figure 3: Orthros 128K Needle in a Haystack task result. Orthros achieves near-perfect accuracy.
Figure 4: The left figure shows the time cost of Orthros and Qwen3 with a single GPU for prefilling, while the right figure presents 8-GPU prefilling time.
128K input / 128 output
127K input / 1K output
120K input / 8K output
Model
TTFT (s)
E2E (s)
Output (tok/s)
Decode-only (tok/s)
TTFT (s)
E2E (s)
Output (tok/s)
Decode-only (tok/s)
TTFT (s)
E2E (s)
Output (tok/s)
Decode-only (tok/s)
Transformer
77.249
84.911
3.0
33.7
76.160
137.715
14.9
33.2
67.770
548.357
29.9
34.1
Orthros
17.016
24.647
10.4
33.3
17.301
76.345
26.8
33.7
15.210
488.830
33.5
34.6
Linear Hybrid
25.566
28.130
9.1
98.8
25.173
46.308
44.2
96.8
22.747
189.302
86.5
98.3
Orthros-H
10.184
12.771
20.0
97.3
9.882
30.724
66.7
105.8
9.302
173.887
94.2
99.5
Table 4: End-to-end inference on a single H100 GPU using SGLang. Time to first token (TTFT) and end-to-end latency (E2E) are measured in seconds; output and decode-only throughput are measured in tokens per second. Output throughput is the total number of generated tokens divided by E2E latency. Lower latency and higher throughput are better.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Item
160M
360M
780M
1.3B
2.7B
6.8B
Model
Architecture
Transformer / Orthros / Orthros w/o SO
Hidden size
768
1024
1536
2048
2560
4096
Layers
16
24
24
24
32
32
Heads
6
8
12
16
20
32
Head dimension
128
Appendix
Table 5: Model configurations and training settings for the scaling and ablation study. All models are trained with a sequence length of 2048 for approximately 10B tokens. Orthros uses a sliding window size of 128.
Item
Llama3.1-8B
Qwen3-8B
Base
Orthros Conv.
Base CPT
Base
Orthros Conv.
Base CPT
Model
Base checkpoint
Llama3.1-8B
Qwen3-8B-Base
Architecture
Transformer
Orthros
Transformer
Transformer
Orthros
Transformer
Hidden size
4096
4096
Layers
32
36
Appendix
Table 6: Model configurations and continual training settings for Transformer-to-Orthros conversion. Base denotes the original pretrained checkpoint without additional training. Orthros Conversion replaces the original full attention modules with Orthros Attention. Base CPT continues training the original Transformer architecture under the same training setting.
Param
Model
ARC-C/Norm
ARC-E/Norm
HS/Norm
OBQA/Norm
PIQA/Norm
Wino
Avg.
160M
Orthros w/o SO
25.86
47.64
33.14
32.20
62.62
50.28
41.96
Orthros w/ SO
26.19
46.68
34.26
30.80
63.33
51.22
42.08
Transformer
25.77
46.25
33.64
30.60
62.08
52.64
41.83
360M
Orthros w/o SO
28.58
51.68
38.01
32.20
64.74
51.46
44.45
Orthros w/ SO
27.90
50.25
38.70
32.80
65.23
51.78
44.44
Transformer
28.41
50.00
38.52
33.80
65.94
50.83
44.58
Appendix
Table 7: Ablation study across different model scales. We report normalized accuracy for ARC-Challenge, ARC-Easy, HellaSwag, OpenBookQA, and PIQA, and accuracy for WinoGrande.
The quadratic complexity of self-attention during the prefill phase impedes long-context inference in large language models. Existing sparse attention methods face a trade-off among context adaptivity, sampling overhead, and fine-tuning costs. We propose VSPrefill, a mechanism requiring lightweight training that uses the vertical-slash structural pattern in attention distributions. Our compact VSIndexer module predicts context-aware importance scores for vertical columns and slash diagonals from key-value representations augmented with RoPE. This approach constructs sparse masks with linear complexity without modifying the backbone parameters. During inference, an adaptive cumulative-threshold strategy allocates sparsity budgets per layer, while a fused kernel executes attention with on-the-fly index merging. Evaluated on Qwen3-4B-Instruct and LLaMA-3.1-8B-Instruct across the LongBench and RULER benchmarks, VSPrefill preserves 98.35% of the full attention accuracy while delivering a 4.95x average speedup at a context length of 128k. These results establish a new Pareto frontier in the trade-off between accuracy and efficiency.
Chen Guanzhong
School of Computer Science and Technology, Tongji University
The quadratic computational complexity of standard attention mechanisms presents a severe scalability bottleneck for LLMs in long-context scenarios. While hybrid attention mechanisms combining Full Attention (FA) and Sparse Attention (SA) offer a potential solution, existing methods typically rely on static allocation ratios that fail to accommodate the variable retrieval demands of different tasks. Furthermore, head-level dynamic sparsity often introduces severe computational load imbalance and synchronization long-tails, which hinder hardware acceleration during autoregressive decoding. To bridge this gap, we introduce Flux Attention, a context-aware framework that dynamically optimizes attention computation at the layer level. By integrating a lightweight Layer Router into frozen pretrained LLMs, the proposed method adaptively routes each layer to FA or SA based on the input context. This layer-wise routing preserves high-fidelity information retrieval while ensuring contiguous memory access, translating theoretical computational reductions into practical wall-clock speedups. As a parameter-efficient approach, our framework requires only 12 hours of training on 8×A800 GPUs. Extensive experiments across multiple long-context and mathematical reasoning benchmarks demonstrate that Flux Attention achieves a superior trade-off between performance and inference speed compared with baseline models, with speed improvements of up to 2.8× and 2.0× in the prefill and decode stages.
Quantong Qiu, Zhiyi Hong, Yi Yang +5
School of Computer Science and Technology, Soochow University · Baidu Inc, China
Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-intensive prefilling phase. While various sparse attention mechanisms have been explored, they typically suffer from either significant search latency or insufficient sparsity. In this paper, we propose FlashPrefill, a framework enabling ultra-fast prefilling via instantaneous pattern discovery and thresholding. FlashPrefill leverages a fast block-searching technique to simultaneously locate dynamic vertical, slash, and block-sparse attention patterns. Crucially, it introduces a dynamic thresholding mechanism that bypasses the prohibitive overhead of sorting or accumulating attention scores while effectively eliminating the long-tail distribution to enhance sparsity. Extensive evaluations demonstrate that FlashPrefill achieves a substantial leap in efficiency, delivering an unprecedented 27.78x speedup on 256K sequences. Notably, unlike existing methods that incur efficiency degradation on shorter contexts, FlashPrefill maintains a 1.71x speedup even at a 4K context length, demonstrating its robustness and practical utility across varying sequence scales.