Post-training hybridization of large language models (LLMs) often replaces quadratic self-attention with sliding-window attention (SWA) to reduce KV cache usage and improve latency. Existing hybridization schemes are typically defined either at the layer level (e.g., interleaving) or at the head level via static rankings from local to global. Layer-level schemes ignore that local and global dependencies are routed through heads within the same layer, while static head-level rankings suffer from entanglement: a head's local/global behavior can change after hybridization. We propose BOSCH, Black-box Binary Optimization for Short-context Head Selection, a training-free method that formulates the problem as a Large Neighborhood Search and decomposes it into three subproblems: (i) layer-importance detection via small-budget black-box probes, (ii) adaptive per-layer SWA-ratio assignment based on these sensitivities, and (iii) grouped head-level optimization within ratio buckets. Extensive experiments on 4 LLMs ranging from 1.7B to 30B parameters, across 4 SWA ratios, show that BOSCH consistently outperforms layer-level heuristics and 6 strong static head-level methods, with larger gains at higher SWA ratios. Under continual pretraining, BOSCH recover original long-context performance faster and to a higher level. Analysis of the selected heads reveals substantial turnover for BOSCH across different SWA ratios, underscoring the importance of performing head-level selection for each target ratio rather than relying on fixed locality rankings.
Figures & tables
Figure 1: Illustrative application of BOSCH to a Transformer with L=8 layers and H=8 heads per layer, targeting ρ=0.5% SWA heads. Left (Stage 1: layer-importance detection): each row is a layer; blue squares are self-attention heads ( z=1 ), red squares are SWA heads ( z=0 ); light-red squares with a loupe indicate the layer(s) currently scored via black-box optimization, from top to bottom. Middle (Stage 2: adaptive ratio assignment): for each layer ℓ we compute the relative drop δℓ (from the original model), normalize it to an importance weight wℓ∈[0,1] , and map it to a per-layer SWA ratio rℓ while respecting the global budget. Right (Stage 3: multi-layer head selection under adaptive ratios): layers are bucketed by ratio into groups gℓ and optimized jointly from more SWA-tolerant to less tolerant; the example highlights searching layers with gℓ=2 using rℓ=0.25 , yielding the final head mask z . " − " indicates that either entire layer is full-attention or SWA ( no optimization is applied).
Method
ρ=0.25
ρ=0.5
ρ=0.75
ρ=0.875
1.7B
8B
14B
30B
1.7B
8B
14B
30B
1.7B
8B
14B
30B
1.7B
8B
14B
30B
Original*
92.7
99.1
99.6
99.4
92.7
99.1
99.6
99.4
92.7
99.1
99.6
99.4
92.7
99.1
99.6
99.4
Layer-level Heuristics
RAND
66.9
45.9
79.0
79.6
38.0
15.4
39.2
16.0
12.7
12.8
17.1
15.0
12.6
13.2
13.0
10.6
BME
40.0
30.8
41.3
47.5
11.5
12.4
11.8
18.9
11.1
12.2
11.9
12.0
11.2
12.7
12.2
10.5
INTR
74.9
72.9
41.1
69.1
19.1
19.0
14.1
12.8
12.8
12.7
11.3
22.8
12.5
12.6
11.4
11.3
Table 1: Zero-shot average scores on the NIAH benchmark for SWA hybridization methods across 4 Qwen3 models (1.7B-30B) under 4 SWA ρ ratios. The highest and second-highest scores for each model under each ratio are highlighted in bold and underline, respectively. *For readability purposes, we repeat the original model’s performance in each column (SWA hybridization is not applied to the original model).
Method
ρ=0.25
ρ=0.5
ρ=0.75
ρ=0.875
1.7B
8B
14B
30B
1.7B
8B
14B
30B
1.7B
8B
14B
30B
1.7B
8B
14B
30B
Original*
45.4
57.1
58.4
54.9
45.4
57.1
58.4
54.9
45.4
57.1
58.4
54.9
45.4
57.1
58.4
54.9
Layer-level Heuristics
RAND
33.7
40.2
48.2
40.2
28.5
31.5
37.6
18.3
23.3
24.1
32.8
15.6
23.1
24.3
31.4
13.8
BME
31.5
31.0
32.6
28.2
21.1
24.4
15.9
16.7
20.7
28.9
29.1
12.3
20.9
26.5
30.2
14.0
INTR
36.0
43.3
43.4
42.3
28.2
34.2
25.0
25.2
19.8
29.1
30.9
14.9
19.7
23.5
30.4
13.1
Table 2: Zero-shot average scores on the LongBench benchmark for SWA hybridization methods across 4 Qwen3 models under 4 SWA ρ ratios. Notations follow those used in Table 1 .
Figure 2: Jaccard distance between SWA heads selected by seven methods for ρ=0.25 (upper plot) and ρ=0.5 (lower plot). The distances for 8B and 14B models are shown in the lower and upper triangles, respectively.
Figure 3: NIAH (first 2 plots) and LongBench (last 2 plots) performances (y-axis) for Qwen3-8B-Base and Qwen3-14B-Base models as a function of intermediate checkpoints during continual pretraining on 2.5B tokens (x-axis). We report scores for two layer-level and two head-level SWA hybridization methods using ρ=0.75 . The performance of the full-attention original models (without additional training) is shown as a straight line with no markers.
Model
0.25→0.5
0.5→0.75
0.75→0.875
T
T~
T
T~
T
T~
1.7B
30%
30%
15%
30%
6%
36%
8B
26%
26%
15%
29%
7%
44%
14B
28%
28%
16%
31%
5%
31%
30B
29%
29%
16%
31%
6%
34%
Table 3: Raw turnover T and geometrically normalized turnover T~ between BOSCH local-head sets at 2 adjacent ratios ρ . All values are reported as percentages.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Model
#P
#L
#QH
#KVH
ML
Q3-1.7B
1.7
28
16
8
32k
Q3-8B
8.2
36
32
8
32k
Q3-14B
14.8
40
40
8
32k
Q3-30B-A3B
30.5
48
32
4
40k
Appendix
Table 4: Qwen3-Base backbones used in our experiments. We report parameter count (#P, in billions), number of layers (#L), number of query attention heads (#QH), number of key/value heads (#KVH), and maximum length (ML, tokens).
Model
NIAH
LongBench
0.25→0.5
0.5→0.75
0.75→0.875
0.25→0.5
0.5→0.75
0.75→0.875
Aρl
Aρl′
Aρl
Aρl′
Aρl
Aρl′
Aρl
Aρl′
Aρl
Aρl′
Aρl
Aρl′
1.7B
78.3
57.1
58.0
47.1
30.3
19.7
32.1
27.4
26.0
23.0
23.6
17.0
8B
90.3
79.5
72.7
68.7
42.5
20.7
41.8
34.3
31.6
29.2
28.6
24.3
14B
94.0
87.9
83.6
73.9
47.2
41.1
47.0
40.4
38.0
34.0
36.1
29.9
30B
86.3
67.2
50.2
36.8
26.9
15.7
36.0
28.4
24.8
22.3
19.3
14.7
Appendix
Table 5: Zero-shot average scores on the NIAH and LongBench benchmarks for BOSCH across 4 Qwen3 models under 3 SWA ratios ρl∈{0.5,0.75,0.875} . The left side of → corresponds to ρs , while the right side corresponds to ρl . Aρl′ indicates results obtained after replacing a randomly selected subset of heads at a given configuration (e.g., ρl=0.75 ) with heads that did not appear in a smaller configuration (e.g., ρs=0.5 ). Results in the Aρl′ columns are averaged over 3 runs with different seeds. The Aρl column shows the results for the originally selected SWA heads set.
Method
single
multikey
Avg.
1.7B
8B
14B
30B
1.7B
8B
14B
30B
1.7B
8B
14B
30B
64k
Original
47.9
49.3
49.1
49.3
23.5
44.7
45.9
45.7
35.7
47.0
47.5
47.5
INTR
2.6
2.9
2.6
17.3
1.7
1.8
1.3
2.5
2.2
2.4
2.0
9.9
B-layer
23.7
18.1
12.1
19.5
8.2
1.6
2.7
3.8
16.0
9.9
7.4
11.7
Fisher
27.7
37.0
37.0
23.9
7.5
8.7
13.3
5.7
17.6
22.9
25.2
14.8
Appendix
Table 6: Detailed zero-shot performance on the NIAH benchmark at 64k and 128k for the original models and four SWA hybridization methods (with SWA ratio ρ=0.75 ) applied to four Qwen3 models (1.7B, 8B, 14B, 30B). Zero-shot length extrapolation is performed using a YaRN factor of 2 and 4 for 64k and 128k, respectively. The Avg. block reports, for each model, the mean of its single and multikey scores. The highest score under each configuration is highlighted in bold.
Figure 4: NIAH (first 2 plots) and LongBench (last 2 plots) zero-shot average performances (y-axis) for Qwen3-8B-Base and Qwen3-14B-Base models as a function of varying the SWA window size (x-axis). We report scores for two layer-level and two head-level SWA hybridization methods using ρ=0.75 . The performance of the full-attention original models (without additional training) is shown as a straight line with no markers.
Figure 5: Latency and memory statistics comparing the original Qwen3-8B-Base model with SWA hybrid variants at different SWA ρ ratios using BOSCH . The left plot shows prefill throughput (tokens/s), the middle plot shows decoding throughput in the same units, and the right plot shows p90 memory allocation (GB). These statistics are measured with input prompts of lengths ranging from 64 to 1M tokens (x-axis). Missing data points correspond to runs where the model encountered out-of-memory (OOM) error.
Figure 6: Latency and memory statistics of BOSCH ρ=0.75 Qwen3-8B-Base SWA hybrid models when varying the SWA window size between 256 and 4096. Notation is the same as in Figure 5 .
Method
single
multikey
Avg.
4k
8k
16k
32k
4k
8k
16k
32k
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
Original
100.0
99.9
100.0
100.0
98.9
94.9
84.1
63.9
92.7
Layer-level Heuristics
RAND
95.2
71.9
32.4
32.0
87.5
58.8
15.7
15.6
76.5
22.3
6.5
7.5
65.4
21.4
2.9
2.8
71.3
51.7
19.9
17.8
47.1
23.8
11.1
10.9
32.9
14.5
4.6
4.5
23.0
7.6
1.5
1.5
66.9
38.0
12.7
12.6
BME
85.9
32.2
32.2
32.2
69.7
15.6
15.5
15.5
60.0
7.5
7.5
7.5
43.9
2.7
2.8
2.8
33.7
15.9
19.5
19.9
16.1
9.6
10.4
10.5
11.1
4.1
3.7
4.3
5.3
1.3
1.1
1.3
40.0
11.5
11.1
11.2
Appendix
Table 7: Detailed zero-shot scores on the NIAH benchmark for SWA hybridization methods for Qwen3-1.7B-Base under 4 SWA ρ ratios. The highest score under each configuration is highlighted in bold.
Method
Single-document QA
Multi-document QA
Summarization
Few-shot learning
Synthetic Tasks
Code Completion
Math Reasoning
Avg.
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
Original
39.8
30.9
26.0
67.4
32.7
48.2
72.6
45.4
Layer-level Heuristics
RAND
27.0
17.6
13.1
14.9
15.7
10.0
9.4
11.1
23.3
19.1
16.3
16.4
56.1
46.3
36.6
36.4
4.2
3.6
2.6
3.2
49.8
48.5
47.2
44.9
60.0
54.4
38.1
34.5
33.7
28.5
23.3
23.1
BME
23.1
12.7
12.9
14.1
13.8
11.3
11.1
10.2
21.1
14.8
15.7
15.9
58.1
36.8
37.8
37.0
3.2
2.9
2.8
2.5
50.0
32.3
34.6
36.7
51.0
36.8
29.9
30.3
31.5
21.1
20.7
20.9
INTR
29.8
19.3
13.6
13.2
13.1
10.8
11.8
12.4
24.4
19.4
15.6
15.5
59.7
44.7
39.0
36.6
7.8
4.1
3.6
3.0
53.3
46.2
26.8
22.5
63.9
53.3
28.4
35.0
36.0
28.2
19.8
19.7
Appendix
Table 8: Detailed zero-shot scores on the LongBench benchmark for SWA hybridization methods for Qwen3-1.7B-Base under 4 SWA ρ ratios. The highest score under each configuration is highlighted in bold.
Method
single
multikey
Avg.
4k
8k
16k
32k
4k
8k
16k
32k
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
Original
100.0
100.0
100.0
99.9
99.5
99.6
99.1
94.5
99.1
Layer-level Heuristics
RAND
68.5
37.0
32.0
32.2
60.4
17.6
15.8
15.6
53.6
8.1
7.5
7.5
49.9
3.3
2.8
2.8
58.8
34.6
26.5
27.6
38.5
14.4
11.7
12.8
22.7
6.0
4.5
5.3
15.0
2.1
1.6
1.7
45.9
15.4
12.8
13.2
BME
68.9
32.3
32.1
32.3
51.3
15.8
15.6
15.6
39.7
7.9
7.4
7.5
25.5
3.7
2.8
2.8
36.6
22.3
22.8
24.7
14.5
11.0
11.5
12.2
7.3
4.6
4.2
4.7
2.5
1.5
1.3
1.7
30.8
12.4
12.2
12.7
Appendix
Table 9: Detailed zero-shot scores on the NIAH benchmark for SWA hybridization methods for Qwen3-8B-Base under 4 SWA ρ ratios. The highest score under each configuration is highlighted in bold.
Method
Single-document QA
Multi-document QA
Summarization
Few-shot learning
Synthetic Tasks
Code Completion
Math Reasoning
Avg.
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
Original
45.5
41.9
26.8
70.3
63.0
66.6
85.8
57.1
Layer-level Heuristics
RAND
25.8
14.6
7.5
9.8
20.4
11.6
5.3
7.4
24.5
18.7
16.6
16.5
62.9
50.0
38.1
36.5
12.8
3.8
3.5
3.0
52.5
43.2
32.9
33.1
82.3
78.8
65.2
64.0
40.2
31.5
24.1
24.3
BME
19.3
10.8
14.4
10.3
15.4
9.8
14.8
12.0
21.0
19.2
20.6
18.8
50.7
42.6
41.4
38.2
4.0
3.8
2.9
2.7
29.9
26.3
45.0
42.0
76.3
58.2
62.9
61.3
31.0
24.4
28.9
26.5
INTR
29.1
18.9
18.1
11.9
26.1
12.5
15.1
8.2
25.4
22.5
20.0
15.6
63.9
48.7
40.7
36.3
12.1
4.1
2.4
3.9
64.2
53.7
44.1
26.4
82.4
78.8
63.4
62.2
43.3
34.2
29.1
23.5
Appendix
Table 10: Detailed zero-shot scores on the LongBench benchmark for SWA hybridization methods for Qwen3-8B-Base under 4 SWA ρ ratios. The highest score under each configuration is highlighted in bold.
Method
single
multikey
Avg.
4k
8k
16k
32k
4k
8k
16k
32k
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
Original
100.0
99.9
100.0
100.0
99.8
99.7
99.3
98.2
99.6
Layer-level Heuristics
RAND
99.9
86.4
42.1
32.3
97.3
59.1
22.8
15.6
92.9
47.8
11.7
7.5
82.9
31.7
8.8
2.8
96.2
54.0
30.7
26.9
74.6
22.4
13.5
12.5
53.0
9.4
5.5
4.9
35.6
3.0
1.6
1.6
79.0
39.2
17.1
13.0
BME
98.5
31.3
32.3
32.3
82.5
15.5
15.6
15.6
43.7
7.1
7.4
7.5
34.1
2.7
2.8
2.8
41.3
22.1
20.6
21.9
20.8
11.1
11.0
11.4
7.1
3.7
4.2
4.6
2.1
1.3
1.6
1.5
41.3
11.8
11.9
12.2
Appendix
Table 11: Detailed zero-shot scores on the NIAH benchmark for SWA hybridization methods for Qwen3-14B-Base under 4 SWA ρ ratios. The highest score under each configuration is highlighted in bold.
Method
Single-document QA
Multi-document QA
Summarization
Few-shot learning
Synthetic Tasks
Code Completion
Math Reasoning
Avg.
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
Original
43.7
43.5
27.0
72.1
66.7
66.6
89.2
58.4
Layer-level Heuristics
RAND
32.1
21.9
17.6
15.8
26.7
17.1
15.2
14.7
26.3
24.2
22.4
21.6
68.0
58.6
52.0
49.8
42.5
9.8
4.0
3.7
58.2
56.4
55.0
57.0
84.0
75.7
63.8
57.0
48.2
37.6
32.8
31.4
BME
25.1
5.3
15.8
14.6
23.2
3.5
10.8
11.7
18.8
11.6
21.4
21.7
47.9
18.5
46.4
50.4
3.5
2.3
3.7
3.8
41.2
25.4
50.0
53.9
68.9
45.0
55.2
55.3
32.6
15.9
29.1
30.2
INTR
20.3
5.3
16.6
18.4
19.0
4.6
14.0
14.1
25.4
11.1
21.7
21.5
65.4
35.3
53.8
53.3
27.8
2.0
2.8
3.0
61.2
48.5
57.8
53.1
84.6
68.0
49.7
49.4
43.4
25.0
30.9
30.4
Appendix
Table 12: Detailed zero-shot scores on the LongBench benchmark for SWA hybridization methods for Qwen3-14B-Base under 4 SWA ρ ratios. The highest score under each configuration is highlighted in bold.
Method
single
multikey
Avg.
4k
8k
16k
32k
4k
8k
16k
32k
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
Original
100.0
100.0
99.9
100.0
99.7
99.6
99.1
96.7
99.4
Layer-level Heuristics
RAND
100.0
31.0
30.8
30.6
99.8
22.2
21.9
18.0
99.9
20.0
19.8
10.8
97.3
9.7
9.8
2.9
87.7
22.0
19.9
19.5
61.9
8.6
7.9
7.9
51.6
4.2
4.3
3.7
35.5
1.3
1.4
1.2
79.6
16.0
15.0
10.6
BME
80.1
45.7
32.3
29.7
87.9
43.3
15.3
12.8
88.1
37.9
6.4
6.2
45.2
23.2
2.8
2.8
38.5
17.9
23.8
23.5
18.3
11.6
10.2
9.9
15.9
6.7
3.6
3.9
6.2
2.3
1.3
1.5
47.5
18.9
12.0
10.5
Appendix
Table 13: Detailed zero-shot scores on the NIAH benchmark for SWA hybridization methods for Qwen3-30B-A3B-Base under 4 SWA ρ ratios. The highest score under each configuration is highlighted in bold.
Method
Single-document QA
Multi-document QA
Summarization
Few-shot learning
Synthetic Tasks
Code Completion
Math Reasoning
Avg.
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
0.25
0.5
0.75
0.875
Original
24.6
39.8
27.6
70.0
64.3
67.8
90.1
54.9
Layer-level Heuristics
RAND
14.3
4.6
4.0
3.2
18.6
3.9
3.9
3.6
26.2
11.4
8.6
7.0
67.7
28.1
19.5
17.8
26.6
3.1
2.7
3.3
58.8
31.3
25.8
25.3
69.4
45.6
44.7
36.4
40.2
18.3
15.6
13.8
BME
9.5
4.2
4.2
3.7
5.9
3.9
3.8
3.7
21.9
10.1
7.3
8.4
55.1
28.9
25.0
20.9
3.1
2.7
3.2
3.0
32.0
23.6
17.4
30.6
70.1
43.1
25.5
27.3
28.2
16.7
12.3
14.0
INTR
12.6
5.2
4.7
3.3
17.5
3.9
4.3
4.0
25.7
15.7
9.8
7.1
68.7
36.0
24.9
24.4
23.9
2.8
3.0
3.2
65.5
52.7
34.0
26.6
82.5
60.2
24.0
23.0
42.3
25.2
14.9
13.1
Appendix
Table 14: Detailed zero-shot scores on the LongBench benchmark for SWA hybridization methods for Qwen3-30B-A3B-Base under 4 SWA ρ ratios. The highest score under each configuration is highlighted in bold.
Method
single
multikey
Avg.
4k
8k
16k
32k
4k
8k
16k
32k
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
Qwen3-1.7B-Base
Original
100.0
99.9
100.0
100.0
98.9
94.9
84.1
63.9
92.7
INTR
95.9
96.4
96.5
95.7
91.6
93.4
95.1
93.6
72.6
75.5
78.7
76.7
52.2
57.3
59.4
56.4
80.3
87.3
88.4
88.9
64.1
74.9
78.9
79.1
36.4
49.5
56.9
57.3
17.9
24.7
30.1
31.7
63.9
69.9
73.0
72.4
B-layer
99.7
99.0
98.7
98.4
99.7
96.1
94.1
94.5
99.4
93.1
89.9
89.8
98.9
90.8
84.2
85.5
98.2
99.2
99.2
99.3
89.5
93.1
93.0
93.5
79.4
85.9
85.5
86.6
70.2
72.5
72.9
72.6
91.9
91.2
89.7
90.0
Appendix
Table 15: Detailed continual pretraining performances on the NIAH benchmark for 4 SWA hybridization methods for 4 Qwen3 models (1.7B, 8B, 14B, 30B) under SWA ratio ρ=0.75 . The highest score under each configuration is highlighted in bold.
Method
Single-document QA
Multi-document QA
Summarization
Few-shot learning
Synthetic Tasks
Code Completion
Math Reasoning
Avg.
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
0.7B
1.3B
2.0B
2.5B
Qwen3-1.7B-Base
Original
39.8
30.9
26.0
67.4
32.7
48.2
72.6
45.4
INTR
25.9
28.1
28.4
29.4
22.6
23.8
25.3
25.2
20.6
21.1
21.3
21.5
55.6
55.8
57.0
57.9
3.8
4.7
4.7
4.7
50.8
47.3
46.8
45.8
58.7
59.9
60.6
61.1
34.0
34.4
34.9
35.1
B-layer
35.4
36.5
37.3
37.2
25.4
27.4
29.4
30.3
21.6
22.4
22.8
22.6
60.8
62.4
62.6
63.0
6.0
10.0
13.0
13.2
48.2
46.9
46.1
44.2
57.8
60.3
59.8
61.1
36.9
38.4
38.7
38.8
Fisher
35.3
36.8
37.9
38.1
24.6
27.1
28.8
28.3
21.3
22.7
23.5
23.5
62.9
65.2
65.4
65.4
7.2
8.3
8.8
9.2
44.3
42.4
43.7
44.2
59.5
61.3
60.4
60.7
36.4
37.7
38.4
38.5
Appendix
Table 16: Detailed continual pretraining performances on the LongBench benchmark for 4 SWA hybridization methods for 4 Qwen3 models (1.7B, 8B, 14B, 30B) under SWA ratio ρ=0.75 . The highest score under each configuration is highlighted in bold.
Method
single
multikey
Avg.
4k
8k
16k
32k
4k
8k
16k
32k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
Qwen3-1.7B-Base
Original
100.0
99.9
100.0
100.0
98.9
94.9
84.1
63.9
92.7
INTR
5.1
13.8
55.6
100.0
3.2
7.3
30.9
55.5
2.4
2.7
14.2
28.9
2.1
2.3
6.8
13.1
4.1
9.5
40.7
98.9
2.6
6.1
16.3
27.6
1.6
2.3
7.9
13.5
0.9
1.2
3.9
5.5
2.8
5.7
22.0
42.9
B-layer
82.3
83.9
89.3
100.0
80.1
81.1
85.7
89.2
73.2
76.2
81.8
86.5
64.2
59.6
59.3
66.7
27.1
27.7
57.7
98.9
18.1
22.2
26.0
36.3
20.3
18.9
20.8
25.3
14.2
13.4
15.8
17.8
47.4
47.9
54.5
65.1
Appendix
Table 17: Detailed zero-shot performances on the NIAH benchmark for 4 SWA hybridization methods for 4 Qwen3 models (1.7B, 8B, 14B, 30B) under SWA ratio ρ=0.75 when SWA windows size is {256,512,2048,4096} . The highest score under each configuration is highlighted in bold.
Method
Single-document QA
Multi-document QA
Summarization
Few-shot learning
Synthetic Tasks
Code Completion
Math Reasoning
Avg.
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
256
512
2k
4k
Qwen3-1.7B-Base
Original
39.8
30.9
26.0
67.4
32.7
48.2
72.6
45.4
INTR
13.2
13.9
16.8
25.1
11.4
11.9
12.6
13.3
11.8
14.6
18.3
20.0
19.2
27.9
47.1
55.8
3.1
3.5
3.8
4.9
24.6
23.9
30.4
39.0
3.9
4.6
71.0
71.0
12.5
14.3
28.6
32.7
B-layer
13.3
15.8
21.6
27.9
8.9
10.9
13.7
16.0
14.0
18.0
21.7
23.1
29.2
34.6
49.5
55.3
3.0
3.7
3.6
5.8
31.2
32.2
37.2
43.6
15.2
19.6
71.0
71.0
16.4
19.3
31.2
34.7
Fisher
10.9
12.8
19.5
26.7
8.9
9.5
12.6
15.3
10.0
12.2
17.5
19.9
32.4
40.8
50.9
55.0
1.8
2.6
3.0
5.2
18.5
17.8
26.3
34.8
6.0
7.5
71.3
71.3
12.6
14.7
28.7
32.6
Appendix
Table 18: Detailed zero-shot performances on the LongBench benchmark for 4 SWA hybridization methods for 4 Qwen3 models (1.7B, 8B, 14B, 30B) under SWA ratio ρ=0.75 when SWA windows size is {256,512,2048,4096} . The highest score under each configuration is highlighted in bold.
Method
single
multikey
Avg.
4k
8k
16k
32k
4k
8k
16k
32k
Qwen3-1.7B-Base
0.25→0.5
81.6
80.4
71.0
70.2
51.9
41.5
31.3
29.1
57.1
0.5→0.75
69.8
61.0
52.9
45.3
53.6
44.5
29.7
20.0
47.1
0.75→0.875
44.8
31.5
13.1
10.7
22.8
18.7
8.0
8.0
19.7
Qwen3-8B-Base
Appendix
Table 19: Zero-shot scores on the NIAH benchmark for BOSCH when using Aρl′ across 4 Qwen3 models under 3 SWA ratios ρ∈{0.5,0.75,0.875} . The left side of → corresponds to ρs , while the right side corresponds to ρl . Aρl′ indicates results obtained after replacing a randomly selected subset of heads at a given configuration (e.g., ρl=0.75 ) with heads that did not appear in a smaller configuration (e.g., ρs=0.5 ). Scores are averaged over three runs with different seeds.
Method
Tasks
Avg.
Single-document QA
Multi-document QA
Summarization
Few-shot learning
Synthetic Tasks
Code Completion
Math Reasoning
Qwen3-1.7B-Base
0.25→0.5
15.0
8.9
18.3
48.3
3.8
47.4
49.8
27.4
0.5→0.75
11.6
9.1
17.1
43.4
2.9
38.4
38.8
23.0
0.75→0.875
4.8
3.2
12.9
39.9
2.2
24.0
31.6
17.0
Qwen3-8B-Base
Appendix
Table 20: Zero-shot scores on the LongBench benchmark for BOSCH when using Aρl′ across 4 Qwen3 models under 3 SWA ratios ρ∈{0.5,0.75,0.875} (right side of → ). Aρl′ indicates results obtained after replacing a randomly selected subset of heads at a given configuration (e.g., ρl=0.75 ) with heads that did not appear in a smaller configuration (e.g., ρs=0.5 ). Scores are averaged over three runs with different seeds.
Figure 7: Jaccard distance between SWA heads selected by seven methods for ρ∈{0.25,0.5,0.75,0.875} . The distances for Qwen3-8B-Base and Qwen3-14B-Base models are shown in the lower and upper triangles, respectively.
Figure 8: Jaccard distance between SWA heads selected by seven methods for ρ∈{0.25,0.5,0.75,0.875} . The distances for Qwen3-1.7B-Base and Qwen3-30B-A3B-Base models are shown in the lower and upper triangles, respectively.
Linearizing pretrained large language models (LLMs) primarily relies on intra-layer hybrid attention mechanisms to alleviate the quadratic complexity of standard softmax attention. Existing methods perform token routing based on sliding-window partitions, resulting in position-based selection and fails to capture token-specific global importance. Meanwhile, linear attention further suffers from distribution shift caused by learnable feature maps that distort pretrained feature magnitudes. Motivated by these limitations, we propose STILL, an intra-layer hybrid linearization framework for efficiently linearizing LLMs. STILL introduces a Self-Saliency Score with strong local-global consistency, enabling accurate token selection using sliding-window computation, and retains salient tokens for sparse softmax attention while summarizing the remaining context via linear attention. To preserve pretrained representations, we design a Norm-Preserved Feature Map (NP-Map) that decouples feature direction from magnitude and reinjects pretrained norms. We further adopt a unified training-inference architecture with chunk-wise parallelization and delayed selection to improve hardware efficiency. Experiments show that STILL matches or surpasses the original pretrained model on commonsense and general reasoning tasks, and achieves up to a 86.2% relative improvement over prior linearized attention methods on long-context benchmarks. Code is available at this URL.
Weikang Meng, Liangyu Huo, Yadan Luo +4
SMULL Group, Harbin Institute of Technology, Shenzhen · Pengcheng Laboratory · UQMM Lab, University of Queensland +1
Hybrid attention models that mix full and sliding-window attention across layers offer a promising approach to efficient long-context inference, but the critical question of which layers should retain full attention remains unsolved. Existing methods use either fixed periodic patterns or attention-based heuristics that may not capture what matters for downstream accuracy. We propose NLL-guided layer selection, a training-free method that directly measures each layer's importance by computing the negative log-likelihood degradation on answer tokens when that layer uses sliding-window instead of full attention. On LongMemEval with Qwen3-4B, our method achieves 64.6% accuracy using only 1/4 full-attention layers, matching the 1/2-FA periodic baseline (65.0%) while halving the computational budget. NLL-guided selection outperforms the SWAA-reported periodic 1/4-FA baseline by 10.4 percentage points and a matched LightTransfer-style baseline by 26.4 percentage points. De-confounding analysis shows the signal is consistent with long-range attention needs rather than generic layer sensitivity. The method requires only ∼15 minutes of one-time calibration, advancing the efficiency-accuracy Pareto frontier for long-context LLM deployment.
The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a layer-wise strategy. Yet, prior work has noted the inherent difficulty of integrating Linear Attention (LA) with Full Attention (FA), suggesting that the design space of attention hybridization remains underexplored. To probe this space, we conduct interpretability analysis and observe that layers exhibit block-wise functional similarity, while individual heads within the same layer display distinct functional specialization despite sharing input features. This head-level heterogeneity suggests that the head dimension provides a natural and principled granularity for fusing heterogeneous attention signals. Building on this insight, we introduce HydraHead, a novel architecture that hybridizes FA and LA along the head axis. HydraHead features two key innovations: (1) an interpretability-driven selection strategy that identifies retrieval-critical heads and preserves FA only for them, and (2) a scale-normalized fusion module that reconciles the distributional gap between FA and LA head outputs. By leveraging a three-stage transfer pipeline with parameter reuse and distillation, we achieve high-performance hybrid models with minimal training overhead. Under a unified training setup, HydraHead outperforms other hybrid designs in long-context tasks while maintaining strong general reasoning. With interpretability-driven head selection, it matches a 3:1 layer-wise hybrid's long-context performance at a 7:1 LA-to-FA ratio. Crucially, trained on only 15B tokens, HydraHead achieves over 69% improvement over the baseline at 512K context length, approaching Qwen3.5, a leading model of comparable size with a native context length of 256K. This highlights the significant scaling potential of head-level hybridization.