Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

Chenhong He; Lei Li; Shicheng Li; Hanglong Lv; Lingpeng Kong; Qi Liu; Tong Yang; Shuhuai Ren

cs.CVAug 28, 2026

Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

Authors: Chenhong He, Lei Li, Shicheng Li, Hanglong Lv, Lingpeng Kong, Qi Liu, Tong Yang, Shuhuai Ren

Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Xiaomi Corporation, LLM-Core Team · Department of Computer Science, The University of Hong Kong

Abstract

Hybrid attention dominates frontier LLMs, yet Vision Transformers (ViTs) in multimodal LLMs lack a satisfactory hybrid design, with no consensus on why certain attention patterns work better. To fill this gap, we study ViT attention heads and find they differentiate into object- and background-specialist roles, a pattern most pronounced under full attention; we call this Semantic Head Specialization (SHS). We propose SHS-Index to quantify this specialization, show that it distinguishes full-attention from chunk-window ViTs, and find that it strongly tracks downstream benchmark performance. We then identify three structural factors that shape SHS---window interaction, token serialization, and local softmax allocation---and use them as design principles for hybrid attention. Guided by these factors, we design Ariadne Attention, a hybrid that matches full attention on 22 image and video tasks at 6.5x less attention compute. Our findings establish head specialization as a measurable property for diagnosing and designing principled hybrid ViT attention at the multimodal-LLM scale.

Figures & tables

Figure 1: SHS-Index separates ViT attention designs. Across 16 open-source visual encoders and VLMs, SHS-Index cleanly separates full-attention and chunk-window ViTs. Within Qwen2.5-VL, changing the LLM size from 3B to 72B leaves SHS-Index nearly unchanged, indicating that the signal comes from the ViT attention design.

Panel of Figure 2 — Figure 2: Semantic Head Specialization (SHS) in ViT attention heads. First column: input image (top) and a foreground (blue) / background (green) segmentation (bottom); each ViT token is labeled foreground or background by the dominant region within its image patch. Columns 2–4: received-attention heatmaps for a representative head at two ViT layers (L11, L17) across three attention designs. Under Full Attention, heads specialize: bright regions concentrate on the foreground object, revealing clear object/background differentiation. Our Ariadne attention preserves this specialization. Under Chunk Window attention (Qwen2.5-VL style), heads show grid-like window-boundary artifacts and fail to differentiate foreground from background.

Panel of Figure 3 — Figure 3: Token arrangements on a $4\!\times\!8$ patch grid ( $2\!\times\!4$ blocks). Numbers indicate position in the 1D sequence; bold borders delimit $2\!\times\!2$ blocks. (a) Raster-scan. (b) Qwen2.5-VL chunk window: blocks grouped into non-overlapping windows (dashed red line); token 15 and token 16 are spatially adjacent but in separate windows. (c) Row-major: all blocks serialized left-to-right, top-to-bottom for SWA. (d) Column-major: blocks serialized top-to-bottom, left-to-right. Ariadne alternates (c) and (d) across layers (§ 4.1 ).

Panel of Figure 4 — Figure 4: Why the 2D token ordering is incompatible with SWA. Numbers indicate position in the 1D sequence. (a) Chunk-2D: neighbors 16–23 map to the top of Window 1, spatially far from the anchor . (b) Row-major 1D: nearly all neighbors fall into surrounding rows, forming a coherent neighborhood.

Figure 5: SHS-Index across structural configurations. The controlled variants differ only in the local attention structure while sharing the same backbone and training recipe. SHS-Index rises under the structure changes validated above, with Full+Sink shown as the highest-SHS diagnostic point and chunk-2D as the lowest.

Figure 6: SHS-Index and benchmark quality. Across the eight diagnostic configurations, SHS-Index correlates with the corrected 22-task average (

r{=}0.858

p{=}0.006

; dashed line = 8-config linear fit). Ariadne is shown as a later design point and is not used to define the primary fit.

Panel of Figure 7 — Figure 7: The 4-row-3-col block architecture of Ariadne Attention. Each block contains 4 row-major SWA + sink layers, 3 column-major SWA + sink layers, and 1 full-attention layer; four repeats form the 32-layer ViT.

Figure 8: Attention compute vs. image-task quality at

896^{2}

. Ariadne (

\bigstar

), an SHS-informed hybrid, comes within about 0.5 points of full attention at

6.5\times

lower attention FLOPs.

	Image tasks (20)	Video tasks (2)
Full Attention	40.92	22.1
Chunk-window Attention	37.35	23.3
Ariadne Attention	40.40	24.3

Table 1: Corrected benchmark averages reported separately for image and video tasks at the 8k primary checkpoint. Ariadne is within 0.52 points of full attention on image tasks; the two video tasks are reported separately rather than folded into the headline claim.

Figure 9: ViT forward-pass time at

896^{2}

(4,096 tokens). Solid bars: end-to-end block time; hatched bars: attention kernel only. In the updated scaling run, Ariadne is 106.8 ms end-to-end versus 123.5 ms for Full, and its attention kernel is 6.7 ms versus 24.6 ms.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Layers	32
Full-attention layers	positions 7, 15, 23, 31
Local-attention layers	28
Hidden dimension	1,280
MLP dimension	3,456
Attention heads	16

Appendix

Table 2: ViT encoder hyperparameters, shared across all 9 configurations.

Short Name	Layout	Attention	Sink	Window Size	SHS-Index	Performance (20-image)
wa2d	2D	Chunk	No	64 tok	.577	37.35
wa2d_sink	2D	Chunk	Yes	64 tok	.579	37.94
swa2d	2D	SWA	No	128 tok	.588	38.09
swa2d_sink	2D	SWA	Yes	128 tok	.597	39.13
swa_row	1D row	SWA	No	128 tok	.600	38.80
swa_sink_row	1D row	SWA	Yes	128 tok	.598	40.14

Appendix

Table 3: Complete model configurations. All models share the same ViT architecture (32 layers, 16 heads,

d_{h}{=}80

) and training setup (Table 8 ); only the local attention operator differs. Bench = macro-average over 20 image tasks.

Window	Context	SHS	20-image	22-task
$w{=}32$	64 tok	.597	38.11	36.95
$w{=}64$	128 tok	.602	40.40	38.93
$w{=}128$	256 tok	.595	38.65	37.27

Appendix

Table 4: Window size ablation (Ariadne, swa_sink_4r3c) at the primary checkpoint.

w{=}64

is optimal under both 20-image and 22-task averages, matching the window size used by GPT-OSS ( OpenAI, 2025 ) and MiMo-V2-Flash ( Xiaomi LLM-Core Team, 2026 ) .

Figure 10: Benchmark trajectory across training for six key models. Architectural rankings are stable across late checkpoints, while SWA+Sink (2D) begins to overfit in the late stage.

Iter	full_s †	full	4r3c	sink_r	swa_r	s2d_s	swa2d	w_s	wa2d
2k	22.25	21.48	24.58	23.15	22.34	22.98	22.67	23.22	23.02
4k	34.44	37.15	36.24	35.77	34.80	35.67	33.30	35.48	34.62
6k	37.70	40.56	38.04	37.72	36.64	36.57	37.64	38.19	36.82
8k	39.95	40.92	40.40	40.14	38.80	39.13	38.09	37.94	37.35
10k	39.94	40.40	40.10	40.11	38.84	38.01	35.21	38.27	37.10

Appendix

Table 5: Benchmark (avg w/o video, 20 tasks) across training iterations. The 8k primary checkpoint uses corrected MMBench-EN scores; the 10k row retains the original evaluation because the re-evaluation was run at 8k only. † full_sink 2k–6k based on general tasks only (10 tasks).

Iter	full_s †	full	4r3c	sink_r	swa_r	s2d_s	swa2d	w_s	wa2d
2k	22.25	21.42	24.41	23.04	22.35	22.85	22.58	23.13	22.92
4k	33.25	35.77	35.16	34.67	33.73	34.67	32.41	34.49	33.61
6k	36.26	38.93	36.85	36.53	35.55	35.45	36.54	37.01	35.62
8k	38.18	39.21	38.93	38.66	37.51	37.76	36.91	36.70	36.07
10k	38.20	38.73	38.66	38.63	37.61	36.84	34.27	37.02	35.91

Appendix

Table 6: Benchmark (avg w/ video, 22 tasks) across training iterations. Includes Video-MME and Video-MMMU. The 8k primary checkpoint uses corrected MMBench-EN scores; the 10k row retains the original evaluation. † full_sink 2k–6k based on general tasks only (10 tasks).

Cat.	Task	Full	Chunk	Ariadne	$\boldsymbol{\Delta}$
General	AI2D	37.2	35.3	37.5	$+2.3$
	BLINK	37.0	39.4	38.3	$-1.1$
	CV-Bench	47.7	43.4	49.5	$+6.1$
	MMMU	29.1	28.7	30.2	$+1.5$
	MMBench-EN	36.34	36.94	38.57	$+1.63$
	MMBench-CN	30.8	27.2	29.9	$+2.7$

Appendix

Table 7: Per-task benchmark results for the three key models at the 8k primary checkpoint. MMBench-EN was re-evaluated with corrected output parsing.

\Delta

= Ariadne

-

Chunk. Bold = best between Chunk and Ariadne. Ariadne recovers most of the full-hybrid gap on vision-centric tasks (V*

+11.0

, DocVQA

+9.8

, OCRBench

+8.9

); grounding tasks, which rely on local spatial reasoning within the sliding window, show near-parity.

Parameter	Value
ViT encoder	Random-init, 32 layers
LLM backbone	Qwen2-0.5B ( Yang et al., 2024 )
Projector	Random-init
ViT / LLM lr	$5{\times}10^{-4}$ / $1{\times}10^{-5}$
LR schedule	Constant w/ 1k warmup
Global batch size	2,048

Appendix

Table 8: Training configuration shared across all nine models.

Cat.	Tasks	N
General	AI2D, BLINK, CV-Bench, MMMU, MMBench-EN/CN, Mantis, MME-RealWorld-CN, V*, VLMs-are-Blind	10
OCR	ChartXiv-RQ/DQ, DocVQA, InfoVQA, ChartQA, OCRBench	6
Ground.	RefCOCO, RefCOCO+, RefCOCOg	3
Count	PixMo-Count	1
Video	Video-MME, Video-MMMU	2

Appendix

Table 9: Benchmark suite: 20 image + 2 video tasks. Video tasks excluded from primary “Avg w/o video”; included in “Avg w/ video.” Citations: General ( Kembhavi et al., 2016 ; Fu et al., 2024 ; Tong et al., 2024 ; Yue et al., 2024 ; Liu et al., 2024a ; Jiang et al., 2024 ; Zhang et al., 2025 ; Wu and Xie, 2024 ; Rahmanzadehgervi et al., 2024 ) ; OCR ( Wang et al., 2024 ; Mathew et al., 2021 ; Mathew et al., 2022 ; Masry et al., 2022 ; Liu et al., 2024c ) ; Grounding ( Yu et al., 2016 ) ; Count ( Deitke et al., 2024 ) ; Video ( Fu et al., 2025 ; Hu et al., 2025 ) .

Panel of Figure 11 — Figure 11: Per-head received-attention box plots corresponding to Figure 2 . Each subplot shows the distribution for all 16 heads, split by foreground (blue) and background (green). Full Attention and Ariadne show clear obj/bg separation; Chunk Window distributions are uniform, indicating suppressed specialization.

Panel of Figure 12 — Figure 12: Additional received-attention heatmaps across four images, laid out as Figure 2 . Each block: first column = input image (top) and foreground mask (bottom); columns 2–4 = representative head heatmaps at L11 and L17 (layer labels on the right). Full Attention and Ariadne consistently show clear object/background differentiation; Chunk Window shows suppressed specialization across all images.

#	Model	Family	Attention	ViT Spec	SHS-all	SHS-core
1	InternVL3.5-38B	InternVL	Full	6B (45L, 3200d, 25h)	.648	.627
2	InternVL3.5-8B	InternVL	Full	300M (24L, 1024d, 16h)	.641	.651
3	Qwen3-VL-32B	Qwen3-VL	Full	SigLIP2-SO-400M	.633	.644
4	InternVL3-8B	InternVL	Full	300M (24L, 1024d, 16h)	.630	.649
5	Qwen3-VL-8B	Qwen3-VL	Full	SigLIP2-SO-400M	.626	.639
6	Qwen3-VL-4B	Qwen3-VL	Full	SigLIP2-Large (300M)	.624	.639

Appendix

Table 10: SHS-Index on 16 open-source visual encoders and VLMs. The full-attention group (mean SHS

\approx

0.631) and chunk-window group (mean SHS

=

0.585) show zero overlap: the lowest full-attention score is .610, above the highest chunk score .586. Qwen3-VL uses SigLIP2-SO-400M for its 8B/32B variants and SigLIP2-Large (300M) for its 2B/4B variants; Qwen2.5-VL uses an identical ViT across 3B–72B LLM sizes (SHS varies by

{<}\,0.002

). DINOv2 and DINOv3 were not trained with a language model, providing an additional check that the separation is not specific to VLM training.

Model	COCO	VOC	OpenImages	Chart
Full+Sink	.611	.615	.618	.634
Full	.606	.609	.614	.627
SWA+Sink (1D)	.598	.596	.608	.625
SWA+Sink (2D)	.597	.600	.598	.626
Chunk	.577	.583	.575	.613
Pearson $r$ (vs. 22-task)	.87	.80	.88	.75

Appendix

Table 11: Cross-domain SHS-Index robustness. Correlations use all eight diagnostic configurations at 8k; the five model rows are representative. The model ranking remains highly stable across domains (cross-domain rank Spearman

\rho{=}0.97

Figure 13: SHS-Index across iterations. High/low separation stabilizes from 4k onward.

Figure 14: Obj-dominant heads across iterations. SWA-1D without sink has the lowest percentage, while full attention remains highest.

Iter	full_s	full	4r3c	sink_r	swa_r	s2d_s	swa2d	w_s	wa2d
2k	.632	.627	.612	.610	.601	.605	.602	.596	.589
4k	.616	.610	.603	.600	.591	.596	.590	.585	.579
6k	.614	.606	.601	.597	.586	.595	.590	.580	.578
8k	.611	.606	.602	.598	.600	.597	.588	.579	.577
10k	.610	.604	.600	.595	.601	.596	.587	.581	.576

Appendix

Table 12: SHS-Index (mean over all 32 layers) across training iterations for all 9 controlled architectures. This is the primary aggregation used throughout the paper (Table 16 , Pearson correlation in § 3 , and Figure 6 ). Column headers abbreviated: full_s = full_sink, sink_r = swa_sink_row, swa_r = swa_row, s2d_s = swa2d_sink, w_s = wa2d_sink.

Iter	full_s	full	4r3c	sink_r	swa_r	s2d_s	swa2d	w_s	wa2d
2k	.651	.644	.612	.615	.604	.605	.606	.597	.586
4k	.636	.629	.602	.603	.593	.598	.596	.583	.577
6k	.635	.624	.604	.600	.588	.597	.599	.578	.576
8k	.632	.623	.604	.599	.603	.601	.592	.580	.576
10k	.630	.622	.605	.594	.601	.598	.593	.580	.574

Appendix

Table 13: SHS-Index (core layers L9–21) across training iterations.

Iter	full_s	full	4r3c	sink_r	swa_r	s2d_s	swa2d	w_s	wa2d
2k	93.8	91.7	83.3	84.4	81.2	83.3	84.4	81.2	90.6
4k	82.3	79.2	69.8	74.0	68.8	74.0	75.0	78.1	75.0
6k	81.2	80.2	68.8	67.7	61.5	69.8	75.0	68.8	76.0
8k	82.3	78.1	71.9	67.7	59.4	69.8	71.9	66.7	72.9
10k	79.2	76.0	67.7	64.6	59.4	72.9	70.8	69.8	71.9

Appendix

Table 14: Percentage of obj-dominant heads (core L9–21, majority vote across analysis images).

Iter	full_s	full	4r3c	sink_r	swa_r	s2d_s	swa2d	w_s	wa2d
2k	.618	.608	.566	.576	.552	.567	.570	.566	.557
4k	.596	.584	.546	.554	.533	.547	.548	.545	.543
6k	.589	.574	.545	.540	.524	.539	.546	.529	.537
8k	.587	.574	.545	.543	.522	.540	.539	.531	.534
10k	.581	.566	.543	.532	.519	.537	.537	.530	.531

Appendix

Table 15: Mean rcv_auroc_obj (core L9–21) across training iterations.

Model	Layout	Attn	Sink	SHS	Entropy	Distance	Bench
full	N/A	Full	No	.606	4.584	14.922	40.92
full_sink	N/A	Full	Yes	.611	4.350	14.824	39.95
4r3c	1D rc	SWA	Yes	.602	2.768	13.950	40.40
swa_sink_row	1D r	SWA	Yes	.598	2.969	12.002	40.14
swa_row	1D r	SWA	No	.600	3.010	11.713	38.80
swa2d_sink	2D	SWA	Yes	.597	2.975	12.802	39.13

Appendix

Table 16: SHS, Entropy, and Distance for all 9 models at the primary checkpoint. SHS-Index is the all-32-layer mean (matching Table 12 ; same convention used for the Pearson correlation in § 3 ). Entropy and Distance are computed only over the 28 non-full-attn layers, since the 4 full-attn layers are identical across all 9 architectures and would dominate these distance/entropy averages.

Metric	$\boldsymbol{\rho}$	$\boldsymbol{p}$
SHS	+0.857	0.007
Distance	+0.833	0.010
Entropy	+0.524	0.183

Appendix

Table 17: Cross-model Spearman

\rho

vs. benchmark (N=8). SHS is the strongest correlate; Entropy is not significant.

Change	$\Delta$ SHS	$\Delta$ Ent.	$\Delta$ Dist.	$\Delta$ Bench
+Sink (Chunk 2D)	+.002	$-0.05$	$+0.46$	$+0.59$
+Sink (SWA 2D)	+.009	$+0.06$	$+1.00$	$+1.04$
+Sink (SWA 1D)	$-.002$	$-0.04$	$+0.29$	$+1.34$
+Sink (Full)	+.005	$-0.23$	$-0.10$	$-0.97$
Chunk $\to$ SWA (2D)	+.011	$+0.44$	$+1.11$	$+0.74$
Chunk $\to$ SWA (2D+s)	+.018	$+0.55$	$+1.66$	$+1.78$

Appendix

Table 18: Pairwise deltas for each architectural change at the corrected 8k checkpoint. The 2D

\to

1D transition improves the benchmark, but individual local changes do not always move SHS and benchmark in the same direction; the relationship is a global cross-architecture pattern rather than a monotonic rule for every intervention. (

\Delta

SHS uses all-32-layer mean as in Table 16 ;

\Delta

Entropy/Distance use the non-full-attn layers.)

Model	bg stl.	obj stl.	$\Delta$	bg $>$ obj
full_sink	0.38%	0.23%	+0.14%	97.9%
swa_sink_row	4.61%	2.86%	+1.74%	96.9%
swa_sink_4r3c	3.79%	2.52%	+1.26%	93.8%
swa2d_sink	2.84%	1.68%	+1.16%	96.9%
wa2d_sink	4.32%	3.69%	+0.64%	78.1%

Appendix

Table 19: Sink steal analysis (all heads, core L9–21 at the primary checkpoint). bg/obj stl. = fraction of received attention redirected to sink. In SWA models, 94–97% of heads steal more from bg than obj, i.e., sink acts as a background suppressor . In wa2d_sink, directionality drops to 78%, explaining why chunk+sink fails.

Setting	High	Low	$p$ (U)	$d$
8 diagnostic @8k	38.80–40.92	37.35–38.09	.018	3.65
9 all @8k	38.80–40.92	37.35–38.09	.012	3.91
9 all @10k	38.01–40.40	35.21–38.27	.024	2.37

Appendix

Table 20: Robustness check:

K

-means (

k{=}2

) on SHS-Index partitions the controlled configurations into high-SHS and low-SHS groups (benchmark ranges shown). Mann-Whitney

U

test on the 20-image-task average confirms significant separation. Cohen’s

d

uses unweighted pooled SD. The 9-model grouping remains significant at 10k.

Panel A: High-8 vs. Low-8 (avg over 23 tasks)
Condition	$\Delta$ vs. base
Remove 8 hi-SHS	$-1.25$
Remove 8 lo-SHS	$-1.07$
hi $-$ lo	$-0.18$	(14/23 tasks hi $<$ lo)
Panel B: All 16 heads removed
Task	Base	$\boldsymbol{\Delta}$

Appendix

Table 21: Head elimination on the full model (core layers). Panel A : removing 8 high-SHS vs. 8 low-SHS heads yields negligible difference; individual heads are interchangeable. Panel B : removing all 16 heads collapses vision-dependent tasks; MMMU (reasoning-heavy) is unaffected. The collective specialization pattern is indispensable. The head-elimination suite (23 tasks) is a superset of the 22-task primary benchmark, with sub-scores (e.g., MME-cognition vs. MME-perception) reported separately for finer-grained per-task analysis.

Model	Tasks	avg (hi $-$ lo)
full	23	$-0.18$
wa2d	23	$+0.20$
swa_sink_row	25	$-0.27$

Appendix

Table 22: High-low SHS head elimination across models. No model shows a significant difference. Task counts vary slightly (23–25) across models because evaluation suites differ in which sub-scores are reported separately at this checkpoint; the conclusion (no high/low gap) is robust to this choice.

	Q	K	V	O	Q $\!{\times}\!$ K
full	+.09	+.11	$-.06$	+.05	+.10
4r3c	+.15	+.16	+.01	+.05	+.16
sink_row	+.00	+.02	$-.03$	+.13	+.01
swa_row	+.14	+.14	+.06	+.16	+.14
s2d_sink	+.33	+.32	+.05	+.19	+.33
swa2d	+.19	+.20	+.01	+.12	+.20

Appendix

Table 23: Spearman

\rho

between per-head weight Frobenius norms and SHS. Q/K show weak positive correlations (8/8 positive); V shows none (mean

-.01

Model	Base SHS	Eq. SHS	$\boldsymbol{\Delta}$
full	.6237	.6178	$-0.0059$
4r3c	.6044	.6001	$-0.0044$
swa_sink_row	.5993	.5958	$-0.0035$
swa_row	.6036	.5997	$-0.0038$
wa2d	.5763	.5751	$-0.0013$

Appendix

Table 24: QK equalization: SHS drops in all 5 models. The positive QK-SHS correlation is a consequence, not a cause, of head specialization.

		Full			wa2d (Chunk)			SWA-row			SWA-sink-row
Res	Tok	FLOP	op	e2e	FLOP	op	e2e	FLOP	op	e2e	FLOP	op	e2e
$224^{2}$	256	0.04	1.2	22.1	0.01	2.8	24.4	0.02	1.2	22.1	0.02	1.2	22.3
$448^{2}$	1024	0.69	2.2	32.1	0.12	4.5	32.9	0.16	1.4	31.4	0.16	1.4	31.3
$672^{2}$	2304	3.48	7.6	68.7	0.52	9.7	68.4	0.60	2.9	64.6	0.60	2.9	64.6
$\mathbf{896^{2}}$	4096	11.0	23.9	124.4	1.52	17.5	115.3	1.68	6.2	109.0	1.68	6.3	109.0
$1120^{2}$	6400	26.8	55.2	211.2	3.59	28.8	178.1	3.83	12.1	168.5	3.83	12.0	168.6

Appendix

Table 25: Four-operator ViT profiling at representative resolutions. FLOP = attention TFLOP; op = FlashAttention kernel time (ms); e2e = full block time (ms). SWA-sink-row

\equiv

SWA-row: sink adds zero overhead in this run (FA3 native s_aux support). The updated Full–Ariadne scaling sweep is reported in Table 26 .

Resolution	Tokens	Full attn.	Ariadne attn.	Full e2e	Ariadne e2e	e2e saving
$448^{2}$	1,024	0.69T	0.16T	31.3	30.6	$-2.2\%$
$672^{2}$	2,304	3.48T	0.60T	66.9	62.8	$-6.1\%$
$896^{2}$	4,096	11.0T	1.68T	123.5	106.8	$-13.5\%$
$1120^{2}$	6,400	26.8T	3.83T	210.1	166.1	$-20.9\%$
$1344^{2}$	9,216	55.7T	7.64T	327.5	240.0	$-26.7\%$
$1568^{2}$	12,544	103.1T	13.8T	500.2	331.0	$-33.8\%$

Appendix

Table 26: Updated Full–Ariadne efficiency sweep on NVIDIA Hopper with FlashAttention-3. Attention columns report theoretical attention FLOPs; e2e columns report complete ViT block time in milliseconds. The end-to-end saving grows with resolution, which is the regime targeted by hybrid attention.

Step	Setting	22-task		20 image-task
		$r$	$p$	$r$	$p$
8k	$N{=}8$ (diagnostic)	.858	.006	.877	.004
8k	$N{=}9$ (incl. Ariadne)	.853	.003	.875	.002
10k	$N{=}8$	.66	.077	.67	.067
10k	$N{=}9$	.67	.048	.69	.041

Appendix

Table 27: Pearson correlation between SHS-Index and benchmark across controlled architectures. The 8k rows use corrected MMBench-EN scores; bootstrap 95% CI for the primary 22-task setting is

[0.51,0.98]

, and an exact two-sided permutation test gives

p{=}0.008

. The 10k rows retain the original evaluation and are included as a sensitivity analysis.

Setting	$r$	Bootstrap 95% CI	Exact $p$
$N{=}8$ , 22-task	.858	[.51, .98]	.008
$N{=}8$ , 20-image	.877	[.63, .98]	.006
$N{=}9$ , 22-task	.853	[.50, .98]	.004
$N{=}9$ , 20-image	.875	[.63, .98]	.002

Appendix

Table 28: Bootstrap intervals (10k resamples) and exact two-sided permutation

p

-values computed from the corrected 8k benchmark scores. The

N{=}8

permutation enumerates all 40,320 assignments.

Vision Transformer

Vision Transformers (ViTs) use self-attention over image patches to learn visual representations for recognition, segmentation, and other tasks.

See all 365 papers

HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization
Jun 18, 2026Zhentao Tan, Wei Chen, Jingyi Shen +4Softmax Attention Long-Context Modeling
Alibaba Group
Mind the Heads: Topological Representation Alignment for Multimodal LLMs
Jun 22, 2026Davide Caffagni, Alberto Compagnoni, Federico Melis +5Cross-Modal Alignment Multimodal Large Language Models
University of Modena and Reggio Emilia · University of Pisa · AMD Silo AI
Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration
Sep 24, 2026Jiaqi Deng, Zonghan Wu, Zhan Heng +3Multimodal Grounding Multimodal Large Language Models
University of Technology Sydney · East China Normal University · The University of New South Wales +2

Jun 18, 2026J/K move · Enter open · S save

HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization

Zhentao Tan, Wei Chen, Jingyi Shen, Yao Liu+3

Alibaba Group

Softmax Attention Long-Context Modeling Linear Attention Multi-Head Attention Hybrid Attention Attention Head Specialization

Open paper AskPDF arXiv

The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a layer-wise strategy. Yet, prior work has noted the inherent difficulty of integrating Linear Attention (LA) with Full Attention (FA), suggesting that the design space of attention hybridization remains underexplored. To probe this space, we conduct interpretability analysis and observe that layers exhibit block-wise functional similarity, while individual heads within the same layer display distinct functional specialization despite sharing input features. This head-level heterogeneity suggests that the head dimension provides a natural and principled granularity for fusing heterogeneous attention signals. Building on this insight, we introduce HydraHead, a novel architecture that hybridizes FA and LA along the head axis. HydraHead features two key innovations: (1) an interpretability-driven selection strategy that identifies retrieval-critical heads and preserves FA only for them, and (2) a scale-normalized fusion module that reconciles the distributional gap between FA and LA head outputs. By leveraging a three-stage transfer pipeline with parameter reuse and distillation, we achieve high-performance hybrid models with minimal training overhead. Under a unified training setup, HydraHead outperforms other hybrid designs in long-context tasks while maintaining strong general reasoning. With interpretability-driven head selection, it matches a 3:1 layer-wise hybrid's long-context performance at a 7:1 LA-to-FA ratio. Crucially, trained on only 15B tokens, HydraHead achieves over 69% improvement over the baseline at 512K context length, approaching Qwen3.5, a leading model of comparable size with a native context length of 256K. This highlights the significant scaling potential of head-level hybridization.

Abstract

Vision Transformer

Explore similar work

HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

Mind What Matters for Reasoning: Aligning Cross-Modal Attention via Selective Probability Mass Concentration