TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs

Wentao Ni; Hong Cai; Mingu Lee; Chihao Yu; Zhongming Yu; Hejia Zhang; Kangqi Zhang; Fatih Porikli

cs.LGOct 30, 2026

TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs

Authors: Wentao Ni, Hong Cai, Mingu Lee, Chihao Yu, Zhongming Yu, Hejia Zhang, Kangqi Zhang, Fatih Porikli,

Organizations: University of California, San Diego · Qualcomm AI Research · Purdue University · Inco AI

Abstract

Smartphone applications increasingly require large language models to process long contexts, whereas high prefill latency and memory demands constrain local inference. Block-sparse attention reduces computation and memory traffic, but its irregular selections conflict with mobile NPUs' preference for large, regular tiles. Efficient execution must exploit cross-query reuse while avoiding redundant computation and preserving selections needed for accuracy. In this paper, we present TileWeave, which partitions sparse attention by this cross-query reuse: highly shared key blocks run in coarse NPU tiles, while low-reuse blocks run on the GPU only for the query blocks that selected them. In addition, we design an NPU-native pooled-key selector that keeps selection inexpensive, and a runtime that publishes GPU partials in the order the NPU consumes them and merges them in its existing output store. On RULER, HeteroSparse matches XAttention's accuracy with Qwen3-1.7B and improves it with Llama-3.2-3B-Instruct; its mean LongBench scores match XAttention as well. On Snapdragon SM8750 and SM8850 phones, it achieves 1.35--3.23 $\times$ faster full-model prefill than XAttention over 1k--16k tokens and up to 1.58 $\times$ faster than dense attention.

Figures & tables

Figure 1: Thin blue frames mark NPU query tiles: 64 queries in (1) and 256 in (2)–(4). Small tiles preserve selections but repeat K/V staging and coordination, limiting NPU efficiency. Coarse tiles improve reuse, but dropping exceptions risks accuracy (2), while coarse masked tiles compute unwanted interactions (3). TileWeave keeps shared keys on coarse NPU tiles and preserves exceptions through fine GPU execution (4).

Figure 2: Hexagon NPU architecture and attention mapping. Larger query tiles amortize K/V staging and coordination.

Query tile $B_{r}$	$S=1024$	$S=1536$	$S=2048$
64	2.62 ( $1.00\times$ )	3.18 ( $1.00\times$ )	4.12 ( $1.00\times$ )
128	2.15 ( $1.22\times$ )	2.53 ( $1.25\times$ )	3.28 ( $1.26\times$ )
256	1.51 ( $1.73\times$ )	1.82 ( $1.74\times$ )	2.34 ( $1.76\times$ )

Table 1: NPU attention latency in ms (speedup over

B_{r}=64

). Larger query tiles improve efficiency, giving up to

1.76\times

speedup at fixed selected work.

Attention policy	Score	Density (%)
Pool + diagonal	85.57	55.42
Pool + exceptions (ours)	86.32	62.19
Union	86.29	73.13

Table 2: Llama-3.2-3B on RULER 4k. Non-diagonal exceptions add 0.75 points; the union retains more pairs for a similar score.

	Ideal speedup
Selection count offloaded	4k	8k
None (baseline)	$1.00\times$	$1.00\times$
1 of 4	$1.21\times$	$1.25\times$
At most 2 of 4	$1.49\times$	$1.59\times$
At most 3 of 4	$1.90\times$	$2.12\times$

Table 3: Ideal NPU speedup for Llama-3.2-3B assuming removed blocks complete for free. Offloading blocks selected by at most two query blocks gives

1.49\times

–

1.59\times

speedup.

Figure 3: TileWeave overview. (1) Four sampled query rows and four pooled key vectors per 64-token block produce compact scores; causal softmax, GQA maximum, and relative thresholding with forced sink/diagonal blocks produce wanted lists

W_{i}

per K/V head. (2) With

c^{*}=3

, reuse across four query blocks assigns key blocks to the shared NPU pool

P

(

c(b)\geq 3

, blue) or query-specific GPU exceptions

E_{i}=W_{i}\setminus P

(

1\leq c(b)<3

, teal). The masks illustrate the partition; plus signs mark shared-pool fill-in. (3) Coarse NPU tiles and fine GPU blocks execute together, and a fused output store rescales, combines, and normalizes their online-softmax partials into

O

Figure 4: Qwen3-1.7B NPU selection cost relative to dense attention (totals in ms). TileWeave reduces selection to 6–10% of dense attention’s time.

Figure 5: Fine-grained schedules on a shared time axis (1024 queries, eight K/V heads). Endpoints are measured two-round means from the historical 4k Qwen3-1.7B/SM8750 study using the earlier threshold scorer; selection finishes before

t=0

and other model operators are excluded. Tile intervals are approximate; store/merge marks encode no duration. GPU completion spans 0.37–1.47 ms from a nearby trace, with intermediate groups interpolated. Timing calibration illustrates overlap and relative latency, not a single event trace.

	Qwen3-1.7B			Llama-3.2-3B-Instruct
Method	4k	8k	16k	4k	8k	16k
Dense	87.12	79.48	70.86	86.77	83.39	80.61
XAttention	86.31	77.16	70.08	86.00	82.64	80.13
ShadowAttn	86.95	78.51	69.54	86.72	83.49	80.71
TileWeave	86.37	77.19	69.99	86.32	82.98	80.19

Table 4: RULER mean accuracy. TileWeave matches XAttention on Qwen3-1.7B and is slightly more accurate on Llama-3.2-3B-Instruct at every tested length.

	Qwen3-1.7B							Llama-3.2-3B-Instruct
Method	Qasper	MFQA	HQA	2WQA	MSQ	QMSum	Avg.	Qasper	MFQA	HQA	2WQA	MSQ	QMSum	Avg.
Dense	16.98	42.23	35.57	29.38	13.75	22.82	26.79	37.71	50.25	50.09	38.20	22.19	24.13	37.10
XAttention	18.13	42.35	36.15	28.33	14.73	22.63	27.06	37.32	50.24	48.15	36.19	22.86	23.47	36.37
ShadowAttn	18.86	42.36	35.38	28.53	14.43	22.73	27.05	39.39	51.04	50.20	38.75	21.89	24.10	37.56
TileWeave	17.46	43.36	35.24	28.27	13.03	22.18	26.59	38.36	48.81	50.41	37.20	21.03	23.41	36.54

Table 5: LongBench accuracy (F1; ROUGE-L for QMSum). TileWeave’s mean stays within 0.6 points of dense attention on both models.

Figure 6: Qwen3-1.7B attention latency, including selection and coordination. TileWeave achieves speedups of up to

2.14\times

over dense attention,

5.82\times

over XAttention,

130.14\times

over ShadowAttn (CPU), and

24.53\times

over ShadowAttn (GPU).

Figure 7: Qwen3-1.7B full-model prefill latency over 1k–16k tokens. TileWeave achieves speedups of up to

1.58\times

over dense attention,

3.23\times

over XAttention,

63.23\times

over ShadowAttn (CPU), and

13.98\times

over ShadowAttn (GPU).

Selector	Score	Density (%)
TileWeave	86.37	57.30
Keys sampled, not pooled	85.81	57.31
Queries pooled as well	86.80	57.21
XAttention’s antidiagonal scores	86.67	57.26
Dense attention	87.12	100.00

Table 6: Selector ablation at matched density (Qwen3-1.7B, RULER 4k). Pooled keys improve accuracy, while pooled queries and antidiagonal scores trade additional NPU time for small gains.

Attention (ms)
		8k prompts		16k prompts
Scorer	Execution	Time	Gain	Time	Gain
XAttention	naive, per K/V head	3,762	2.24 $\times$	12,978	2.37 $\times$
XAttention	naive, per head	7,263	4.33 $\times$	25,207	4.61 $\times$
Pooled keys	naive	2,552	1.52 $\times$	8,892	1.63 $\times$
Pooled keys	split, $c^{*}{=}2$	1,826	1.09 $\times$	6,007	1.10 $\times$

Table 7: Qwen3-1.7B scorer and execution ablations; gain is TileWeave’s speedup over each row. The scorer and heterogeneous split make comparable contributions to the latency reduction.

	RULER 8k		Attention (ms)		Prefill (s)
Execution	Score	Dens.	8k	16k	8k	16k
TileWeave	82.98	58.0	2,375	7,864	6.77	16.60
Masked tile	82.64	51.1	3,066	10,537	7.63	19.63
Union	83.16	70.1	2,949	10,092	7.52	19.18
Speedup over the masked tile			1.29 $\times$	1.34 $\times$	1.13 $\times$	1.18 $\times$
Speedup over the union			1.24 $\times$	1.28 $\times$	1.11 $\times$	1.16 $\times$

Table 8: Llama-3.2-3B executions at a fixed selector and threshold. TileWeave gives 1.11–1.18

\times

faster prefill with competitive RULER 8k accuracy.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Dense		XAttention		TileWeave		Group mean
Task	Score	Density	Score	Density	Score	Density	Score	Density
cwe	1.66	100.0	2.10	39.2	2.44	48.5	2.80	40.6
fwe	91.67	100.0	93.53	44.7	93.07	54.2	93.40	45.9
niah_multikey_1	96.80	100.0	93.40	32.9	94.40	40.1	92.60	33.9
niah_multikey_2	50.40	100.0	51.80	45.7	49.20	53.6	43.00	46.0
niah_multikey_3	37.20	100.0	32.20	42.3	34.00	50.9	30.80	43.3

Appendix

Table 9: RULER 16k task scores and density (%) for Qwen3-1.7B. Taking the GQA-group maximum improves retrieval over the group mean at the same threshold.

	1k		2k
Method	Score	Dens.	Score	Dens.
Dense	97.30	100	91.13	100
XAttention	96.88	86.3	90.73	68.9
TileWeave	97.00	82.1	90.69	69.2

Appendix

Table 10: RULER 1k/2k mean scores and density (%) for Qwen3-1.7B. TileWeave remains within 0.12 points of XAttention.

	Dense		XAttention		ShadowAttn		TileWeave
Task	Score	Density	Score	Density	Score	Density	Score	Density
cwe	95.78	100.0	93.22	51.2	95.08	51.9	94.48	57.4
fwe	64.13	100.0	64.73	58.9	64.67	52.1	64.20	66.1
niah_multikey_1	99.80	100.0	99.80	54.6	99.20	55.8	99.80	57.1
niah_multikey_2	85.00	100.0	83.60	62.0	84.60	52.3	84.20	67.4
niah_multikey_3	92.20	100.0	88.60	55.3	91.00	50.9	89.00	60.7

Appendix

Table 11: RULER 4k task scores and density (%) for Qwen3-1.7B. TileWeave gains on multi-key retrieval but loses on question answering relative to XAttention.

	Dense		XAttention		ShadowAttn		TileWeave
Task	Score	Density	Score	Density	Score	Density	Score	Density
cwe	33.10	100.0	30.76	52.1	28.90	51.5	27.84	63.0
fwe	82.40	100.0	86.40	55.5	82.33	52.3	84.80	66.4
niah_multikey_1	99.20	100.0	99.40	51.0	99.40	52.8	99.60	60.4
niah_multikey_2	99.80	100.0	99.60	59.5	99.80	51.6	99.80	71.1
niah_multikey_3	99.20	100.0	95.80	55.0	99.00	50.3	99.00	65.1

Appendix

Table 12: RULER 4k task scores and density (%) for Llama-3.2-3B-Instruct. TileWeave improves variable tracking and multi-key retrieval over XAttention.

	Method	1k	2k	4k	8k	16k	LB
RULER	TileWeave	82.1	69.2	57.3	49.6	43.8	41.4
	$c^{*}{=}2$	84.7	72.5	60.4	52.7	46.9	44.8
	group mean	80.9	66.1	53.2	45.2	39.6	38.2
	XAttention	86.3	68.9	53.4	43.6	36.2	34.4
	ShadowAttn	–	–	53.4	51.2	49.8	50.1
Phone	TileWeave	76.8	67.7	56.8	48.5	43.7	–

Appendix

Table 13: Density (%) of retained causal pairs and executed phone blocks for Qwen3-1.7B (LB: LongBench). Both block methods become sparser on longer prompts, TileWeave more slowly.

	RULER 4k		Attention (ms)
Execution	Score	Dens.	4k	8k
TileWeave, $c^{*}=3$	86.10	53.1	374	1277
Masked tile	86.15	53.1	542 (1.45)	1973 (1.54)
Union	85.12	53.2	426 (1.14)	1426 (1.12)

Appendix

Table 14: Llama-3.2-3B at matched density: TileWeave matches masked-tile accuracy and reduces attention latency. Parentheses show latency relative to TileWeave.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,147 papers

QUILT: Rethinking Sparse-Attention Prefill through Shared Query Execution
Oct 8, 2026Zhenduo Zhao, Qihui Zhou, Mingcong Song +7LLM Inference Acceleration Sparse Attention
Huawei Technologies Co., Ltd. China
TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration
Aug 18, 2026Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4LLM Inference Acceleration Long-Context Language Model Inference
LLaVi Lab, Computer Science & Engineering, University of North Texas · Computer Science, Saint Louis University · Data Science, University of North Texas
STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU
Jul 10, 2026Victor J. B. Jung, Gagandeep Singh, Joseph Melber +3Efficient Transformer Inference Energy-Efficient ML
Integrated Systems Laboratory (IIS), ETH Zürich, Switzerland. · AMD Research and Advanced Development (RAD). · Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna, Italy.

Oct 8, 2026J/K move · Enter open · S save

QUILT: Rethinking Sparse-Attention Prefill through Shared Query Execution

Zhenduo Zhao, Qihui Zhou, Mingcong Song, Zhiyi Chen+6

Huawei Technologies Co., Ltd. China

LLM Inference Acceleration Sparse Attention GPU Kernel Optimization

Open paper AskPDF arXiv

Sparse attention reduces the cost of long-context attention, but existing kernels typically process queries independently, repeatedly loading and dequantizing KV entries shared across queries. We observe substantial overlap in the KV entries selected by neighboring queries, creating opportunities for cross-query reuse. We present QUILT, a workload-aware sparse-attention execution mechanism that jointly processes neighboring queries and reuses shared KV entries to reduce redundant memory traffic and computation. QUILT introduces Shift-and-Compare Set Decomposition (SCSD), which transforms irregular set operations into regular data-parallel primitives suitable for modern accelerators, and pipelines SCSD with attention computation to hide its overhead. Cascaded sharing captures reuse hierarchically at multiple granularities. A tile-aware execution strategy balances sharing granularity with hardware tile utilization and selectively removes low-importance query-specific tails to eliminate underutilized tiles. We evaluate QUILT on LongBench using GLM-5.3 and DeepSeek-3.2 under both tensor and sequence parallelism. Compared with the state-of-the-art sparse-attention kernel, QUILT reduces average kernel latency by up to 55.1% and processed KV data by up to 55.9%, while reducing time-to-first-token (TTFT) latency by up to 36.8% with negligible accuracy degradation.

Abstract

LLM Inference Acceleration

Explore similar work

QUILT: Rethinking Sparse-Attention Prefill through Shared Query Execution

TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration

STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU