Long-context LLM inference is costly because attention repeatedly reads a growing key-value (KV) history. We introduce Chunk-Indexed Sparse Attention (CISA), which retrofits pretrained dense attention models without gradient updates. CISA selects fixed-size KV chunks using compact summaries of existing keys, avoiding token-level indexing and a separate full-length index cache. It uses one indexing query per KV group and shares selected chunks across queries in that group. During prefill, QShare reuses selections across adjacent queries to reduce indexing cost and improve KV reuse and GPU utilization. We evaluate CISA on NVIDIA Nemotron 3 Nano and Ultra, and Inkling-Small. In one-layer BF16 benchmarks at batch size 32 using Ultra's per-device attention dimensions, core attention is 11.7x faster for prefill queries near the end of a 1M-token prompt and 13.0x faster during decode. Including chunk scoring, selection, and all other measured GPU work, speedups are 8.6x and 3.0x. On Ultra NVFP4 with a 4,096-chunk budget, aggregate scores stay within 1.5 percentage points of dense attention on MRCR, LongBench v2, and GraphWalks. RULER sweeps on Nano and Ultra BF16 up to one million tokens show scores within 2.05 percentage points of dense attention with 8-16x fewer KV reads. Serving million-token prompts with 32 concurrent requests and 4K output tokens per request, CISA achieves 2.29x the end-to-end throughput of dense attention using Nemotron 3 Ultra NVFP4 on four NVIDIA B300 GPUs. Selecting fewer chunks increases throughput to 2.61x; in separate GraphWalks evaluations, the smaller budget widens the aggregate score gap from 1.49 to 3.59 percentage points. These results expose a practical quality-speed tradeoff without sparse training.
Figures & tables
Figure 1: CISA retrofit and online sparse execution. Within each KV group, QShare reuses one page plan across adjacent queries; attention uses the original Q/K/V.
Figure 2: Query sharing and attention-path efficiency on Inkling-Small at approximately one million context tokens, batch size 16, and kchunks=4,096 . (a) Terminal prefill with G∈{4,16,32} . Query sharing increases reuse of loaded KV blocks across query positions and can improve attention-tile utilization, depending on the number of query heads per KV head. (b) Single-token decode with effective positional sharing G=1 . Bars show mean GPU busy time over ten retained forwards; labels give total milliseconds and speedup relative to dense attention. Measurements use a one-layer BF16 model. Component definitions, full timings, and setup appear in Appendix Table 11 .
Figure 3: Budget-matched RULER results for Nano and Ultra (BF16, G=4 ) and Inkling-Small (NVFP4, G=16 ). Each sparse run keeps the labeled budget fixed. Panels share the same accuracy scale; signed labels show CISA minus dense accuracy in percentage points. Exact scores, budgets, and remote-compression factors appear in Appendix Table 5 .
Benchmark
Dense
CISA k=2,048
Δ
Verb. Δ
CISA k=4,096
Δ
Verb. Δ
Ultra NVFP4 ( G=4 )
MRCR
47.34
46.64
-0.70
+2.9%
46.96
-0.38
+3.1%
LongBench v2
61.38
61.38
+0.00
+8.0%
62.48
+1.09
+4.7%
GraphWalks
53.42
49.84
-3.59
+14.1%
51.93
-1.49
+10.6%
Inkling-Small NVFP4 ( G=32 )
MRCR
41.46
43.33
+1.87
-3.8%
42.41
+0.94
-3.6%
Table 1: Ultra and Inkling-Small NVFP4 long-context accuracy and generation length at two fixed budgets. Ultra uses G=4 ; Inkling-Small uses G=32 . Scores are percentages, and score Δ is CISA minus the corresponding dense baseline in percentage points. Verbosity Δ is the relative change in mean generated tokens. The two budgets select 32,768 and 65,536 remote historical tokens, respectively, in addition to the fixed recent window and current-page tail. Sampling and cache settings appear in Appendix A.1 .
Model
Benchmark
k
Dense
CISA
Δ
Verbosity Δ
Nano BF16
AA-LCR
1,024
34.94
34.25
-0.69
-9.1%
Ultra NVFP4
AA-LCR
1,024
65.00
65.88
+0.88
+1.8%
SWE-bench Verified
2,048
69.20
69.93
+0.73
-0.2%
Terminal-Bench 2.0
2,048
48.03
49.44
+1.40
-0.7%
Terminal-Bench Hard
1,024
35.94
35.42
-0.52
+2.8%
GDPval
2,048
1395.59
1408.71
+13.12
-2.8%
Table 2: Capability preservation for Nano BF16 and Ultra NVFP4 at their target budgets. Scores use each benchmark’s native percentage metric except GDPval, reported in Elo; score Δ is in percentage points or Elo, respectively, and verbosity Δ is the relative change in mean generated tokens.
Figure 4: Quality–cost operating points for (a) Nemotron Ultra and (b) Inkling-Small, both NVFP4. Quality is the equal-weight mean score across RULER 1M, MRCR200, GraphWalks-200, and AA-LCR. The horizontal axis divides controlled, fixed-output total token throughout speedup by the mean benchmark-relative output-length ratio. It is an estimate that accounts for differences in output length, not a direct measurement of natural completion throughput. Reduced profiles halve the base recall budgets. Lines connect point estimates as G increases within each profile. Exact coordinates and measurement details appear in Appendix Table 7 and Appendix B.1 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Ultra NVFP4 scores by dense-reference prompt length, comparing dense attention with CISA at kchunks∈{2,048,4,096} and G=4 . Labels give the number of distinct prompts. Error bars show 95% task-bootstrap confidence intervals, retaining all four repeats of each sampled task. Table 4 reports exact scores and cohort sizes.
Figure 6: Inkling-Small NVFP4 scores by dense-reference prompt length, comparing dense attention with CISA at kchunks∈{2,048,4,096} and G=32 . Labels give the number of distinct prompts; shaded buckets contain no prompts. Error bars show pointwise 95% confidence intervals from 20,000 task-bootstrap resamples, retaining all four repeats of each sampled task.
Benchmark
Input length
Prompts
Rollouts
Dense
k=2,048
Δ
k=4,096
Δ
MRCR
<32 K
863
3,452
84.67
84.85
+0.18
84.73
+0.06
32–128K
601
2,404
46.58
46.47
-0.11
46.07
-0.51
128–256K
282
1,128
18.36
17.78
-0.58
18.34
-0.02
256–512K
316
1,264
10.36
7.64
-2.72
9.50
-0.86
512–768K
136
544
5.66
4.96
-0.70
5.20
-0.46
768K–1M
153
612
4.76
4.07
-0.69
4.69
-0.07
Appendix
Table 3: Ultra BF16 scores by dense-reference prompt length (%). Both CISA configurations use 16-token chunks, a 128-page recent window, and G=4 . Δ is CISA minus dense in percentage points. Empty MRCR final answers are scored as zero: one for dense attention and two for kchunks=4,096 .
Benchmark
Input length
Prompts
Rollouts
Dense
k=2,048
Δ
k=4,096
Δ
MRCR
<32 K
863
3,452
84.32
84.30
-0.02
84.64
+0.32
32–128K
601
2,404
45.28
44.13
-1.15
44.64
-0.64
128–256K
282
1,128
19.65
17.93
-1.72
18.39
-1.26
256–512K
316
1,264
14.87
13.17
-1.70
13.17
-1.70
512–768K
136
544
11.09
10.70
-0.39
10.60
-0.50
768K–1M
153
612
9.59
10.13
+0.54
10.33
+0.74
Appendix
Table 4: Ultra NVFP4 scores by dense-reference prompt length (%), with BF16 KV state. Both CISA configurations use 16-token chunks, a 128-page recent window, and G=4 . Δ is CISA minus dense in percentage points.
(a) Nano and Ultra BF16: matched budgets, G=4
Model
Context
k (chunks)
Remote comp.
Dense
CISA
Δ
Nano
128K
1,024
8 ×
94.61
92.93
-1.68
Nano
256K
2,048
8 ×
92.75
90.70
-2.05
Nano
512K
4,096
8 ×
91.37
89.52
-1.85
Nano
1M
4,096
16 ×
86.60
85.87
-0.74
Quest bound + CISA index, Nano 1M
G=1 : 84.95; G=4 : 84.41
Appendix
Table 5: RULER accuracy across context lengths, recall budgets, and query-sharing groups. (a) Nano and Ultra BF16 controls at G=4 ; Δ is CISA minus dense. (b) Inkling-Small NVFP4 results at G=16 and G=32 ; Δ16−32 is the score at G=16 minus that at G=32 . K counts selected remote 16-token chunks; remote compression excludes the recent window and current tail. Scores are percentages and differences are percentage points, calculated before rounding. Daggers identify the G=16 settings plotted in Figure 3 ; dashes denote unavailable results. The Nano 1M Quest-bound controls use the same serving build, budget, window, and page size at G=1 and 4 ; the CISA reference uses a different build.
Scale
Precision
Benchmark
Metric
k
Dense
CISA
Δ
Verbosity Δ
Nano
BF16
AA-LCR
pass@1 (%)
1,024
34.94
34.25
-0.69
-9.1%
Ultra
BF16
AA-LCR
pass@1 (%)
1,024
65.13
65.38
+0.25
+5.3%
Ultra
NVFP4
AA-LCR
pass@1 (%)
1,024
65.00
65.88
+0.88
+1.8%
Ultra
BF16
SWE-bench Verified
resolved (%)
2,048
70.00
69.07
-0.93
-4.2%
Ultra
NVFP4
SWE-bench Verified
resolved (%)
2,048
69.20
69.93
+0.73
-0.2%
Ultra
BF16
Terminal-Bench 2.0
pass@1 (%)
2,048
49.86
49.30
-0.56
-19.1%
Appendix
Table 6: Nano and Ultra capability results at fixed chunk budgets. Scores are percentages except GDPval, for which we report failure-adjusted Elo. Verbosity Δ is 100(LˉCISA/Lˉdense−1) , where Lˉ is mean generated tokens per rollout on the matched cohort. For multi-turn agents, rollout length sums generated tokens across all model calls. 2 Unavailable results reflect incomplete evaluations or unrecoverable token counts for the matched cohort. 3 GDPval includes all 220 tasks and two rollouts per task under the failure-adjustment rule below.
Figure 7: Unadjusted quality–throughput results for Ultra and Inkling-Small NVFP4. Quality scores and profiles match Figure 4 ; the horizontal axis shows measured total-token throughput relative to each model’s dense baseline, using MRCR200 with 4,096 output tokens per request. Lines connect increasing G within each budget profile. Table 7 reports the exact coordinates.
Profile
G
Mean [95% CI]
Measured Sp
ΔL (%)
Adjusted Sadj,p
Ultra NVFP4
Dense
–
48.04 [45.75, 50.30]
1.000×
+0.0
1.000×
Base
4
47.68 [45.39, 49.93]
2.285×
+17.9
1.939×
Reduced
4
45.40 [43.14, 47.64]
2.615×
+34.1
1.950×
Base
8
46.77 [44.48, 49.02]
2.594×
+13.9
2.277×
Inkling-Small NVFP4
Appendix
Table 7: Operating points for Figures 4 and 7 . Mean is the equally weighted four-benchmark score (%) with its pointwise 95% task-bootstrap interval. Sp is measured fixed-output throughput relative to dense; ΔL=100(Rp−1) is the mean benchmark-relative output-length change. Sadj,p=Sp/Rp is the generation-cost sensitivity proxy. Base budgets are {4096,4096,4096,1024} chunks for RULER 1M, MRCR200, GraphWalks-200, and AA-LCR, respectively; reduced budgets halve these values.
Profile
RULER
MRCR200
GraphWalks-200
AA-LCR
Mean
Dense
94.53
9.21
24.22
64.19
48.04
G=4 , base
93.76
9.99
22.43
64.56
47.68
G=4 , reduced
91.88
10.48
16.99
62.25
45.40
G=8 , base
92.69
9.81
20.69
63.88
46.77
Appendix
Table 8: Ultra NVFP4 benchmark scores underlying the operating-point means in Table 7 . Scores use each benchmark’s native percentage metric; the mean weights all four equally.
(a) Quality
Setting
k (chunks)
Score
95% CI
Dense
–
24.22
[20.02,28.64]
G=1
2,048
18.38
[14.67,22.30]
G=4
2,048
16.99
[13.44,20.73]
G=4
4,096
22.43
[18.32,26.73]
G=8
4,096
20.69
[16.61,24.90]
Appendix
Table 9: Ultra NVFP4 results with and without query sharing. (a) GraphWalks-200 node-set F1 (%) with pointwise 95% intervals from 20,000 task-bootstrap resamples (seed 20260901). (b) MRCR200 total-token throughput on four B300 GPUs in TP2DP2 with concurrency 32 and 4,096 generated tokens per request. The G=1 throughput result is provisional and uses one allocation; other rows average three, with the observed range shown.
Figure 8: Offline routing diagnostics on 64 held-out 131K-token Ultra documents ( kchunks=1,024 , 16-token pages). (a) Dense-attention-mass loss when one selection with W=128 is shared across G adjacent queries over the final 2,048 positions. (b) Mass gain relative to W=0 when recent pages are added to kchunks selected pages or replace the same number at a fixed total, sampling within the final 8,192-position region. Whiskers are pointwise 95% document-bootstrap intervals (2,000 draws, seed 42); window differences are paired with W=0 . These are mechanism diagnostics, not benchmark-quality or throughput measurements.
Local model shape
Phase
Metric
B=16
B=24
B=32
Geom. mean
Nano TP1 ( HKV=2 )
Terminal prefill
GPU busy
4.49×
4.57×
4.56×
4.54×
Core attention
5.95×
6.40×
6.07×
6.13×
Steady decode
GPU busy
2.72×
3.24×
3.39×
3.11×
Core attention
12.37×
13.18×
12.94×
12.83×
Ultra TP2/rank ( HKV=1 )
Terminal prefill
GPU busy
8.61×
8.09×
8.61×
8.43×
Core attention
11.63×
11.57×
11.70×
11.63×
Appendix
Table 10: CISA speedup over matched dense GQA at one million tokens (higher is better). The geometric mean is across the three executed batch sizes. GPU busy is the union of kernel, memcpy, and memset intervals, with overlaps counted once. Core attention includes only the FlashAttention forward and combine kernels; GPU busy also includes projections, scoring, selection, metadata, and copies.
Phase
Variant
Attention
Scoring
Top- k
Setup
Other
GPU busy
Terminal prefill
Dense
14.176
0.000
0.000
0.018
0.912
15.106
Terminal prefill
CISA G4
9.154
1.174
1.004
0.152
0.972
12.456
Terminal prefill
CISA G16
2.626
0.589
0.325
0.069
0.968
4.577
Terminal prefill
CISA G32
1.198
0.587
0.147
0.060
0.962
2.954
Steady decode
Dense
1.218
0.000
0.000
0.019
0.092
1.329
Steady decode
CISA
0.104
0.536
0.118
0.047
0.112
0.918
Appendix
Table 11: Inkling-Small attention-path timings for Figure 2 . Values are mean milliseconds over ten forwards from one capture. GPU busy counts overlapping kernel and memory-operation intervals once and excludes idle gaps. Component sums may differ from totals because of rounding.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Department of Computer Science, Rice University, TX, USA. · Department of Electrical Engineering and Computer Sciences, UC Berkeley, CA, USA. · Department of Computer Science, Stanford University, CA, USA. +1