Attention in Constant Time: Vashista Sparse Attention for Long-Context Decoding with Exponential Guarantees
Organizations: Datar Consulting
Abstract
Large language models spend most of their inference cost on attention over long contexts, yet empirical behavior suggests that only a small subset of tokens meaningfully contributes to each query. We formalize this phenomenon by modeling attention as a projection onto the convex hull of key vectors and analyzing its entropic (softmax-like) relaxation. Our main theoretical contribution is a face-stability theorem showing that, under a strict complementarity margin (a support gap (Δ) certified by KKT multipliers), entropic attention concentrates on a constant-size active face: the total mass assigned to inactive tokens decays exponentially as (\exp(-Ω(Δ/\varepsilon))), while the error on the active face scales linearly in the temperature/regularization parameter (\varepsilon). This yields a practical criterion for when sparse long-context decoding is safe and provides a principled knob to trade accuracy for compute. Building on these guarantees, we introduce Vashista Sparse Attention, a drop-in mechanism that maintains a small candidate set per query through a paging-style context selection strategy compatible with modern inference stacks. Across long-context evaluations, we observe stable constant-size effective support, strong wall-clock speedups, and minimal quality degradation in the regimes predicted by the support-gap diagnostics. Finally, we discuss deployment implications for privacy-sensitive and air-gapped settings, where interchangeable attention modules enable predictable latency and cost without external retrieval dependencies.
Figures & tables
| Symbol | Meaning |
|---|---|
| Number of tokens/atoms (dictionary size) | |
| Embedding dimension | |
| Dictionary / value vectors (columns) | |
| Query-dependent target vector | |
| Probability simplex | |
| Euclidean projection objective |
| Context | Dense ms | Sparse ms | Dense/Sparse |
|---|---|---|---|
| 8192 | 0.80 | 21.89 | 0.04 |
| 16384 | 1.65 | 22.06 | 0.07 |
| 32768 | 3.36 | 22.58 | 0.15 |
| 65536 | 6.80 | 23.38 | 0.29 |
| 131072 | 13.67 | 24.72 | 0.55 |
| P | Solver | Dense ms | Sparse ms | Dense/Sparse | |
|---|---|---|---|---|---|
| 32 | 64 | eg | 6.71 | 12.47 | 0.54 |
| 32 | 128 | eg | 6.79 | 12.46 | 0.54 |
| 32 | 192 | eg | 6.58 | 12.63 | 0.52 |
| 64 | 64 | eg | 6.85 | 23.26 | 0.29 |
| 64 | 128 | eg | 6.37 | 23.29 | 0.27 |
| 64 | 192 | eg | 6.76 | 23.38 | 0.29 |
Sparse Attention
Sparse attention reduces the token interactions computed by Transformers to lower the cost of long-context modeling and inference.
See all 271 papers