To mitigate the quadratic complexity bottleneck of the Transformer, sparse attention has emerged as a pivotal technology. Despite the extensive empirical success of sparse Transformers, the theoretical understanding of sparse attention remains fragmented. In particular, two fundamental questions remain unclear: (1) How does sparsification affect the information fidelity of attention mechanisms? (2) How does this information loss interact with the generalization behavior of the model? To bridge this gap, this paper proposes a systematic analysis of the Jensen-Shannon (JS) divergence and of the generalization gap of sparse attention mechanisms. Specifically, we first characterize the approximation error via the JS divergence. Through an order-statistics-based concentration analysis of the truncation mass alpha --- where the attention scores are assumed to be independent and identically distributed sub-Gaussian random variables with parameter sigma --- the JS divergence between the full attention distribution and the sparse attention distribution is shown to admit the closed form log 2 + ((1 - alpha)/2) log(1 - alpha) - ((2 - alpha)/2) log(2 - alpha). Subsequently, we derive a generalization bound through Rademacher complexity, quantified by O(gamma * sqrt(M/n) * (sqrt(log(3eL/M)) + sqrt(pi)/2)). Furthermore, building on a mutual-information-based generalization bound together with an entropy and covering-number analysis of the sparse hypothesis class, we obtain the sparsity-dependent generalization bound O(sqrt((M/(2n)) * (log(eL/M) + log(1 + 2/epsilon)))). Our analysis shows that sparsity reduces the complexity of the considered hypothesis class while introducing approximation error that can be quantified by the JS divergence. These findings provide a theoretical characterization of the trade-off between information fidelity and generalization in sparse Transformer architectures.
Figures & tables
Figure 1: Upper bound of R versus the sequence length L
Figure 2: Verification of Theorem 3.5 on GPT-2 attention. (a) Empirical DJS(pi∥qi) against the closed form f(α) , over 2.7×105 attention rows with α∈[0,0.943] . (b) Histogram of the absolute difference; the maximum is 1.5×10−7 .
Figure 3: Truncation mass α (a) and information loss DJS=f(α) (b) versus the sequence length L , for six sparsity strategies with their own budget M(L) . Error bars are ±1 standard deviation over 12×12×8 attention rows. The dashed line in (a) is the lower bound on α implied by Theorem 3.8 .
Figure 4: Generalization bound versus the sparsity degree M .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The JS divergence f(α) as a function of the discarded mass α .
Figure 6: Distribution of BERT self-attention Q/K spectral norms.
Figure 7: Distribution of BERT word-embedding norms.