A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts
Organizations: ⋄Hanoi University of Science and Technology · ♢Purdue University · ‡Trivita AI · †The University of Texas at Austin
Abstract
Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to improve performance, a gated attention model that leverages a gating mechanism within the multi-head self-attention has recently been proposed as a promising alternative. Gated attention has been empirically demonstrated to increase the expressiveness of low-rank mapping in standard attention and even to eliminate the attention sink phenomenon. Despite its efficacy, a clear theoretical understanding of gated attention's benefits remains lacking in the literature. To close this gap, we rigorously show that each entry in a gated attention matrix or a multi-head self-attention matrix can be written as a hierarchical mixture of experts. By recasting learning as an expert estimation problem, we demonstrate that gated attention is more sample-efficient than multi-head self-attention. In particular, while the former needs only a polynomial number of data points to recover the model parameters, the latter requires exponentially many data points to achieve the same estimation error. Furthermore, our analysis also provides a theoretical justification for why gated attention is more effective at mitigating the attention sink phenomenon and the low-rank bottleneck when a gate is placed at the output of the scaled dot product attention or the value map rather than at other positions in the multi-head self-attention architecture.
Figures & tables
| Mechanisms | Expert estimation rates | Sample complexity | Theorems |
|---|---|---|---|
| Multi-head self-attention | Slower than any polynomial order | Theorem 1 | |
| Gated attention (Setting I) | to | Theorem 2 | |
| Gated attention (Setting II) | to | Theorem 3 |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Head | Expert | Channel | ||
|---|---|---|---|---|
| 0 | 0 | 0 | 1.0 | |
| 0 | 0 | 1 | 0.5 | |
| 0 | 1 | 0 | 1.0 | |
| 0 | 1 | 1 | 0.5 | |
| 1 | 0 | 0 | 0.8 | |
| 1 | 0 | 1 | 0.3 |
| Head | Expert | Channel | ||
|---|---|---|---|---|
| 0 | 0 | 0 | 1.0 | |
| 0 | 0 | 1 | 0.5 | |
| 0 | 0 | 2 | 0.6 | |
| 0 | 1 | 0 | 1.0 | |
| 0 | 1 | 1 | 0.5 | |
| 0 | 1 | 2 | 0.6 |
| Configuration | Value |
|---|---|
| Transformer layers / hidden size | / |
| Attention heads / head dimension | / |
| Feed-forward experts / active experts | / |
| Expert hidden dimension | |
| Parameters | |
| Context length / vocabulary size | / |
| Tokens (M) | Attention | Perplexity | HellaSwag | CBT | BLiMP | RACE |
|---|---|---|---|---|---|---|
| 327.68 | Control | |||||
| VG | ||||||
| SG | ||||||
| 655.36 | Control | |||||
| VG | ||||||
| SG |
Gated Attention
Gated attention uses learned, often input-dependent controls to regulate which tokens, heads, or features contribute to attention.
See all 37 papers