Laplacian Heads Improve Transformers by Smoothing Token Representations
Organizations: University of Toronto, Vector Institute
Abstract
Transformers update token representations through multi-head attention and residual connections: , where is the softmax probability matrix in head . We propose replacing some 's with , giving . We call heads that use Laplacian heads. Our proposal has two motivations. First, it allows attention heads to update the mean of token representations, while Laplacian heads can directly control within-sequence variance. Second, if tokens are viewed as nodes in a graph with normalized edge weights , then is a graph Laplacian operator, and the update can be interpreted as one step of heat diffusion on the graph. We show that this simple modification improves performance across supervised learning, causal language modeling, and vision self-supervised learning tasks. To investigate why, we examine the token representations learned with and without Laplacian heads. In supervised learning, Laplacian heads collapse token representations within the same sequence and align the sequence means with the geometry of Neural Collapse. In language modeling, they increase the separability of token representations that share the same next-token prediction. In vision self-supervised learning, they produce token representations whose top principal components better capture segmentation structure. Across modalities, they also lead to faster-decaying spectra, indicating stronger token smoothing. Overall, our findings challenge the prevailing view that token oversmoothing is inherently harmful, showing instead that certain forms of smoothing can be beneficial.
Figures & tables
| Model | CIFAR10 | CIFAR100 | ImageNet |
|---|---|---|---|
| Baseline (no Laplacian) | |||
| 3 Laplacian heads | |||
| 6 Laplacian heads | |||
| 9 Laplacian heads | |||
| 11 Laplacian heads | |||
| 12 Laplacian heads |
| Task | ViT-S | 1L | 3L | 5L | 6L |
|---|---|---|---|---|---|
| ImageNet Top-1 | |||||
| ADE20K mIoU |
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
| CIFAR-10 | CIFAR-100 | ImageNet | |
| Loss | Cross Entropy | Cross Entropy | Binary Cross Entropy |
| Optimizer | AdamW | AdamW | LAMB |
| AdamW | 0.9 | 0.9 | 0.9 |
| AdamW | 0.999 | 0.999 | 0.999 |
| Starting Learning Rate | 1e-6 | 1e-6 | 1e-6 |
| Peak Learning Rate | 3e-4 | 3e-4 | 3e-3 |
| Hyperparameter | 135M | 477M | 1.45B |
|---|---|---|---|
| Blocks / width / heads | |||
| Training tokens (billions) | |||
| Optimizer steps | |||
| Global batch size (tokens) | |||
| Peak LR: transformer (Muon) | |||
| Peak LR: embedding (AdamW) |
| Hyperparameter | Value |
|---|---|
| Architecture | |
| Backbone | ViT-S/16 |
| Position embedding | RoPE (base ) |
| Drop path rate | |
| LayerScale initialization | |
| Precision | BF16 parameters / FP32 gradient reduction |
Transformer
Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.
See all 574 papers