cs.LGFeb 10, 2026

Laplacian Heads Improve Transformers by Smoothing Token Representations

Authors: Yuchong Zhang, Vardan Papyan

Organizations: University of Toronto, Vector Institute

Abstract

Transformers update token representations through multi-head attention and residual connections: X←X+∑iP(i)Norm(X)WViWoiX \leftarrow X + \sum_{i} P^{(i)}\text{Norm}(X)W_{V_i}W_{o_i}, where P(i)P^{(i)} is the softmax probability matrix in head ii. We propose replacing some P(i)P^{(i)}'s with I−P(i)I - P^{(i)}, giving X←X+∑i∈AP(i)Norm(X)WViWoi+∑i∈L(I−P(i))Norm(X)WViWoiX \leftarrow X + \sum_{i \in \mathcal{A}} P^{(i)}\text{Norm}(X)W_{V_i}W_{o_i} + \sum_{i \in \mathcal{L}} (I - P^{(i)})\text{Norm}(X)W_{V_i}W_{o_i}. We call heads that use I−P(i)I - P^{(i)} Laplacian heads. Our proposal has two motivations. First, it allows attention heads to update the mean of token representations, while Laplacian heads can directly control within-sequence variance. Second, if tokens are viewed as nodes in a graph with normalized edge weights P(i)P^{(i)}, then I−P(i)I - P^{(i)} is a graph Laplacian operator, and the update can be interpreted as one step of heat diffusion on the graph. We show that this simple modification improves performance across supervised learning, causal language modeling, and vision self-supervised learning tasks. To investigate why, we examine the token representations learned with and without Laplacian heads. In supervised learning, Laplacian heads collapse token representations within the same sequence and align the sequence means with the geometry of Neural Collapse. In language modeling, they increase the separability of token representations that share the same next-token prediction. In vision self-supervised learning, they produce token representations whose top principal components better capture segmentation structure. Across modalities, they also lead to faster-decaying spectra, indicating stronger token smoothing. Overall, our findings challenge the prevailing view that token oversmoothing is inherently harmful, showing instead that certain forms of smoothing can be beneficial.

Figures & tables

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LapMerge: Differentiable Exact-Budget Token Merging in Linear Time

    Oct 9, 2026Magdalena Trędowicz, Aleksandra Nowak, Tomasz Steifer +3Token MergingVisual Token Merging

  2. LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel

    Apr 22, 2026Zhe Feng, Sen Lian, Changwei Wang +5Nyström MethodEfficient ViTs

  3. Can Capability Enhancement be Localized?

    Oct 15, 2026Anh Tuan Nguyen, Anna Bair, Zico KolterAttention Head AnalysisMulti-Head Attention