cs.LGFeb 10, 2026

Laplacian Heads Improve Transformers by Smoothing Token Representations

Authors: Yuchong Zhang, Vardan Papyan

Organizations: University of Toronto, Vector Institute

Abstract

Transformers update token representations through multi-head attention and residual connections: X←X+∑iP(i)Norm(X)WViWoiX \leftarrow X + \sum_{i} P^{(i)}\text{Norm}(X)W_{V_i}W_{o_i}, where P(i)P^{(i)} is the softmax probability matrix in head ii. We propose replacing some P(i)P^{(i)}'s with I−P(i)I - P^{(i)}, giving X←X+∑i∈AP(i)Norm(X)WViWoi+∑i∈L(I−P(i))Norm(X)WViWoiX \leftarrow X + \sum_{i \in \mathcal{A}} P^{(i)}\text{Norm}(X)W_{V_i}W_{o_i} + \sum_{i \in \mathcal{L}} (I - P^{(i)})\text{Norm}(X)W_{V_i}W_{o_i}. We call heads that use I−P(i)I - P^{(i)} Laplacian heads. Our proposal has two motivations. First, it allows attention heads to update the mean of token representations, while Laplacian heads can directly control within-sequence variance. Second, if tokens are viewed as nodes in a graph with normalized edge weights P(i)P^{(i)}, then I−P(i)I - P^{(i)} is a graph Laplacian operator, and the update can be interpreted as one step of heat diffusion on the graph. We show that this simple modification improves performance across supervised learning, causal language modeling, and vision self-supervised learning tasks. To investigate why, we examine the token representations learned with and without Laplacian heads. In supervised learning, Laplacian heads collapse token representations within the same sequence and align the sequence means with the geometry of Neural Collapse. In language modeling, they increase the separability of token representations that share the same next-token prediction. In vision self-supervised learning, they produce token representations whose top principal components better capture segmentation structure. Across modalities, they also lead to faster-decaying spectra, indicating stronger token smoothing. Overall, our findings challenge the prevailing view that token oversmoothing is inherently harmful, showing instead that certain forms of smoothing can be beneficial.

Figures & tables

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Oct 9, 2026

LapMerge: Differentiable Exact-Budget Token Merging in Linear Time

Modern vision and language models process ever longer token sequences, and reducing them is the main lever on downstream cost. Token compression, however, faces a fundamental trade-off: selecting tokens discards information, while existing merging methods preserve it only through pairwise token-to-token matching or MNMN token-to-output interactions. We introduce LapMerge, a differentiable merging operator that resolves this trade-off for ordered sequences: a learned mass distribution partitions the sequence into exactly NN equal-mass segments under a Laplace-kernel density, and prefix/suffix scans aggregate each segment, so both the forward and the backward pass run in O(M+N)O(M+N) time. This combination (an exact output budget, end-to-end differentiability, and linear cost) is precisely what previous reducers trade off against one another. The budget can be changed without changing operator parameters, and the learned mass doubles as a token-level map of where the budget is allocated. Across vision and vision-language tasks, LapMerge matches or improves on strong token reducers while being up to two orders of magnitude faster at large sequence lengths; it is particularly effective when token order reflects spatial or temporal locality, and works both as a drop-in reduction module and as a trainable architectural component.
Apr 22, 2026

LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel

The quadratic complexity of softmax attention presents a major obstacle for scaling Transformers to high-resolution vision tasks. Existing linear attention variants often replace the softmax with Gaussian kernels to reduce complexity, but such approximations lack theoretical grounding and tend to oversuppress mid-range token interactions. We propose LaplacianFormer, a Transformer variant that employs a Laplacian kernel as a principled alternative to softmax, motivated by empirical observations and theoretical analysis. To address expressiveness degradation under low-rank approximations, we introduce a provably injective feature map that retains fine-grained token information. For efficient computation, we adopt a Nyström approximation of the kernel matrix and solve the resulting system using Newton--Schulz iteration, avoiding costly matrix inversion and SVD. We further develop custom CUDA implementations for both the kernel and solver, enabling high-throughput forward and backward passes suitable for edge deployment. Experiments on ImageNet show that LaplacianFormer achieves strong performance-efficiency trade-offs while improving attention expressiveness.
Oct 15, 2026

Can Capability Enhancement be Localized?

Transformer-based language models have shown strong capabilities across many tasks and domains. However, prior work shows that, for a given task, not all attention heads matter equally. Specifically, they suggest that many redundant heads can be safely removed with little-to-no effect, while a small set of important heads, when ablated, severely damages the model's performance on the given task. In this work, we investigate a complementary aspect, called Capability Enhancement Localization (CEL): given a model and a task, does there exist a small set of heads whose removal improves the task performance? We first investigate this question empirically across models from different families. Using a simple two-stage routine that screens candidate heads and evaluates their joint ablations, we identify sparse head combinations whose removal improves task-specific performance without further fine-tuning. Motivated by these observations, we ask theoretically whether such a phenomenon can naturally arise from training dynamics, and under what conditions. To this end, we construct a simplified multi-head softmax attention model and two tasks with conflicting features, and show that gradient descent from small random initialization provably concentrates the conflicting features into particular heads. We further show that ablating such heads improves performance on one task while harming the other, even though training converges to a global optimum of the joint objective. This establishes a possible training-induced theoretical explanation for CEL.