A Geometric Invariant of Attention Predicts Head Specialization from Weights Alone
Organizations: Feedzai, Lisbon, Portugal
Abstract
Transformers are built with continuous symmetries in the attention mechanism. We derive , a scalar invariant that measures each head's capacity to couple to rotary positional encodings. This framework allows us to analyze heads without forward passes, opening a new research direction into data-independent analysis of modern LLMs. Empirically, we find that predicts induction heads, the most-studied circuit type in mechanistic interpretability, with Spearman correlation between and across four bias-free RoPE models spanning M to B parameters. As a preliminary proof of concept, we use to apply YaRN to only the top-50% positional heads, which degrades perplexity less than uniform application. Code is available at https://github.com/MPedraBento/rho-invariant.
Figures & tables
| Model | Params | Spearman | top-10 within |
|---|---|---|---|
| SmolLM2-135M | 135M | bottom 14% | |
| TinyLlama-1.1B | 1.1B | bottom 7% | |
| SmolLM2-1.7B | 1.7B | bottom 17% | |
| Mistral-7B | 7B | bottom 30% |
| Method | Short PPL (512) | Long PPL (4096) |
|---|---|---|
| Baseline ( ) | 7.63 | 5.44 |
| YaRN (all heads) | 19.37 | 162.96 |
| YaRN + Targeted (ours) | 12.92 | 130.34 |
Attention Head Analysis
Attention head analysis studies how individual Transformer heads route information, support computations, and affect model behavior.
See all 87 papers