cs.LGNov 9, 2026

A Geometric Invariant of Attention Predicts Head Specialization from Weights Alone

Authors: Miguel P. Bento

Organizations: Feedzai, Lisbon, Portugal

Abstract

Transformers are built with continuous symmetries in the attention mechanism. We derive ρρ, a scalar invariant that measures each head's capacity to couple to rotary positional encodings. This framework allows us to analyze heads without forward passes, opening a new research direction into data-independent analysis of modern LLMs. Empirically, we find that ρρ predicts induction heads, the most-studied circuit type in mechanistic interpretability, with Spearman correlation between −0.55-0.55 and −0.74-0.74 across four bias-free RoPE models spanning 135135M to 77B parameters. As a preliminary proof of concept, we use ρρ to apply YaRN to only the top-50% positional heads, which degrades perplexity less than uniform application. Code is available at https://github.com/MPedraBento/rho-invariant.

Figures & tables

Explore similar work

CardsList
  1. Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalization

    May 29, 2026Felipe Urrutia, Juan José Alegría, Cinthia Sanchez Macias +3Rotary Positional EmbeddingsAttention Head Analysis

  2. Quantifying LLM Attention-Head Stability: Implications for Circuit Universality

    Feb 17, 2026Karan Bali, Jack Stanley, Praneet Suresh +1Transformer InterpretabilityAttention Head Analysis