Toward Manifest Relationality in Transformers via Symmetry Reduction

J. François; L. Ravera

cs.LGFeb 21, 2026

Toward Manifest Relationality in Transformers via Symmetry Reduction

Authors: J. François, L. Ravera

Organizations: Mons University (UMONS), 20 Place du Parc, 7000 Mons, Belgium. · Grupo de Investigación en Física Teórica, Universidad Católica De La Santísima Concepción, Alonso de Ribera 2850, Concepción, Chile.

Abstract

Transformer models contain substantial internal redundancy arising from coordinate-dependent representations and continuous symmetries, in model space and in head space, respectively. While recent approaches address this by explicitly breaking symmetry, we propose a complementary framework based on symmetry reduction. We reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction. This perspective yields architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.

Figures & tables

Figure 1: Schematic summary of the main content of the work.

Redundancy type	Invariant variable
Model-space frame	$G=XX^{\top}$ , $XAX^{\top}$ , $\hat{X}=Xu[X]^{-1}$
QK head-space symm.	$G_{QK}=W_{Q}^{\top}W_{K}$
VO head-space symm.	$G_{VO}=W_{O}W_{V}$

Table 1: Redundancies and corresponding invariants.

Explore similar work

A Smaller Transformer in Your Transformer
Sep 17, 2026Dhananjay Tomar, Marius Aasan, Andreas Kleppe +1Transformer Inference Vision Transformer
Institute for Cancer Genetics and Informatics Oslo University Hospital Oslo, Norway · Department of Informatics University of Oslo Oslo, Norway · SFI Visual Intelligence UiT The Arctic University of Norway Tromsø, Norway
How Do Transformers Learn to Represent Symmetries?
Oct 7, 2026Eduardo Santos-Escriche, Valerie Engelmayer, Ya-Wei Eileen Lin +1Invariant Representation Learning Transformer
Technical University of Munich Munich Center for Machine Learning · Massachusetts Institute of Technology
GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression
Sep 22, 2026Baher Mohammad, Ammar Ali, Stamatios LefkimmiatisModel Compression Low-Rank Compression
MWS AI, ITMO University · MWS AI

Sep 17, 2026J/K move · Enter open · S save

A Smaller Transformer in Your Transformer

Dhananjay Tomar, Marius Aasan, Andreas Kleppe, Adín Ramírez Rivera

Institute for Cancer Genetics and Informatics Oslo University Hospital Oslo, Norway · Department of Informatics University of Oslo Oslo, Norway · SFI Visual Intelligence UiT The Arctic University of Norway Tromsø, Norway

Transformer Inference Vision Transformer Efficient ViTs Neural Network Compression

Open paper AskPDF arXiv

Recent findings indicate that Vision Transformers settle into locally similar computational phases, implying a level of depthwise computational redundancy. However, existing methods to exploit this redundancy either fail to reduce inference compute or severely degrade model expressivity. In this work, we formalise a unified view of block redundancy that decouples the geometry from specific surrogate interventions. We then introduce Transformer-Within-Transformer (TWT), a post-hoc method that fuses contiguous groups of redundant layers into a single learned surrogate layer. TWT reduces parameter count and inference compute while remaining competitive with original models using half the depth on natural images, and in several downstream histopathology settings, TWT matches or even improves on the original baseline.

Abstract

Explore similar work

A Smaller Transformer in Your Transformer

How Do Transformers Learn to Represent Symmetries?

GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compression