DDCL-INCRT: A Self-Organising Transformer with Hierarchical Prototype Structure (Theoretical Foundations)

Giansalvo Cirrincione

cs.LGApr 2, 2026

DDCL-INCRT: A Self-Organising Transformer with Hierarchical Prototype Structure (Theoretical Foundations)

Authors: Giansalvo Cirrincione

Organizations: Laboratory LTI, Université de Picardie Jules Verne, Chemin du Thil, Amiens, 80025, France

Abstract

Modern neural networks of the transformer family require the practitioner to decide, before training begins, how many attention heads to use, how deep the network should be, and how wide each component should be. These decisions are made without knowledge of the task, producing architectures that are systematically larger than necessary: empirical studies find that a substantial fraction of heads and layers can be removed after training without performance loss. This paper introduces DDCL-INCRT, an architecture that determines its own structure during training. Two complementary ideas are combined. The first, DDCL (Deep Dual Competitive Learning), replaces the feedforward block with a dictionary of learned prototype vectors representing the most informative directions in the data. The prototypes spread apart automatically, driven by the training objective, without explicit regularisation. The second, INCRT (Incremental Transformer), controls the number of heads: starting from one, it adds a new head only when the directional information uncaptured by existing heads exceeds a threshold. The main theoretical finding is that these two mechanisms reinforce each other: each new head amplifies prototype separation, which in turn raises the signal triggering the next addition. At convergence, the network self-organises into a hierarchy of heads ordered by representational granularity. This hierarchical structure is proved to be unique and minimal, the smallest architecture sufficient for the task, under the stated conditions. Formal guarantees of stability, convergence, and pruning safety are established throughout. The architecture is not something one designs. It is something one derives.

Figures & tables

Result	Status	Source
Prototype anti-collapse guarantee	Inherited	[ 1 ]
Algebraic decomposition $\mathcal{L}_{q}=L_{\mathrm{OLS}}+V$	Inherited	[ 1 ]
Two-timescale Lyapunov (frozen architecture)	Inherited	[ 2 ]
Channel Destruction Theorem	Inherited	[ 2 ]
Spectral growth criterion	Inherited	[ 3 ]
Finite convergence (standalone INCRT)	Inherited	[ 3 ]

Table 1: Origin of the main results. Inherited : used without modification from cited work. Extended : strengthened or generalised here. New : proved for the first time in this paper.

Assump.	Informal meaning	First use	Required by
(A1)	Prototype bank non-degenerate: $\mathcal{S}(P)>0$	Sec. 4	Thm. 1 , 2 , 6
(A2’)	$\exists\,k,k^{\prime}$ : prototypes differ along $u^{*}$	Sec. 4	Thm. 1 , 2
(A3)	$T$ large enough for soft assignments	Sec. 4	Thm. 1
(B1)	Each head’s prototype bank non-degenerate	Sec. 5	Thm. 3 , 4 , 5
(B2)	Prototype subspaces mutually orthogonal	Sec. 5	Thm. 3 , 4
(B3)	Heads have distinct initial gate activities	Sec. 5	Thm. 3

Table 2: Complete list of assumptions. (H1)–(H5) inherited from [ 3 ] .

Panel of Figure 1 — Figure 1: Synthetic experiments 1–3. Left (Exp. 1): Spectral ordering and directional coverage: $\lambda^{(h)}$ decreasing in $h$ (bars match theoretical eigenvalues, circles); coverage fraction exceeds the theoretical lower bound $1-\varepsilon_{\theta_{w}}=0.899$ (dashed). Centre (Exp. 2): Temperature divergence: all eight heads converge to $T_{\min}=0.1$ in order of decreasing $\lambda^{(h)}$ ; Spearman rank correlation $=-1.00$ (prediction: $\leq-0.9$ ). Right (Exp. 3): Separation force monotonicity: fractional $\mathcal{F}^{(h)}$ strictly decreasing (bars); empirical ratios lie on the diagonal, confirming the bound of Lemma 27 .

Figure 2: Experiment 4 (Colab, BERT/SST-2, frozen encoder, 3 epochs). (a) Directional energy

\Gamma_{h}

per head: values range from 0.15 to 0.32, confirming head differentiation on real data. (b)

\max|\mathcal{S}(P^{(h^{\prime})})\text{ after}-\mathcal{S}(P^{(h^{\prime})})\text{ before}|=0

at all six pruning events (log scale; bars touch

10^{-15}

), verifying Theorem 6 (i) with machine precision. (c) Actual

F_{\rm sep}

drop (blue) equals

F_{\rm sep}

of the pruned head (red) exactly at every event (all labelled OK ), verifying Lemma 14 : prototype subspace orthogonality makes the drop decompose exactly.

Encoder architecture
Hyperparameter	Value
Embedding dimension $d$	64
Attention heads (initial)	2
Dropout	0.4
Weight decay	$10^{-3}$
DDCL-INCRT layer

Table 3: Hyperparameters for Experiment 5 (end-to-end SST-2).

Figure 3: Experiment 5 (end-to-end, SST-2,

d=64

\theta_{w}=0.8

\lambda_{\rm reg}=0.05

, 10 epochs). Left: training loss and validation accuracy over 10 epochs; final validation accuracy

69.4\%

. Centre: residual content

\lambda_{\max}(A_{\rm res})

at the five growth events, strictly decreasing (

4.36\to 2.38\to 2.15\to 1.35\to 0.84

), confirming the spectral ordering prediction. Right: separation force

F_{\rm sep}^{(h)}

(bars) and prototype spread

\mathcal{S}(P^{(h)})

(green curve) per head;

\mathcal{S}(P^{(h)})>0

for all heads confirms assumption (A1);

F_{\rm sep}

is partially ordered, with deviations attributable to end-to-end co-adaptation after growth events.

Exp.	Prediction	Metric	Theorem
1	$\lambda^{(h)}$ decreasing in $h$	Growth order	Sec. 12
1	Coverage $\geq 1-\varepsilon_{\theta_{w}}$	$\sum\lambda^{(h)}/\\|M_{a}\\|_{F}^{2}$	Sec. 12
2	$T^{(h)}_{\infty}$ increasing in $h$	Temperature order	Sec. 5
2	$t^{(h)}_{\min}$ decreasing in $\lambda^{(h)}$	Rank corr. $\leq-0.9$	Sec. 5
3	$\mathcal{F}^{(h)}$ decreasing in $h$	Bar chart	Sec. 12
3	Ratio bound saturated	Scatter vs. diagonal	Lem. 27

Table 4: Experimental predictions and corresponding theoretical results (Experiments 1–3 and 5). Experiment 4 results are tabulated in the caption of Figure 2 .

Transformer

Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.

See all 528 papers

INCRT: An Incremental Transformer That Determines Its Own Architecture
Apr 12, 2026Giansalvo CirrincioneTransformer Neural Architecture Search
Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France
Choosing the Number of Attention Heads from Intrinsic Dimension: A Small-Scale Study
Oct 14, 2026Arjun Boya, Charan Sai Bandaru, Masthanaiah Pillimetla +1Multi-Head Attention
Institute of Cognitive Science University of Osnabrück · Department of Social Sciences and Technology Technical University of Munich
Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing
Jun 25, 2026Shubham AggarwalHierarchical Representation Learning Transformer Attention

Apr 12, 2026J/K move · Enter open · S save

INCRT: An Incremental Transformer That Determines Its Own Architecture

Giansalvo Cirrincione

Laboratoire LTI, Université de Picardie Jules Verne, Amiens, France

Transformer Neural Architecture Search Attention Head Pruning

Open paper AskPDF arXiv

Transformer architectures are designed by trial and error: the number of attention heads, the depth, and the head size are fixed before training begins, with no mathematical principle to guide the choice. The result is systematic structural redundancy -- between half and four-fifths of all heads in a trained model can be removed without measurable loss -- because the architecture allocates capacity without reference to the actual requirements of the task.This paper introduces INCRT (Incremental Transformer), an architecture that determines its own structure during training. Starting from a single head, INCRT adds one attention head at a time whenever its current configuration is provably insufficient, and prunes heads that have become redundant. Each growth decision is driven by a single, online-computable geometric quantity derived from the task's directional structure, requiring no separate validation phase and no hand-tuned schedule. Two theorems form the theoretical backbone. The first (homeostatic convergence) establishes that the system always reaches a finite stopping configuration that is simultaneously minimal (no redundant heads) and sufficient (no uncaptured directional energy above the threshold). The second (compressed-sensing analogy) provides a geometric upper bound on the number of heads that this configuration can contain, as a function of the spectral complexity of the task. Experiments on SARS-CoV-2 variant classification and SST-2 sentiment analysis confirm both results: the predicted and observed head counts agree within 12% across all benchmarks, and the final architectures match or exceed BERT-base on distribution-specific tasks while using between three and seven times fewer parameters and no pre-training.

Abstract

Transformer

Explore similar work

INCRT: An Incremental Transformer That Determines Its Own Architecture

Choosing the Number of Attention Heads from Intrinsic Dimension: A Small-Scale Study

Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing