All You Need Is Non-Commutative Words

Carla M. Quispe Flores; Stanley Salvatierra; Renan Cabrera

cs.CLAug 29, 2026

All You Need Is Non-Commutative Words

Authors: Carla M. Quispe Flores, Stanley Salvatierra, Renan Cabrera

Organizations: Department of Physics, Colorado School of Mines, Golden, Colorado 80401, USA · Lookia MX, OR 0401, BO · Yardley, PA 19067, USA

Abstract

We represent lexical tokens as unitary matrices and encode each sentence as their ordered product. The noncommutativity of matrix product captures word order without positional encodings (PEs). The same algebra yields several capabilities, including antisymmetric self-attention with no query, key, or value projections, and parallel composition of variable-length text chunks at a reduced attention cost. Furthermore, it provides a canonical-coset readout layer that encodes all true unitary degrees of freedom compactly, while supporting continual learning through nested group extensions that enlarge the operator space with each new task preserving prior representations exactly. Across standard text-classification benchmarks, the method matches or exceeds bag-of-words baselines. Achieving higher accuracy on IMDB and comparable performance on AG News. Notably, this is accomplished by replacing the conventional $\sim$ 30,000-dimensional vocabulary space with a dense, 64-parameter real-valued encoding, highlighting the expressive efficiency of our parameterization.

Figures & tables

Figure 1: Word order is captured natively by representing each word as a unitary matrix and composing the sentence via their ordered product. Because matrix multiplication is noncommutative, the resulting document state $P_{L}$ preserves token ordering without positional encodings (PEs). A flattened readout followed by a single linear head then maps this state to the target label.

Figure 2: Conventional self-attention mechanics (blue) versus our QKV-Free Self-Attention (orange). The former needs positional encodings and the projections $W_{Q}$ , $W_{K}$ , $W_{V}$ to score token pairs; the latter scores unitary prefix products directly through one Hermitian matrix $A$ , with order supplied by non-commutativity and antisymmetry of $S_{ij}$ built in.

Description	IMDB	AG News
Global $\varepsilon=2.2$	85.28%	87.45%
Predicted $\varepsilon_{w}$ as in Eq.( 5 )	86.53%	87.68%
BoW baseline	85.25%	87.45%

Table 1: Accuracy comparison of adaptive per-word rotation budget arms on the IMDB and AG News datasets.

Model	IMDB	AG News
QKV-Free Self-Attention Eq.( 7 )	84.4%	87.1%
Matched transformer	84.6%	84.5%
PaTH-lite [ 31 ]	83.4%	84.1%
BoW baseline	85.3%	87.6%

Table 2: Accuracy comparison of attention variants on the IMDB and AG News datasets.

Configuration	Mean accuracy	Rel. score-pair cost
$c=1$ (flat case)	85.7%	1
$c=16$ (naive full $\varepsilon$ )	51.7%	1/256
$c=16$ (wrap-safe $\varepsilon/c$ )	85.8%	1/256
$c=64$ (wrap-safe $\varepsilon/c$ )	85.6%	1/4096
BoW baseline	84.1%	—

Table 3: Chunked attention on the IMDB dataset (full reviews, capped at 1024 words). Wrap-safe composites are built at rotation budget

\varepsilon/c

; at the full

\varepsilon

the composite aliases past

2\pi

and collapses to chance.

Representation	IMDB	AG News
Flatten-128	82.6%	84.9%
Coset-64	79.4%	84.8%

Table 4: Effect of the readout representation on mean accuracy at a fixed rotation budget

\varepsilon=0.15

: the canonical-coset readout (64 parameters) against the standard flattened 128-dimensional readout

Coset tower
Stage (trainable block)	Trainable coords.	Task A	Task B
1. Train A in $U(8)$ , freeze ( $H_{A}$ )	64	83.5%	—
2. Embed in $U(12)$ , train B ( $M,H_{k}$ )	80	83.5%	69.2%
3. Delete B: $(M,H_{k})\to 0$ (none)	0	83.5%	28.8%
Reference points
Standard finetuning on B (all of $H$ )	144	53.2%	—

Table 5: Coset-tower continual learning on a two-task sequence,

U(8)\subset U(12)

(shell

k=4

2nk+k^{2}=80

new real coordinates by Eq. ( 12 )). Task A is trained first and then frozen:

H_{A}

receives zero gradient thereafter, and task B is trained entirely inside the shell

(M,H_{k})

of Eq. ( 16 ). Deletion is the linear projection

(M,H_{k})\to 0

of Section II.7 , not a retraining step. Accuracy on A is unchanged at every stage, whereas standard finetuning of the same model on B, with all of

H

free, costs

30.3

pp on A and lands at chance. Trainable coords. counts the real coordinates of the generator of Eq. ( 16 ) that receive gradient at that stage, per word operator:

n^{2}=64

for

H_{A}

2nk+k^{2}=80

for the shell

(M,H_{k})

, and

(n+k)^{2}=144

for all of

H

. The tower thus trains

80

144

coordinates and never moves the

64

that carry task A. Dashes mark stages where the task is undefined: B has not yet been added (stage 1), or has been deleted (stage 3). Chance is

50\%

on A and

25\%

on B. Single seed.

Panel of Figure 3 — Figure 3: Coset-Tower and Continual Learning: The $\mathrm{U}(8)$ block trained on domain A (blue, inner ring) is frozen and embedded in $\mathrm{U}(12)$ ; domain B is learned in the new $80$ -coordinate shell (orange) without changing the inner block, such that A is preserved exactly ( $83.5\%\!\to\!83.5\%$ ). Whereas, standard finetuning erase it ( $\to 53.2\%$ ). Projecting the shell back to zero deletes B ( $\to$ chance) and leaves A unchanged.

Compositional Representation Learning

Compositional representation learning studies how models encode reusable parts and relations to understand combinations they have not seen.

See all 86 papers

Commutator Memory: Sparse, Path-Local Reading and Steering in Language Models
Sep 28, 2026John SweeneyLLM Training
Sideplane AI
Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers
Sep 26, 2026Ilya Koziev, Ivan OseledetsTransformer LLM Inference Acceleration
DAIMLD, Moscow, Russia · Artificial Intelligence Research Institute, Moscow, Russia
Why Attend to Everything? Focus is the Key
Mar 12, 2026Hengshuai Yao, Xing Chen, Ahmed Murtadha +8Attention Mechanisms Efficient Attention
Sapient Intelligence · University of Alberta · Tsinghua University

Sep 28, 2026J/K move · Enter open · S save

Commutator Memory: Sparse, Path-Local Reading and Steering in Language Models

John Sweeney

Sideplane AI

LLM Training

Open paper AskPDF arXiv

Gradient updates on different data generally do not commute: training a language model on two data sources in opposite orders gives different weights, even with the same data and total exposure. Loss or benchmark deltas show that the models differ, not where. We ask whether this path dependence leaves a parametric training-history memory: a weight component that flips sign when the two sources are swapped, is localized in output space, changes the held-out loss gap between the two orders under targeted interventions, and reveals which trained model came from which order. For one small SGD step of size

η

on each of sources

A

and

B

, the weight difference

θ_{AB}-θ_{BA}

is, to leading order,

η^2 b_{AB}

, where

b_{AB}=H_Bg_A-H_Ag_B

is the Lie bracket of the two gradient fields at the base model. We define commutator memory by projecting the bracket through the logits into one score per vocabulary token; the scores sum to the bracket's prediction of the gap. The scores are localized: on three models, the same readout of the measured

θ_{AB}-θ_{BA}

, or of a bracket from disjoint batches, shares 82-99% of the original top-20 tokens, versus 35-49% for norm-matched random directions. They are causally actionable: in Qwen-3-4B SFT, downweighting the ten tokens with the largest predicted share of the gap closes a median 32% of the measured gap, while frequency-matched tokens with near-zero scores have almost no effect. The weights themselves carry the component: projecting the difference between the two trained models onto

b_{AB}

identifies which came from which order in 92% of cases across four LLMs (chance 50%). Controlled tests also cover matched-batch DPO, a frozen-rollout GRPO-style objective, and an AdamW endpoint check. The memory is defined per source pair, not per example, and its projection on

b_{AB}

decays with further training.

Abstract

Compositional Representation Learning

Explore similar work

Commutator Memory: Sparse, Path-Local Reading and Steering in Language Models

Change the Product, Keep the Parameters: Associative Algebra Layers for Transformers

Why Attend to Everything? Focus is the Key