Tracking States or Tracking Cosets? An Algebraic Account of Learned State Tracking

Zhiyu Zhang; Yupeng Li

cs.LGSep 24, 2026

Tracking States or Tracking Cosets? An Algebraic Account of Learned State Tracking

Authors: Zhiyu Zhang, Yupeng Li

Organizations: Conflux Labs Ltd · Michigan State University

Abstract

State tracking requires composing a sequence of updates, but accuracy alone does not reveal what a model has learned. We study neural networks trained to predict the running product of group elements. We identify quotient solutions in Transformers, where models recover the quotient class while predicting nearly uniformly among its members. The reciprocal of class size predicts partial accuracy without a fitted parameter, extending parity-based accounts to non-parity quotients. Our baseline Transformers' predictions change little under prefix reordering beyond the exact-tracking frontier. We prove that, for finite groups under uniform i.i.d. full-group inputs, optimal order-blind exact accuracy converges to the reciprocal of abelianization class size as prefix length grows, consistent with the observed abelianization plateaus. Sequential updates permit more: any partition into right cosets of a subgroup, normal or not, survives sequential updates. In our census of standard Transformers, every recovered coset partition comes from a normal subgroup, whereas parameter-matched recurrent networks pass through both normal and non-normal right-coset stages during training. On $A_5$ , we identify low-dimensional subspaces of the recurrent state that encode non-normal cosets. In the three-dimensional cases, coset mean vectors form approximate dodecahedra, and swapping the state components in these subspaces transfers the donor's coset state through a shared input suffix. Our results connect partial accuracy, learning stages, and internal computation through the subgroup cosets that models learn to track.

Figures & tables

Figure 1: Models track classes of states and carry them in memory. (a) A possible refinement on $S_{4}$ , not a universal training sequence. The surveyed Transformers track the abelianization or finer quotients, while recurrent networks can pass through non-normal coset stages. (b) An $A_{5}$ GRU’s 20 coset mean vectors form an approximate dodecahedron in a three-dimensional subspace. Each vertex represents a non-normal $C_{3}$ coset with three possible states. Edges illustrate selected possible transitions between cosets, not neural connections.

$G$	$\|G\|$	$f$	$1/f$	Cond. (%)	Quot. (%)	$\times$ chance
$C_{8}$	8	1	1	–	–	7.88–7.96
$D_{4}^{*}$	8	2	$1/2$	49.95–50.05	99.94–99.99	4.00
$Q_{8}^{*}$	8	2	$1/2$	49.91–49.96	99.92–99.93	3.99
$S_{3}$	6	3	$1/3$	33.32–33.39	99.89–99.92	2.00
$D_{6}^{*}$	12	3	$1/3$	33.31–33.47	99.86–99.97	3.99–4.01
$\mathrm{Dic}_{3}$	12	3	$1/3$	33.25–33.36	99.51–99.93	3.97–3.99

Table 1: Quotient-class size and partial accuracy, min–max over the same runs. Here

f=|[G,G]|

is the size of an abelianization class. Cond. is

A_{\mathrm{exact}}/A_{\mathrm{quotient}}

. We report

\times

chance as

|G|A_{\mathrm{exact}}

. With accurate quotient tracking, the class-size prediction gives

\times

chance approximately

|G|/f

C_{8}

has singleton classes and

A_{5}

a single class; we omit conditional and quotient accuracies for these controls.

A_{5}

and

S_{5}

f

but differ twofold in the predicted

\times

chance. ∗ 148,438 training steps; unmarked rows use 74,219. We use all available doubled-budget runs where possible (App. I ). † The

G_{5}

window includes an early transition; exact accuracy at positions 70–100 is .1995–.1997.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Model	Seed	Sustained subgroup orders	Final exact accuracy
$A_{5}$	LSTM	42	$3$	.9996
	LSTM	43	$3$	.9923
	LSTM	44	$3$ (two intervals)	.0877 ∗
	GRU	42	$3$	.9938
	GRU	43	$3$	.9961
	GRU	44	$3$	.9950

Appendix

Table 2: Recurrent stages in the reported learning-rate arms. Orders refer to the unresolved subgroup

H

, not the number of cosets. Only the order-60 stages are nontrivial normal quotients. ∗ The third

A_{5}

LSTM reaches approximately .99 accuracy and subsequently loses it; its final accuracy must not be read as failure to reach an accurate state earlier.

Group	$\|H\|$	Stages	$R$ : mean $\pm$ SD	$R$ : range	$L$ : mean $\pm$ SD	$L$ : range
$S_{4}$	$2$	9	$0.9896\pm 0.0068$	$0.8935$ – $1.0000$	$0.3337\pm 0.0023$	$0.3055$ – $0.3382$
$A_{5}$	$3$	7	$0.9718\pm 0.0178$	$0.8510$ – $0.9998$	$0.0979\pm 0.0024$	$0.0856$ – $0.1025$
$S_{5}$	$2$	1	$0.9903$ (—)	$0.9589$ – $0.9999$	$0.0660$ (—)	$0.0638$ – $0.0676$
$S_{5}$	$2^{\prime}$	1	$0.9922$ (—)	$0.9215$ – $0.9997$	$0.0989$ (—)	$0.0951$ – $0.1010$
$S_{5}$	$3$	3	$0.9920\pm 0.0029$	$0.8800$ – $1.0000$	$0.1003\pm 0.0003$	$0.0962$ – $0.1016$
$S_{5}$	$4$	1	$0.9912$ (—)	$0.8880$ – $0.9994$	$0.1997$ (—)	$0.1832$ – $0.2018$

Appendix

Table 3: Stage summaries and full evaluation ranges for Fig. 3 d.

R

is the fraction of errors with

pq^{-1}\in H

;

L

uses

q^{-1}p\in H

. Means and sample SDs weight stages equally. Ranges include every evaluation within accepted stages. A dash indicates a single stage, for which SD is undefined. Primes match the subgroup types in the figure.

Configuration	Seed	Sustained subgroup	Last census exact
No feedback	42 / 43 / 44	$A_{4}$	.0825 / .0836 / .0808
One round	42 / 43	$A_{4}$	.0849 / .0840
One round	44	$V_{4}$	.2466
Four rounds, restart	42	$D_{4}$ then $V_{4}$	.2709
Four rounds	43	none	.1398
Four rounds	44	$V_{4}$	.5412

Appendix

Table 4: Recirculation census on

S_{4}

. Completed runs reach 74,219 updates; the interrupted run’s last census is at 38,750. A listed stage can precede the last evaluation. The shuffled control cyclically shifts source vectors across the batch, preserving the number of feedback rounds.

Model	Step	$A_{\rm coset}$	$k$	Top $k$	Rest	Full	Residual
LSTM 42	1,500	0.987	7	0.987	0.985	0.988	0.00172
LSTM 42	1,900	0.995	7	0.994	0.992	0.994	0.00155
LSTM 42	2,100	0.993	7	0.993	0.989	0.992	0.00106
LSTM 43	1,450	0.993	3	0.985	0.987	0.989	0.00056
LSTM 43	1,600	0.999	3	0.999	0.997	0.999	0.00051
LSTM 43	1,700	0.999	3	0.999	0.997	0.999	0.00047

Appendix

Table 5: All 18

A_{5}

mechanism evaluations.

A_{\rm coset}

is the fraction of full-state argmax predictions in the correct right coset.

k

is the number of fitted directions. “Top

k

” reports donor agreement after replacing those components; “Rest” reports recipient agreement after replacing their complement; “Full” reports donor agreement after replacing the full selected layer state. Residual is the mean relative squared error of orthogonal fits to the class-mean action.

Group	Seed	Updates	Full (%)	Quotient (%)	$100/f$	KL	$\Delta$ LL
$D_{4}$	43	148,438	50.063	99.948	50.000	0.313	-0.329
$Q_{8}$	42	148,438	49.910	99.927	50.000	0.230	-0.251
$S_{3}$	42	74,219	33.353	99.906	33.333	0.346	-0.337
$D_{6}$	44	148,438	33.364	99.968	33.333	0.236	-0.179
$\mathrm{Dic}_{3}$	44	74,219	33.235	99.933	33.333	0.593	-0.667
$D_{5}$	42	74,219	20.078	99.993	20.000	0.244	-0.212

Appendix

Table 6: The other 12 groups, supplementing the

A_{4}

main figure. All values use positions 17–100 and all examples of the same 8,192-sequence FP32 evaluation protocol. KL and

\Delta

LL are in

10^{-3}

bits. Endpoint selection for nontrivial proper quotients requires quotient accuracy

\geq 99.9\%

, preferring the smaller eligible completed budget and then the smaller eligible seed; full-state accuracy is not used for selection.

C_{8}

(

Q=G

) and

A_{5}

(trivial

Q

) are controls, fixed to seed 42 at 74,219 updates.

C_{8}

singleton-fiber diagnostics are omitted. This table describes selected endpoints, not success rates over seeds.

Figure S1: Quotient stages in the $S_{4}$ comparison. (a) The long Transformer run retained in the main figure. (b) LSTM exact accuracy for individual seeds and quotient accuracy means with seed ranges. The horizontal scale changes at 3k updates, where dense replay gives way to the original logs. Beige shading marks the shared $1/12$ stage; dashed references give $1/12$ and $1/4$ .

Figure S2: Additional training and architecture comparisons. (a) The original $D_{4}$ long-training display. (b) $D_{4}$ endpoints at positions 70–100: quotient accuracy and exact accuracy conditional on a correct quotient, under different budgets and recipes. (c) Original $S_{4}$ trajectories for a bilinear RNN (TC/RNN), LSTM, and GRU. Lines and bands show means and ranges across three seeds; thin blue lines retain individual full-state runs. TC/RNN selects the most probable class after summing its members’ probabilities; LSTM and GRU project the full-state argmax.

Model	Input / recipe	Updates	Seeds	$Q$ (%)	Full $\mid Q$ (%)
NeoX	full / A	74,219	3	81.727	50.043
NeoX	full / A	148,438	2	99.932	49.987
GPT-2	full / A	74,219	3	99.945	49.990
Llama	full / A	74,219	3	99.990	49.902
NeoX	gen. / A	74,219	3	72.575	50.015
GPT-2	full / B	312,500	3	94.214	49.959

Appendix

Table 7: All 20 available candidate endpoints considered for Fig. S2 b, grouped by condition and budget. Entries are seed means on positions 70–100. Shorter budgets and partial quotient channels are retained. The table uses the same evaluator and per-run ratio as the main panel; budgets are not pooled.

$G$	$f$	$t=3$	$t=4$	$t=8$	$t=12$	$t=17$	non-sweep mass
$S_{3}$	3	0.3009	0.2130	0.0516	0.0120	0.0019	8.4e-06 ( $t=17$ )
$D_{4}$	2	0.2188	0.1484	0.0323	0.0070	0.0010	2.3e-05 ( $t=17$ )
$Q_{8}$	2	0.2188	0.1484	0.0323	0.0070	0.0010	2.3e-05 ( $t=17$ )
$D_{6}$	3	0.3009	0.2130	0.0516	0.0120	0.0027 $\pm$ 6.8e-04	3.9e-04 ( $t=12$ )
$\mathrm{Dic}_{3}$	3	0.3009	0.2130	0.0516	0.0120	0.0031 $\pm$ 9.4e-04	3.9e-04 ( $t=12$ )
$A_{4}$	4	0.2500	0.1568	0.0200	0.0023	1.6e-04 $\pm$ 5.2e-05	1.1e-05 ( $t=12$ )

Appendix

Table 8: Excess of the exact order-blind ceiling over

1/|[G,G]|

, by prefix length. Entries at

t=17

with a standard error are Monte Carlo over 100 multisets with exact orderings; all other entries are exact over all multisets. The last column is the probability mass of multisets whose orderings do not sweep their coset, at the largest exactly enumerated length.

Figure S6: Prefix-count perturbations have signed, approximately additive effects. Three $H_{3}(\mathbb{Z})$ models trained at $10^{-3}$ . Here $\Delta\hat{z}$ is the change in predicted mean $z$ . (a) Responses per unit perturbation at position 64, averaged over outputs at positions 64, 72, 96, and 100. Filled dots show seeds, bars their mean, and open dots random-direction controls. (b) Joint effects versus summed single-site effects, with $\delta=+4$ at both sites. Each of 18 points is a mean over test words for one seed, token pattern, and pair type; the diagonal denotes additivity. Directions are fitted globally in (a), by position and pattern in (b) (App. P.5 ).

Architecture	Recipe / measure	Groups; seeds	Outcome	Status; location
GPT-NeoX 4L, $d$ =256, 4 heads, quarter rotary (recipe A)	A; full alphabet	13 groups; 3	plateau law	completed; § 3
Recipe A at 2 and 6 layers	A; full alphabet	$D_{4}$ ; 3+3	ratio .4986–.5034; frontier 2–3 at two layers, 5–12 at six	completed; § 5
Recipe A at 2/3/5/6 layers, widths 364/296/228/208 (parameter-matched) and 256 (shared grid)	A; full alphabet	$D_{4}$ ; 3 each	ratio .4991–.5017, frontier 3–8, no escape	completed; § 5
Recipe A at 2 and 6 layers, and at the doubled budget	A; full alphabet	$A_{4}$ , $\mathrm{SL}(2,3)$ ; 3 each	ratio $1/f\pm.0011$ ; frontier $+3$ / $+5$ on $A_{4}$ , $+2$ / $+1$ on $\mathrm{SL}(2,3)$	completed; § 5
Recipe A at 4, 8, 12 layers	A; full alphabet	$S_{5}$ ; 36 pairs	parity-first or no progress; 0 of 36 associative	completed; § 5
Recipe A at $d$ =512, 8 heads	A; full alphabet	$D_{4}$ ; 3	plateau, ratio .4998–.5013	completed; § 5

Appendix

Table 9: Architecture coverage, part 1 of 3. Status refers to the evaluation package; completed packages need not reach full tracking. All listed outcomes and seed counts are retained.

Architecture	Recipe / measure	Groups; seeds	Outcome	Status; location
NeoX 4L/256, no position embedding	A; full alphabet	$D_{4}$ ; 1	frontier 5, quotient unfinished, ratio .5002	completed; § 5
Recipe A on the generator stream	A; $\{r,s\}$	$D_{4}$ ; 3	plateau, ratio .499–.516	completed; App. O
GPT-2 sinusoidal 3L, $d$ =512, 8 heads (recipe B)	Liu; $\{r,s\}$	$D_{4}$ ; 3 (+1 at depth 4, +1 at $3\times 10^{-4}$ )	escape 3/3; depth 4 escapes; $3\times 10^{-4}$ partial	completed; App. O
Recipe B minus PE; on full alphabet; with recipe A optimizer; MLP 2048	Liu / A block	$D_{4}$ ; 3 each	chance; plateau 2/3; frozen frontier; escape	completed; App. O
GPT-2 rotary; NeoX sinusoidal; NeoX rotary, all 3L/512	Liu; $\{r,s\}$	$D_{4}$ ; 3 each	escape 1 / partial 2 (ratio .82–.998); escape 1 / partial 2 (.86–.99); plateau (.524–.532)	completed; NeoX-rotary cell in § 5 , the other two only here
NeoX rotary 3L/512 at 4 and 5 layers	Liu; $\{r,s\}$	$D_{4}$ ; 3 each	ratio .814, .951; frontier 31–32, 51–58	completed; § 5

Appendix

Table 10: Architecture coverage, part 2 of 3. Status refers to the evaluation package; completed packages need not reach full tracking. All listed outcomes and seed counts are retained.

Architecture	Recipe / measure	Groups; seeds	Outcome	Status; location
Looped transformer, 2 shared blocks, no position encoding, $d$ =128	curriculum to 16	$D_{4}$ ; 1	one position per loop; no quotient shelf	pilot; not used
Expert-split MLP; MoE readout	A	$D_{4}$ ; 1 pair each	frontier 11 vs 6; fiber .498–.506	exploratory; not used
Bilinear recurrence, state 128	Adam $10^{-3}$ ; full alphabet	10 groups; 3 (+10)	$1/\|N\|$ floors 13/13; acquired across positions in 16/16	completed; § 5
Two-layer polynomial net on group-Fourier input	6 configs, 30k	$D_{4}$ ; 1	amplitude sets learning order	exploratory; not used
Pooled (order-blind by construction) MLP; pooled + recurrent sum	Adam $10^{-3}$ , 20k	$S_{3}\times C_{3}$ ; 1	quotient only .40 at 20k; registered ceiling test missed (gap .255)	exploratory; not used
Two-layer LSTM, width 444 (parameter-matched to recipe A)	A; full alphabet	$D_{4}$ , $A_{4}$ , $S_{4}$ ; 3 per arm, 2 arms	escape 18/18; transient stages at $1/\|N\|$ levels	completed; § 5

Appendix

Table 11: Architecture coverage, part 3 of 3. Status refers to the evaluation package; completed packages need not reach full tracking. All listed outcomes and seed counts are retained.

Training support / encoding	Recipe	Groups; seeds	Outcome	Location
Full alphabet, i.i.d. uniform	A, C, LSTM, bilinear, 160M	all	quotient stages and $1/\|N\|$ levels	§ 3
Generator stream $\{r,s\}$ , i.i.d.	B; A	$D_{4}$ ; 3 per arm	escape (B); plateau (A)	App. O
Two $[G,G]$ -cosets of $D_{4}$ , $\{rK,sK\}$ , lift bias $\varepsilon\in\{0,.5,.9,.95,.99\}$	B	$D_{4}$ ; 3 at $\varepsilon$ =0 and .99, 1 otherwise	crawl or escape at $\varepsilon$ =0; the $\varepsilon$ axis is retired as an explanatory variable	App. O
Three $[G,G]$ -cosets of $D_{4}$	B	$D_{4}$ ; 3	flat, frontier 7–11	App. O
Two cosets of $Q_{8}$ , $\{iK,jK\}$ ; matched full-support baseline	B	$Q_{8}$ ; 3+1	all crawl; baseline flat	App. O
Single reflection coset of $D_{3}$ , $D_{5}$ , $D_{7}$	B	3, 1, 3	escape at 3; crawl at 5, 7	App. O

Appendix

Table 12: Training-support and encoding coverage, part 1 of 2. Outcomes refer to the listed recipes and budgets. Pilot and inconclusive results are retained.

Training support / encoding	Recipe	Groups; seeds	Outcome	Location
Frozen Peter–Weyl encoding, tied scores	A	$S_{3}\times C_{3}$ ; 3	characters learned, 2-dim irrep not	App. N
$S_{3}\times C_{3}$ encoding: frozen / trainable $\times$ amplitude 1, 2	A	1 each, 10k steps	no rescue of the 2-dim irrep	pilot; incomplete; not used
Equal-dimension embedding rows, four arms	12L/256, Li schedule	$S_{3}$ , $S_{5}$ ; 1k steps	frontier 2 in every arm	inconclusive; not used
Coset-union flows of $S_{3}\times C_{3}$ , $D_{5}$ , $D_{7}$ , $Q_{8}$ , $S_{4}$ , $S_{5}$	3L/128 pilot, 6k steps, length 64	6 groups; 1	quotient learned; exact $\mid$ quotient stays at $1/\|[G,G]\|$ ; gain vanishes on the full stream	exploratory; not used
Exact Fourier degree of the fiber and quotient sectors under a measure family from the full alphabet to the generator stream	zero training	$D_{4}$ , $S_{3}$ , $D_{6}$ , $Q_{8}$	pure degree $t$ under the full alphabet; reduced degree only on unpaired-lift supports	completed; App. O
Released pipeline of Li et al. [2025] on $S_{5}$ : fixed 1M-sequence dataset repeated for 20 epochs, bf16	their code and architecture	$S_{5}$ ; 1	remained at chance under bf16; excluded from mechanism analysis	not used

Appendix

Table 13: Training-support and encoding coverage, part 2 of 2. Outcomes refer to the listed recipes and budgets. Pilot and inconclusive results are retained.

$G$	seed	budget	exact $\mid$ quotient	quotient	exact
$C_{8}$	42	74k	0.9918	–	0.9918
$C_{8}$	43	74k	0.9945	–	0.9945
$C_{8}$	44	74k	0.9849	–	0.9849
$D_{4}$	42	74k	0.5002	0.990	0.4950
$D_{4}$	43	74k	0.4992	0.951	0.4748
$D_{4}$	44	74k	0.4999	0.849	0.4245

Appendix

Table 14: Original finite-group runs retained for reproducibility.

$G$	Seed	Exact $\mid$ quotient	Quotient	Exact
$H_{3}(\mathbb{Z}/3)$	42	0.3343	0.9991	0.3339
$H_{3}(\mathbb{Z}/3)$	43	0.3332	0.9982	0.3326
$H_{3}(\mathbb{Z}/3)$	44	0.3336	0.9996	0.3334
$H_{3}(\mathbb{Z}/5)$	42	0.1999	0.9966	0.1993
$H_{3}(\mathbb{Z}/5)$	43	0.1997	0.9786	0.1954
$H_{3}(\mathbb{Z}/5)$	44	0.1986	0.9458	0.1879

Appendix

Table 15: Original Heisenberg runs. All values use positions 17–100 and the original final evaluations; each condition has 74,219 updates. Seed identifiers are retained here for reproducibility.

$G$	Seed	Updates	Exact $\mid$ quotient	Quotient	Exact
$A_{4}$	42	148k	0.2502	0.9997	0.2501
$A_{4}$	43	148k	0.2502	0.9997	0.2501
$A_{4}$	44	148k	0.2500	0.9997	0.2500
$\mathrm{SL}(2,3)$	42	148k	0.1249	0.9998	0.1249
$\mathrm{SL}(2,3)$	43	148k	0.1245	0.9998	0.1245
$\mathrm{SL}(2,3)$	44	148k	0.1246	0.9995	0.1245

Appendix

Table 16: Additional doubled-budget runs used in Table 1 . All values use positions 17–100 of the independent final evaluation. The doubled-budget

D_{4}

Q_{8}

, and

D_{6}

runs are listed above with the original finite-group suite.

$G$	$\|G\|$	$f$	$1/f$	Cond. (%)	Quot. (%)	Exact (%)	Chance (%)	$\times$ chance	Updates
$C_{8}$	8	1	1	–	–	98.49–99.45	12.50	7.88–7.96	74k
$D_{4}^{*}$	8	2	$1/2$	49.95–50.05	99.94–99.99	49.94–50.02	12.50	4.00	148k
$Q_{8}^{*}$	8	2	$1/2$	49.91–49.96	99.92–99.93	49.87–49.93	12.50	3.99	148k
$S_{3}$	6	3	$1/3$	33.32–33.39	99.89–99.92	33.29–33.36	16.67	2.00	74k
$D_{6}^{*}$	12	3	$1/3$	33.31–33.47	99.86–99.97	33.26–33.45	8.33	3.99–4.01	148k
$\mathrm{Dic}_{3}$	12	3	$1/3$	33.25–33.36	99.51–99.93	33.11–33.26	8.33	3.97–3.99	74k

Appendix

Table 17: Budget-specific cohorts underlying Table 1 . Measured accuracies are in percent; 74k and 148k denote 74,219 and 148,438 updates. † The common

G_{5}

window includes an early transition (App. J.1 ).

Group	Exact 17–100	Quotient 17–100	Exact 70–100	Exact 101–128
$D_{4}$	53.33	99.55	49.65	15.04
$A_{4}$ (43)	35.47	99.96	25.13	19.87
$A_{4}$ (44)	33.13	99.98	25.01	18.09
$S_{4}$	8.30	99.69	8.31	4.43

Appendix

Table 18: Additional five-million-update final accuracies (%). Quotient readouts use the abelianization. Parentheses distinguish the two

A_{4}

seeds.

Seed	$C_{3}$	Sign	$S_{3}\mid\mathrm{sign}$	$F_{C_{3}}(.90)$	$F_{S_{3}}(.75)$
42	99.88	99.89	33.37	100	3
43	99.58	99.66	33.35	100	2
44	98.60	99.80	33.36	100	2

Appendix

Table 19: Matched-score encoding control on

S_{3}\times C_{3}

at 74,219 updates. Accuracies (%) are averaged over positions 17–100. Each frontier is the largest

F\leq 100

for which every position

1,\ldots,F

reaches its stated accuracy threshold.

Group	Layers	Initial LR	Seed	Exact	Quotient	Exact $\mid$ quotient
$D_{4}$	2	$5\times 10^{-5}$	42	31.17	62.54	49.84
$D_{4}$	2	$5\times 10^{-5}$	43	30.29	60.69	49.90
$D_{4}$	2	$5\times 10^{-5}$	44	20.83	41.70	49.97
$D_{4}$	2	$10^{-3}$	42	49.06	98.11	50.01
$D_{4}$	2	$10^{-3}$	43	49.84	99.85	49.91
$D_{4}$	2	$10^{-3}$	44	49.86	99.91	49.91

Appendix

Table 20: All matched Mamba-2 endpoints after 74,219 updates. Accuracies are percentages on positions 17–100. “Quotient” is the abelianization readout; “Exact

\mid

quotient” conditions on its correctness. Reference conditional accuracies are 50%, 25%, and

100/12\%

D_{4}

A_{4}

, and

S_{4}

. Each row is one completed run, without selection by accuracy.

Alphabet	Cosets	Form	Runs	Outcome (recipe B)
$\{r,s\}$ (preregistered)	2	yes	3	conditional .9997–1.000 at 25–45k updates
$\{r,r^{3},s,r^{2}s\}$ (two cosets)	2	yes	3	frontiers 44, 81, 100
$\{r,r^{3},s\}$	2	yes	2	frontier 38 rising; 24 stalled (300k)
$\{r,s,e\}$ (preregistered)	3	no	3	frontiers 25, 26, 23; conditional .50
$\{r,s,rs\}$	3	no	1	frontier 20
$\{r,s,r^{2}\}$	3	no	1	frontier 14

Appendix

Table 21: Support criterion against

D_{4}

outcomes under recipe B. Frontiers are the number of leading positions with exact accuracy at least .75, read at 150k updates unless noted; earlier rows come from the support interventions above. The two preregistered rows use seeds 47–49 under a frozen protocol; an earlier exploratory set (seeds 42–44) gave the same classification for both alphabets, and the remaining new rows are exploratory. The criterion characterizes availability of the stated counting form; it does not guarantee that training learns it.

Seed	Joint 17–100	Quotient 17–100	Joint at 100	Joint 101–128
42	98.10	100.00	92.90	36.82
43	98.17	100.00	93.74	37.32
44	98.07	100.00	92.81	27.32

Appendix

Table 22: Integer Heisenberg final accuracy (%). All models use learning rate

10^{-3}

. Joint accuracy requires all coordinates; quotient accuracy requires

(x,y)

	$a$ count		$a^{-1}$ count		Net $a$ count
Position	MAE	Exact (%)	MAE	Exact (%)	MAE	Exact (%)
32	0.0381	100.00	0.0366	100.00	0.0040	100.00
64	0.0887	100.00	0.0847	100.00	0.0069	100.00
100	0.1513	99.22	0.1465	99.42	0.0088	100.00

Appendix

Table 23: First-block readout of preceding counts at a

b

token. MAE is in count units; exact denotes accuracy after rounding to the nearest integer. Test subsets contain 532, 499, and 514 examples at positions 32, 64, and 100.

Seed	8	12	16	24	32	48	64	80	100
42	0.9786	0.9919	0.9942	0.9962	0.9977	0.9983	0.9984	0.9988	0.9986
43	0.9767	0.9926	0.9948	0.9977	0.9979	0.9982	0.9984	0.9988	0.9985
44	0.9775	0.9911	0.9935	0.9967	0.9973	0.9974	0.9969	0.9980	0.9972

Appendix

Table 24: Shared count readout: test

R^{2}

at each position for a single probe per model on normalized second-block outputs. All models use learning rate

10^{-3}

. No position-specific refitting is used.

Seed	MLP $C$ readout	Isolated $C$	Layer-3 KV	Exact shift (%)
42	.861	.00021	.909	83.08
43	.983	-.00021	.638	27.44
44	.651	-.00001	.812	61.21

Appendix

Table 25: Count-sensitive readouts and interventions. The MLP column measures the change in the decoded local contribution after an input replacement; isolated

C

and KV columns measure changes in the predicted mean

z

. All three are divided by the task-level increment change

\delta v_{j}

. Exact shift is the fraction of argmax predictions changing by the expected

\pm 2

under KV replacement. All models use learning rate

10^{-3}

Group	Seed	$t_{*}$	Accuracy at $t_{*}$	$\\|\bar{h}_{1}\\|/\\|\bar{h}_{t_{*}}\\|$	$\\|\bar{h}_{40}\\|/\\|\bar{h}_{t_{*}}\\|$
$A_{4}$	42	5	53.44	30.33	1.93
$A_{4}$	43	5	58.79	38.58	2.27
$A_{4}$	44	4	81.93	45.25	2.53
$\mathrm{SL}(2,3)$	42	4	50.21	40.95	2.34
$\mathrm{SL}(2,3)$	43	4	36.27	46.01	3.51
$\mathrm{SL}(2,3)$	44	3	88.59	37.97	1.13

Appendix

Table 26: Per-model net-gradient profiles behind Fig. S9 .

t_{*}

maximizes

\|\bar{h}_{t}\|

over the measured grid. Ratios and accuracy are percentages. Accuracy at position 1 is 100% in every row.

$G$	$A$	$n$	$r$	$r_{\min}$	$Q$	Sub.	Comp.	Rand.
$C_{8}$	$C_{8}$	3	2	2	0.997–1.000	0.930–0.991	0.000–0.009	0.000–0.001
$D_{4}$	$C_{2}^{2}$	5	2	2	1.000	0.999–1.000	0.000–0.001	0.000–0.001
$A_{4}$	$C_{3}$	3	2	2	1.000	1.000	0.000	0.000
$Q_{8}$	$C_{2}^{2}$	4	1,2	2	0.510–1.000	0.482–1.000	0.000–0.172	0.000–0.166
$C_{7}\rtimes C_{3}$	$C_{3}$	3	2	2	1.000	1.000	0.000	0.000
$D_{6}$	$C_{2}^{2}$	4	2	2	1.000	1.000	0.000–0.001	0.000–0.001

Appendix

Table 27: Geometry and same-position transfer across all 40 Transformer endpoints.

n

counts endpoints,

r

is the retained class-mean rank, and

r_{\min}

is the minimum faithful real dimension of

A

Q

is quotient accuracy at

t=51

. The final three columns give donor-class agreement after replacing the class-mean subspace (Sub.), its orthogonal complement (Comp.), or a same-rank random subspace (Rand.). Values are ranges across endpoints; each random value first averages five controls. No endpoint is excluded, including the low-accuracy

Q_{8}

endpoint. The protocol and agreement metric are defined below.

Explore similar work

May 8, 2026cs.LG

Rethinking State Tracking in Recurrent Models Through Error Control Dynamics

The theory of state tracking in recurrent architectures has predominantly focused on expressive capacity: whether a fixed architecture can theoretically realize a set of symbolic transition rules. We argue that equally important is error control, the dynamics governing hidden-state drift along the directions that distinguish symbolic states. We prove that affine recurrent networks, a class of models encompassing State-Space Models and Linear Attention, cannot correct errors along state-separating subspaces once they preserve state representations. Consequently, practical affine trackers do not learn robust state tracking; rather, they learn finite horizon solutions governed by accumulated state-relevant error. We characterize the mechanics of this failure, showing that tracking remains readable only while the accumulating within-class spread remains small relative to the initial between-class separation. We demonstrate empirically on group state-tracking tasks that this breakdown is predictable: tracking collapses when the distinguishability ratio crosses the readability threshold of the trained decoder. Across trained models, the point of this crossing predicts the horizon at which downstream accuracy fails. These results establish that robust state tracking is determined not only by an architecture's theoretical expressivity but crucially by its error control.

Jiwan Chung, Heechan Choi, Seon Joo Kim

Yonsei University

Recurrent Neural Networks Recurrent Model State-Tracking State Estimation

Ask PDF

Jun 5, 2026cs.LG

A Held-Out Transition-Pair Falsifier for Long-Horizon Non-Abelian State Tracking

State tracking exposes a sharp limitation of sequence models: the relevant signal is often not a summary of observed tokens, but an ordered latent state that evolves through non-commutative transformations. We introduce a held-out transition-pair falsifier for finite non-Abelian group tracking. The protocol forbids selected ordered generator pairs during training and requires the same local patterns during evaluation, blocking one direct local-transition memorization pathway. In a controlled

S_3 \times S_3

benchmark, a projected recurrent state model trained only on length-8 sequences produces error-free final-state predictions (perfect 250/250 per horizon) through evaluation horizons up to 1,048,576 tokens across five seeds. Matched native-readout baselines, including bag, GRU, and a single-configuration structured state-space model, remain near floor under the same protocol. Projection-matched GRU, structured SSM, and bag baselines equipped with analogous finite-group prototype readouts also remain near chance under the same split. Mechanism diagnostics show that hard projection coincides with low homomorphism error, low state-consistency drift, and non-trivial commutator separation, while softened projection collapses final-state accuracy. Clean-split audits verify zero verbatim reduced-word overlap and zero structural-template overlap between training and evaluation partitions. The evidence is scoped to this controlled finite-group falsifier rather than to a general architecture ranking. Within that regime, explicit projected non-commutative state composition acts as a useful inductive bias for long-horizon hidden-state tracking.

Jeonghoon Lee

Attractor Dynamics

Recurrent State State-Tracking Finite Groups

Ask PDF

Feb 16, 2026cs.LG

Learning State-Tracking from Code Using Linear RNNs

Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers and RNNs (linear and non-linear). However, these are often sequence-to-sequence tasks: learning to map actions (permutations) to states, which is incompatible with the next-token prediction setting commonly used to train language models. We address this gap by converting permutation composition into code via REPL traces that interleave state-reveals through prints and variable transformations. We show that linear RNNs capable of state-tracking excel also in this setting, while Transformers still fail. Motivated by this representation, we investigate why tracking states in code is generally difficult: actions are not always fully observable. We frame this as tracking the state of a probabilistic finite-state automaton with deterministic state reveals and show that linear RNNs can be worse than non-linear RNNs at tracking states in this setup.

Julien Siems, Riccardo Grazzi, Korbinian Pöppel +3

University of Freiburg · Prior Labs · Microsoft Research +2

State-Tracking Sequence Modeling Automata Recurrent Neural Networks

Ask PDF