Elastic Selective Spectral Hybrids for Train-Once, Export-Many Budgeted Inference

Dachuan Song; Chuchu Chen; Xuan Wang

cs.LGSep 26, 2026

Elastic Selective Spectral Hybrids for Train-Once, Export-Many Budgeted Inference

Authors: Dachuan Song, Chuchu Chen, Xuan Wang

Organizations: George Mason University · George Washington University

Abstract

Deploying a language model under different computing and latency budgets calls for compact models with different quality-cost trade-offs. Towards this end, elastic spectral state space models provide ordered and temporally decomposed channels that can be truncated, but they are associated with linear time-invariant filters that cannot selectively preserve relevant past information or forget irrelevant information as the context evolves. To address this, we introduce the Elastic Selective Spectral Hybrid (ESSH), which realizes each Hankel spectral channel as an independent recurrent unit using fitted damped rotation modes. It also features an input-dependent decay and write/read gates that make temporal retention and state update input-dependent while preserving channel-wise truncation and a structured recurrence for efficient execution. ESSH combines these selective spectral mixers with sliding-window attention and jointly trains multiple capacities by reducing spectral-channel count and feed-forward width at different rates through a two-rate capacity map with full-model distillation. The resulting models support chunked parallel training and fused recurrent decoding while avoiding computation for discarded channels. At full capacity, ESSH achieves language-modeling quality comparable to similarly sized independently trained models, while smaller exports exhibit a smooth quality-cost trade-off. We validate the effectiveness of the proposed framework using language understanding, retrieval, cross-domain text, and DNA experiments, by assessing quality retention and the trade-off against independently trained and elastic baselines. At the 1.53B model configuration, fused batch-one decoding takes 1.37 ms per token on a B300, providing a 2.14-2.80x speedup over the tested Mamba-2 and Mamba-3 implementations and 3.03x over Transformer++ at matched parameter counts.

Figures & tables

Figure 1: Overview of ESSH and its elastic axes. (A) The hybrid backbone combines spectral mixers with three sliding-window attention layers. (B) At tier

T

, the input produces channel values and token-dependent clock and write/read controls. Each retained spectral channel updates its

m

recurrent modes; channels

k>K_{T}

are removed before the outputs are aggregated. (C) MatFormer varies feed-forward width, MatMamba varies mixer inner width, while ESSH varies both spectral-channel count and feed-forward width. The inset shows the quality–cost trade-off across capacities.

Figure 2: Efficient execution of the selective mixer. (A) A chunked parallel scan evaluates within-chunk contributions in parallel, while mode states connect consecutive chunks. (B) Mixer-level recurrent decoding is compared with Mamba-2. (C) Inference efficiency of ESSH versus parameter-matched Mamba-2 on B300. Full efficiency results: Appendices P and R .

(a) Model	Params (B)	PPL $\downarrow$	LAMB	HellaS	PIQA	ARC-e	ARC-c	Wino	OBQA	Mean
ESSH	1.53	10.11	46.8	60.6	72.9	73.7	40.7	58.3	32.2	55.0
Transformer++ †	$\sim 1.5$	10.51	50.3	60.6	73.8	74.0	40.4	58.7	29.6	55.4
Gated DeltaNet †	$\sim 1.5$	10.45	49.2	61.3	74.3	75.3	41.2	58.0	31.6	55.8
Mamba-2 †	$\sim 1.5$	10.47	47.8	61.4	73.6	75.3	41.8	57.5	32.6	55.7
Mamba-3 SISO †	1.49	10.35	49.4	61.9	73.6	75.9	42.7	59.4	32.0	56.4
Mamba-3 MIMO †	1.50	10.24	51.7	62.3	75.3	76.5	44.5	60.6	32.6	57.6

Table 1: Quality and inference cost: (a) 1.5B full models; (b) 1.5B ESSH tiers; (c) 370M models. Pretraining uses 100B FineWeb-Edu tokens in (a,b) and 7.4B in (c), with ESSH tiers sharing training.

\dagger

: published values from Table 1 of Lahoti et al. (2026) . PPL uses each source’s validation shard. Accuracies: percentages in (a,b), fractions in (c); full-key retrieval uses 2K contexts. B300 inference in (b) uses batch one and 2K context. Protocols: Appendix E . Results: Appendices F – H , N.1 , and R .

Figure 3: Quality and training cost at the 370M scale. Connected points belong to one model family, isolated markers are single full-size models, and

T_{k}

denotes an ESSH tier. (a) PPL versus model size; (b) seven-task accuracy against independently trained Transformer++; (c) estimated A100-hours to obtain four sizes from four Transformer++ runs (green) or one shared 371M ESSH run (red), each labelled by size; (d) PPL at longer contexts. Protocols and full results: Appendices N.1 , N.8 , and H.3 .

Figure 4: B300 deployment at matched full-model parameter counts (1.526B–1.536B). (a) Decode throughput; (b) batch-one latency; (c) state+KV per sequence; (d) batch-one prefill. Decode uses a 2K context, and compact ESSH tiers use their sizes. Matched full-model values are three-session medians; compact prefill uses single-session measurements; full measurements are in Appendix R .

Appendix figures & tables68 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$L$ , $t,s$ , $\tau=t-s$	sequence length, target and source positions, and lag
$u(t)$ , $h(t)$ , $y(t)$	input, latent state, and output of a generic state-space model
$A,B,C,D$	state transition, input, readout, and direct input–output matrices
$N$ , $j$	LTI state dimension and eigenvalue index
$d_{\mathrm{in}},d_{\mathrm{out}}$	input and output dimensions of the generic SSM
$\mathcal{P}_{K}$ , $\Gamma_{j}$	rank- $K$ Hankel projection and spectral residue matrix

Appendix

Table 2: Notation for the generic SSM and Hankel spectral construction.

Symbol	Meaning
$u(t)$	mixer input after RMSNorm
$d$ , $d_{\mathrm{ff}}$	residual width and full FFN hidden width
$\bar{K}$ , $K_{T}$	full spectral-channel count and channel count retained by tier $T$ , with $\bar{K}=32$ in the default recipe
$m$ , $i$ , $P$	modes per channel, mode index, and channel value width; default language models use $m=8$ and $P=d/\bar{K}$
$H_{k,i}(t)$	mode state in $\mathbb{R}^{2\times P}$
$v_{k}^{0}(t)$ , $v_{k}(t)$	projected channel value before and after the channel-local normalization blend

Appendix

Table 3: Notation for the selective spectral mixer and elastic training.

Component	Parameters	1.5B ( $d{=}2048$ , $d_{\mathrm{ff}}{=}5632$ )	370M ( $d{=}896$ , $d_{\mathrm{ff}}{=}4608$ )
value in / out projections	$2d^{2}$	8,388,608	1,605,632
write, read gates	$2(\bar{K}m\,d+\bar{K}m)$	1,049,088	459,264
clock	$\bar{K}m\,d+\bar{K}m$	524,544	229,632
spectral tables $(\rho,\theta,\kappa^{\mathrm{c}},\kappa^{\mathrm{s}})$	$4\bar{K}m$	1,024	1,024
zero-lag coefficient $a$ , value blend $\beta$	$\bar{K}m+\bar{K}$	288	288
SwiGLU feed-forward ( Shazeer, 2020 )	$3d\,d_{\mathrm{ff}}$	34,603,008	12,386,304

Appendix

Table 4: Parameters per block for the two language-model configurations, with

\bar{K}=32

m=8

, and

P=d/\bar{K}

. Whole-model counts include tied embeddings.

Block	projection MACs + scan cost	decode cost per token	persistent state
Spectral mixer	$2d^{2}+3\bar{K}md$ $+\,O(Q\bar{K}(P+m)+\bar{K}mP)$	$O(d^{2}+\bar{K}md+md)$	$2\bar{K}mP=2md$
Window attention	$4d^{2}+O(wd)$	$O(d^{2}+wd)$	$2wd$
SwiGLU feed-forward	$3d\,d_{\mathrm{ff}}$	$O(d\,d_{\mathrm{ff}})$	$0$
Mamba-2 block	$O(d\,d_{\mathrm{inner}})$ $+\,O((Q+N_{\mathrm{state}})d_{\mathrm{inner}})$	$O(d\,d_{\mathrm{inner}}+N_{\mathrm{state}}d_{\mathrm{inner}})$	$N_{\mathrm{state}}d_{\mathrm{inner}}+O(d_{\mathrm{inner}})$
Full attention	$4d^{2}+O(Ld)$	$O(d^{2}+Ld)$	$2Ld$

Appendix

Table 5: Per-token arithmetic and per-sequence persistent state at full capacity. Compact-tier spectral and feed-forward costs follow by replacing

\bar{K}

and

d_{\mathrm{ff}}

with

K_{T}

and

n_{\mathrm{ff},T}

, while attention remains shared. The Mamba-2 row assumes a shared

B/C

group and

N_{\mathrm{state}}\leq d_{\mathrm{inner}}

, as in the timed configuration ( Dao and Gu, 2024 ) . One multiply-accumulate counts as two FLOPs.

Figure 5: Two-rate elastic training. The two-rate map reduces spectral-channel count and feed-forward width at different rates;

\operatorname{Align}_{64}(x)=64\max\{1,\lfloor x/64\rfloor\}

. The optimizer-step diagram shows joint training of full and sampled capacities with full-tier distillation. The right-hand boxes identify the shared attention and embeddings, reduced-capacity gradient scales, and the capacity warm-up and FFN-ordering schedule.

tier	$K_{T}$	$n_{\mathrm{ff},T}$ , 1.5B	$n_{\mathrm{ff},T}$ , 370M	params, 1.5B (B)	params, 370M (M)	modes $K_{T}m$
$T_{1}$	8	576	448	0.474	92.1	64
$T_{2}$	11	1024	832	0.575	118.9	88
$T_{3}$	15	1664	1344	0.716	154.7	120
$T_{4}$	18	2176	1792	0.828	185.3	144
$T_{5}$	20	2688	2240	0.931	214.5	160
$T_{6}$	23	3392	2752	1.076	248.8	184

Appendix

Table 6: The ten deployment tiers and measured parameter counts. Channel count and FFN width follow the two-rate map. Fixed embedding parameters cause the exported parameter fractions to differ between the two model scales.

	370M	1.5B
blocks / width / feed-forward width	22 / 896 / 4608	28 / 2048 / 5632
channels $\bar{K}$ / modes $m$ / value width $P$	32 / 8 / 28	32 / 8 / 64
sliding-window layers (zero-based; window 2048, RoPE ( Su et al., 2024 ) , head dim. 64)	blocks 12, 17, 21	blocks 15, 22, 27
tokenizer / embedding	GPT-2, tied	Llama-3.1, tied
parameters at full capacity	370.9M	1.531B
pretraining tokens	7.4B main / 3.7B ablations	100B

Appendix

Table 7: Language-model configurations. Common algorithmic choices are listed alongside scale-specific depth, width, tokenizer, learning rate, and training budget.

Model	Used in	Matching rule
Transformer++, Gated DeltaNet, Mamba-2, Mamba-3 (SISO, MIMO), 1.5B, published †	Section 5.1 , Appendix F	published source configurations and source-evaluation metrics; same corpus, tokenizer, context, and token budget where reported
Transformer++, own, 370M	Section 5.2 , Appendices F , J	same depth, data, token budget, and schedule; parameter count within $2\%$
Gated DeltaNet + SWA, own, 370M	same	parameter count within $2\%$ ; its own depth, width, and SWA configuration
Mamba-3 + SWA ⋆ , own, 370M	same	dedicated full-capacity training; parameter count within $2\%$ , with its own backbone configuration
MatFormer, own, 370M	Section 5.2 , Appendix N	Transformer++ backbone with nested FFN widths and four sampled training capacities
MatMamba, own, 370M	same	Mamba-2 backbone with head-prefix nesting and four sampled capacities

Appendix

Table 8: Baseline roles and matching rules. Each method keeps its stated architecture and training budget. Provenance markers are defined in Appendix E.9 .

Model	Params (M)	Layers	Width	FFN	Attention blocks	Pretrain KL
ESSH	370.9	22	896	4608	12, 17, 21	0.5
Mamba-3 + SWA	377.1	24	1024	2304	10, 22	–
Gated DeltaNet + SWA	378.1	24	1024	2432	10, 22	–
Transformer++	365.6	22	896	4224	All 22	–
MatFormer	365.6	22	896	4224	All 22	0
MatMamba	369.9	48	1024	–	None	0

Appendix

Table 9: Configurations used for the approximately 370M quality comparison. Attention positions are zero based. Pretraining KL denotes the full-tier distillation coefficient.

Model / tier	Params (M)	$r$	PPL $\downarrow$	$\Delta$ NLL $\downarrow$	7-task $\uparrow$	Full key $\uparrow$	ms/token $\downarrow$
ESSH, $T_{1}$	92.1	0.248	23.61	0.473	0.376	0.241	0.91
ESSH, $T_{2}$	118.9	0.321	20.04	0.309	0.388	0.342	0.94
ESSH, $T_{3}$	154.7	0.417	18.45	0.227	0.402	0.355	1.01
ESSH, $T_{4}$	185.3	0.500	17.68	0.184	0.405	0.370	1.04
ESSH, $T_{5}$	214.5	0.578	16.98	0.144	0.413	0.366	1.10
ESSH, $T_{6}$	248.8	0.671	16.25	0.100	0.417	0.362	1.11

Appendix

Table 10: Complete 370M ESSH family and full-capacity local references. ESSH quality uses the final shared checkpoint after 7.4B-token pretraining and brief shared refinement; local full-model references use 7.4B training tokens. Retained fraction

r

and

\Delta

NLL are relative to ESSH

T_{10}

. Full-key is teacher-forced full-key accuracy. Latency is a separate A100 batch-one shape benchmark on the exact ESSH export architectures with synthetic weights, a 2048-token context, and CUDA Graph execution.

		PPL $\downarrow$		7-task $\uparrow$
Tier	M params	Shared	Independent	Shared	Independent
$T_{1}$	92.1	23.61	19.79	0.376	0.385
$T_{4}$	185.3	17.68	16.25	0.405	0.412
$T_{7}$	274.3	15.91	15.10	0.420	0.424
$T_{10}$	370.9	14.71	14.44	0.428	0.425

Appendix

Table 11: Shared versus independently trained ESSH models at identical deployment shapes. The shared exports come from one 7.67B-token training run, while each independent model is trained separately for 7.4B tokens at the corresponding shape.

Tier	Params (B)	FW-Edu PPL $\downarrow$	LAMB PPL $\downarrow$	LAMB acc. $\uparrow$	7-task $\uparrow$
ESSH
$T_{10}$	1.53	10.11	12.96	46.8	55.0
$T_{7}$	1.17	10.97	15.39	44.9	53.0
$T_{4}$	0.83	12.26	18.27	42.1	50.7
$T_{1}$	0.47	16.49	36.87	34.0	43.8

Appendix

Table 12: Final 1.5B ESSH at four deployment tiers. Accuracy and seven-task means are percentages. PPL uses the 1.5B validation stream. Published full-model references are in Table 1 (a).

Model / tier	Params (B)	LAMB	HellaS	PIQA	ARC-e	ARC-c	Wino	OBQA
ESSH, $T_{10}$	1.53	46.8	60.6	72.9	73.7	40.7	58.3	32.2
ESSH, $T_{7}$	1.17	44.9	56.6	71.9	70.6	38.1	58.4	30.6
ESSH, $T_{4}$	0.83	42.1	51.9	69.4	67.5	35.1	58.6	30.4
ESSH, $T_{1}$	0.47	34.0	39.6	66.5	59.8	29.5	52.7	24.6

Appendix

Table 13: Seven-task accuracy (%) for the final 1.5B ESSH model. HellaSwag and ARC-Challenge use normalized accuracy; other columns use accuracy. Published full-model results are listed in Table 1 (a).

tier	params (B)	PPL $\downarrow$	7-task $\uparrow$	Full key $\uparrow$	prefill ktok/s $\uparrow$	decode ms/token $\downarrow$	memory MB $\downarrow$
$T_{1}$	0.474	16.49	0.438	0.370	75.0	1.13	51.15
$T_{2}$	0.575	14.08	0.462	0.384	79.8	1.12	51.46
$T_{3}$	0.716	12.87	0.486	0.367	77.9	1.13	51.87
$T_{4}$	0.828	12.26	0.507	0.379	81.4	1.20	52.17
$T_{5}$	0.931	11.78	0.515	0.371	74.6	1.18	52.38
$T_{6}$	1.076	11.23	0.526	0.323	76.1	1.28	52.69

Appendix

Table 14: 1.5B deployment family. Quality uses one ESSH checkpoint across all ten tiers; runtime uses B300 shape benchmarks. Prefill is batch one at 2K; decode is batch one after 2K. Decode uses three sessions at every tier; prefill uses three sessions at

T_{10}

and one session at compact tiers. Persistent memory is state+KV, excluding weights. Selected tiers are summarized in Table 1 (b).

		accuracy $\uparrow$
Model	params (M)	LAMB acc	HellaS	PIQA	ARC-e	ARC-c	Wino	OBQA	mean
ESSH, $T_{1}$	92.1	0.230	0.308	0.626	0.508	0.242	0.535	0.180	0.376
ESSH, $T_{2}$	118.9	0.253	0.322	0.644	0.540	0.259	0.515	0.186	0.388
ESSH, $T_{3}$	154.7	0.269	0.344	0.656	0.561	0.275	0.516	0.190	0.402
ESSH, $T_{4}$	185.3	0.275	0.355	0.660	0.555	0.275	0.512	0.200	0.405
ESSH, $T_{5}$	214.5	0.293	0.365	0.653	0.574	0.282	0.513	0.210	0.413

Appendix

Table 15: Zero-shot accuracy for the 370M comparison. ESSH rows use the final shared checkpoint and correspond to the ten deployment tiers; local reference models are trained independently. HellaSwag and ARC-Challenge use normalized accuracy. Scores are reported to three decimals.

Figure 6: Task-level zero-shot accuracy across ESSH capacities (

T_{1}

T_{4}

T_{7}

T_{10}

) and three independently trained full-size references. Cell values report absolute accuracy, and Mean is the unweighted average of the seven tasks. Shading for ESSH tiers represents the absolute score change relative to

T_{10}

within each task; reference rows are unshaded. Table 15 gives parameter counts and complete numerical results.

Model / tier	Params (B)	BoolQ	SIQA	SciQ	COPA	RACE	LogiQA	Mean
ESSH, $T_{10}$	1.531	0.613	0.419	0.911	0.790	0.344	0.214	0.549
ESSH, $T_{7}$	1.168	0.594	0.422	0.904	0.810	0.344	0.209	0.547
ESSH, $T_{4}$	0.828	0.596	0.400	0.890	0.740	0.344	0.198	0.528
ESSH, $T_{1}$	0.474	0.617	0.395	0.870	0.710	0.297	0.220	0.518

Appendix

Table 16: Additional zero-shot task accuracies (fractions,

\uparrow

) for the final 1.5B ESSH model. Mean averages the six task columns. The seven reference tasks are in Table 13 .

Model	tier	params (M)	margin win $\uparrow$	first token $\uparrow$	full key $\uparrow$	95% half-width
ESSH	$T_{10}$	1531.1	0.826	0.332	0.330	0.039
ESSH	$T_{7}$	1168.3	0.831	0.341	0.339	0.039
ESSH	$T_{4}$	827.6	0.857	0.380	0.379	0.040
ESSH	$T_{1}$	474.5	0.845	0.372	0.370	0.038
ESSH 370M	$T_{10}$	370.9	0.826	0.389	0.389	0.036
ESSH 370M	$T_{7}$	274.3	0.824	0.366	0.366	0.034

Appendix

Table 17: NIAH retrieval at context 2048, using 256 key pairs at five depths. Full-key accuracy uses teacher forcing. First-token accuracy and candidate margin are separate fields. The final column is the half-width of the 95% item-cluster bootstrap interval for full-key accuracy (2000 resamples of key pairs, keeping their five depths together). The 1.5B block uses the final ESSH model. Full key is also reported in Table 1 (b), in percent.

	Depth at 2048 tokens					Sequence length (tokens)
Tier	0.1	0.3	0.5	0.7	0.9	512	1024	2048
ESSH
$T_{1}$	0.004	0.242	0.355	0.496	0.754	0.742	0.582	0.355
$T_{4}$	0.051	0.270	0.352	0.500	0.723	0.711	0.582	0.352
$T_{7}$	0.027	0.246	0.324	0.457	0.641	0.668	0.547	0.324
$T_{10}$	0.035	0.227	0.309	0.445	0.633	0.664	0.512	0.309

Appendix

Table 18: Teacher-forced full-key accuracy for the final 1.5B ESSH model. Left: five needle depths in a 2048-token sequence. Right: sequence lengths with the needle at depth 0.5. Each depth uses 256 trials; 2048 is the same depth-0.5 measurement in both panels.

		validation PPL $\downarrow$ by window length
Model	params (M)	2048	4096	8192	16384
ESSH, $T_{10}$	370.9	14.91	14.97	14.76	15.05
ESSH, $T_{7}$	274.3	16.03	16.13	15.92	16.23
ESSH, $T_{4}$	185.3	17.76	17.83	17.62	17.95
ESSH, $T_{1}$	92.1	23.37	23.42	23.22	23.70
Mamba-3 + SWA 370M ⋆ (reference hybrid)	377.1	14.34	14.41	14.19	14.46

Appendix

Table 19: Perplexity beyond the 2048-token training context. Each length is evaluated on its own set of 64 non-overlapping windows at each scale. Both ESSH scales use their final models.

1.5B ESSH
		contains-match accuracy $\uparrow$
Model	Params (M)	SWDE	FDA	SQuAD	TriviaQA	NQ	DROP	mean
ESSH, $T_{10}$	1531.1	0.491	0.358	0.446	0.629	0.189	0.251	0.394
ESSH, $T_{7}$	1168.3	0.452	0.282	0.446	0.618	0.173	0.251	0.371
ESSH, $T_{4}$	827.6	0.411	0.297	0.453	0.578	0.151	0.259	0.358
ESSH, $T_{1}$	474.5	0.365	0.216	0.394	0.515	0.111	0.208	0.302

Appendix

Table 20: Contains-match accuracy on six open-book tasks at context 2048. Locally trained 370M rows share data and pretraining tokens. The 1.5B group uses ESSH; model scales are ranked separately.

Figure 7: Retrieval and question answering from a supplied 2048-token context. (a) SWDE and FDA test extraction of values. (b) four open-book QA tasks test answers recovered from the context. Scores use contains-match accuracy, with higher values better. Generation is greedy, stops at a newline and is capped at 48 tokens; inputs are left-truncated to at most 2000 tokens within the 2048-token evaluation budget. Panel (c) plots the mean of these six tasks against parameter count for the 370M and 1.5B ESSH families, so quality should be compared at similar sizes; the nearly coincident GDN + SWA and Mamba-3 + SWA points are offset slightly along the x-axis. The reference hybrid denotes Mamba-3 + SWA. Table 20 gives the complete 1.5B and 370M results.

Sequence length $L=256$ , scaled accuracy $\uparrow$
Model	Capacity	Params (M)	8 pairs	16 pairs	32 pairs	64 pairs
ESSH	$T_{10}$	1.81	0.999	0.998	0.999	1.000
ESSH	$T_{4}$	1.43	0.998	0.998	0.998	1.000
ESSH	$T_{1}$	1.24	0.998	0.999	0.999	1.000
ESSH (mixer only)	$T_{10}$	2.04	0.124	0.063	0.024	0.004
ESSH (mixer only)	$T_{4}$	1.53	0.128	0.062	0.023	0.003

Appendix

Table 21: MQAR scaled accuracy after 20k training steps, seed 0. Three-seed results for 32 and 64 pairs are in Table 22 . For each length/pair setting, ESSH uses one matched run with a preselected learning rate and evaluates

T_{1}

T_{4}

, and

T_{10}

from the same checkpoint. Baseline rows report their best learning-rate sweep result. Columns vary sequence length and key–value pair count; parameter counts use the task-specific vocabulary. Bold marks the highest displayed accuracy within each setting.

32 key–value pairs
Model / tier	$L=256$	$L=512$	$L=1024$
ESSH $T_{10}$	0.937 [0.818, 0.999]	0.668 [0.019, 0.994]	0.917 [0.811, 0.988]
ESSH $T_{4}$	0.846 [0.549, 0.998]	0.664 [0.014, 0.993]	0.854 [0.659, 0.980]
ESSH $T_{1}$	0.792 [0.632, 0.999]	0.661 [0.012, 0.991]	0.867 [0.728, 0.966]
Mixer only $T_{10}$	0.024 [0.022, 0.025]	0.005 [0.002, 0.009]	0.005 [0.002, 0.010]
Mixer only $T_{4}$	0.022 [0.017, 0.024]	0.002 [0.002, 0.003]	0.003 [0.002, 0.004]

Appendix

Table 22: MQAR across training seeds 0, 1, and 2: mean [minimum, maximum] scaled accuracy after 20k steps. The learning rate is fixed across seeds within each model/length/pair setting. ESSH tiers share each seed’s checkpoint. Bold marks the highest displayed mean within each length/pair setting.

Model	Capacity	Params (M)	$L=256$	$L=512$	$L=1024$
Hybrid	$T_{10}$	0.765	1.000	1.000	1.000
Hybrid	$T_{4}$	0.389	1.000	1.000	0.997
Hybrid	$T_{1}$	0.198	0.999	0.996	0.974
Mixer only	$T_{10}$	0.996	0.892	0.428	0.120
Mixer only	$T_{4}$	0.491	0.088	0.111	0.040
Mixer only	$T_{1}$	0.207	0.147	0.105	0.044

Appendix

Table 23: Palindrome token accuracy after 20k training steps, seed 0. Tier rows use the retained standalone model shape. Depth, residual width, and training steps are shared across the compared task-specific models; parameter counts differ.

1.5B ESSH; Llama-3.1 tokenizer
Model	Params (M)	PG-19	arXiv	GitHub	CC-News	Proof-Pile	WikiText
ESSH, $T_{10}$	1531.1	27.59	28.35	43.65	13.89	36.00	13.57
ESSH, $T_{7}$	1168.3	30.11	31.16	50.63	14.93	39.21	14.88
ESSH, $T_{4}$	827.6	33.30	34.36	58.86	16.70	43.92	16.71
ESSH, $T_{1}$	474.5	48.40	48.61	97.24	22.79	63.94	23.35
370M models; GPT-2 tokenizer

Appendix

Table 24: Cross-domain perplexity (

\downarrow

), compared within each tokenizer group. The 1.5B block uses ESSH; the 370M group uses the same training stages as the main comparison. Each group scores fixed non-overlapping 2048-target-token windows, up to 400 per domain; WikiText supplies 139 windows for GPT-2 and 141 for Llama-3.1. Bold compares models within a group.

	$T_{1}$	$T_{4}$	$T_{7}$	$T_{10}$
Params (M)	0.135	0.240	0.362	0.534
Bits per base $\downarrow$	1.775	1.732	1.729	1.662

Appendix

Table 25: Held-out chromosome-21 next-nucleotide loss. Every tier is evaluated on the same 4000 non-overlapping windows of 1024 bases. Parameter counts exclude task-specific classification heads.

Model / tier	Params (M)	Mouse enh.	Coding	Human/ worm	Cohn	Ensembl enh.	Regul.	OCR	Non- TATA
$T_{1}$	0.135	0.759	0.905	0.962	0.729	0.834	0.879	0.765	0.887
$T_{4}$	0.240	0.774	0.906	0.960	0.721	0.836	0.885	0.761	0.913
$T_{7}$	0.362	0.775	0.906	0.960	0.725	0.846	0.884	0.766	0.925
$T_{10}$	0.534	0.760	0.923	0.961	0.726	0.869	0.879	0.773	0.942
No pretraining	0.534	0.716	0.871	0.949	0.682	0.773	0.745	0.683	0.808
CNN †	0.264	0.715	0.892	0.942	0.702	0.744	0.872	0.698	0.861

Appendix

Table 26: Genomic Benchmarks test accuracy (

\uparrow

). ESSH rows average three task-adaptation seeds (0, 197, 511); each export is adapted separately. No pretraining uses the full architecture.

\dagger

: results reported by Schiff et al. (2024) , with the CNN from Grešová et al. (2023) ; bidir.: bidirectional. Published recipes differ from ours. Bold marks the best measured score within each block. OCR denotes open chromatin regions.

	validation PPL $\downarrow$				needle first-token acc. $\uparrow$		7-task $\uparrow$	params
Arm	$T_{1}$	$T_{4}$	$T_{7}$	$T_{10}$	$T_{4}$	$T_{10}$	$T_{10}$	$T_{10}$ (M)
ESSH (control)	25.32	19.15	17.38	16.46	0.259	0.336	0.408	370.9
ESSH (no selective clock)	25.69	19.47	17.70	16.80	0.139	0.266	0.410	370.9
ESSH (neutral gates)	26.04	19.79	17.99	17.09	0.102	0.140	0.406	370.9
ESSH (non-oscillatory init., no fit)	26.18	19.92	18.01	17.00	0.140	0.180	0.405	370.9
ESSH (random damped modes)	26.00	19.64	17.68	16.69	0.234	0.280	0.406	370.9

Appendix

Table 27: Selective-dynamics controls at 370M after 3.7B pretraining tokens. Compact PPL and needle first-token accuracy use the final checkpoints; full-capacity PPL uses the corresponding raw checkpoint. Increasing

m

also increases parameter count and recurrent-state size.

Figure 8: Effects of selective-mixer controls on language modeling and retrieval. (a) PPL changes relative to the control; positive values are worse, and shading shows the three-seed range of the control’s full-capacity PPL (0.09; Table 29 ). (b) Needle first-token accuracy at full capacity and

T_{4}

. The

m=16

arm also increases parameter count and recurrent-state size. Table 27 gives the numerical values and checkpoint stages.

	relative 2-norm error $\downarrow$
Fit	median ( $k\leq 24$ )	$k=1$	$k=4$	$k=8$	$k=16$	$k=24$
rotations, $m=8$ (initialization)	0.59	$1.4\times 10^{-6}$	$2.2\times 10^{-4}$	0.04	0.75	0.81
rotations, $m=16$	$8.9\times 10^{-6}$	$2.4\times 10^{-10}$	$7.8\times 10^{-8}$	$1.8\times 10^{-6}$	$8.8\times 10^{-7}$	0.56
real poles, $m=8$	0.76	$9.2\times 10^{-3}$	0.06	0.70	0.95	1.00
real poles, $m=16$	0.30	$4.2\times 10^{-4}$	$4.1\times 10^{-3}$	0.06	0.54	0.95
real poles, $m=32$	$5.6\times 10^{-4}$	$3.9\times 10^{-7}$	$1.2\times 10^{-5}$	$2.6\times 10^{-4}$	$8.1\times 10^{-3}$	0.11

Appendix

Table 28: Relative filter-fit error at

L=2048

for positive real decays and damped rotations. Medians use the first 24 numerically resolved channels. Each damped mode uses two real state coordinates. Errors refer to the filter-fitting comparison on float64 Hankel eigenvectors.

Figure 9: Finite-mode approximation of the Hankel spectral filters. Panels (a,b) show relative

\ell_{2}

error for eight and 16 modes over the first 24 numerically resolved channels; panel (c) counts channels below the stated error threshold as mode budget increases. A damped rotation uses two real state coordinates, while a positive real pole uses one. Panels (a,b) compare equal mode counts; eight rotations and sixteen real poles have equal state size. Table 28 reports the numerical summaries.

Initialization	Seed	PPL $T_{10}$	PPL $T_{4}$	PPL $T_{1}$	Full-key acc. $T_{4}$
Hankel control	197	16.46	19.59	26.19	0.249
Hankel control	313	16.55	19.67	26.28	0.030
Hankel control	511	16.52	19.64	26.18	0.134
Random damped modes	197	16.68	20.11	26.91	0.253
Random damped modes	313	16.84	20.26	27.14	0.177
Shuffled slots	197	16.51	19.73	26.49	0.129

Appendix

Table 29: Initialization controls after 3.7B pretraining tokens, evaluated from raw checkpoints with the common frontier protocol. Full-key accuracy at

T_{4}

is teacher forced. Rows report independent training seeds; retrieval varies substantially across seeds and is shown without ranking.

Arm	Change	$\Delta$ PPL $T_{10}\downarrow$	$\Delta$ PPL $T_{1}\downarrow$
ESSH (control)	–	0	0
ESSH (non-oscillatory init.)	identical real-decay modes, zero readout, no fit	$+0.54$	$+0.90$
ESSH (random damped modes)	fitted channel structure removed	$+0.23$	$+0.72$
ESSH (shuffled slots)	fitted channels placed in random slots	$+0.05$	$+0.30$
ESSH (reversed slots)	fitted channels placed in reverse order	$+0.07$	$+0.30$
ESSH (all decays $1-10^{-4}$ )	uniform initial decay across modes	$+0.88$	$+1.21$

Appendix

Table 30: Initialization controls at 370M and 3.7B tokens. PPL differences use end-of-pretraining validation from the trainer (seed 197). Table 27 instead reports reduced-capacity PPL after shared refinement, while Table 29 uses the common PPL evaluator for independent seeds.

Model	energy	knock-out	energy	knock-out
	Negative Spearman $\uparrow$		first-8 share
initialization (Hankel, untrained)	0.86	–	0.92	–
ESSH (control, seed 197)	0.66	0.82	0.46	0.64
ESSH (control, seed 313)	0.84	0.75	0.74	0.64
ESSH (alternative map)	0.74	0.70	0.51	0.57
ESSH (mixer-only nesting)	0.80	0.89	0.57	0.77

Appendix

Table 31: Channel importance at 370M after 3.7B tokens. Rows use seed 197 unless stated. Ordering is

-\operatorname{Spearman}(k,\mathrm{importance}_{k})

, so larger values indicate a stronger decrease with channel index. Shares use the first eight channel indices. Output energy and knock-out importance are reported separately. The alternative map uses exponents

(1.0,0.7)

; mixer-only nesting keeps FFN width fixed.

Figure 10: Concentration of predictive importance toward low-index spectral channels. (a) Knock-out importance measures the PPL increase after zeroing one channel. (b) Cumulative output energy measures activation magnitude and is distinct from knock-out importance. (c) Larger

-\operatorname{Spearman}(k,\mathrm{importance}_{k})

indicates a stronger decrease of importance with channel index. Open circles denote seed 313; panel (b) uses seed 197. Table 31 gives the corresponding statistics.

		validation PPL $\downarrow$				full-key acc. $\uparrow$	params, full
Arm	$P$	$T_{1}$	$T_{4}$	$T_{7}$	$T_{10}$	NIAH	(M)
ESSH ( $\bar{K}=32$ )	28	25.32	19.15	17.38	16.46	0.334	370.9
ESSH ( $\bar{K}=16$ )	56	–	18.95	17.35	16.49	0.316	364.3
ESSH ( $\bar{K}=64$ )	14	23.46	19.16	17.78	16.60	0.312	384.0
ESSH ( $\bar{K}=16$ , dense control)	56	–	–	–	16.09	0.304	364.3
ESSH ( $\bar{K}=32$ , dense control)	28	–	–	–	15.98	0.181	370.9

Appendix

Table 32: Channel-count controls at 370M after 3.7B pretraining tokens. Channel value width

P

is adjusted so that the full recurrent-state size is constant. Compact elastic PPL uses final checkpoints; full PPL is measured from raw checkpoints. The retrieval column reports teacher-forced full-key accuracy. For both

\bar{K}=64

rows, retrieval uses

\xi=1/2

(45 retained spectral channels); the other retrieval values use full capacity. Dashes denote configurations without a reported evaluation; dense PPL entries use their full-capacity models.

			validation PPL $\downarrow$
Arm	exponents (channel, FFN)	$r$ at $T_{1}$	$T_{1}$	$T_{4}$	$T_{7}$	$T_{10}$
ESSH (control)	$(0.5,\,0.8)$	0.248	25.32	19.15	17.38	16.46
ESSH (alternative map)	$(1.0,\,0.7)$	0.257	25.54	19.50	17.50	16.46
ESSH (steeper channels)	$(0.65,\,0.8)$	0.237	26.34	19.40	17.50	16.50
ESSH (flatter channels)	$(0.35,\,0.8)$	0.263	24.41	19.02	17.35	16.53
ESSH (flattest channels)	$(0.25,\,0.8)$	0.278	23.74	18.76	17.26	16.47

Appendix

Table 33: Capacity-map and ordering controls. In the upper block, compact PPL uses final checkpoints and full PPL uses raw checkpoints. Different maps produce different parameter counts at the same nominal tier, so rows are not parameter matched. The lower diagnostic block reports the recorded

T_{4}

needle first-token accuracy and raw

T_{1}

PPL. For the dense control, the raw

T_{1}

value is before compact recovery; the upper block reports the recovered compact checkpoint.

Reduced-capacity objective	PPL $T_{1}\downarrow$	PPL $T_{4}\downarrow$	PPL $T_{7}\downarrow$	PPL $T_{10}\downarrow$
$\mathrm{CE}$	27.61	20.08	18.03	16.48
$1.5\,\mathrm{CE}$	26.76	19.90	18.01	16.58
$\mathrm{CE}+0.5\,\mathrm{KL}$	26.19	19.59	17.83	16.46

Appendix

Table 34: Reduced-capacity objective controls at 370M, 3.7B tokens and seed 197. All four PPL values use the end-of-pretraining model and the common 16-batch evaluator. Full-capacity micro-batches always use CE. All three runs use 128 sequences per optimizer update.

	validation PPL $\downarrow$				needle first-token acc. $\uparrow$	7-task $\uparrow$
Arm	$T_{1}$	$T_{4}$	$T_{7}$	$T_{10}$	$T_{4}$	$T_{10}$
ESSH (control)	25.32	19.15	17.38	16.46	0.259	0.408
ESSH ( $f=0.5$ )	23.50	18.67	17.44	17.01	0.216	0.407
ESSH ( $f=0.875$ )	27.56	19.92	17.53	16.26	0.199	0.410
ESSH ( $\alpha_{\mathrm{ff}}=0.25$ )	26.25	19.59	17.59	16.52	0.135	0.410
ESSH ( $\alpha_{\mathrm{ff}}=\alpha_{\mathrm{mix}}=0.5$ )	25.72	19.31	17.48	16.50	0.162	0.413

Appendix

Table 35: One-setting controls at 370M and 3.7B pretraining tokens. Compact PPL uses final checkpoints; full PPL and the seven-task mean use the corresponding full-model evaluations. The retrieval column reports NIAH first-token accuracy at the exact

T_{4}

shape.

Schedule	$T_{1}$	$T_{2}$	$T_{4}$	$T_{7}$	$T_{10}$
ESSH (fixed share $f=0.75$ , control)	0	0	0	0	0
ESSH (fixed share $f=0.5$ )	$-2.16$	$-1.30$	$-0.58$	$-0.02$	$+0.55$
ESSH (adaptive share, 0.5 to 0.9)	$-2.59$	$-0.96$	$-0.25$	$+0.17$	$+0.52$
ESSH (sampled capacities = the ten tiers)	$+1.38$	$+1.37$	$+0.01$	$-0.28$	$-0.08$
ESSH (sampled capacities = five tiers $T_{1},T_{3},T_{5},T_{7},T_{10}$ )	$+0.10$	$+2.59$	$+0.23$	$-0.21$	$-0.03$
ESSH (progressive capacity reduction, four stages)	$+0.92$	$+0.11$	$-0.10$	$-0.05$	$+0.07$

Appendix

Table 36: Raw PPL differences relative to the fixed-share control for alternative 370M sampling schedules after 3.7B tokens. Negative values indicate lower PPL than the control. Columns evaluate the exact deployment shapes.

Figure 11: Selected capacity-sampling schedules. Panel (a) shows PPL differences at the five evaluated tiers, panel (b) tracks

T_{1}

during training, and panel (c) shows the realized full-capacity share. The progressive schedule’s early maximum of +1324 PPL is clipped by the plotting range. The complete schedule comparison, including the share ramp and gap-proportional variant, is in Table 36 .

Model (370M scale)	tokens (B)	PPL $T_{10}$ $\downarrow$	seed range
ESSH	3.7	16.46	0.09
ESSH (dense control, no capacity mixing)	3.7	15.98	0.06
PPL gap (elastic $-$ dense)	3.7	$+0.48$	–
ESSH	7.4	14.85	0.05
ESSH (dense control, no capacity mixing)	7.4	14.44	0.05
PPL gap (elastic $-$ dense)	7.4	$+0.41$	–

Appendix

Table 37: Full-capacity PPL at the end of pretraining. ESSH comparisons use the same architecture and token budget within each block. Mamba-3 + SWA is a separate independently trained reference. PPL reports seed 197; range is the maximum minus the minimum across three seeds for 3.7B elastic ESSH and two for each other training condition.

$d_{\mathrm{ff}}$	params $T_{1}/T_{10}$ (M)	PPL $T_{1}$ $\downarrow$	PPL $T_{4}$ $\downarrow$	PPL $T_{7}$ $\downarrow$	PPL $T_{10}$ $\downarrow$	needle first-token acc. $T_{4}\uparrow$
3072	84.6 / 280.0	26.44	20.26	18.29	17.12	0.191
4608 (ESSH)	92.1 / 370.9	25.32	19.15	17.38	16.46	0.259
6144	103.5 / 461.7	24.18	18.59	16.91	16.09	0.319

Appendix

Table 38: FFN-width sweep at fixed depth, residual width, and spectral mixer after 3.7B pretraining tokens. Exported parameter counts vary with FFN width. Compact PPL uses final checkpoints; full-capacity PPL uses the corresponding raw checkpoints.

Tier	Params (M)	$r$	PPL $\downarrow$	$\Delta$ NLL $\downarrow$	7-task $\uparrow$	ms/token $\downarrow$
$T_{2}$	146.0	0.400	19.37	0.142	0.390	3.14
$T_{3}$	168.7	0.462	18.48	0.095	0.396	3.21
$T_{4}$	191.5	0.524	17.82	0.058	0.401	3.29
$T_{5}$	214.2	0.586	17.50	0.041	0.404	3.27
$T_{6}$	248.2	0.679	17.02	0.012	0.408	3.34
$T_{7}$	270.9	0.741	16.98	0.011	0.410	3.37

Appendix

Table 39: MatFormer at the ten ESSH deployment budget fractions, using its end-of-pretraining 7.4B-token model. Architecture-specific retained dimensions and parameter counts differ from ESSH.

r

is the retained parameter fraction;

\Delta

NLL is relative to the full model.

T_{1}

lies below the smallest capacity used during MatFormer training. Seed 197. Latency is a batch-one CUDA-graph measurement after a 2K context; see Appendix E.5 .

Tier	Params (M)	$r$	PPL $\downarrow$	$\Delta$ NLL $\downarrow$	7-task $\uparrow$	ms/token $\downarrow$
$T_{2}$	102.4	0.277	22.06	0.262	0.371	3.21
$T_{3}$	131.1	0.354	20.53	0.190	0.379	3.03
$T_{4}$	159.7	0.432	19.10	0.118	0.386	3.19
$T_{5}$	188.4	0.509	18.47	0.084	0.390	3.18
$T_{6}$	226.6	0.613	17.47	0.028	0.398	3.24
$T_{7}$	255.3	0.690	17.38	0.023	0.400	3.36

Appendix

Table 40: MatMamba at the ten ESSH deployment budget fractions, using its end-of-pretraining 7.4B-token model. Architecture-specific retained dimensions and parameter counts differ from ESSH.

r

is the retained parameter fraction;

\Delta

NLL is relative to the full model.

T_{1}

lies below the smallest capacity used during MatMamba training. Seed 197. Latency is a batch-one CUDA-graph measurement after a 2K context; see Appendix E.5 .

	Params (M)		PPL $\downarrow$		7-task $\uparrow$
Target	ESSH	TF++	ESSH	TF++	ESSH	TF++
$T_{1}$	92.1	92.7	23.61	20.36	37.56	38.41
$T_{4}$	185.3	186.3	17.68	17.76	40.45	39.84
$T_{7}$	274.3	273.6	15.91	16.69	42.00	40.79
$T_{10}$	370.9	365.6	14.71	15.88	42.84	41.38

Appendix

Table 41: PPL and seven-task mean accuracy (%) at four deployment sizes. ESSH follows the complete shared-training procedure; Transformer++ uses four independent runs. Both use 7.4B pretraining tokens with the same data, tokenizer and sequence length. ESSH’s additional shared stage is specified in Appendix E.3 .

Family	Seed	PPL $T_{1}\downarrow$	PPL $T_{4}\downarrow$	PPL $T_{7}\downarrow$	PPL $T_{10}\downarrow$
ESSH	197	23.61	17.68	15.91	14.71
ESSH	511	23.77	17.76	15.96	14.73
MatFormer	197	5844.15	17.82	16.98	16.80
MatFormer	313	106.56	18.08	17.30	17.14
MatFormer	511	112.01	18.05	17.25	17.10
MatMamba	197	3029.37	19.10	17.38	16.98

Appendix

Table 42: PPL across independent training seeds at four deployment shapes. ESSH uses final shared checkpoints; MatFormer and MatMamba use end-of-pretraining checkpoints. Baseline

T_{1}

lies below the smallest capacity sampled during their training and is not used for the common-range comparison.

3.7B pretraining tokens
	Additional training		$T_{4}$ , 185.3M			$T_{7}$ , 274.3M
Method	Tokens (B)	Steps	PPL $\downarrow$	7-task $\uparrow$	Full key $\uparrow$	PPL $\downarrow$	7-task $\uparrow$	Full key $\uparrow$
ESSH	0.27	2060	19.15	0.394	0.258	17.38	0.401	0.256
Dense control, truncated and recovered	0.27	2060	23.75	0.376	0.131	18.63	0.397	0.116
Dense control, label-only recovery	0.54	4120	20.07	0.386	0.146	17.22	0.407	0.114
7.4B pretraining tokens

Appendix

Table 43: Models at the exact ESSH

T_{4}/T_{7}

shapes, grouped by pretraining budget. Additional tokens and steps specify the subsequent shared-training or recovery stage. Full-key accuracy uses the common 2K NIAH protocol.

Configuration	tokens (B)	params (M)	7-task mean $\uparrow$	Full-key acc. $\uparrow$
parent, dense Mamba-3 + SWA ⋆	7.4	377.1	0.445	0.230
$100\%$ mask, after elastification	7.9	377.1	0.438	0.248
mask targeting $T_{4}$	7.9	185.0	0.397	0.034
ESSH, $T_{4}$	7.67	185.3	0.405	0.370
mask targeting $T_{7}$	7.9	273.8	0.425	0.147
ESSH, $T_{7}$	7.67	274.3	0.420	0.366

Appendix

Table 44: Post-hoc masks targeting

T_{4}/T_{7}

sizes. Recovered Mamba-3 models use 7.90B total tokens; ESSH uses 7.67B. Seven-task and full-key metrics use the same evaluation protocols. Actual parameter counts are shown.

Model / target	Params (M)	First token $\uparrow$	Full key $\uparrow$	7-task $\uparrow$
Mamba-3 + SWA parent	377.1	0.241	0.230	0.445
Original mask, $T_{4}$ target	185.0	0.036	0.034	0.397
Preserve attention, $T_{4}$ target	185.3	0.025	0.023	0.395
ESSH, $T_{4}$	185.3	0.370	0.370	0.405
Original mask, $T_{7}$ target	273.8	0.152	0.147	0.425
Preserve attention, $T_{7}$ target	276.5	0.154	0.148	0.426

Appendix

Table 45: Original and attention-preserving post-hoc masks near the

T_{4}/T_{7}

sizes. Both recovery strategies use the same parent and 0.5B recovery tokens. Task and retrieval scores use the common protocols.

Configuration	tokens (B)	params (M)	PPL $\downarrow$	7-task mean $\uparrow$	Full-key acc. $\uparrow$
parent: ESSH (dense control, 3.7B)	3.7	370.9	15.98	0.414	0.181
$100\%$ mask, after elastification	4.2	370.9	16.15	0.417	0.113
Post-hoc $T_{4}$	4.2	185.3	19.64	0.394	0.154
ESSH (3.7B), $T_{4}$	3.97	185.3	19.15	0.394	0.258
ESSH (dense control, reduced), $T_{4}$	3.97	185.3	23.75	0.376	0.131
Post-hoc $T_{7}$	4.2	274.3	17.35	0.404	0.107

Appendix

Table 46: Same-architecture compact-model comparison. Post-hoc elastification uses 4.20B total tokens; capacity-mixed ESSH and the directly reduced dense control use 3.97B. Compact rows use the exact deployment shapes. Full-key is teacher-forced full-key accuracy.

Stage	Tokens (B)	ktok/s	A100-h $\downarrow$
Shared pretraining	7.4	41.9	49.1
Shared refinement	0.27	–	4.81
Independent $T_{1}$	7.4	90.2	22.8
Independent $T_{4}$	7.4	62.1	33.1
Independent $T_{7}$	7.4	50.2	40.9
Independent $T_{10}$	7.4	40.66	50.6

Appendix

Table 47: Internal ESSH cost comparison for obtaining

T_{1}/T_{4}/T_{7}/T_{10}

on one A100-80GB. Pretraining A100-hours are computed from 7.4B tokens and measured steady-state throughput; shared refinement uses the measured 2060-step wall-clock time. The four deployment sizes are 92.1, 185.3, 274.3, and 370.9M parameters.

Transformer++ target / total	Rate (ktok/s)	A100-hours
$T_{1}$	141.32	14.55
$T_{4}$	91.98	22.35
$T_{7}$	70.85	29.01
$T_{10}$	57.87	35.52
Four independent Transformer++ runs	–	101.43
One shared ESSH run	–	53.90

Appendix

Table 48: Training cost for the four sizes in Table 41 . All rates use one A100-80GB. Independent costs are estimated as 7.4B tokens divided by each model’s steady-state rate. Shared ESSH includes the teacher computation and measured shared refinement stage. The reported costs are estimates of training work at the measured rates.

Check	Error metric	Maximum
CPU recurrence, chunk boundaries and zero-lag identities	absolute	$3.55\times 10^{-15}$
CPU recurrence gradients versus centered finite differences	absolute	$2.28\times 10^{-9}$
CPU toy export versus same-tier shared model	absolute	$0$
GPU fused decoder versus chunked forward	relative	$1.37\times 10^{-6}$
GPU fused decoder versus reference stepper	relative	$2.11\times 10^{-5}$

Appendix

Table 49: Numerical verification of the recurrent and export implementations. CPU entries report fp64 maximum absolute error with random seed 2333; centered finite differences use step

10^{-6}

. GPU entries report relative error on the recorded reduced test shapes.

Device	Model	Params (M)	$B$	$A$	ktok/s $\uparrow$
370M-scale full models
A100	ESSH 370M	370.9	8	16	41.7
A100	Mamba-3 + SWA 377M	377.1	8	16	34.1
H200	ESSH 370M	370.9	8	16	75.5
H200	Mamba-3 + SWA 377M	377.1	8	16	88.6
B300	ESSH 370M	370.9	8	16	82.3

Appendix

Table 50: Full-capacity training throughput on one GPU at context 2048. Params include tied embeddings.

B/A

denote micro-batch size/accumulation; rates use steps 20–35 of 40-step probes. Only the 370M group provides near-size cross-model comparisons. The 1.53B block reports ESSH device throughput alone. ESSH remains in full-capacity warm-up, before capacity mixing and teacher forwards.

		prefill ktok/s $\uparrow$
Model	Params (B)	2K	8K	32K	128K	Prefill ms (2K) $\downarrow$
ESSH, $T_{1}$	0.474	44.4	46.7	48.5	49.1	46.1
ESSH, $T_{4}$	0.828	36.4	39.3	41.1	41.6	56.3
ESSH, $T_{7}$	1.168	32.1	35.1	36.2	36.5	63.8
ESSH, $T_{10}$	1.531	28.3	32.0	33.5	34.0	72.3
Mamba-2 + FFN	1.526	47.5	53.6	55.0	54.7	43.1

Appendix

Table 51: A100 batch-one full-logit prefill. Throughput is ktok/s and latency ms; peak allocations are reported in Table 53 . Compact ESSH rows use the tier sweep; full-model rows use matched three-session measurements. Each session uses three warm-ups and five timed calls. Configurations are in Table 54 ; Mamba-3 MIMO uses the compiler-compatible repeat. Bold compares measured entries within each ruled group.

Model	Params (B)	$B=1$	16	128	512	Eager $B=1$
ESSH, $T_{1}$	0.474	1.46	2.60	7.53	23.76	3.98
ESSH, $T_{2}$	0.575	1.67	2.66	7.35	24.89	4.07
ESSH, $T_{3}$	0.716	1.95	2.96	7.75	25.91	4.20
ESSH, $T_{4}$	0.828	2.12	3.02	8.29	27.31	4.27
ESSH, $T_{5}$	0.931	2.22	3.13	8.40	27.73	4.25
ESSH, $T_{6}$	1.076	2.41	3.28	8.66	28.82	4.28

Appendix

Table 52: A100 whole-model decode latency (ms/step), with state/cache sized for a 2K context. Batch columns use CUDA graphs; the last uses eager execution. ESSH weights/KV are bf16 and recurrent state fp32. Full-model comparisons use matched three-session measurements, with MIMO from the compiler-compatible repeat. Compact tiers and the fused/unfused control use the separate protocols in Appendix E.5 . Configurations: Table 54 . Compact and full-model blocks are ranked separately; OOM denotes out of memory.

		$n_{\mathrm{b}}{=}1$ (MB) $\downarrow$			total $n_{\mathrm{b}}{=}128$ $\downarrow$	prefill peak (GB) $\downarrow$
Model	Params (B)	state	KV	total	(GB)	2K	8K	32K	128K
ESSH, $T_{1}$	0.474	0.82	50.33	51.15	6.55	1.5	3.1	9.6	35.7
ESSH, $T_{4}$	0.828	1.84	50.33	52.17	6.68	2.2	3.8	10.3	36.4
ESSH, $T_{7}$	1.168	2.56	50.33	52.89	6.77	3.0	4.6	11.1	37.1
ESSH, $T_{10}$	1.531	3.28	50.33	53.61	6.86	3.6	5.2	11.7	37.8
Mamba-2 + FFN	1.526	30.33	0.00	30.33	3.88	3.6	5.3	11.8	37.8

Appendix

Table 53: A100 memory at the stated model sizes. State/cache totals exclude weights; prefill peaks include weights, logits, and temporaries. ESSH state is fp32 and cache bf16. MB/GB are decimal. Full-model allocations use the matched A100 measurements; compact rows use the tier sweep. Bold compares measured total/peak allocations separately within compact and full-model blocks.

Model	Params (M)	Gap (%)	FFN width	SWA layers
ESSH	1531.1	+0.00	5632	3
Mamba-2 + FFN	1526.0	-0.33	3136	0
Mamba-3 SISO + FFN	1534.9	+0.25	3136	0
Mamba-3 MIMO + FFN	1526.7	-0.29	2816	0
Transformer++	1536.3	+0.34	4672	0

Appendix

Table 54: Configurations for the A100, H200, and B300 comparisons. Parameter counts include tied embeddings. All use vocabulary 128256, residual width 2048, and 28 blocks. Baseline FFN widths are matched in multiples of 64; official Mamba mixer settings stay fixed. Gap is relative to ESSH. A100 Mamba-3 MIMO uses the compiler-compatible repeat.

Constant	370M	1.5B	fixed by
$\bar{K}$ , $m$	32, 8	32, 8	divisibility, Appendices L.5 , L.4
window layers, window	3 late, 2048	3 late, 2048	Appendix E.2
channel / feed-forward exponents	0.5 / 0.8	same	Appendix M.1
full share $f$ , slice scale $\alpha_{\mathrm{ff}}$	0.75, 0.5	0.75, 0.5	Appendix M.2
distillation objectives and coefficients	KL at 0.5 (pretraining), TV at 5 (final stage)	same	Appendix M.2
pretraining capacities	$1/16$ , $3/32$ , $1/8$ , $5/32$ , $3/16$ , $1/4$ , $3/8$ , $1/2$ , $3/4$ , $1$	same	design

Appendix

Table 55: Constants used in the main recipe and the experiment sections that evaluate alternative settings.

Stage	Devices	GPU-hours
1.5B main training	A100, H100, H200, H800, B200	1808
1.5B final stage, export and evaluation	H200, H800, RTX 5090/3090	47
370M shared recipe, 7.4B	A100, H100, RTX 5090/3090	104
370M comparisons, 7.4B	A100, H200, RTX 5090/3090	772
370M ablations, 3.7B	A100, RTX 5090/3090	1633
Synthetic tasks and DNA	A100, RTX 5090/3090	129

Appendix

Table 56: Reconstructed raw GPU-hours for this study. Totals include failed, aborted, and repeated work for these experiments (105.2 GPU-hours), while excluding unreported exploration. Device hours are summed without hardware normalization and rounded to integers.

Parameter-matched full models
Model	Params (B)	$B=1$	16	128	512	Eager $B=1$
ESSH, $T_{10}$	1.531	1.67	2.15	5.35	17.63	3.44
Mamba-2 + FFN	1.526	2.91	3.93	7.81	22.05	15.97
Mamba-3 SISO + FFN	1.535	3.52	4.78	8.85	24.12	27.65
Mamba-3 MIMO + FFN	1.527	3.85	5.30	9.87	26.46	28.71
Transformer++	1.536	4.15	7.32	30.03	OOM	12.54

Appendix

Table 57: H200 decode latency (ms/step), with a 2K cache. Batch columns use CUDA graphs; the last uses eager execution. All five models use the parameter-matched three-session measurements and the same attention-backend policy. ESSH state is fp32 and weights/KV bf16; baselines retain native state precision. OOM records allocation failure in all three sessions. Bold ranks within each group.

Parameter-matched full models
		Prefill ktok/s $\uparrow$				ms $\downarrow$	Peak GB $\downarrow$
Model	Params (B)	2K	8K	32K	128K	2K	2K	128K
ESSH, $T_{10}$	1.531	54.9	58.9	60.6	60.7	37.3	3.64	37.79
Mamba-2 + FFN	1.526	73.8	126.0	127.4	125.4	27.8	3.65	37.80
Mamba-3 SISO + FFN	1.535	71.8	81.1	83.8	83.9	28.5	3.65	37.80
Mamba-3 MIMO + FFN	1.527	39.6	42.1	42.6	42.2	51.7	3.64	37.79

Appendix

Table 58: H200 batch-one full-logit prefill. Throughput: ktok/s; latency: ms; peak allocation: decimal GB. All five models use three sessions with three warm-ups and five timed calls per session. cuDNN SDPA is disabled for the shared comparison. Bold ranks separately within each group.

Model	Params (B)	$B=1$	16	128	512	Eager $B=1$
ESSH, $T_{1}$	0.474	1.13	1.35	3.30	10.25	3.43
ESSH, $T_{2}$	0.575	1.12	1.33	3.53	10.62	3.41
ESSH, $T_{3}$	0.716	1.13	1.36	3.55	10.91	3.46
ESSH, $T_{4}$	0.828	1.20	1.42	3.63	11.18	3.49
ESSH, $T_{5}$	0.931	1.18	1.42	3.61	11.13	3.49
ESSH, $T_{6}$	1.076	1.28	1.49	3.83	11.62	3.46

Appendix

Table 59: B300 decode latency (ms/step), after a 2K context. Batch columns use CUDA graphs; the last uses eager execution. All full-model entries and compact

B=1

entries use three-session measurements. Compact

B=16/128/512

entries use a separate tier sweep. ESSH state is fp32; weights/KV are bf16. Mamba rows use the official mixers with a residual FFN block. Bold is ranked separately within each ruled group.

		Prefill ktok/s $\uparrow$				ms $\downarrow$	Peak GB $\downarrow$
Model	Params (B)	2K	8K	32K	128K	2K	2K	128K
ESSH, $T_{1}$	0.474	75.0	108.5	110.4	109.7	27.3	1.51	35.67
ESSH, $T_{2}$	0.575	79.8	106.0	107.6	107.6	25.7	1.70	35.86
ESSH, $T_{3}$	0.716	77.9	101.9	103.2	103.4	26.3	1.99	36.15
ESSH, $T_{4}$	0.828	81.4	99.1	101.0	100.9	25.2	2.21	36.36
ESSH, $T_{5}$	0.931	74.6	96.8	98.4	98.5	27.5	2.42	36.57

Appendix

Table 60: B300 batch-one full-logit prefill. Throughput: ktok/s; latency: ms; peak allocation: decimal GB. Full models use three sessions, with three warm-ups and five timed calls per session. Compact rows use single-session measurements. Full-model parameter counts differ by at most

0.34\%

from ESSH. Bold is ranked separately within each ruled group.

(a) Tier sweeps on A100 and B300
Family	Tier	Params (M)	Graph ms $\downarrow$		Eager ms $\downarrow$
			A100	B300	A100	B300
ESSH	$T_{1}$	92.1	0.91	0.60	3.21	2.27
ESSH	$T_{2}$	118.9	0.94	0.65	3.30	2.32
ESSH	$T_{3}$	154.7	1.01	0.68	3.40	2.43
ESSH	$T_{4}$	185.3	1.04	0.69	3.49	2.42

Appendix

Table 61: Batch-one decode at the 370M scale, with a 2K cache. (a) A100/B300 tier sweeps, with each family ranked internally before the full-model comparison. (b) H200 full-model repeat at the stated parameter counts. Values are three-session medians. ESSH state is fp32 and KV bf16; baselines retain native precision. Bold marks the best value within each comparison group and device/execution column. Equal tier labels across families need not have equal sizes.

Full models near 370M: graph decode, batch one (ms $\downarrow$ )
Measurement / model	Params (M)	A100	H200	B300
ESSH	370.9	1.307	0.910	0.735
MatFormer	365.6	3.464	2.819	2.443
MatMamba	369.9	3.416	2.334	2.562
Full models near 1.53B: graph decode, batch 1 (ms $\downarrow$ )
ESSH, $T_{10}$	1531.1	3.06	1.67	1.37

Appendix

Table 62: Cross-device efficiency at the stated parameter counts. Graph decode uses a 2K cache; prefill uses batch one. Full-model rows use three-session medians at matched parameter counts. A100 MIMO uses a separate matched repeat; OOM is measured allocation failure. Bold ranks within each group and device. Memory is decimal; ESSH state is fp32 and KV bf16, with native baseline state precision.

Selective SSMs

Selective state-space models (SSMs) use input-dependent recurrent dynamics to filter sequence information with linear-time scans.

See all 70 papers

May 8, 2026

Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control

Training a family of large language models (LLMs), either from scratch or via iterative compression, is prohibitively expensive and inefficient, requiring separate training runs for each model in the family. In this paper, we introduce Star Elastic, a novel LLM post-training method that adds N nested submodels to a given parent reasoning model using the compute of one run (N-fold savings) via a single post-training job. Beyond reducing training costs, Star Elastic also addresses a fundamental limitation of efficient reasoning: the rigidity of static architectures, which forces the allocation of constant resources regardless of token difficulty. By unlocking elastic budget control, Star Elastic enables a novel inference scheme that uses different submodels for each reasoning phase (thinking and answering). Star Elastic supports (1) nesting along the SSM, embedding channel, MoE, and FFN axes, (2) learning nested submodels via an end-to-end trainable router, and (3) curriculum-based knowledge distillation. Building on the Nemotron Elastic framework, we apply Star Elastic to the NVIDIA Nemotron Nano models, with a particular focus on hybrid Mixture-of-Experts (MoE) architectures: from Nemotron Nano v3 (30B/3.6A), we generate 23B (2.8A) and 12B (2.0A) variants with 160B training tokens. All nested models match or outperform independently trained baselines of comparable size and achieve a 360x reduction versus pretraining from scratch and a 7x reduction over state-of-the-art compression. Crucially, elastic budget control advances the accuracy-latency Pareto frontier, achieving up to 16% higher accuracy and 1.9x lower latency via dynamic per-phase model selection. We further extend Star Elastic to quantized regimes via Quantization-Aware Distillation (QAD), producing nested NVFP4 and FP8 elastic checkpoints that preserve zero-shot slicing while delivering smaller deployment footprints.

Ali Taghibakhshi, Ruisi Cai, Saurav Muralidharan +17

NVIDIA

LLM Compression LLM Post-Training

Ask PDF

Oct 8, 2026

One Model to Loop Them All: Elastic Latent Recurrence for Language Models

A latent recurrent language model applies a shared block of layers repeatedly, so one checkpoint can trade inference compute for accuracy by changing the loop count. At one loop, however, such models typically fall well short of a non-recurrent model of the same size, a one-loop tax that makes a separate non-recurrent model the better choice whenever latency matters. We present Elastic Latent Recurrence (ELR), a recipe for training looped language models without this tax. At one loop, an ELR model costs the same to serve as a non-recurrent model of the same size and, trained on the same data, reaches similar accuracy. Each additional loop then buys more accuracy for more inference compute. Three choices make this work: a gated state update with a learned step size, loop counts mixed within each optimizer update, and conditioning each pass on its loop index and budget. The first two preserve one-loop accuracy, and the third lowers loss at every additional loop. Mixed depths can be introduced late in pretraining to limit the added training cost, and with zero-initialized conditioning added at the switch, this late schedule reaches lower loss than mixed-depth training from scratch. In MoE models, reusing first-loop expert routes on later loops cuts mixed-depth update time by 11.7% with no measurable change in accuracy. At inference, keeping only the last loop's attention KV cache holds attention KV memory at its one-loop level while retaining most of the gain from additional loops. On 4B dense and 30B-A3B MoE hybrid models, ELR at one loop is 0.52 points below and 1.04 points above its matched baseline on a ten-task average, and four loops add 7.25 and 4.75 points. Carried through post-training, the recipe yields Koi-4B, a single checkpoint that scores an estimated 18 on the Artificial Analysis Intelligence Index at one loop, matching Qwen3.5-4B, and 29 at four loops.

Faisal Ladhak, Shantanu Acharya, Santiago Akle Serano +3

NVIDIA

Language Modeling Recurrent Language Models Recurrent Transformers Efficient Language Model Inference Adaptive Computation

Ask PDF

Oct 5, 2026

Stepped MoE: Segment-Level Routing with Configurable Inference Complexity

Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures enable flexible model deployment and sparsely activated models allow input-adaptive computation, existing approaches treat these dimensions independently. Moreover, models catered towards on-device edge inference need to conform to the memory and compute limitations of the serving devices. In this paper, we introduce a unified framework that combines elastic structures with sparsely gated architectures to create models that adapt simultaneously to both deployment constraints and task requirements. Our approach employs a model backbone that conditions on both the context and target efficiency specifications, enabling fine-grained control over the accuracy-efficiency trade-off at inference time. The model learns to activate task-relevant parameters within elastically-nested sub-networks, allowing a single model to span multiple capacity points while maintaining input-adaptive routing. Through experiments we demonstrate that we can create a model that allows the flexibility to use 1,2,3,4 billion parameters while being more accurate than their dense counter-parts (2-5% on knowledge-intensive benchmarks) and at par with their static versions while delivering similar latency metrics as dense models. Overall, we save on device disk space by sharing the model parameters, allow flexibility of serving based on DRAM and compute available while delivering more accurate results.

Arnav Kundu, Zhaoyang Xu, Bairu Hou +3

Apple

On-Device Language Model Inference Dynamic Neural Networks Mixture of Experts Elastic Neural Networks LLM Routing Efficient Language Model Inference

Ask PDF

Abstract

Selective SSMs

Explore similar work

Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control

One Model to Loop Them All: Elastic Latent Recurrence for Language Models

Stepped MoE: Segment-Level Routing with Configurable Inference Complexity