Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs

Maria Lomeli; Antoine Groudiev; Matthijs Douze; Loïc Cabannes; Pierre Emmanuel Mazaré; François Fleuret; Maximilian Beck; Gergely Szilvasy

cs.LGSep 25, 2026

Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs

Authors: Maria Lomeli, Antoine Groudiev, Matthijs Douze, Loïc Cabannes, Pierre Emmanuel Mazaré, François Fleuret, Maximilian Beck, Gergely Szilvasy,

Organizations: Meta FAIR · École Normale Supérieure – PSL · École Normale Supérieure Paris-Saclay

Abstract

This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference time, we first compute the output of the gating matrix and, thanks to its high sparsity, we avoid computations with the two other matrices, reducing the FLOP count by up to 3x. While this theoretical speedup is an upper bound, model casting translates into significant speedups both on CPU and GPU. We then introduce LoPA Gating, a new FFN design that increases the maximum theoretical speedup. It is a low-FLOPs parameterization of the gating matrix that overcomes the 3x cap by allocating fewer FLOPs and parameters to the gating matrix, compared to the two other FFN matrices that are sparsely activated. We consider two cases: (i) we cast a pre-trained model with a sparsity inducing activation; (ii) we train with LoPA from scratch. In all settings, we significantly outperform existing pruning solutions and regular RELU-fication. For instance, at matched quality, we achieve a 3.2x FLOP speedup with LoPA Casting, against 1.6x at best for competing methods top-p and TEAL. Using dedicated kernels, we achieve an actual 3.31x speed-up on GPU at 90% sparsity, past the 3x ceiling of standard gating; RELU-fication, meanwhile, plateaus below 80% sparsity.

Figures & tables

Figure 1: Model casting speedup at comparable quality, vs the baseline dense model (reference

\times 1

and NLL =

2.145

) and existing sparsification techniques: RELU -fication and the training-free top-

p

and TEAL. At a

90\%

sparsity target, LoPA Casting provides a

5.8\times

speed-up for a

0.93\%

NLL increase over the dense baseline. See detailed results in Tables 1 and 2 , and practical speed-up in Section 5.3 .

Figure 2: Fewer FLOPs with sparsity and LoPA Gating. FLOPs per linear layer at

D\,{=}\,2560

H\,{=}\,7168

and rank

r\,{=}\,400

, with

90\%

sparse R-S+ gates. In (c), the hidden dimension grows from

H

H^{\prime}{=}\,9787

to keep a constant parameter count. The FLOPs reductions are due to sparsity in (b,c) and to LoPA Gating in (c).

Setup (pre-training $\rightarrow$ mid-training)	NLL	$\Delta$ NLL	Sparsity	Speedup
Baselines: mid-training and RELU -fication
Base[ SILU ] $\to$ Base[ SILU ] (dense baseline)	2.145	—	—	$1.00\times$
Base[ SILU ] $\to$ Base[ RELU ] ( RELU -fication)	2.159	$+0.65\%$	76%	$2.00\times$
(a) Activation Casting
Base[ SILU ] $\to$ Base[ R-S+ ]	2.153	$+0.36\%$	69%	$1.84\times$
Base[ SILU ] $\to$ Base[ R-S+ , $\ell_{1}$ @90%]	2.160	$+0.68\%$	89%	$2.48\times$

Table 1: Model casting enables significant FLOPs speedups at low NLL impact. Results for 0.81B-parameter models (level 8 in Table 12 ).

\Delta

NLL is computed relative to the first row in bold . We indicate the Base or LoPA architecture with the activation [in square brackets].

		Theoretical FLOP speedup $\uparrow$
FFN Architecture	Ref. NLL $\downarrow$	Model casting @75% (ours)	top- $p$	TEAL
Standard Gating	2.154	2.0	1.4	1.4
LoPA Gating	2.153	3.2	1.3	1.3

Table 2: Trained-in sparsity gives a larger FLOP speedup than post-hoc pruning at matched quality. FFN theoretical FLOP speedup (

1/\text{FLOP-ratio}

; dense

=1\times

, higher is better), at batch size 1, with all methods matched at the reference NLL. See Appendix B for other training budgets.

Method	Budget	Decay	Sparsity (%)	NLL	Speedup
Pretrained (no casting)	0T	—	—	1.843	$1.00\times$
	1T	✗	89.6	1.912	$2.48\times$
Activation Casting	2T	✗	89.6	1.908	$2.48\times$
PT $\rightarrow$ Base[ R-S+ , $\ell_{1}$ @90%]	4T	✗	89.6	1.907	$2.48\times$
	5T	✓	89.5	1.875	$2.48\times$
	1T	✗	89.6	2.020	$8.01\times$

Table 3: Qwen3-1.7B base model mid-trained with model casting,

\gamma\,{=}\,7.4{\times}10^{-5}

. The 0T row is the pretrained checkpoint that we try to recover with sparse models. “Decay” indicates if the 1T cosine annealing is applied.

Figure 5: Sparsity translates to wall time improvements on CPU and GPU. Throughput as a function of activation sparsity, for a 5.79B non-embedding parameters model (level 13 in the scaling ladder from Table 12 ) in float32. CPU times are end-to-end; GPU times cover the FFN only, with the memory-bandwidth roofline shown as an upper bound. At 90% sparsity, the speedup on GPU is

2.06\times

for Standard Gating and

3.31\times

for LoPA Gating.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Illustration of the non-learnable component. We depict, for $H=12$ and $D=6$ , the fixed component $M\in\mathbb{R}^{H\times D}$ of three low-rank variants. (a) Low-rank + Identity uses the padded identity $I_{\text{pad}}$ . (b) Low-rank + Diagonal uses the repeated diagonal $G$ . (c) Low-rank + Tiled Identity uses the tiled identity $T$ , that is $(Tx)_{i}=x_{\,i\bmod D}$ . All non-zeros are fixed $1$ s (gray). The last two variants, or any permutation of those, correspond to what we call LoPA. The simple identity $I_{\text{pad}}$ differs from LoPA as some rows only contain zeros.

	Average validation NLL
	Level 2				Level 8
Parameterization	$r{=}64$	$r{=}128$	$r{=}256$	$r{=}512$	$r{=}128$	$r{=}256$	$r{=}512$	$r{=}1024$
Full matrix (baseline)	$2.800\pm 0.005$				$2.152\pm 0.002$
Full matrix, double activation	$2.795$				$2.153$
Low-rank	$2.829$	$2.813$	$2.804$	$2.806$	$2.183$	$2.167$	$2.154$	$2.153$
Low-rank + Identity	$2.810$	$2.799$	$2.795$	$2.805$	$2.159$	$2.156$	$2.152$	$2.154$

Appendix

Table 4: Ablations of different design choices for the

W_{1}

matrix. Unless stated otherwise,

W_{1}

is parameterized using LoPA Gating consisting of a Low-rank + Diagonal matrices, with double activation (activations on the outputs of both

W_{1}

and

W_{3}

). All models are trained with SILU for a duration of

8T

, without any sparsity loss, and have the same parameter count. We report the standard deviation of the NLL of the baseline, computed over 4 seeds. † Low-rank + Diag and Low-rank + Tiled Identity are identical up to a permutation. We include their comparison as a sanity check. Their minor differences in NLL are to be attributed to training noise.

Symbol	Tensor	Dim	Produced by	Skips
$s_{g}$	input $x\!\to\!W_{1}$	$D$	TEAL/top-p	$W_{1}$ input read ( $V$ )
$s_{u}$	input $x\!\to\!W_{3}$	$D$	TEAL/top-p	$W_{3}$ input read
$s_{\mathrm{pre}}$	$A(W_{1}x)$	$H^{\prime}$	ours (trained), top-p (gate)	$W_{3}$ cols $+$ $W_{2}$ rows
$s_{\mathrm{gated}}$	$\mathrm{out2}\cdot x_{3}$	$H^{\prime}$	ours, TEAL/top-p	$W_{2}$ rows

Appendix

Table 5: Summary of sparsification approaches. The two FFN sparsity mechanisms and the matmuls each reduces. Input sparsity (

s_{g},s_{u}

; dim

D

) skips input coordinates; hidden-activation sparsity (

s_{\mathrm{pre}},s_{\mathrm{gated}}

; dim

H^{\prime}

) skips hidden units.

				Theoretical FLOP speedup $\uparrow$
FFN Architecture	MT	Decay	Ref. NLL $\downarrow$	Ours @75%	top- $p$	TEAL
Activation Casting	1T	✗	2.326	2.0	1.6	1.6
	2T	✗	2.307	2.0	1.6	1.5
	4T	✗	2.294	2.0	1.5	1.4
	4T	✓	2.154	2.0	1.4	1.4
LoPA Casting	1T	✗	2.331	3.2	1.5	1.6

Appendix

Table 6: Full version of Table 2 : trained-in sparsity gives a larger theoretical FLOP speedup than post-hoc pruning at matched quality. FFN theoretical FLOP speedup, at a shared reference NLL equal to our method at 75% sparsity. Models are pre-trained

4T

then cast for the listed mid-training (MT) budget; ✓ adds a

1T

decay. Table 2 reports the decayed

4T

row only; here we give every mid-training budget, with and without decay. Casting stays near

2.0\times

for Activation Casting and

3.2\times

for LoPA Casting, while top-

p

and TEAL fall toward

1.3\times

as the model is mid-trained longer.

			Model casting @90% (ours)			RELU -fication
FFN Architecture	MT	Decay	NLL $\downarrow$	FLOP sp. $\uparrow$	Spars. $\uparrow$	NLL $\downarrow$	FLOP sp. $\uparrow$	Spars. $\uparrow$
Standard Gating	1T	✗	2.340	2.5	89%	2.330	2.0	75%
	2T	✗	2.315	2.5	89%	2.312	2.1	77%
	4T	✗	2.298	2.5	89%	2.299	2.1	79%
	4T	✓	2.160	2.5	89%	2.159	2.0	76%
LoPA Gating	1T	✗	2.346	5.8	89%	2.326	2.3	62%

Appendix

Table 7: Full version of the RELU -fication comparison of Table 1 : model casting reaches higher sparsity at a larger theoretical FLOP speedup. Both are trained-activation methods reported at their own operating points. Table 1 reports the decayed

4T

operating point only; here we give every mid-training budget, with and without decay. At comparable NLL, model casting attains

89\%

sparsity for both Standard Gating and LoPA Gating, whereas RELU -fication plateaus at

75

–

79\%

and

57

–

62\%

respectively.

Router	$\lambda$ (PT)	$\lambda$ (MT)	Train	MT	Avg NLL	Avg sparsity
R-S+	0	0	1T	4T	1.981	5%
R-S+ , fixed $\lambda$	0.1	0.1	1T	4T	1.999	49%
Sigmoid, top- $k$	—	—	1T	4T	2.007	75%
R-S+ , fixed $\lambda$	0.3	0.1	1T	4T	2.011	65%
R-S+ , adaptive $\lambda$	0.3	@70%	1T	4T	2.023	70%
R-S+ , adaptive $\lambda$	0.1	@70%	1T	4T	2.027	71%

Appendix

Table 8: MoE’s Average NLL and router sparsity . We group runs by MT budget (4T, 2T, 1T), and rank them by NLL within each group. As expected, R-S+ without sparsity loss on the router results in the best NLL but very little sparsity (the model using all the experts most of the time); best sparsity is obtained using the usual sigmoid and top-

k

routing, but model casting can find intermediate operating points.

Figure 7: MoE FFNs sparsity vs. NLL for different values for the

\ell_{1}

penalty weight, using cosine schedule. The baseline corresponds to model casting of the FFN to R-S+ with

\lambda=0

. The sparsity can be increased by about 25%, but at the cost of considerable NLL degradation, greater than in the non-MoE case.

Activation	MT	$\lambda$	NLL	$\Delta$ NLL	Sparsity (FFN)
SILU (pre-trained)	0T	0	2.008	+1.36%	–
SILU	4T	0	1.981	+0.00%	–
R-S+	4T	0	2.020	+1.97%	56.4%
R-S+	4T	$10^{-3}$	2.014	+1.65%	59.0%
R-S+	4T	$10^{-2}$	2.017	+1.82%	66.2%
R-S+	4T	@75%	2.022	+2.04%	72.8%

Appendix

Table 9: NLL degradation for MoE FFN Activation Casting. We report the average NLL and sparsity for different penalty weights for

\ell_{1}

loss with fixed or adaptive

\lambda

, in the FFN Activation Casting setup for MoEs. Rows with @

x\%

use an adaptive

\ell_{1}

loss for a given sparsity target of

x\%

. We mid-train the MoE with 8 experts top-

2

with model casting on the experts FFN blocks using a WSD learning rate schedule. All models start from the same pre-trained checkpoint, trained for 4T.

Activation	top- $k$	MT	NLL	Sparsity (FFN)
SILU (pre-trained)	2	0T	2.008	–
R-S+ @90%	2	4T	2.064	88.6%
SILU (pre-trained)	1	0T	2.091	–
SILU	1	4T	2.073	–

Appendix

Table 10: FFN post-activation sparsity vs. lower number of experts. Performance of MoE top-

1

vs MoE top-

2

, out of 8 experts. R-S+ @90% with 2 active experts results in lower NLL and lower FLOPs than SILU with 1 active expert.

Component	Precision	FP8 format
Attention / norms / embeddings / residual	bf16	—
Router GEMM + gating ( SILU ) + routing losses	fp32	—
Expert FFN — forward	fp8 ( $1{\times}128$ act, $128{\times}128$ wt)	E4M3
Expert FFN — dgrad (incl. grad. quant.)	fp8 (tile-wise $1{\times}128$ )	E4M3
Expert FFN — wgrad ( SILU )	fp8	E4M3
Expert FFN — wgrad ( R-S+ )	bf16	—

Appendix

Table 11: Mixed-precision training setup. All fp8 tensors use the E4M3 format (no E5M2).

Figure 8: Impact of a fixed

\ell_{1}

loss. NLL as a function of sparsity for different fixed

\lambda

values, for level 2, 4, 6, 8 and 10 dense models mid-trained with Activation Casting. The numbers on the markers indicate

\lambda

, the weight of the

\ell_{1}

loss.

		Architecture				Training budget
Scale	Non-Embed Params (B)	$D$	$H$	$L$	Peak LR ( $\gamma$ )	Train FLOPs	Total Tokens (B)
0	$0.03$	$640$	$2048$	$7$	$2.06\text{\times}{10}^{-3}$	$7.3\text{\times}{10}^{18}$	$7.65$
1	$0.05$	$768$	$2048$	$8$	$1.87\text{\times}{10}^{-3}$	$1.3\text{\times}{10}^{19}$	$10.73$
2	$0.07$	$768$	$2048$	$11$	$1.70\text{\times}{10}^{-3}$	$2.2\text{\times}{10}^{19}$	$14.86$
3	$0.12$	$1024$	$3072$	$10$	$1.50\text{\times}{10}^{-3}$	$5.1\text{\times}{10}^{19}$	$24.33$
4	$0.17$	$1024$	$3072$	$14$	$1.35\text{\times}{10}^{-3}$	$8.9\text{\times}{10}^{19}$	$33.97$

Appendix

Table 12: Scaling ladder. Non-embedding parameters, architecture (FFN input dimension

D

, FFN hidden dimension

H

, number of layers

L

), peak learning rate

\gamma

and training budget per compute level. The cosine schedule decays to

\gamma/100

			Score: NLL				$\Delta$ NLL vs reduced params
Training duration			no $\ell_{1}$		$\ell_{1}$	Reduced params	no $\ell_{1}$		$\ell_{1}$
PT	MT	Total	SILU	R-S+	R-S+	SILU	SILU	R-S+	R-S+
4T	0T	5T	2.240	2.255	2.276	2.217	+1.05%	+1.73%	+2.66%
4T	0.5T	5.5T	2.216	2.226	2.240	2.207	+0.42%	+0.87%	+1.53%
4T	1T	6T	2.203	2.211	2.222	2.200	+0.16%	+0.50%	+1.01%
4T	2T	7T	2.197	2.204	2.212	2.199	-0.09%	+0.22%	+0.61%

Appendix

Table 13: Architecture Casting produces models with competitive NLL for their parameter count. Impact of casting time (“MT” column) on the NLL of models. We compare level 8 models trained with Architecture Casting (three first NLL columns) against a dense SILU model trained from scratch with hidden dimension reduced to

H=3712

so that its FFN parameter count matches that of the cast models.

(a) Activation Casting
Training duration		Score: NLL			R-S+ sparsity
Casting	Total	SILU	R-S+	R-S+ , $\ell_{1}$	no $\ell_{1}$	$\ell_{1}$
0T	5T	2.178	2.190	2.214	61%	89%
0.5T	5.5T	2.169	2.179	2.197	63%	89%
1T	6T	2.162	2.171	2.185	65%	89%
2T	7T	2.161	2.170	2.179	66%	89%

Appendix

Table 14: Evolution of NLL during casting. Detailed results when training a 0.81B dense model (level 8 in the scaling ladder from Table 12 ), performing the three model casting recipes. All models are pre-trained for 4T; the “Casting” column gives the mid-training budget and “Total” counts every token spent, including the

1T

of decay. For LoPA and Architecture Casting, the rank ratio

\alpha

is set to

1/8

. The

\ell_{1}

loss targets

90\%

sparsity at training time; the sparsity measured on the validation set is about one point lower, which is why the

\ell_{1}

columns read

89\%

. In addition to R-S+ , we simulate a model casting with SILU as a control experiment.

Setup (pre-training $\rightarrow$ mid-training)	NLL	$\Delta$ NLL	Sparsity	Speedup
Base[ SILU ] $\to$ Base[ SILU ] (dense baseline)	1.781	—	—	$1.00\times$
LoPA[ SILU ] $\to$ LoPA[ R-S+ , $\ell_{1}$ @90%]	1.792	$+0.65\%$	89%	$5.86\times$

Appendix

Table 15: Scaling up model casting. Results for 5.79B-parameter models (level 13 in Table 12 ).

\Delta

NLL is computed relative to the first row in bold , corresponding to a control experiment conducted with SILU throughout training. All models are pretrained for 4T tokens, and midtrained for 4T tokens of stable learning rate, followed by 1T tokens of learning rate decay.

(a) Activation Casting
		Casting without $\ell_{1}$			Casting with $\ell_{1}$
Model scale	NLL	Duration	NLL	Sparsity	Duration	NLL	Sparsity
2	3.013	1.00T	3.011	66%	1.50T	3.004	89%
4	2.738	0.75T	2.725	67%	2.00T	2.735	89%
6	2.521	1.00T	2.518	67%	1.50T	2.516	89%
8	2.321	1.25T	2.317	70%	1.75T	2.313	89%

Appendix

Table 16: Mid-training duration required to catch up with the SILU model’s performance, without the decay phase, for different model sizes with pre-trained models of 4T duration. Models are evaluated every 0.25T, and we report the first time at which the finetuned model catches up to the original NLL, rounded up by at most 0.25T. When the

\ell_{1}

loss is applied (rightmost columns), fixing the target sparsity at

90\%

during training leads to a measured sparsity around

89\%

. For LoPA Casting, the rank ratio

\alpha

is set to

1/8

		FP32		int8		int4
Casting family	Variant	NLL	$\Delta$ NLL (%)	NLL	$\Delta$ NLL (%)	NLL	$\Delta$ NLL (%)
Activation	SILU	2.145	—	2.160	+0.70%	2.178	+1.54%
Activation	R-S+ + $\ell_{1}$ @90%	2.160	+0.68%	2.173	+1.29%	2.188	+2.01%
LoPA	SILU	2.147	+0.09%	2.162	+0.76%	2.180	+1.62%
LoPA	R-S+ + $\ell_{1}$ @90%	2.166	+0.99%	2.179	+1.57%	2.194	+2.29%
Architecture	SILU	2.180	+1.63%	2.193	+2.21%	2.213	+3.16%

Appendix

Table 17: Impact on NLL of quantization and casting. After Activation, LoPA, and Architecture casting, we apply QAT and quantize models in a weight-only int4 and int8 setting. Degradation is relative to the activation-casting SILU model at full precision in bold .

Figure 9: Measured speedups for quantized sparse models. FFN-bound throughput as a function of the activation sparsity, for a 5.79B non-embedding parameters model (level 13 in the scaling ladder from Table 12 ) at different quantization levels ( int8 and int4 ).

Panel of Figure 10 — Figure 10: $\ell_{1}$ loss without model casting induces higher performance degradation for the same sparsity penalty $\lambda$ with level 2 (top) and level 8 (bottom) models.

Level 2
		$\ell_{1}$ loss weight ( $\lambda$ )
Metric	$\ell_{1}$ during PT	$10^{-4}$	$3\times 10^{-4}$	$10^{-3}$	$3\times 10^{-3}$	$10^{-2}$	$3\times 10^{-2}$
NLL	✓	$\mathbf{2.749}$	$\mathbf{2.748}$	$\mathbf{2.747}$	$2.750$	$2.751$	$2.754$
NLL	✗	$2.750$	$2.750$	$2.750$	$\mathbf{2.749}$	$\mathbf{2.750}$	$\mathbf{2.753}$
Sparsity	✓	$\mathbf{65.3\%}$	$65.6\%$	$66.7\%$	$68.1\%$	$71.0\%$	$73.6\%$
Sparsity	✗	$65.2\%$	$\mathbf{65.7\%}$	$\mathbf{66.8\%}$	$\mathbf{69.3\%}$	$\mathbf{74.5\%}$	$\mathbf{80.9\%}$

Appendix

Table 18: Using

\ell_{1}

loss during pre-training degrades final sparsity. Ablations of using the

\ell_{1}

loss during pre-training. Models are pre-trained and cast with the same

\ell_{1}

weight

\lambda

. Adding the

\ell_{1}

loss on the output of SILU during pre-training results in similar or worse NLL (up to training noise), and reduces the sparsity level after casting.

	Level 2					Level 8
	NLL $\downarrow$				Spars.	NLL $\downarrow$				Spars.
Activation	$1T$	$2T$	$4T$	+D	+D	$1T$	$2T$	$4T$	+D	+D
R-S+	3.011	2.984	2.966	2.804	65%	2.326	2.307	2.295	2.153	69%
SILU	2.999	2.970	2.945	2.794	—	2.326	2.307	2.295	2.153	—
G-S+	2.999	2.974	2.944	2.808	—	2.314	2.298	2.287	2.148	—
GELU	2.999	2.973	2.955	2.808	—	2.314	2.297	2.287	2.147	—

Appendix

Table 19: SILU and GELU result in the best performances for casting. NLL and FFN activation sparsity when casting with different activation functions. Both models are pre-trained for

4T

; each NLL column is a mid-training budget, and “+D” adds the final

1T

decay. Sparsity varies little with the budget, so we report it at the decayed checkpoint only. SILU , G-S+ and GELU are not sparsifying activations and have no exact zeros to report; they bound the attainable NLL rather than competing on sparsity. Bold marks the best NLL per column.

Core-12 short-context benchmarks
	Peak LR $\gamma$
Task	$1.00\times 10^{-5}$	$7.40\times 10^{-5}$ $\ast$	$1.05\times 10^{-4}$	$1.48\times 10^{-4}$	$2.50\times 10^{-4}$	$7.40\times 10^{-4}$
HSwag	.6279	.6451	.6458	.6439	.6359	.5969
ARC-e	.7268	.7281	.7328	.7230	.7087	.6283
ARC-c	.4489	.4575	.4592	.4403	.4206	.3519
PIQA	.7361	.7503	.7470	.7448	.7503	.7388

Appendix

Table 20: Short context evaluations for different peak-lr for mid-training Qwen3-1.7B. The evaluation is computed post-cooldown at step 1,250. All arms branch from Qwen3-1.7B-Base with the R-S+AO activation swap. Bold marks the best arm per task; ∗ marks the chosen LR.

Peak LR $\gamma$	In-mixture	$\Delta$ vs. base	All 9	GovReport	Sparsity (%)
Pretrained Base	1.711	—	1.843	2.152	—
Swap, 0 steps	10.256	$+8.544$	9.996	9.788	76.77
$1.00\times 10^{-5}$	1.761	$+0.050$	1.927	2.212	79.79
$7.40\times 10^{-5}$ $\ast$	1.724	$\mathbf{+0.013}$	1.895	2.188	78.59
$1.05\times 10^{-4}$	1.727	$+0.015$	1.898	2.190	78.60
$1.48\times 10^{-4}$	1.734	$+0.022$	1.906	2.195	78.76

Appendix

Table 21: Validation NLLs using different peak-lr for mid-training Qwen3-1.7B. Held-out NLL per token and FFN activation sparsity for Qwen3-1.7B. Every arm runs

0.093T

(1,250 steps, 2.62 B tokens) without

\ell_{1}

loss. Bold marks the best arm per NLL column; ∗ marks the chosen learning rate.

Run	Budget	In-mixture	$\Delta$ vs. base	All 9	GovReport	Sparsity (%)
						(All 9)
Pretrained Base	—	1.711	—	1.843	2.152	—
Swap, 0 steps	$0T$	10.256	$+8.544$	9.996	9.788	76.77
R-S+	$1T$	1.702	$-0.010$	1.881	2.163	76.84
	$2T$	1.698	$-0.013$	1.881	2.155	76.43
	$3T$	1.699	$-0.012$	1.884	2.151	76.25

Appendix

Table 22: Mid-training Qwen3-1.7B with model casting. Held-out NLL per token and FFN activation sparsity for Qwen3-1.7B mid-training at the swept-optimum peak

\gamma=7.40\times 10^{-5}

. Runs differ only in FFN activation and sparsity penalty: R-S+ is the swap with no penalty, “Activation Casting @90%” adds an adaptive-

\ell_{1}

penalty targeting 90% sparsity, and SILU , no swap is plain mid-training. Bold marks the best trained checkpoint per NLL column.

Figure 11: Measured speedups in bfloat16. Throughput as a function of the activation sparsity, for a 5.79B non-embedding parameters model (level 13 in the scaling ladder from Table 12 ) in bfloat16. At 90% sparsity the speedup on GPU is

1.71\times

for Standard Gating and

2.41\times

for LoPA Gating compared to the standard dense case.

Benchmark	Metric	Type
HellaSwag	acc_char	Choice
WinoGrande	acc_char	Choice
ARC-e	acc_char	Choice
ARC-c	acc_char	Choice
PIQA	acc_char	Choice
OBQA	acc_char	Choice

Appendix

Table 23: Summary of benchmarks used for evaluation. We prompt level 8 models in a few-shot setting, and report a metric that depends on the benchmark, as well as the average performance. acc_char is the Character-Length Normalized Accuracy.

			$\Delta$ vs. SILU control (points)
Task	Qwen3-1.7B-Base	SILU , no swap	R-S+	Activation Casting
	(pre-trained)	(control)	(no $\ell_{1}$ )	@0.90
Core-12 short-context
HellaSwag	66.5	69.7	$-0.7$	$-1.2$
ARC-e	68.1	72.7	$-1.6$	$-1.6$
ARC-c	45.1	45.0	$-0.2$	$\mathbf{-2.5}$

Appendix

Table 24: Effect of casting mid-trained Qwen3-1.7B downstream performance. Change in downstream performance (percentage points) at the decayed

5T

checkpoint (

100

TPP), at peak LR

\gamma=7.40\times 10^{-5}

. Deltas are taken against the SILU , no swap control (the same model mid-trained on the same mixture without the activation swap) so they isolate the cost of casting from the cost of the mid-training mixture.

Task	0T	4T	75%	90%	0T	4T	75%	90%
	Activation Casting				LoPA Casting
HS	57.0	59.1	$-0.4$	$-1.1$	56.7	59.1	$-1.0$	$-1.6$
WG	59.1	59.4	$-0.2$	$-0.8$	57.1	58.2	$-0.3$	$+1.5$
ARC-e	60.3	61.5	$-0.8$	$-1.4$	57.3	57.8	$\mathbf{+2.1}$	$\mathbf{+2.6}$
ARC-c	31.2	34.2	$-1.0$	$-1.5$	31.0	31.9	$+0.6$	$+0.3$
PIQA	72.6	73.0	$+0.5$	$+0.5$	73.0	74.0	$-0.5$	$-0.5$

Appendix

Table 25: Effect of casting mid-trained level-8 models on downstream performance. Within each family the deltas are taken against that family’s 4T column, that is the same model mid-trained for the same budget without casting. The 0T and 4T columns are absolute %: 0T is the pre-trained checkpoint (4T WS + 1T D) before mid-training, so comparing the two gives the gain from mid-training alone. All rows are decayed. Bold marks changes of at least 2 points. Per-benchmark metrics are given in Table 23 .

Setup (pre-training $\to$ mid-training)	pre-LCFT	LCFT	SFT	$\Delta_{\mathrm{pre}}$	$\Delta_{\mathrm{SFT}}$	$\Delta_{\mathrm{corr}}$
	Core-12 average			relative to the dense baseline
Base[ SILU ] $\to$ Base[ SILU ]	39.6	39.7	36.6	—	—	—
Base[ SILU ] $\to$ Base[ R-S+ , $\ell_{1}$ @75%]	39.5	39.0	36.8	$-0.1$	$+0.2$	$+0.3$
Base[ SILU ] $\to$ Base[ R-S+ , $\ell_{1}$ @90%]	39.2	38.7	36.0	$-0.4$	$-0.6$	$-0.2$
LoPA[ SILU ] $\to$ LoPA[ R-S+ , $\ell_{1}$ @75%]	38.8	38.6	36.1	$-0.8$	$-0.5$	$+0.3$
LoPA[ SILU ] $\to$ LoPA[ R-S+ , $\ell_{1}$ @90%]	39.1	38.1	36.0	$-0.5$	$-0.6$	$-0.1$

Appendix

Table 26: Average downstream performance across post-training, relative to the SILU dense baseline.

\Delta_{\mathrm{pre}}

is the gap before post-training,

\Delta_{\mathrm{SFT}}

the gap after

8000

SFT steps, and

\Delta_{\mathrm{corr}}\,{=}\,\Delta_{\mathrm{SFT}}-\Delta_{\mathrm{pre}}

isolates what post-training itself costs each variant. All values are rounded to one decimal, so the columns reconcile exactly.

Setup (pre-training $\to$ mid-training)	score	$\Delta$	score	$\Delta$
	inst_acc		prompt_acc
Base[ SILU ] $\to$ Base[ SILU ]	33.9	—	22.9	—
Base[ SILU ] $\to$ Base[ R-S+ , $\ell_{1}$ @75%]	33.9	$+0.0\%$	22.0	$-3.9\%$
Base[ SILU ] $\to$ Base[ R-S+ , $\ell_{1}$ @90%]	32.0	$-5.6\%$	21.1	$-7.9\%$
LoPA[ SILU ] $\to$ LoPA[ R-S+ , $\ell_{1}$ @75%]	35.1	$+3.5\%$	23.3	$+1.7\%$
LoPA[ SILU ] $\to$ LoPA[ R-S+ , $\ell_{1}$ @90%]	32.5	$-4.1\%$	19.2	$-16.2\%$

Appendix

Table 27: IFEval after

8000

SFT steps. Scores are loose ; the strict variant penalizes formatting alone and is largely common-mode across variants.

\Delta

is measured against the Activation Casting dense control in the first row, a single reference for the whole table, so the LoPA Casting rows also carry the architecture difference.

Efficient Neural Network Inference

Efficient neural network inference reduces the time, memory, and energy needed to run trained models across hardware and deployment settings.

See all 470 papers

Jul 30, 2026

Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

Feed-forward networks (FFNs) dominate memory traffic and computation in large language model (LLM) inference, making them a primary target for activation sparsification. However, existing training-free methods suffer substantial model-quality degradation at high sparsity due to limitations in their channel-selection strategies. We observe that the SwiGLU intermediate state provides a highly effective channel-selection signal, but obtaining it requires costly dense computation. To address this, we present Prox, a two-stage training-free framework for sparse SwiGLU FFNs. Prox hinges on the key insight: sparse execution requires only the channel mask induced by the intermediate state, which can be constructed from the magnitude ranking of its entries rather than their exact values. Specifically, Stage 1 uses input sparsity and quantized proxy weights to construct a shared mask; Stage 2 computes the selected channels exactly, enabling sparse execution of all three projections. Across ten LLMs from six model families, Prox outperforms training-free baselines at all sparsity levels, achieves up to a

1.99\times

end-to-end decoding speedup at 70% FFN sparsity, and is compatible with quantization and sparse attention.

Jinyi Liu, Wei Chen, Pengyu Chen +4

Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Computer Science, Nanjing University of Posts and Telecommunications

Activation Sparsity Structured Sparsity LLM Inference Acceleration Neural Network Pruning

Ask PDF

May 27, 2026

PrunePath: Towards Highly Structured Sparse Language Models

Feed-forward networks (FFNs) dominate the parameter count and computation of modern language models, yet existing pruning methods often struggle to convert sparsity into hardware-friendly inference efficiency gains. We introduce PrunePath, a budget-adaptive structured sparsification framework for FFN layers. Built on MoEfication, PrunePath replaces independent expert-wise thresholding with a softmax-normalized routing distribution and activates important experts under a cumulative-mass threshold. This formulation imposes a token-level probability budget, enabling adaptive expert counts and a direct inference-time sparsity knob from a single checkpoint. Across NLU, NLG, and instruction-tuning evaluations, PrunePath achieves a favorable sparsity--performance trade-off compared with existing static pruning and MoEfication-based methods. We further implement Triton kernels for KV-cache decoding to translate the resulting structured sparsity into practical memory savings and measurable decoding-speed improvements. These results demonstrate the superior performance of PrunePath for building highly sparse, deployment-friendly large language models.

Zhexuan Gu, Zixun Fu, Yancheng Yuan

Department of Applied Mathematics, The Hong Kong Polytechnic University

Mixture-of-Experts Pruning LLM Pruning Mixture-of-Experts Language Models Structured Sparsity Neural Network Pruning Structured Pruning+3

Ask PDF

Oct 15, 2026

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

Reducing memory use in large language model (LLM) training is an ongoing challenge. Activation checkpointing saves memory by recomputation, while lossy compression changes the stored activations. The reemergence of sparse

\relu

and

\relu^2

activations creates a third opportunity: losslessly compressing the activation cache itself. We present BitSparse, a GPU-resident lossless compression format for sparse activations. Compression and reconstruction run in custom Triton kernels fused into an autograd function, shrinking the activation cache while recovering it exactly for the backward pass. Bfloat16 (BF16) and FP8 activations are supported. Optional sign-bit packing further increases compression up to close to the theoretical limit. We analyze its compression ratio and compare it to the theoretical optimum. BitSparse lowers per-token VRAM by 31% and 30% Modded-NanoGPT and Nemotron-H respectively, with overheads of no more than 1% at practical sequence lengths. Additionally, a new preallocated buffer system avoids host synchronization during training when variable size tensors are allocated.

Max Zhu, Bohang Sun, Pietro Lio +1

University of Cambridge · University of Oxford

Activation Compression Memory Compression Memory-Efficient Training Neural Network Compression Activation Sparsity

Ask PDF

Abstract

Efficient Neural Network Inference

Explore similar work

Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

PrunePath: Towards Highly Structured Sparse Language Models

BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks