cs.LGSep 25, 2026

Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs

Authors: Maria Lomeli, Antoine Groudiev, Matthijs Douze, Loïc Cabannes, Pierre Emmanuel Mazaré, François Fleuret, Maximilian Beck, Gergely Szilvasy, 

Organizations: Meta FAIR · École Normale Supérieure – PSL · École Normale Supérieure Paris-Saclay

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference time, we first compute the output of the gating matrix and, thanks to its high sparsity, we avoid computations with the two other matrices, reducing the FLOP count by up to 3x. While this theoretical speedup is an upper bound, model casting translates into significant speedups both on CPU and GPU. We then introduce LoPA Gating, a new FFN design that increases the maximum theoretical speedup. It is a low-FLOPs parameterization of the gating matrix that overcomes the 3x cap by allocating fewer FLOPs and parameters to the gating matrix, compared to the two other FFN matrices that are sparsely activated. We consider two cases: (i) we cast a pre-trained model with a sparsity inducing activation; (ii) we train with LoPA from scratch. In all settings, we significantly outperform existing pruning solutions and regular RELU-fication. For instance, at matched quality, we achieve a 3.2x FLOP speedup with LoPA Casting, against 1.6x at best for competing methods top-p and TEAL. Using dedicated kernels, we achieve an actual 3.31x speed-up on GPU at 90% sparsity, past the 3x ceiling of standard gating; RELU-fication, meanwhile, plateaus below 80% sparsity.

Figures & tables

Figure 1: Model casting speedup at comparable quality, vs the baseline dense model (reference ×1\times 1 and NLL = 2.1452.145 ) and existing sparsification techniques: RELU -fication and the training-free top- pp and TEAL. At a 90%90\% sparsity target, LoPA Casting provides a 5.8×5.8\times speed-up for a 0.93%0.93\% NLL increase over the dense baseline. See detailed results in Tables 1 and 2 , and practical speed-up in Section 5.3 .
Figure 2: Fewer FLOPs with sparsity and LoPA Gating. FLOPs per linear layer at D = 2560D\,{=}\,2560 , H = 7168H\,{=}\,7168 and rank r = 400r\,{=}\,400 , with 90%90\% sparse R-S+ gates. In (c), the hidden dimension grows from HH to H′= 9787H^{\prime}{=}\,9787 to keep a constant parameter count. The FLOPs reductions are due to sparsity in (b,c) and to LoPA Gating in (c).
Figure 3
Setup (pre-training →\rightarrow mid-training)NLLΔ\Delta NLLSparsitySpeedup
Baselines: mid-training and RELU -fication
Base[ SILU ] →\to Base[ SILU ] (dense baseline)2.145——1.00×1.00\times
Base[ SILU ] →\to Base[ RELU ] ( RELU -fication)2.159+0.65%+0.65\%76%2.00×2.00\times
(a) Activation Casting
Base[ SILU ] →\to Base[ R-S+ ]2.153+0.36%+0.36\%69%1.84×1.84\times
Base[ SILU ] →\to Base[ R-S+ , ℓ1\ell_{1} @90%]2.160+0.68%+0.68\%89%2.48×2.48\times
Table 1: Model casting enables significant FLOPs speedups at low NLL impact. Results for 0.81B-parameter models (level 8 in Table 12 ). Δ\Delta NLL is computed relative to the first row in bold . We indicate the Base or LoPA architecture with the activation [in square brackets].
  Theoretical FLOP speedup ↑\uparrow
FFN ArchitectureRef. NLL ↓\downarrowModel casting @75% (ours)top- ppTEAL
Standard Gating2.1542.01.41.4
LoPA Gating2.1533.21.31.3
Table 2: Trained-in sparsity gives a larger FLOP speedup than post-hoc pruning at matched quality. FFN theoretical FLOP speedup ( 1/FLOP-ratio1/\text{FLOP-ratio} ; dense =1×=1\times , higher is better), at batch size 1, with all methods matched at the reference NLL. See Appendix B for other training budgets.
MethodBudgetDecaySparsity (%)NLLSpeedup
Pretrained (no casting)0T——1.8431.00×1.00\times
 1T✗89.61.9122.48×2.48\times
Activation Casting2T✗89.61.9082.48×2.48\times
PT →\rightarrow Base[ R-S+ , ℓ1\ell_{1} @90%]4T✗89.61.9072.48×2.48\times
 5T✓89.51.8752.48×2.48\times
 1T✗89.62.0208.01×8.01\times
Table 3: Qwen3-1.7B base model mid-trained with model casting, γ = 7.4×10−5\gamma\,{=}\,7.4{\times}10^{-5} . The 0T row is the pretrained checkpoint that we try to recover with sparse models. “Decay” indicates if the 1T cosine annealing is applied.
Figure 5: Sparsity translates to wall time improvements on CPU and GPU. Throughput as a function of activation sparsity, for a 5.79B non-embedding parameters model (level 13 in the scaling ladder from Table 12 ) in float32. CPU times are end-to-end; GPU times cover the FFN only, with the memory-bandwidth roofline shown as an upper bound. At 90% sparsity, the speedup on GPU is 2.06×2.06\times for Standard Gating and 3.31×3.31\times for LoPA Gating.
Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Illustration of the non-learnable component. We depict, for H=12H=12 and D=6D=6 , the fixed component M∈RH×DM\in\mathbb{R}^{H\times D} of three low-rank variants. (a) Low-rank + Identity uses the padded identity IpadI_{\text{pad}} . (b) Low-rank + Diagonal uses the repeated diagonal GG . (c) Low-rank + Tiled Identity uses the tiled identity TT , that is (Tx)i=x i mod D(Tx)_{i}=x_{\,i\bmod D} . All non-zeros are fixed 11 s (gray). The last two variants, or any permutation of those, correspond to what we call LoPA. The simple identity IpadI_{\text{pad}} differs from LoPA as some rows only contain zeros.
 Average validation NLL
 Level 2Level 8
Parameterizationr=64r{=}64r=128r{=}128r=256r{=}256r=512r{=}512r=128r{=}128r=256r{=}256r=512r{=}512r=1024r{=}1024
Full matrix (baseline)2.800±0.0052.800\pm 0.0052.152±0.0022.152\pm 0.002
Full matrix, double activation2.7952.7952.1532.153
Low-rank2.8292.8292.8132.8132.8042.8042.8062.8062.1832.1832.1672.1672.1542.1542.1532.153
Low-rank + Identity2.8102.8102.7992.7992.7952.7952.8052.8052.1592.1592.1562.1562.1522.1522.1542.154
Appendix
Table 4: Ablations of different design choices for the W1W_{1} matrix. Unless stated otherwise, W1W_{1} is parameterized using LoPA Gating consisting of a Low-rank + Diagonal matrices, with double activation (activations on the outputs of both W1W_{1} and W3W_{3} ). All models are trained with SILU for a duration of 8T8T , without any sparsity loss, and have the same parameter count. We report the standard deviation of the NLL of the baseline, computed over 4 seeds. † Low-rank + Diag and Low-rank + Tiled Identity are identical up to a permutation. We include their comparison as a sanity check. Their minor differences in NLL are to be attributed to training noise.
SymbolTensorDimProduced bySkips
sgs_{g}input x ⁣→ ⁣W1x\!\to\!W_{1}DDTEAL/top-pW1W_{1} input read ( VV )
sus_{u}input x ⁣→ ⁣W3x\!\to\!W_{3}DDTEAL/top-pW3W_{3} input read
spres_{\mathrm{pre}}A(W1x)A(W_{1}x)H′H^{\prime}ours (trained), top-p (gate)W3W_{3} cols ++ W2W_{2} rows
sgateds_{\mathrm{gated}}out2⋅x3\mathrm{out2}\cdot x_{3}H′H^{\prime}ours, TEAL/top-pW2W_{2} rows
Appendix
Table 5: Summary of sparsification approaches. The two FFN sparsity mechanisms and the matmuls each reduces. Input sparsity ( sg,sus_{g},s_{u} ; dim DD ) skips input coordinates; hidden-activation sparsity ( spre,sgateds_{\mathrm{pre}},s_{\mathrm{gated}} ; dim H′H^{\prime} ) skips hidden units.
    Theoretical FLOP speedup ↑\uparrow
FFN ArchitectureMTDecayRef. NLL ↓\downarrowOurs @75%top- ppTEAL
Activation Casting1T✗2.3262.01.61.6
2T✗2.3072.01.61.5
4T✗2.2942.01.51.4
4T✓2.1542.01.41.4
LoPA Casting1T✗2.3313.21.51.6
Appendix
Table 6: Full version of Table 2 : trained-in sparsity gives a larger theoretical FLOP speedup than post-hoc pruning at matched quality. FFN theoretical FLOP speedup, at a shared reference NLL equal to our method at 75% sparsity. Models are pre-trained 4T4T then cast for the listed mid-training (MT) budget; ✓ adds a 1T1T decay. Table 2 reports the decayed 4T4T row only; here we give every mid-training budget, with and without decay. Casting stays near 2.0×2.0\times for Activation Casting and 3.2×3.2\times for LoPA Casting, while top- pp and TEAL fall toward 1.3×1.3\times as the model is mid-trained longer.
   Model casting @90% (ours)RELU -fication
FFN ArchitectureMTDecayNLL ↓\downarrowFLOP sp. ↑\uparrowSpars. ↑\uparrowNLL ↓\downarrowFLOP sp. ↑\uparrowSpars. ↑\uparrow
Standard Gating1T✗2.3402.589%2.3302.075%
2T✗2.3152.589%2.3122.177%
4T✗2.2982.589%2.2992.179%
4T✓2.1602.589%2.1592.076%
LoPA Gating1T✗2.3465.889%2.3262.362%
Appendix
Table 7: Full version of the RELU -fication comparison of Table 1 : model casting reaches higher sparsity at a larger theoretical FLOP speedup. Both are trained-activation methods reported at their own operating points. Table 1 reports the decayed 4T4T operating point only; here we give every mid-training budget, with and without decay. At comparable NLL, model casting attains 89%89\% sparsity for both Standard Gating and LoPA Gating, whereas RELU -fication plateaus at 7575 – 79%79\% and 5757 – 62%62\% respectively.
Routerλ\lambda (PT)λ\lambda (MT)TrainMTAvg NLLAvg sparsity
R-S+001T4T1.9815%
R-S+ , fixed λ\lambda0.10.11T4T1.99949%
Sigmoid, top- kk——1T4T2.00775%
R-S+ , fixed λ\lambda0.30.11T4T2.01165%
R-S+ , adaptive λ\lambda0.3@70%1T4T2.02370%
R-S+ , adaptive λ\lambda0.1@70%1T4T2.02771%
Appendix
Table 8: MoE’s Average NLL and router sparsity . We group runs by MT budget (4T, 2T, 1T), and rank them by NLL within each group. As expected, R-S+ without sparsity loss on the router results in the best NLL but very little sparsity (the model using all the experts most of the time); best sparsity is obtained using the usual sigmoid and top- kk routing, but model casting can find intermediate operating points.
Figure 7: MoE FFNs sparsity vs. NLL for different values for the ℓ1\ell_{1} penalty weight, using cosine schedule. The baseline corresponds to model casting of the FFN to R-S+ with λ=0\lambda=0 . The sparsity can be increased by about 25%, but at the cost of considerable NLL degradation, greater than in the non-MoE case.
ActivationMTλ\lambdaNLLΔ\Delta NLLSparsity (FFN)
SILU (pre-trained)0T02.008+1.36%–
SILU4T01.981+0.00%–
R-S+4T02.020+1.97%56.4%
R-S+4T10−310^{-3}2.014+1.65%59.0%
R-S+4T10−210^{-2}2.017+1.82%66.2%
R-S+4T@75%2.022+2.04%72.8%
Appendix
Table 9: NLL degradation for MoE FFN Activation Casting. We report the average NLL and sparsity for different penalty weights for ℓ1\ell_{1} loss with fixed or adaptive λ\lambda , in the FFN Activation Casting setup for MoEs. Rows with @ x%x\% use an adaptive ℓ1\ell_{1} loss for a given sparsity target of x%x\% . We mid-train the MoE with 8 experts top- 22 with model casting on the experts FFN blocks using a WSD learning rate schedule. All models start from the same pre-trained checkpoint, trained for 4T.
Activationtop- kkMTNLLSparsity (FFN)
SILU (pre-trained)20T2.008–
R-S+ @90%24T2.06488.6%
SILU (pre-trained)10T2.091–
SILU14T2.073–
Appendix
Table 10: FFN post-activation sparsity vs. lower number of experts. Performance of MoE top- 11 vs MoE top- 22 , out of 8 experts. R-S+ @90% with 2 active experts results in lower NLL and lower FLOPs than SILU with 1 active expert.
ComponentPrecisionFP8 format
Attention / norms / embeddings / residualbf16—
Router GEMM + gating ( SILU ) + routing lossesfp32—
Expert FFN — forwardfp8 ( 1×1281{\times}128 act, 128×128128{\times}128 wt)E4M3
Expert FFN — dgrad (incl. grad. quant.)fp8 (tile-wise 1×1281{\times}128 )E4M3
Expert FFN — wgrad ( SILU )fp8E4M3
Expert FFN — wgrad ( R-S+ )bf16—
Appendix
Table 11: Mixed-precision training setup. All fp8 tensors use the E4M3 format (no E5M2).
Figure 8: Impact of a fixed ℓ1\ell_{1} loss. NLL as a function of sparsity for different fixed λ\lambda values, for level 2, 4, 6, 8 and 10 dense models mid-trained with Activation Casting. The numbers on the markers indicate λ\lambda , the weight of the ℓ1\ell_{1} loss.
  Architecture Training budget
ScaleNon-Embed Params (B)DDHHLLPeak LR ( γ\gamma )Train FLOPsTotal Tokens (B)
00.030.0364064020482048772.06\times10−32.06\text{\times}{10}^{-3}7.3\times10187.3\text{\times}{10}^{18}7.657.65
10.050.0576876820482048881.87\times10−31.87\text{\times}{10}^{-3}1.3\times10191.3\text{\times}{10}^{19}10.7310.73
20.070.077687682048204811111.70\times10−31.70\text{\times}{10}^{-3}2.2\times10192.2\text{\times}{10}^{19}14.8614.86
30.120.12102410243072307210101.50\times10−31.50\text{\times}{10}^{-3}5.1\times10195.1\text{\times}{10}^{19}24.3324.33
40.170.17102410243072307214141.35\times10−31.35\text{\times}{10}^{-3}8.9\times10198.9\text{\times}{10}^{19}33.9733.97
Appendix
Table 12: Scaling ladder. Non-embedding parameters, architecture (FFN input dimension DD , FFN hidden dimension HH , number of layers LL ), peak learning rate γ\gamma and training budget per compute level. The cosine schedule decays to γ/100\gamma/100 .
 Score: NLLΔ\Delta NLL vs reduced params
Training durationno ℓ1\ell_{1}ℓ1\ell_{1}Reduced paramsno ℓ1\ell_{1}ℓ1\ell_{1}
PTMTTotalSILUR-S+R-S+SILUSILUR-S+R-S+
4T0T5T2.2402.2552.2762.217+1.05%+1.73%+2.66%
4T0.5T5.5T2.2162.2262.2402.207+0.42%+0.87%+1.53%
4T1T6T2.2032.2112.2222.200+0.16%+0.50%+1.01%
4T2T7T2.1972.2042.2122.199-0.09%+0.22%+0.61%
Appendix
Table 13: Architecture Casting produces models with competitive NLL for their parameter count. Impact of casting time (“MT” column) on the NLL of models. We compare level 8 models trained with Architecture Casting (three first NLL columns) against a dense SILU model trained from scratch with hidden dimension reduced to H=3712H=3712 so that its FFN parameter count matches that of the cast models.
Training durationScore: NLLR-S+ sparsity
CastingTotalSILUR-S+R-S+ , ℓ1\ell_{1}no ℓ1\ell_{1}ℓ1\ell_{1}
(a) Activation Casting
0T5T2.1782.1902.21461%89%
0.5T5.5T2.1692.1792.19763%89%
1T6T2.1622.1712.18565%89%
2T7T2.1612.1702.17966%89%
Appendix
Table 14: Evolution of NLL during casting. Detailed results when training a 0.81B dense model (level 8 in the scaling ladder from Table 12 ), performing the three model casting recipes. All models are pre-trained for 4T; the “Casting” column gives the mid-training budget and “Total” counts every token spent, including the 1T1T of decay. For LoPA and Architecture Casting, the rank ratio α\alpha is set to 1/81/8 . The ℓ1\ell_{1} loss targets 90%90\% sparsity at training time; the sparsity measured on the validation set is about one point lower, which is why the ℓ1\ell_{1} columns read 89%89\% . In addition to R-S+ , we simulate a model casting with SILU as a control experiment.
Setup (pre-training →\rightarrow mid-training)NLLΔ\Delta NLLSparsitySpeedup
Base[ SILU ] →\to Base[ SILU ] (dense baseline)1.781——1.00×1.00\times
LoPA[ SILU ] →\to LoPA[ R-S+ , ℓ1\ell_{1} @90%]1.792+0.65%+0.65\%89%5.86×5.86\times
Appendix
Table 15: Scaling up model casting. Results for 5.79B-parameter models (level 13 in Table 12 ). Δ\Delta NLL is computed relative to the first row in bold , corresponding to a control experiment conducted with SILU throughout training. All models are pretrained for 4T tokens, and midtrained for 4T tokens of stable learning rate, followed by 1T tokens of learning rate decay.
  Casting without ℓ1\ell_{1}Casting with ℓ1\ell_{1}
Model scaleNLLDurationNLLSparsityDurationNLLSparsity
(a) Activation Casting
23.0131.00T3.01166%1.50T3.00489%
42.7380.75T2.72567%2.00T2.73589%
62.5211.00T2.51867%1.50T2.51689%
82.3211.25T2.31770%1.75T2.31389%
Appendix
Table 16: Mid-training duration required to catch up with the SILU model’s performance, without the decay phase, for different model sizes with pre-trained models of 4T duration. Models are evaluated every 0.25T, and we report the first time at which the finetuned model catches up to the original NLL, rounded up by at most 0.25T. When the ℓ1\ell_{1} loss is applied (rightmost columns), fixing the target sparsity at 90%90\% during training leads to a measured sparsity around 89%89\% . For LoPA Casting, the rank ratio α\alpha is set to 1/81/8 .
  FP32int8int4
Casting familyVariantNLLΔ\Delta NLL (%)NLLΔ\Delta NLL (%)NLLΔ\Delta NLL (%)
ActivationSILU2.145—2.160+0.70%2.178+1.54%
R-S+ + ℓ1\ell_{1} @90%2.160+0.68%2.173+1.29%2.188+2.01%
LoPASILU2.147+0.09%2.162+0.76%2.180+1.62%
R-S+ + ℓ1\ell_{1} @90%2.166+0.99%2.179+1.57%2.194+2.29%
ArchitectureSILU2.180+1.63%2.193+2.21%2.213+3.16%
Appendix
Table 17: Impact on NLL of quantization and casting. After Activation, LoPA, and Architecture casting, we apply QAT and quantize models in a weight-only int4 and int8 setting. Degradation is relative to the activation-casting SILU model at full precision in bold .
Figure 9: Measured speedups for quantized sparse models. FFN-bound throughput as a function of the activation sparsity, for a 5.79B non-embedding parameters model (level 13 in the scaling ladder from Table 12 ) at different quantization levels ( int8 and int4 ).
Figure 10: ℓ1\ell_{1} loss without model casting induces higher performance degradation for the same sparsity penalty λ\lambda with level 2 (top) and level 8 (bottom) models.
  ℓ1\ell_{1} loss weight ( λ\lambda )
Metricℓ1\ell_{1} during PT10−410^{-4}3×10−43\times 10^{-4}10−310^{-3}3×10−33\times 10^{-3}10−210^{-2}3×10−23\times 10^{-2}
Level 2
NLL✓2.749\mathbf{2.749}2.748\mathbf{2.748}2.747\mathbf{2.747}2.7502.7502.7512.7512.7542.754
NLL✗2.7502.7502.7502.7502.7502.7502.749\mathbf{2.749}2.750\mathbf{2.750}2.753\mathbf{2.753}
Sparsity✓65.3%\mathbf{65.3\%}65.6%65.6\%66.7%66.7\%68.1%68.1\%71.0%71.0\%73.6%73.6\%
Sparsity✗65.2%65.2\%65.7%\mathbf{65.7\%}66.8%\mathbf{66.8\%}69.3%\mathbf{69.3\%}74.5%\mathbf{74.5\%}80.9%\mathbf{80.9\%}
Appendix
Table 18: Using ℓ1\ell_{1} loss during pre-training degrades final sparsity. Ablations of using the ℓ1\ell_{1} loss during pre-training. Models are pre-trained and cast with the same ℓ1\ell_{1} weight λ\lambda . Adding the ℓ1\ell_{1} loss on the output of SILU during pre-training results in similar or worse NLL (up to training noise), and reduces the sparsity level after casting.
 Level 2Level 8
 NLL ↓\downarrowSpars.NLL ↓\downarrowSpars.
Activation1T1T2T2T4T4T+D+D1T1T2T2T4T4T+D+D
R-S+3.0112.9842.9662.80465%2.3262.3072.2952.15369%
SILU2.9992.9702.9452.794—2.3262.3072.2952.153—
G-S+2.9992.9742.9442.808—2.3142.2982.2872.148—
GELU2.9992.9732.9552.808—2.3142.2972.2872.147—
Appendix
Table 19: SILU and GELU result in the best performances for casting. NLL and FFN activation sparsity when casting with different activation functions. Both models are pre-trained for 4T4T ; each NLL column is a mid-training budget, and “+D” adds the final 1T1T decay. Sparsity varies little with the budget, so we report it at the decayed checkpoint only. SILU , G-S+ and GELU are not sparsifying activations and have no exact zeros to report; they bound the attainable NLL rather than competing on sparsity. Bold marks the best NLL per column.
 Peak LR γ\gamma
Task1.00×10−51.00\times 10^{-5}7.40×10−57.40\times 10^{-5} ∗\ast1.05×10−41.05\times 10^{-4}1.48×10−41.48\times 10^{-4}2.50×10−42.50\times 10^{-4}7.40×10−47.40\times 10^{-4}
Core-12 short-context benchmarks
HSwag.6279.6451.6458.6439.6359.5969
ARC-e.7268.7281.7328.7230.7087.6283
ARC-c.4489.4575.4592.4403.4206.3519
PIQA.7361.7503.7470.7448.7503.7388
Appendix
Table 20: Short context evaluations for different peak-lr for mid-training Qwen3-1.7B. The evaluation is computed post-cooldown at step 1,250. All arms branch from Qwen3-1.7B-Base with the R-S+AO activation swap. Bold marks the best arm per task; ∗ marks the chosen LR.
Peak LR γ\gammaIn-mixtureΔ\Delta vs. baseAll 9GovReportSparsity (%)
Pretrained Base1.711—1.8432.152—
Swap, 0 steps10.256+8.544+8.5449.9969.78876.77
1.00×10−51.00\times 10^{-5}1.761+0.050+0.0501.9272.21279.79
7.40×10−57.40\times 10^{-5} ∗\ast1.724+0.013\mathbf{+0.013}1.8952.18878.59
1.05×10−41.05\times 10^{-4}1.727+0.015+0.0151.8982.19078.60
1.48×10−41.48\times 10^{-4}1.734+0.022+0.0221.9062.19578.76
Appendix
Table 21: Validation NLLs using different peak-lr for mid-training Qwen3-1.7B. Held-out NLL per token and FFN activation sparsity for Qwen3-1.7B. Every arm runs 0.093T0.093T (1,250 steps, 2.62 B tokens) without ℓ1\ell_{1} loss. Bold marks the best arm per NLL column; ∗ marks the chosen learning rate.
RunBudgetIn-mixtureΔ\Delta vs. baseAll 9GovReportSparsity (%)
      (All 9)
Pretrained Base—1.711—1.8432.152—
Swap, 0 steps0T0T10.256+8.544+8.5449.9969.78876.77
R-S+1T1T1.702−0.010-0.0101.8812.16376.84
 2T2T1.698−0.013-0.0131.8812.15576.43
 3T3T1.699−0.012-0.0121.8842.15176.25
Appendix
Table 22: Mid-training Qwen3-1.7B with model casting. Held-out NLL per token and FFN activation sparsity for Qwen3-1.7B mid-training at the swept-optimum peak γ=7.40×10−5\gamma=7.40\times 10^{-5} . Runs differ only in FFN activation and sparsity penalty: R-S+ is the swap with no penalty, “Activation Casting @90%” adds an adaptive- ℓ1\ell_{1} penalty targeting 90% sparsity, and SILU , no swap is plain mid-training. Bold marks the best trained checkpoint per NLL column.
Figure 11: Measured speedups in bfloat16. Throughput as a function of the activation sparsity, for a 5.79B non-embedding parameters model (level 13 in the scaling ladder from Table 12 ) in bfloat16. At 90% sparsity the speedup on GPU is 1.71×1.71\times for Standard Gating and 2.41×2.41\times for LoPA Gating compared to the standard dense case.
BenchmarkMetric# ShotsType
HellaSwagacc_char0Choice
WinoGrandeacc_char0Choice
ARC-eacc_char0Choice
ARC-cacc_char0Choice
PIQAacc_char0Choice
OBQAacc_char0Choice
Appendix
Table 23: Summary of benchmarks used for evaluation. We prompt level 8 models in a few-shot setting, and report a metric that depends on the benchmark, as well as the average performance. acc_char is the Character-Length Normalized Accuracy.
   Δ\Delta vs. SILU control (points)
TaskQwen3-1.7B-BaseSILU , no swapR-S+Activation Casting
 (pre-trained)(control)(no ℓ1\ell_{1} )@0.90
Core-12 short-context
HellaSwag66.569.7−0.7-0.7−1.2-1.2
ARC-e68.172.7−1.6-1.6−1.6-1.6
ARC-c45.145.0−0.2-0.2−2.5\mathbf{-2.5}
Appendix
Table 24: Effect of casting mid-trained Qwen3-1.7B downstream performance. Change in downstream performance (percentage points) at the decayed 5T5T checkpoint ( 100100 TPP), at peak LR γ=7.40×10−5\gamma=7.40\times 10^{-5} . Deltas are taken against the SILU , no swap control (the same model mid-trained on the same mixture without the activation swap) so they isolate the cost of casting from the cost of the mid-training mixture.
 Activation CastingLoPA Casting
Task0T4T75%90%0T4T75%90%
HS57.059.1−0.4-0.4−1.1-1.156.759.1−1.0-1.0−1.6-1.6
WG59.159.4−0.2-0.2−0.8-0.857.158.2−0.3-0.3+1.5+1.5
ARC-e60.361.5−0.8-0.8−1.4-1.457.357.8+2.1\mathbf{+2.1}+2.6\mathbf{+2.6}
ARC-c31.234.2−1.0-1.0−1.5-1.531.031.9+0.6+0.6+0.3+0.3
PIQA72.673.0+0.5+0.5+0.5+0.573.074.0−0.5-0.5−0.5-0.5
Appendix
Table 25: Effect of casting mid-trained level-8 models on downstream performance. Within each family the deltas are taken against that family’s 4T column, that is the same model mid-trained for the same budget without casting. The 0T and 4T columns are absolute %: 0T is the pre-trained checkpoint (4T WS + 1T D) before mid-training, so comparing the two gives the gain from mid-training alone. All rows are decayed. Bold marks changes of at least 2 points. Per-benchmark metrics are given in Table 23 .
 Core-12 averagerelative to the dense baseline
Setup (pre-training →\to mid-training)pre-LCFTLCFTSFTΔpre\Delta_{\mathrm{pre}}ΔSFT\Delta_{\mathrm{SFT}}Δcorr\Delta_{\mathrm{corr}}
Base[ SILU ] →\to Base[ SILU ]39.639.736.6———
Base[ SILU ] →\to Base[ R-S+ , ℓ1\ell_{1} @75%]39.539.036.8−0.1-0.1+0.2+0.2+0.3+0.3
Base[ SILU ] →\to Base[ R-S+ , ℓ1\ell_{1} @90%]39.238.736.0−0.4-0.4−0.6-0.6−0.2-0.2
LoPA[ SILU ] →\to LoPA[ R-S+ , ℓ1\ell_{1} @75%]38.838.636.1−0.8-0.8−0.5-0.5+0.3+0.3
LoPA[ SILU ] →\to LoPA[ R-S+ , ℓ1\ell_{1} @90%]39.138.136.0−0.5-0.5−0.6-0.6−0.1-0.1
Appendix
Table 26: Average downstream performance across post-training, relative to the SILU dense baseline. Δpre\Delta_{\mathrm{pre}} is the gap before post-training, ΔSFT\Delta_{\mathrm{SFT}} the gap after 80008000 SFT steps, and Δcorr = ΔSFT−Δpre\Delta_{\mathrm{corr}}\,{=}\,\Delta_{\mathrm{SFT}}-\Delta_{\mathrm{pre}} isolates what post-training itself costs each variant. All values are rounded to one decimal, so the columns reconcile exactly.
 inst_accprompt_acc
Setup (pre-training →\to mid-training)scoreΔ\DeltascoreΔ\Delta
Base[ SILU ] →\to Base[ SILU ]33.9—22.9—
Base[ SILU ] →\to Base[ R-S+ , ℓ1\ell_{1} @75%]33.9+0.0%+0.0\%22.0−3.9%-3.9\%
Base[ SILU ] →\to Base[ R-S+ , ℓ1\ell_{1} @90%]32.0−5.6%-5.6\%21.1−7.9%-7.9\%
LoPA[ SILU ] →\to LoPA[ R-S+ , ℓ1\ell_{1} @75%]35.1+3.5%+3.5\%23.3+1.7%+1.7\%
LoPA[ SILU ] →\to LoPA[ R-S+ , ℓ1\ell_{1} @90%]32.5−4.1%-4.1\%19.2−16.2%-16.2\%
Appendix
Table 27: IFEval after 80008000 SFT steps. Scores are loose ; the strict variant penalizes formatting alone and is largely common-mode across variants. Δ\Delta is measured against the Activation Casting dense control in the first row, a single reference for the whole table, so the LoPA Casting rows also carry the architecture difference.
More in this topic

Efficient Neural Network Inference

Efficient neural network inference reduces the time, memory, and energy needed to run trained models across hardware and deployment settings.

See all 470 papers

Explore similar work

CardsList

  1. Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

    Jul 30, 2026Jinyi Liu, Wei Chen, Pengyu Chen +4Activation SparsityStructured Sparsity

    Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Computer Science, Nanjing University of Posts and Telecommunications

  2. PrunePath: Towards Highly Structured Sparse Language Models

    May 27, 2026Zhexuan Gu, Zixun Fu, Yancheng YuanMixture-of-Experts PruningLLM Pruning

    Department of Applied Mathematics, The Hong Kong Polytechnic University

  3. BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks

    Oct 15, 2026Max Zhu, Bohang Sun, Pietro Lio +1Activation CompressionMemory Compression

    University of Cambridge · University of Oxford