Do Frontier Model Ingredients Add Up? A Replicated Factorial Study of Small Language Models
Organizations: LLM Suite Team, JPMorgan Chase & Co.
Abstract
Building a small language model requires choosing which published improvements to combine. We compare five changes in the same model, using the same data and training budget. We test selected combinations and repeat each one with a second random seed. Mixture-of-experts layers, Muon optimisation and a Gated DeltaNet hybrid give the largest gains. Extra normalisation also helps, while an attention gate with a matched parameter budget adds little on average. The gains mostly combine, but small interactions between changes appear again in the repeated runs. Repetition also reveals benefits that an initial statistical estimate treated as noise. These findings apply to our small-model training setting; the expert models also have greater total capacity. The practical lesson is to test promising components together and repeat the comparisons before deciding which ones to keep.
Figures & tables
| Switch | Change from the baseline |
|---|---|
| G | Attention-output sigmoid gate; narrower feed-forward layers to keep active size similar. |
| M | Muon for hidden matrices; AdamW for other groups, with the chosen family settings. |
| S | Embedding RMSNorm and output RMSNorm before each sub-layer joins the residual stream. |
| E | Layers 2–8: 32 routed experts, top-3 selection, one shared expert; width 352 each. |
| H | Gated DeltaNet in layers 1–3 and 5–7; global attention in layers 4 and 8. |
| Contrast | Seed 0 | Seed 1 | Mean |
|---|---|---|---|
| E | |||
| M | |||
| H | |||
| S | |||
| G | |||
| SE / GMH |
Small Language Models
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
See all 302 papers