cs.CLNov 18, 2026

Do Frontier Model Ingredients Add Up? A Replicated Factorial Study of Small Language Models

Authors: Biswa Sengupta

Organizations: LLM Suite Team, JPMorgan Chase & Co.

Abstract

Building a small language model requires choosing which published improvements to combine. We compare five changes in the same model, using the same data and training budget. We test selected combinations and repeat each one with a second random seed. Mixture-of-experts layers, Muon optimisation and a Gated DeltaNet hybrid give the largest gains. Extra normalisation also helps, while an attention gate with a matched parameter budget adds little on average. The gains mostly combine, but small interactions between changes appear again in the repeated runs. Repetition also reveals benefits that an initial statistical estimate treated as noise. These findings apply to our small-model training setting; the expert models also have greater total capacity. The practical lesson is to test promising components together and repeat the comparisons before deciding which ones to keep.

Figures & tables

Explore similar work

CardsList
  1. Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

    Oct 14, 2026Joel Dag, Nikit Srivastava, Hamada M. Zahera +1Multilingual Language ModelsMixture-of-Experts Language Models