cs.CLNov 5, 2026

Different Corpora, Different Skills? A Paired Study of Small Language Models

Authors: Edoardo Vaira, Georgios Chavales, Pierre Nugues

Organizations: Department of Computer Science, Faculty of Engineering, Lund University

Abstract

Small language models are often trained under tight data budgets, where the choice of pretraining corpus may matter as much as model size or training recipe. We compare two contrasting corpora at matched 10-million- and 37-million-word budgets: the 1911 Encyclopaedia Britannica and the BabyLM 2026 strict pretraining corpus. For each corpus, we train teacher models, distilled 58-million-parameter Baby-Llama students, and no-distillation student ablations, using the same model families, training recipes, and evaluation protocol across corpora. Across BLiMP, BLiMP Supplement, EWoK, Entity Tracking, and COMPS, the corpora produce different skill profiles rather than a single quality ranking. BabyLM 2026 is consistently stronger on linguistic acceptability benchmarks, while Britannica is often stronger on Entity Tracking, a synthetic state-tracking task. The contrast appears across teacher and student models and at both budgets, with the cleanest cross-seed replication in the distilled student comparison. Targeted controls show that the central contrast persists after matching training exposure and at a longer diagnostic context; direct teacher ensembling also does not reproduce the 10M Britannica student's Entity Tracking behavior. Entity Tracking magnitudes nevertheless remain sensitive to training dynamics. For this corpus pair, corpus choice can therefore produce different relative skill profiles at small scale, motivating multi-benchmark evaluation when comparing pretraining data.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. BLiMP-RO: A Benchmark of Linguistic Minimal Pairs for Romanian

    Nov 13, 2026Radu-Stefan Popa, Valentin Feroiu, Adina Camelia Bleotu +1LLM EvaluationLanguage Model Pretraining

  2. Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models

    Nov 12, 2026Yik Siu Chan, Jing Huang, Yanai Elazar +1Language Model PretrainingShortcut Learning in Language Models

  3. LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure

    Aug 13, 2026Fanfei Li, Jana Zeller, Manuel Prada-Corral +4Language Model PretrainingKnowledge Augmentation for Language Models