Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison
Authors: Marianna Nezhurina, Harsh Raj, Victor May, Jörg Franke, Taishi Nakamura, Timur Carstensen, Niccolò Ajroldi, Ville Komulainen, David Salinas, Jenia Jitsev
Organizations: LAION · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ) · Open-Ψ (Open-Sci) Collective · OpenEuroLLM team · ELLIS Institute Tübingen · University of Freiburg · Institute of Science Tokyo · University of Turku
Reliable progress in language modeling depends on knowing whether a new dataset, architecture, or training recipe is genuinely better than existing alternatives. To foster reproducible learning procedure comparison, we present open-sci-ref, a family of dense transformers trained as open reference baselines at four scales (0.13B-1.7B parameters) on nine main reference datasets, with token budgets up to 1T, using a single tuned and fully documented training procedure. To facilitate a shared experimentation and measurement infrastructure, we release model weights, intermediate checkpoints, training logs, code, and downstream evaluations. When comparing training procedures, we take into account total compute spent. Placing training procedures on a common compute axis helps identify reference points that are weak or strong for the compute they use, without requiring a full scaling law study for every comparison. We identify Nemotron-CC-HQ as the strongest reference dataset on our evaluation suite: our 1.7B model trained on 1T of its tokens matches SmolLM2-1.7B with roughly 11× less compute. Shared baselines expose pitfalls of scaling trend comparisons: when a small-scale point is unusually weak, the resulting slope can overstate the scalability of a training procedure. We illustrate this diagnostically with EuroLLM-1.7B, which underperforms 300B token FineWeb-Edu baseline despite 13× more training compute. As an additional case study, the suite supports controlled comparison between permissively licensed and mixed-license datasets: the permissive-first MixtureVitae dataset changes rank substantially with scale. This also illustrates why data ranking should be evaluated through multi-scale reference trajectories rather than single proxy runs. All artifacts are publicly released.
Figures & tables
Figure 1: Why shared reference baselines are needed. Left : published models differ on every axis at once, so an observed gap cannot be attributed to any single cause. Right : varying one factor (here the pretraining data) while matching the rest and placing runs on a common compute axis isolates its effect under a shared protocol. Rank order may not be preserved across scales.
Params (B) (Non-Emb + Emb)
Layers
Hidden
Heads
FFN Hidden
GPU Memory Usage
FLOPs/token (6N estimate)
0.1 + 0.03 = 0.13
22
512
8
2256
0.89GB
7.8⋅108
0.35 + 0.05 = 0.40
22
1024
16
3840
2.32GB
2.4⋅109
1.21 + 0.10 = 1.31
24
2048
32
5440
7.54GB
7.9⋅109
1.61 + 0.10 = 1.71
24
2048
32
8192
9.88GB
1.0⋅1010
Table 2: open-sci-ref model architecture and scales.
Figure 2: Average downstream performance over training for full reference trajectories at four model scales. Rankings are broadly stable among conventional datasets, with gaps widening at larger scales. MixtureVitae is the main exception: it begins near the bottom at 0.13B but rises to the HPLT/FineWeb-Edu group at 1.7B, illustrating why single-scale dataset rankings can be misleading.
Figure 3: Downstream performance across 11 tasks of open-sci-ref models with 1.7B parameters trained on 300B tokens over different datasets. While some tasks yield clear dataset rankings (e.g., ARC, HellaSwag, LAMBADA), others provide insufficient signal for meaningful dataset comparison.
Figure 4: Downstream performance on 11 tasks for open-sci-ref 1.7B models trained for 1T tokens on the top three reference datasets. Nemotron-CC-HQ has the strongest average endpoint, while DCLM is strongest on LAMBADA and competitive on several continuation-style tasks.
Table 3: Per-benchmark endpoint results for selected open-sci-ref-0.02 1.7B reference baselines and external models. All rows in the upper block are open-sci-ref-1.7B runs trained on the listed dataset, where architecture, tokenizer, training recipe, evaluation protocol, and compute accounting are fixed. Bold marks the best value per column within this controlled block (ties bolded jointly). The upper block covers the three 1T endpoints, the 300B FineWeb-Edu reference, and the three permissively licensed datasets. Figure 2 shows all nine datasets at 300B. The lower block reports external models for orientation, with their training dataset in parentheses where known. These rows are not controlled comparisons because they differ in data, tokenizer, architecture details, and training recipe, and are therefore not included in the bolding. Training tokens are given in trillions and training compute in units of 1021 FLOPs, estimated as 6ND from the parameter count N and the training token count D . Scores are accuracies reported without the leading zero. Avg. is the unweighted mean over the 11 listed evaluations, including CommonSenseQA. LAMB. denotes LAMBADA, OBQA denotes OpenBookQA, Wino. denotes WinoGrande, CSQA denotes CommonSenseQA, and HSwag denotes HellaSwag.
Figure 5: Scaling trends for open-sci-ref-0.02 on a common estimated training compute axis. Solid lines show controlled open-sci-ref runs, where architecture, tokenizer, training recipe, evaluation protocol, and compute accounting are fixed. Dashed lines show selected external model families: these are not controlled comparisons because the models differ in data, tokenizer, architecture details, and training recipe.
Figure 6: Performance of English-dominant open-sci-ref on MMMLU, which is a version of the MMLU benchmark translated to multiple languages by OpenAI employing human professional translators.
Figure 7: Average performance on Belebele for different languages while training a 1.7B model on Nemotron-CC-HQ.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Token budget
Global batch
Iterations
Learning rate
Warmup
50B
2.65M
18,839
3×10−3
6,000
50B
4.03M
12,406
6×10−3
1,000
300B
2.09M
143,052
3×10−3
5,000
Appendix
Table 4: Cosine learning rate configurations used during recipe tuning. These runs were used to establish validation loss targets before switching to the WSD schedule.
Token budget
Global batch
Iterations
Learning rate
Warmup
Cooldown
50B
2.65M
18,839
2×10−3
6,000
3,767
50B
4.12M
11,921
4×10−3
1,000
2,384
300B
2.09M
143,052
1×10−3
5,000
28,610
300B
4.12M
72,661
4×10−3
25,000
14,532
1T
4.12M
242,204
4×10−3
25,000
48,440
Appendix
Table 5: WSD learning rate configurations. Cooldown is 20% of the total training duration. The 4.12M-token global batch rows define the main reference recipe used for the longer 300B- and 1T-token runs.
Figure 8: Comparing training with different context length. Training run for open-sci-ref-0.02 1.7B model on 100B tokens of Nemotron-CC-HQ, using various context length of 4096 (RoPE base 100k), 8192 (RoPE base 500k) and 16384 (RoPE base 1M). We observe no difference in average performance, ensuring models with larger context length have no drop in performance compared to reference context length of 4096.
Table 7: open-sci-ref reference baseline distributed training runs. The TFLOP/s/GPU column follows the same accounting as Tab. 6 .
OpenLanguageModel (OLM) is an open-source PyTorch library for building and pretraining small language models while keeping their machinery visible. In OLM, model code reads like the architecture: components are ordinary modules, while Block, Residual, Repeat, and Parallel describe how they are wired. The resulting model can move unchanged from a teaching notebook to a complete pretraining run or a research ablation. OLM connects this readable model layer to tokenizers, local and streaming datasets, optimization, mixed precision, callbacks, checkpoints, and hardware-aware CPU, single-GPU, and single-node multi-GPU execution. We demonstrate the full path by tracing GPT-2 from diagram to code, launching a FineWeb-Edu training script, replacing one attention component, and letting AutoTrainer configure the available machine. The package includes 27 presets across nine familiar model families and documentation that progresses from LM fundamentals to architecture research. Validation shows close agreement with independent reference implementations, 90.6% four-GPU weak-scaling efficiency for a 348M-parameter workload, compact architecture edits, and positive early usability results. OLM is MIT-licensed and available through PyPI, GitHub, and its documentation site.
Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, it is surprising that no systematic study evaluates the methodology to obtain scaling laws and prescriptions across different model types. To shed light on this crucial blind spot and facilitate future research, we introduce the surrogate benchmarks ScAn-Bench-LLM and ScAn-Bench-VLM based on 4524 and 8024 checkpoints of language and vision-language model pipelines. On our benchmarks, we perform the first systematic evaluation of both data acquisition and extrapolation methodology for scaling analysis across different data modalities.
Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75%, unique data budgets up to 2.6B tokens, and total training tokens up to 41.6B over 16 epochs; we further validate extrapolation on held-out dense-equivalent models up to 7.68B parameters. We find that: 1. Sparse scaling in data-limited settings: We introduce a scaling law that models loss as a function of active parameters, unique tokens, data repetition, and sparsity, accurately predicting performance across compute and data budgets. 2. Delayed data saturation: sparse training postpones diminishing returns from repeated data, making multi-epoch training more effective. 3. Resource trade-offs: With fixed data, loss-optimal sparsity is moderate ~ 50%, while compute-optimal sparsity is higher and grows with data scale. Overall, sparsity is not just a tool for efficiency, but a mechanism for improving scaling trade-offs under data scarcity. Our code is available at: https://github.com/boqian333/sparse-dc-scaling.
Boqian Wu, Qiao Xiao, Patrik Okanovic +6
University of Luxembourg · University of Twente · Eindhoven University of Technology +1