cs.CLOct 22, 2026
SaveKolibri: A Sovereign European Model on the Pareto Frontier
Abstract
This report introduces Kolibri, an English-German mixture-of-experts transformer with 78.1B total parameters and 3.46B active parameters per token, released as open weights under the Apache 2.0 licence. We built Kolibri for sovereign and specialised deployment, with a particular focus on German and on regulated domains such as public administration, industry, and aerospace. To support these settings, we control the full model-development process, including data, architecture, training infrastructure, post-training, and evaluation. We train Kolibri on 24T tokens across pre-training, mid-training, and long-context extension, with German accounting for more than 20% of the data mix by bytes. To reach this share, we complement German datasets with more than 2T German tokens that we curate from the web with our own pipeline and generate synthetically. Post-training combines supervised fine-tuning on curated German and English data with reinforcement learning on more than 1.2M internally curated tasks, which cover reasoning, tool use, instruction following, code, and retrieval. The model activates 4.4% of its parameters per token, and among the open models evaluated in this report, it lies on the Pareto frontier of quality and serving cost in English and German. Kolibri builds on Kolibri Origin, an earlier iteration that validated our full pipeline at scale. Both models come from our Model Factory, which connects data, experimentation, training, evaluation, and deployment through version-controlled workflows.
Figures & tables
Figure 1: Average English and German benchmark scores of base (top) and post-trained (bottom) models against decoded bytes of text per second and GPU, where higher and further right is better. Every model runs in its fastest configuration on eight B200 GPUs at a sequence length of k (base) or k (post-trained). Scores from Tables 17 , 3.3.3 and 29 ; evals in Sections 2.4.1 and 3.3.1 ; plot details in Appendix A .
Figure 2: Post-training benchmark scores of Kolibri, Kolibri Origin and three open models in four domains, from Sections 3.3.3 and 29 . Badges mark the English (EN) and German (DE) version of a benchmark; the scale runs from at the centre to at the outer ring.
Figure 3: Training loss of the three stages of Kolibri, each against the tokens of that stage and on its own token axis. Mid-training starts from the pre-training checkpoint at step and long-context extension from the mid-training checkpoint at step . Thin lines are the per-step loss at a sample of the steps, thick lines its mean over bins of B, B and B tokens. Each stage trains on its own mix, so the loss levels of different stages are not comparable.
Figure 4: Architecture of one Kolibri block, which all blocks share. The attention and \tl_set:Ne MoE sublayers each read from the residual stream through an RMSNorm and write back through a second one. Orange marks what only the \tl_set:Ne SWA blocks have: \tl_set:Ne RoPE on queries and keys, and a mask over the preceding tokens and the query token. The strip on the left shows which blocks use full attention. The router scores all routed experts, sends the token to the with the highest scores and weights their outputs by those sigmoid scores without renormalisation.
Figure 5: Compression of the Kolibri tokenisers and other tokenisers on German text from FineWeb-2 ( Penedo et al., 2025 ) and English text from FineWeb ( Penedo et al., 2024 ) . The compression is given in UTF-8 bytes per token (higher is better); the wide hatched bars are German, the narrow ones English. The number under each name denotes the vocabulary size. \tl_set:Ne BPE is plain \tl_set:Ne BPE trained on the same corpus with the same vocabulary size as Kolibri. Tekken is Mistral’s tokeniser ( Mistral AI, 2024 ) , also used by Apertus and Nemotron, and Gemini the tokeniser shared by Gemma 4 and Gemini ( Gemma Team, 2026 ) . GPT-5, DeepSeek V4 and EuroLLM denote the tokenisers of these models ( OpenAI, 2023 ; DeepSeek-AI, 2026 ; Martins et al., 2024 ) .
Figure 6: Proxy at scale M trained with the Kolibri tokeniser and with plain \tl_set:Ne BPE of the same vocabulary size. Left: \tl_set:Ne BPB averaged over the English and German pre-training benchmarks after pre-training. Middle and right: recall by context length after the long-context stage, the recall of assigned variables for RULER \tl_set:Ne VT and the ROUGE-L recall of the retrieved value for HELMET JSON \tl_set:Ne KV.
Figure 7: Average accuracy along the pre-training stage (left) and long-context score after a B long-context stage (right). We evaluate a model with full attention in every block and a hybrid model with 512$$ , both approximately \tl_set:Ne FLOP-matched at proxy scale M ( Table 3 ). Full attention uses heads per block and pre-trains on sequences of k tokens, the hybrid on sequences of k tokens. During the long-context stage, the full-attention model is trained on sequences of k tokens with \tl_set:Ne YaRN ( Peng et al., 2024 ) , while the hybrid model is trained on sequences of k tokens (without any \tl_set:Ne RoPE extension). Left: log-likelihood accuracy averaged over \tl_set:Ne ARC, HellaSwag and \tl_set:Ne MMLU in English and German at raw pre-training checkpoints. Right: mean RULER score ( \tl_set:Ne NIAH, \tl_set:Ne VT, \tl_set:Ne WE and \tl_set:Ne QA) and HELMET score (JSON-KV and RAG-QA), up to the k tokens the full-attention model was trained on.
Figure 8: Pooled long-context scores by window size after the long-context stage, in the resource-matched sweep at scale M (solid) and in the validation at scale L (dashed). Each point denotes the average of RULER \tl_set:Ne NIAH, \tl_set:Ne VT, \tl_set:Ne WE and \tl_set:Ne QA, and the HELMET JSON-KV and RAG-QA F1-scores evaluated at that length. At the longest evaluated context, window size leads both sweeps. Axis labels use tokens.
Figure 9: Accuracy mean during pre-training (left) and maximum pure-decode throughput by live context length (right) of three configurations with B active parameters and different total parameters. Accuracy averages the benchmarks of Section B.3 . Throughput is measured on eight H100 SXM5 GPUs in BF16, with the best tested parallel configuration for each model and context length. Kolibri Origin is shown for reference over its supported context range.
Figure 10: Learning-rate transfer across width and training horizon.
Figure 12: Documents entering each stage of the \tl_set:Ne CC pipeline, split into the documents the stage kept and removed. Each stage receives what the previous one kept. Substring deduplication removes text within documents and drops a document only when fewer than bytes of it remain, so its effect on bytes is larger than on documents.
Figure 14: The general-knowledge, math, and code means of the pre-training suite per language along the production chain, at the M4 merge at the end of pre-training, the M4 and M20 merges at the end of mid-training, and the M20 merge at the end of long-context extension, which is the final base model. Section D.2.7 lists the values.
Figure 15: Length distribution of the long-context mix candidates, as the materialised token budget of each share per document-length band in percent of the mix’s tokens. is the share of the mix drawn from the mid-training mix with the length reweighting, and the portion of naturally long documents is identical across the three candidates. The last panel is the production mix, the recipe rebuilt on the production mid-training pool at proxy scale.
Figure 16: Domain weights of the part of the production long-context mix drawn from the mid-training pool, compared with those of the original mid-training mix.
Figure 17: General-capability performance on the standard non-long-context evaluation suite.
Figure 18: RULER scores of Kolibri Base and Nemotron 3 Nano Base from k to M tokens. For question answering, the end-of-sequence token is blocked at the first generated position, so no answer is empty.
Figure 19: Overall English and German pre-training aggregates of Table 17 vs the active parameters per token on a logarithmic axis, where higher and further left is better. The dashed line connects the models on the Pareto frontier. A model lies on the frontier iff every other model with fewer active parameters scores lower and every other model with as many active parameters scores at most as high.
Figure 20: Score of the Kolibri \tl_set:Ne SFT checkpoint on a selection of benchmarks against the median number of tokens it generates per sample, with one point per reasoning effort from none to high . The marker size grows with the reasoning effort. The German counterparts of the English benchmarks are depicted by a dashed line and square markers.
Figure 21: Artefacts that machine translation carries from an English reasoning trace into a German one. Left, the English trace of a reasoning model on a German problem; right, its machine translation into German. Highlights mark four kinds of artefact, numbered at their first occurrence in each trace. Meta-language: the trace states that the problem is in German and that the answer has to be German. Repetition: glosses of the German prompt for an English reader become tautologies such as “Brötchen sind Brötchen” . Decimal-comma aside: the trace explains the German decimal comma, which a German reader does not need. English notation: the arithmetic keeps 0.45 and 16.2, while the prompt and the answer write 0,45 and 16,20.
Figure 22: Steering the language of the teacher’s reasoning on German prompts, shown on a math problem. Both variants send the same system and user turns. Left, the system prompt instructs the model to reason in German, and the model opens the think block itself; its trace shows the case that switches to English partway. Right, the assistant turn is sent already opened with <think> and a German opener, which the model continues. We checked the language consistency of both approaches on more than completions each.
Figure 23: Generation of one long-context conversation from a seed document. A seed document with a length between k and k tokens is split into about sections to . The generator reads the document, an index of its sections, and the prompt as one message. Four requests to are parsed from its response. Each request is answered in its own call returning a reasoning trace and an answer . Turns are appended in order to a conversation that holds the document, ensuring conversations are shorter than k tokens.
Figure 26: Assignment of reasoning-effort labels to the exchanges of each sample from the \tl_set:Ne SFT data pool. We first grade the difficulty of each conversation by its prompt with Poacher-2B, a grader distilled from a \tl_set:Ne LLM judge. We then count the reasoning length of each exchange, the number of reasoning tokens from one user turn to the next. For each reference group (a combination of a dataset and a difficulty), we compute the 30th and 70th percentiles and of . To label an exchange post hoc, we assign the effort level based on the relative position of to the 30th and 70th percentiles and . Rather than a hard cut-off, we use the soft draw of Equation 12 around the bucket edges and . This effort label is then rendered as part of the system prompt; an exchange without reasoning is labelled none .
Figure 27: Reasoning tokens per effort label in the training data of the released Kolibri \tl_set:Ne SFT checkpoint, read back from a sample of its dataset artefacts and pooled per domain. Dashed lines mark the label medians. The labels overlap within a domain because the thresholds differ between the datasets of a domain and because the draw of Equation 12 is soft; the same label covers different lengths in different domains.
Figure 28: Reasoning length of the released Kolibri \tl_set:Ne SFT checkpoint against the difficulty Poacher-2B assigns to the evaluation prompts. Left: length by judged difficulty over the three reasoning efforts, dashed lines at the medians. Right: the median length per judged difficulty, one line per effort; at a fixed difficulty the three lines show how much the effort adds, and along one line how much the difficulty adds. none is left out, as it produces an empty reasoning block.
Figure 30: Domain weights of the three mixes trained for steps (left) and the scores of their models and of the soup of each pair (right), in . The domain weights are shares of the trained tokens. In each score panel, the diagonal holds the three models and each cell below it the soup of the models named by its row and column; colours are scaled per panel, and the best cell is framed. AIME 2026, LiveCodeBench and AA-LCR are the math, code and long-context benchmarks whose scores spread most over the mixes and soups. The mean is over benchmarks: math (AIME 2025, AIME 2026 and AIME 2026 IID of Section 3.1.3 ), knowledge (GPQA Diamond and KulturBank, Section H.5 ), agentic tool use ( -bench telecom, multi-turn \tl_set:Ne BFCL v3 and BrowseComp Lite), code (LiveCodeBench), instruction following (IFBench, loose accuracy) and long context (AA-LCR and LongBench Pro).
Figure 31: Logit-gradient norm of a token at unit advantage , over its inference probability and training probability . Left: The diagonal marks on-policy probabilities, and the dashed square the bottom left corner 0.5 . The other panels show the norm in that corner under \tl_set:Ne CISPO’s importance-weight cap at $r_{t}=$2 ( MiniMax, 2025 ) , under our norm bound and under \tl_set:Ne CISPO’s cap at 5$$ . Above \tl_set:Ne CISPO’s dashed line, the norm is limited to the cap times , and left of the dark parabola , scales down the token’s gradient to exactly the bound.
Figure 32: Our \tl_set:Ne RL environment mix. Every point represents one environment or a pool of related environments, plotted over its number of train samples and the weight we assign during training (cf. Section 3.2.2.5 ). Colour and mark shape denote the environment’s group; the percentage in the legend is the group’s share of the summed sampling weight. Both axes are logarithmic. The train splits hold rows in total. We deliberately upweight several agentic environments as well as competitive programming, while keeping the total weight of the Merlin–Arthur environments comparatively small. Hallucination reduction holds the answerable questions from four datasets and unanswerable ones from two. Table 55 shows exact per-environment values with step and token budgets.
Figure 33: The verifier of every \tl_set:Ne RL environment and the shape of its reward. A filled disc marks a binary reward, a hollow one the fraction of checks passed. A half-filled one indicates a graded score, i.e. the token F1 against the gold answers or a judge’s grade of , or . A column with several marks combines verifiers, either within one environment or across the environments it pools. The Merlin–Arthur reward is signed ( Section 3.2.2.3 ). Columns are grouped and coloured as in Figure 32 .
| Scope | Signal and clean-output condition |
|---|---|
| Universal | Non-empty output. The final assistant step contains visible text or a parsed tool call. |
| No leaked tool call. No literal tool-call markup leaks into the visible output. | |
| No script leak. No \tl_set:Ne CJK text leaks into a response that is otherwise non- \tl_set:Ne CJK, and no generated token matches the banned-script pattern. | |
| Tool Use | Tool in schema. Every called tool was offered in the row’s tool schema. |
| Well-formed tool call. The emitted tool-call structure is well formed. | |
| Arguments in schema. Tool arguments use the declared names and types. |
Table 24: Format signals active during \tl_set:Ne RL training. Conditional signals are omitted when they do not apply. The signals lie in , where denotes a clean output and a detected defect.
Figure 34: What the German environments buy: held-out benchmark scores of the \tl_set:Ne SFT checkpoint and of ten \tl_set:Ne RL checkpoints trained with the German environments in the mix. Left: share of responses whose reasoning and answer are in the language of the prompt. Right: accuracy on the same benchmarks. AIME 2026 is problems sampled times each; both panels average over all samples. German AIME 2026 moves from to in language consistency and to in accuracy by step , while English AIME 2026 accuracy moves from to and its language consistency stays at .
Figure 35: One workplace-assistant task under three group seeds. The seed draws the system prompt from a table of equivalent phrasings and its language, English or German, independently of the language of the request. About of the words of the request receive a typo ( highlighted ); the words the verifier matches against ( bold ) are never touched. The agentic environments draw their coding harness from the same seed.
Figure 36: Placement of the \tl_set:Ne RL run on hosts with eight B300 GPUs each. Each small square is one GPU and each box of eight squares one host. Host runs trainer ranks and the processes in the dashed box. Every other host runs one rollout worker and one sandbox service. The global router on host picks the inference replica for each group. The rollout worker then sends the group’s requests to that replica directly.
Figure 37: Weight push from the trainer to the inference replicas. At startup, the receive buffers are mapped to the trainer parameters, and each parameter is assigned a trainer rank such that the bytes per rank are balanced. After every optimiser step, the trainer ranks push the new weights via \tl_set:Ne RDMA into the single receiving replica of each node, from where they are broadcast to the node’s other replicas over NVLink. The conversion on the trainer side is batched and pipelined, and the intra-node NVLink distribution reduces load on the inference node’s \tl_set:Ne NIC. The single-GPU replicas of our deployment skip the split of the weights into a per-replica layout.
Figure 38: Sandbox stack of the \tl_set:Ne RL run on one worker node. Task images are converted once to single-file Apptainer images on shared scratch. Sandboxes of the same image share its read-only SquashFS layer on the node, and each adds its own writable tmpfs layer. The right panel shows one sandbox: the OverlayFS root, the long-running Apptainer instance, and the egress proxy for the allowlisted hosts.
Figure 39: Validation reward of the Kolibri \tl_set:Ne RL run by environment group. Each line is the mean environment reward of the validation sequences that finish in an optimiser step, averaged over reasoning efforts and over the environments of a group of Figure 32 , weighted by their sampling weight, as a centred moving average over steps. The retrieval and long-context line also averages the Hallucination Reduction environments, whose rewards lie in . The grey line is the same weighted mean over all environments.
Figure 40: Training dynamics over optimiser steps of the Kolibri \tl_set:Ne RL run. Left: entropy of the training policy over the trained tokens of each optimiser step, divided by the logarithm of the vocabulary size. Middle: share of tokens removed by the \tl_set:Ne BTV mask and scaled down by the gradient-norm bound of Section 3.2.1.2 , i.e. with . Right: mean absolute log-ratio and forward \tl_set:Ne KL divergence between the training policy and the inference policy. Both metrics include a median policy staleness of optimiser steps from our asynchronous setup ( Section 3.2.1.5 ).
Figure 41: Difficulty buckets of the \tl_set:Ne RL curriculum over training, summed over all environments except the Hallucination Reduction environments, which bypass the curriculum. Left: share of all prompts in the hard, trainable and solved buckets. Right: share of sampled prompts per bucket, divided by its target bucket weight . During initial training steps, the curriculum compensates for missing and inaccurate offline pass rates, which populates the hard and solved buckets across environments. The model then improves over the course of training, causing more prompts to become solved. In particular, several environments become entirely solved or trainable, draining their share of hard prompts.
Figure 42: Targeted \tl_set:Ne OPSD over the Kolibri \tl_set:Ne RL run. Left: Detected failure cases across the terminal environments, counted at the optimiser step whose weights produced the sequence. Right: Moving average over steps of the share of distilled tokens for which the teacher’s most likely token is the sampled token.
Figure 43: Benchmark scores of the \tl_set:Ne RL checkpoints of Kolibri Origin and Kolibri over time. Grey lines indicate median evaluation scores for open models. Each coloured point corresponds to a finished evaluation run that improved at least one of the four depicted benchmarks. Vertical lines mark the first evaluation of an \tl_set:Ne RL checkpoint of Kolibri. The lines track the best score of their model generation, evaluated with their respective setup at the time they were run. These intermediate points do not indicate representative benchmark performance, but are meant as a visual representation of Kolibri’s progress over time. A star marks the released Kolibri checkpoint, sampled at top- and top- as in production. SWE-bench Verified is scored with the mini-swe-agent harness ( Yang et al., 2024b ) and shown for Kolibri only, as we did not optimise Kolibri Origin towards agentic capabilities. The release (star) is scored with the OpenCode harness, without web access.
Figure 44: Agentic retrieval scores of the Kolibri \tl_set:Ne RL checkpoints evaluated in September 2026. Each point is one eval run, each solid line the mean of the runs of a day, and the dashed line Kolibri Origin; under each title, the score of Kolibri Origin and of the released Kolibri checkpoint. MuSiQue and Honeypot are scored on the test splits of the two training environments. The five Industry \tl_set:Ne RAG benchmarks of Section I.1.4 are in-house proxies for deployed customer assistants and share no documents or questions with the training data.
Figure 45: Example sequences that led to exploits, with ellipses marking omitted text. Left: The OpenCode harness exposes a web-fetch tool, and the policy resolves an SWE-bench instance to its upstream pull request, downloads the patch and applies it. Right: The sandbox does not let apt switch to its unprivileged download user. The reasoning concludes that the process runs as root and can therefore write to kernel settings, and the next tool call disables unprivileged user namespaces on the host kernel.
| Benchmark | Description | German version |
|---|---|---|
| Knowledge | ||
| GPQA Diamond ( Rein et al., 2024 ) | Chain-of-thought prompts, avg@ , since a single run leaves several points of sampling noise | Translation |
| Humanity’s Last Exam ( Phan et al., 2025 ) | Text-only questions, no output cap | Translation |
| AA-Omniscience ( Jackson et al., 2025 ) | Public set. Its accuracy scores a wrong answer no better than abstaining, its index between and scores a wrong answer and an abstention or partial answer , and its non-hallucination rate is the share of non-correct answers that abstained or answered partially rather than guessed. | – |
| \tl_set:Ne MMLU-Pro ( Wang et al., 2024a ) | Chain-of-thought prompts | MMLU-ProX ( Xuan et al., 2025 ) |
| Math | ||
Table 27: Post-training benchmarks by category, with their settings and German versions. Every German version marked as a translation comes from ellamind (2025) , and an en dash marks a benchmark without a German version. The score avg@ is the mean over runs of every sample.
Figure 46: Overall English and German post-training aggregates of Section 3.3.3 against the active parameters per token on a logarithmic axis, where higher and further left is better. The dashed line marks the convex hull of the Pareto frontier. Apertus 70B Instruct is excluded because its deployment does not produce comparable scores.
Figure 47: Agentic retrieval and customer-proxy benchmark scores ( ) of the released Kolibri checkpoint and the three open models of Figure 2 , from Table 29 . MuSiQue, Honeypot and Agentic Wiki QA (DE) are agentic retrieval benchmarks; the five customer proxies are internal benchmarks built from customer use cases, named by vertical.
Table 28: Post-training benchmark scores of Kolibri, Kolibri Origin, and comparable open models (part I/II). The first two rows give each model’s total and active parameters in billions, with bars on a common scale. Bold is the best score among \tl_set:Ne MoE models , underlined is best in class , incomplete or non-percentage scores are greyed out . Averaging is performed by language and over the category averages, with details given in the caption of Table 29 .
More in this topic
Language Modeling
Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.
See all 519 papers