cs.CLOct 22, 2026

Kolibri: A Sovereign European Model on the Pareto Frontier

Authors: Aleph Alpha, :, Moritz Althaus, Artur Baranowski, Michael Barlow, Fabien C. Y. Benureau, Felix Berkenkamp, Amrita Bhattacherjee, 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

This report introduces Kolibri, an English-German mixture-of-experts transformer with 78.1B total parameters and 3.46B active parameters per token, released as open weights under the Apache 2.0 licence. We built Kolibri for sovereign and specialised deployment, with a particular focus on German and on regulated domains such as public administration, industry, and aerospace. To support these settings, we control the full model-development process, including data, architecture, training infrastructure, post-training, and evaluation. We train Kolibri on 24T tokens across pre-training, mid-training, and long-context extension, with German accounting for more than 20% of the data mix by bytes. To reach this share, we complement German datasets with more than 2T German tokens that we curate from the web with our own pipeline and generate synthetically. Post-training combines supervised fine-tuning on curated German and English data with reinforcement learning on more than 1.2M internally curated tasks, which cover reasoning, tool use, instruction following, code, and retrieval. The model activates 4.4% of its parameters per token, and among the open models evaluated in this report, it lies on the Pareto frontier of quality and serving cost in English and German. Kolibri builds on Kolibri Origin, an earlier iteration that validated our full pipeline at scale. Both models come from our Model Factory, which connects data, experimentation, training, evaluation, and deployment through version-controlled workflows.

Figures & tables

Figure 1: Average English and German benchmark scores of base (top) and post-trained (bottom) models against decoded bytes of text per second and GPU, where higher and further right is better. Every model runs in its fastest configuration on eight B200 GPUs at a sequence length of 44 k (base) or 1616 k (post-trained). Scores from Tables 17 , 3.3.3 and 29 ; evals in Sections 2.4.1 and 3.3.1 ; plot details in Appendix A .
Figure 2: Post-training benchmark scores of Kolibri, Kolibri Origin and three open models in four domains, from Sections 3.3.3 and 29 . Badges mark the English (EN) and German (DE) version of a benchmark; the scale runs from 0 %0\text{\,}\mathrm{\%} at the centre to 100 %100\text{\,}\mathrm{\%} at the outer ring.
Figure 3: Training loss of the three stages of Kolibri, each against the tokens of that stage and on its own token axis. Mid-training starts from the pre-training checkpoint at step 264 750264\,750 and long-context extension from the mid-training checkpoint at step 34 20034\,200 . Thin lines are the per-step loss at a sample of the steps, thick lines its mean over bins of 2020 B, 55 B and 11 B tokens. Each stage trains on its own mix, so the loss levels of different stages are not comparable.
Figure 4: Architecture of one Kolibri block, which all 5050 blocks share. The attention and \tl_set:Ne MoE sublayers each read from the residual stream through an RMSNorm and write back through a second one. Orange marks what only the 4040 \tl_set:Ne SWA blocks have: \tl_set:Ne RoPE on queries and keys, and a mask over the 512512 preceding tokens and the query token. The strip on the left shows which blocks use full attention. The router scores all 384384 routed experts, sends the token to the 66 with the highest scores and weights their outputs by those sigmoid scores gig_{i} without renormalisation.
Figure 5: Compression of the Kolibri tokenisers and other tokenisers on German text from FineWeb-2 ( Penedo et al., 2025 ) and English text from FineWeb ( Penedo et al., 2024 ) . The compression is given in UTF-8 bytes per token (higher is better); the wide hatched bars are German, the narrow ones English. The number under each name denotes the vocabulary size. \tl_set:Ne BPE is plain \tl_set:Ne BPE trained on the same corpus with the same vocabulary size as Kolibri. Tekken is Mistral’s tokeniser ( Mistral AI, 2024 ) , also used by Apertus and Nemotron, and Gemini the tokeniser shared by Gemma 4 and Gemini ( Gemma Team, 2026 ) . GPT-5, DeepSeek V4 and EuroLLM denote the tokenisers of these models ( OpenAI, 2023 ; DeepSeek-AI, 2026 ; Martins et al., 2024 ) .
Figure 6: Proxy at scale M trained with the Kolibri tokeniser and with plain \tl_set:Ne BPE of the same vocabulary size. Left: \tl_set:Ne BPB averaged over the English and German pre-training benchmarks after pre-training. Middle and right: recall by context length after the long-context stage, the recall of assigned variables for RULER \tl_set:Ne VT and the ROUGE-L recall of the retrieved value for HELMET JSON \tl_set:Ne KV.
Figure 7: Average accuracy along the pre-training stage (left) and long-context score after a 55 B long-context stage (right). We evaluate a model with full attention in every block and a hybrid model with w=w=512$$ , both approximately \tl_set:Ne FLOP-matched at proxy scale M ( Table 3 ). Full attention uses 1414 heads per block and pre-trains on sequences of 44 k tokens, the hybrid on sequences of 1616 k tokens. During the long-context stage, the full-attention model is trained on sequences of 6464 k tokens with \tl_set:Ne YaRN ( Peng et al., 2024 ) , while the hybrid model is trained on sequences of 256256 k tokens (without any \tl_set:Ne RoPE extension). Left: log-likelihood accuracy averaged over \tl_set:Ne ARC, HellaSwag and \tl_set:Ne MMLU in English and German at raw pre-training checkpoints. Right: mean RULER score ( \tl_set:Ne NIAH, \tl_set:Ne VT, \tl_set:Ne WE and \tl_set:Ne QA) and HELMET score (JSON-KV and RAG-QA), up to the 6464 k tokens the full-attention model was trained on.
Figure 8: Pooled long-context scores by window size ww after the long-context stage, in the resource-matched sweep at scale M (solid) and in the validation at scale L (dashed). Each point denotes the average of RULER \tl_set:Ne NIAH, \tl_set:Ne VT, \tl_set:Ne WE and \tl_set:Ne QA, and the HELMET JSON-KV and RAG-QA F1-scores evaluated at that length. At the longest evaluated context, window size 512512 leads both sweeps. Axis labels use k=1024k=1024 tokens.
Figure 9: Accuracy mean during pre-training (left) and maximum pure-decode throughput by live context length (right) of three configurations with 3.43.4 B active parameters and different total parameters. Accuracy averages the 1717 benchmarks of Section B.3 . Throughput is measured on eight H100 SXM5 GPUs in BF16, with the best tested parallel configuration for each model and context length. Kolibri Origin is shown for reference over its supported context range.
Figure 10: Learning-rate transfer across width and training horizon.
Figure 11
Figure 12: Documents entering each stage of the \tl_set:Ne CC pipeline, split into the documents the stage kept and removed. Each stage receives what the previous one kept. Substring deduplication removes text within documents and drops a document only when fewer than 100100 bytes of it remain, so its effect on bytes is larger than on documents.
Figure 13
Figure 14: The general-knowledge, math, and code means of the pre-training suite per language along the production chain, at the M4 merge at the end of pre-training, the M4 and M20 merges at the end of mid-training, and the M20 merge at the end of long-context extension, which is the final base model. Section D.2.7 lists the values.
Figure 15: Length distribution of the long-context mix candidates, as the materialised token budget of each share per document-length band in percent of the mix’s tokens. ss is the share of the mix drawn from the mid-training mix with the length reweighting, and the portion of naturally long documents is identical across the three candidates. The last panel is the production mix, the s=1/3s=1/3 recipe rebuilt on the production mid-training pool at proxy scale.
Figure 16: Domain weights of the part of the production long-context mix drawn from the mid-training pool, compared with those of the original mid-training mix.
Figure 17: General-capability performance on the standard non-long-context evaluation suite.
Figure 18: RULER scores of Kolibri Base and Nemotron 3 Nano Base from 128128 k to 11 M tokens. For question answering, the end-of-sequence token is blocked at the first generated position, so no answer is empty.
Figure 19: Overall English and German pre-training aggregates of Table 17 vs the active parameters per token on a logarithmic axis, where higher and further left is better. The dashed line connects the models on the Pareto frontier. A model lies on the frontier iff every other model with fewer active parameters scores lower and every other model with as many active parameters scores at most as high.
Figure 20: Score of the Kolibri \tl_set:Ne SFT checkpoint on a selection of benchmarks against the median number of tokens it generates per sample, with one point per reasoning effort from none to high . The marker size grows with the reasoning effort. The German counterparts of the English benchmarks are depicted by a dashed line and square markers.
Figure 21: Artefacts that machine translation carries from an English reasoning trace into a German one. Left, the English trace of a reasoning model on a German problem; right, its machine translation into German. Highlights mark four kinds of artefact, numbered at their first occurrence in each trace. Meta-language: the trace states that the problem is in German and that the answer has to be German. Repetition: glosses of the German prompt for an English reader become tautologies such as “Brötchen sind Brötchen” . Decimal-comma aside: the trace explains the German decimal comma, which a German reader does not need. English notation: the arithmetic keeps 0.45 and 16.2, while the prompt and the answer write 0,45 and 16,20.
Figure 22: Steering the language of the teacher’s reasoning on German prompts, shown on a math problem. Both variants send the same system and user turns. Left, the system prompt instructs the model to reason in German, and the model opens the think block itself; its trace shows the case that switches to English partway. Right, the assistant turn is sent already opened with <think> and a German opener, which the model continues. We checked the language consistency of both approaches on more than 50005000 completions each.
Figure 23: Generation of one long-context conversation from a seed document. A seed document with a length between 128128 k and 224224 k tokens is split into about 4040 sections S0S_{0} to S39S_{39} . The generator reads the document, an index of its sections, and the prompt as one message. Four requests r1r_{1} to r4r_{4} are parsed from its response. Each request is answered in its own call returning a reasoning trace tit_{i} and an answer aia_{i} . Turns are appended in order to a conversation that holds the document, ensuring conversations are shorter than 256256 k tokens.
Figure 24
Figure 26: Assignment of reasoning-effort labels to the exchanges of each sample from the \tl_set:Ne SFT data pool. We first grade the difficulty of each conversation by its prompt with Poacher-2B, a grader distilled from a \tl_set:Ne LLM judge. We then count the reasoning length LL of each exchange, the number of reasoning tokens from one user turn to the next. For each reference group gg (a combination of a dataset and a difficulty), we compute the 30th and 70th percentiles ℓg30\ell_{g}^{30} and ℓg70\ell_{g}^{70} of LL . To label an exchange post hoc, we assign the effort level based on the relative position of LL to the 30th and 70th percentiles ℓg30\ell_{g}^{30} and ℓg70\ell_{g}^{70} . Rather than a hard cut-off, we use the soft draw of Equation 12 around the bucket edges 0.30.3 and 0.70.7 . This effort label is then rendered as part of the system prompt; an exchange without reasoning is labelled none .
Figure 27: Reasoning tokens per effort label in the training data of the released Kolibri \tl_set:Ne SFT checkpoint, read back from a sample of its dataset artefacts and pooled per domain. Dashed lines mark the label medians. The labels overlap within a domain because the thresholds differ between the datasets of a domain and because the draw of Equation 12 is soft; the same label covers different lengths in different domains.
Figure 28: Reasoning length of the released Kolibri \tl_set:Ne SFT checkpoint against the difficulty Poacher-2B assigns to the evaluation prompts. Left: length by judged difficulty over the three reasoning efforts, dashed lines at the medians. Right: the median length per judged difficulty, one line per effort; at a fixed difficulty the three lines show how much the effort adds, and along one line how much the difficulty adds. none is left out, as it produces an empty reasoning block.
Figure 28
Figure 30: Domain weights of the three mixes trained for 40004000 steps (left) and the scores of their models and of the soup of each pair (right), in %\mathrm{\%} . The domain weights are shares of the trained tokens. In each score panel, the diagonal holds the three models and each cell below it the soup of the models named by its row and column; colours are scaled per panel, and the best cell is framed. AIME 2026, LiveCodeBench and AA-LCR are the math, code and long-context benchmarks whose scores spread most over the mixes and soups. The mean is over 1212 benchmarks: math (AIME 2025, AIME 2026 and AIME 2026 IID of Section 3.1.3 ), knowledge (GPQA Diamond and KulturBank, Section H.5 ), agentic tool use ( τ2\tau^{2} -bench telecom, multi-turn \tl_set:Ne BFCL v3 and BrowseComp Lite), code (LiveCodeBench), instruction following (IFBench, loose accuracy) and long context (AA-LCR and LongBench Pro).
Figure 31: Logit-gradient norm GtG_{t} of a token at unit advantage ∣At∣=1\lvert A_{t}\rvert=1 , over its inference probability pμ,tp_{\mu,t} and training probability pθ,tp_{\theta,t} . Left: The diagonal marks on-policy probabilities, and the dashed square the bottom left corner pμ,t,pθ,t≤p_{\mu,t},p_{\theta,t}\leq0.5 . The other panels show the norm in that corner under \tl_set:Ne CISPO’s importance-weight cap at $r_{t}=$2 ( MiniMax, 2025 ) , under our norm bound Gmax⁡=2G_{\max}=2 and under \tl_set:Ne CISPO’s cap at rt=r_{t}=5$$ . Above \tl_set:Ne CISPO’s dashed line, the norm GtG_{t} is limited to the cap times 1−pθ,t1-p_{\theta,t} , and left of the dark parabola Gt=Gmax⁡G_{t}=G_{\max} , ct<1c_{t}<1 scales down the token’s gradient to exactly the bound.
Figure 32: Our \tl_set:Ne RL environment mix. Every point represents one environment or a pool of related environments, plotted over its number of train samples and the weight we assign during training (cf. Section 3.2.2.5 ). Colour and mark shape denote the environment’s group; the percentage in the legend is the group’s share of the summed sampling weight. Both axes are logarithmic. The train splits hold 1 209 9651\,209\,965 rows in total. We deliberately upweight several agentic environments as well as competitive programming, while keeping the total weight of the Merlin–Arthur environments comparatively small. Hallucination reduction holds the answerable questions from four datasets and unanswerable ones from two. Table 55 shows exact per-environment values with step and token budgets.
Figure 33: The verifier of every \tl_set:Ne RL environment and the shape of its reward. A filled disc marks a binary reward, a hollow one the fraction of checks passed. A half-filled one indicates a graded score, i.e. the token F1 against the gold answers or a judge’s grade of 00 , 0.50.5 or 11 . A column with several marks combines verifiers, either within one environment or across the environments it pools. The Merlin–Arthur reward is signed ( Section 3.2.2.3 ). Columns are grouped and coloured as in Figure 32 .
ScopeSignal and clean-output condition
UniversalNon-empty output. The final assistant step contains visible text or a parsed tool call.
 No leaked tool call. No literal tool-call markup leaks into the visible output.
 No script leak. No \tl_set:Ne CJK text leaks into a response that is otherwise non- \tl_set:Ne CJK, and no generated token matches the banned-script pattern.
Tool UseTool in schema. Every called tool was offered in the row’s tool schema.
 Well-formed tool call. The emitted tool-call structure is well formed.
 Arguments in schema. Tool arguments use the declared names and types.
Table 24: Format signals active during \tl_set:Ne RL training. Conditional signals are omitted when they do not apply. The signals si,e,ks_{i,e,k} lie in [0,1][0,1] , where 11 denotes a clean output and 00 a detected defect.
Figure 34: What the German environments buy: held-out benchmark scores of the \tl_set:Ne SFT checkpoint and of ten \tl_set:Ne RL checkpoints trained with the German environments in the mix. Left: share of responses whose reasoning and answer are in the language of the prompt. Right: accuracy on the same benchmarks. AIME 2026 is 3030 problems sampled 1616 times each; both panels average over all samples. German AIME 2026 moves from 89 %89\text{\,}\mathrm{\%} to 99 %99\text{\,}\mathrm{\%} in language consistency and 75 %75\text{\,}\mathrm{\%} to 91 %91\text{\,}\mathrm{\%} in accuracy by step 10001000 , while English AIME 2026 accuracy moves from 92 %92\text{\,}\mathrm{\%} to 96 %96\text{\,}\mathrm{\%} and its language consistency stays at 100 %100\text{\,}\mathrm{\%} .
Figure 35: One workplace-assistant task under three group seeds. The seed draws the system prompt from a table of equivalent phrasings and its language, English or German, independently of the language of the request. About 5 %5\text{\,}\mathrm{\%} of the words of the request receive a typo ( highlighted ); the words the verifier matches against ( bold ) are never touched. The agentic environments draw their coding harness from the same seed.
Figure 36: Placement of the \tl_set:Ne RL run on 3232 hosts with eight B300 GPUs each. Each small square is one GPU and each box of eight squares one host. Host 00 runs trainer ranks 0707 and the processes in the dashed box. Every other host runs one rollout worker and one sandbox service. The global router on host 00 picks the inference replica for each group. The rollout worker then sends the group’s requests to that replica directly.
Figure 37: Weight push from the trainer to the inference replicas. At startup, the receive buffers are mapped to the trainer parameters, and each parameter is assigned a trainer rank such that the bytes per rank are balanced. After every optimiser step, the trainer ranks push the new weights via \tl_set:Ne RDMA into the single receiving replica of each node, from where they are broadcast to the node’s other replicas over NVLink. The conversion on the trainer side is batched and pipelined, and the intra-node NVLink distribution reduces load on the inference node’s \tl_set:Ne NIC. The single-GPU replicas of our deployment skip the split of the weights into a per-replica layout.
Figure 38: Sandbox stack of the \tl_set:Ne RL run on one worker node. Task images are converted once to single-file Apptainer images on shared scratch. Sandboxes of the same image share its read-only SquashFS layer on the node, and each adds its own writable tmpfs layer. The right panel shows one sandbox: the OverlayFS root, the long-running Apptainer instance, and the egress proxy for the allowlisted hosts.
Figure 39: Validation reward of the Kolibri \tl_set:Ne RL run by environment group. Each line is the mean environment reward of the validation sequences that finish in an optimiser step, averaged over reasoning efforts and over the environments of a group of Figure 32 , weighted by their sampling weight, as a centred moving average over 2121 steps. The retrieval and long-context line also averages the Hallucination Reduction environments, whose rewards lie in [−1,1][-1,1] . The grey line is the same weighted mean over all 3838 environments.
Figure 40: Training dynamics over optimiser steps of the Kolibri \tl_set:Ne RL run. Left: entropy of the training policy over the trained tokens of each optimiser step, divided by the logarithm of the vocabulary size. Middle: share of tokens removed by the \tl_set:Ne BTV mask and scaled down by the gradient-norm bound of Section 3.2.1.2 , i.e. with ct<1c_{t}<1 . Right: mean absolute log-ratio ∣log⁡rt∣\lvert\log r_{t}\rvert and forward \tl_set:Ne KL divergence KL(pμ ∥ pθ)≈rt−1−log⁡rt\mathrm{KL}(p_{\mu}\,\|\,p_{\theta})\approx r_{t}-1-\log r_{t} between the training policy and the inference policy. Both metrics include a median policy staleness of 10.010.0 optimiser steps from our asynchronous setup ( Section 3.2.1.5 ).
Figure 41: Difficulty buckets of the \tl_set:Ne RL curriculum over training, summed over all environments except the Hallucination Reduction environments, which bypass the curriculum. Left: share of all prompts in the hard, trainable and solved buckets. Right: share of sampled prompts per bucket, divided by its target bucket weight cbc_{b} . During initial training steps, the curriculum compensates for missing and inaccurate offline pass rates, which populates the hard and solved buckets across environments. The model then improves over the course of training, causing more prompts to become solved. In particular, several environments become entirely solved or trainable, draining their share of hard prompts.
Figure 42: Targeted \tl_set:Ne OPSD over the Kolibri \tl_set:Ne RL run. Left: Detected failure cases across the 55 terminal environments, counted at the optimiser step whose weights produced the sequence. Right: Moving average over 2525 steps of the share of distilled tokens for which the teacher’s most likely token is the sampled token.
Figure 43: Benchmark scores of the \tl_set:Ne RL checkpoints of Kolibri Origin and Kolibri over time. Grey lines indicate median evaluation scores for open models. Each coloured point corresponds to a finished evaluation run that improved at least one of the four depicted benchmarks. Vertical lines mark the first evaluation of an \tl_set:Ne RL checkpoint of Kolibri. The lines track the best score of their model generation, evaluated with their respective setup at the time they were run. These intermediate points do not indicate representative benchmark performance, but are meant as a visual representation of Kolibri’s progress over time. A star marks the released Kolibri checkpoint, sampled at top- pp 0.970.97 and top- kk 128128 as in production. SWE-bench Verified is scored with the mini-swe-agent harness ( Yang et al., 2024b ) and shown for Kolibri only, as we did not optimise Kolibri Origin towards agentic capabilities. The release (star) is scored with the OpenCode harness, without web access.
Figure 44: Agentic retrieval scores of the Kolibri \tl_set:Ne RL checkpoints evaluated in September 2026. Each point is one eval run, each solid line the mean of the runs of a day, and the dashed line Kolibri Origin; under each title, the score of Kolibri Origin and of the released Kolibri checkpoint. MuSiQue and Honeypot are scored on the test splits of the two training environments. The five Industry \tl_set:Ne RAG benchmarks of Section I.1.4 are in-house proxies for deployed customer assistants and share no documents or questions with the training data.
Figure 45: Example sequences that led to exploits, with ellipses marking omitted text. Left: The OpenCode harness exposes a web-fetch tool, and the policy resolves an SWE-bench instance to its upstream pull request, downloads the patch and applies it. Right: The sandbox does not let apt switch to its unprivileged download user. The reasoning concludes that the process runs as root and can therefore write to kernel settings, and the next tool call disables unprivileged user namespaces on the host kernel.
BenchmarkDescriptionGerman version
Knowledge
GPQA Diamond ( Rein et al., 2024 )Chain-of-thought prompts, avg@ 88 , since a single run leaves several points of sampling noiseTranslation
Humanity’s Last Exam ( Phan et al., 2025 )Text-only questions, no output capTranslation
AA-Omniscience ( Jackson et al., 2025 )Public set. Its accuracy scores a wrong answer no better than abstaining, its index between −100-100 and 100100 scores a wrong answer −100-100 and an abstention or partial answer 00 , and its non-hallucination rate is the share of non-correct answers that abstained or answered partially rather than guessed.–
\tl_set:Ne MMLU-Pro ( Wang et al., 2024a )Chain-of-thought promptsMMLU-ProX ( Xuan et al., 2025 )
Math
Table 27: Post-training benchmarks by category, with their settings and German versions. Every German version marked as a translation comes from ellamind (2025) , and an en dash marks a benchmark without a German version. The score avg@ NN is the mean over NN runs of every sample.
Figure 46: Overall English and German post-training aggregates of Section 3.3.3 against the active parameters per token on a logarithmic axis, where higher and further left is better. The dashed line marks the convex hull of the Pareto frontier. Apertus 70B Instruct is excluded because its deployment does not produce comparable scores.
Figure 47: Agentic retrieval and customer-proxy benchmark scores ( %\mathrm{\%} ) of the released Kolibri checkpoint and the three open models of Figure 2 , from Table 29 . MuSiQue, Honeypot and Agentic Wiki QA (DE) are agentic retrieval benchmarks; the five customer proxies are internal benchmarks built from customer use cases, named by vertical.
Table 28: Post-training benchmark scores of Kolibri, Kolibri Origin, and comparable open models (part I/II). The first two rows give each model’s total and active parameters in billions, with bars on a common scale. Bold is the best score among \tl_set:Ne MoE models , underlined is best in class , incomplete or non-percentage scores are greyed out . Averaging is performed by language and over the category averages, with details given in the caption of Table 29 .
More in this topic

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

Explore similar work

CardsList
Jul 10, 2026

A Sovereign, Open-Source Foundation Model for German and English

We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense models for long-context, high-concurrency deployment. Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while achieving the best code aggregates in both languages among 17 open base models, and outperforms every European sovereign baseline in our comparison, including ones far larger in active parameters. Among fully open models, Soofi S obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC scale AI infrastructure operated by Deutsche Telekom in Munich. Soofi S will be released under highly permissive, open-access terms: weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code. Where source licenses permit, data-construction artifacts are released under permissive licenses; commercially licensed sources are documented with aggregate statistics and exact mixture accounting.

The Soofi-Team, :, Benedikt Droste +29

ellamind · Fraunhofer IAIS · Lamarr +8

Language Model PretrainingMultilingual Language ModelsMixture-of-Experts Language ModelsHybrid Mamba-Transformer ArchitecturesLanguage ModelingEfficient Language Model Inference
AskPDF
Jun 23, 2026

Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment

Transformer-based language models have become the default substrate for natural language processing and the pace of new releases has made it hard for practitioners to separate durable ideas from the noise of incremental announcements. This review works at two levels. At the level of mechanism, we organise the main transformer families into a working taxonomy, covering encoder-only, decoder-only, encoder-decoder, long-context, permutation-based, and generator-discriminator variants. We then extend the discussion to post-2023 developments that changed the picture in practice: instruction tuning, reinforcement learning from human feedback, direct preference optimisation, mixture-of-experts scaling, retrieval augmentation and the current flagship model families from OpenAI, Anthropic, Google, Meta, Mistral and DeepSeek. At the level of use, we survey deployments across healthcare, finance, legal, education, customer service, creative writing and scientific work. Based on this we link each to the specific capabilities that make a transformer the appropriate tool. The contribution of this paper is a critical assessment that is based on the survey. We compare architectures on four axes that matter to deployment decisions, we quantify the trade-off between parameter count and energy cost. We also discuss how alignment methods, data provenance and benchmark saturation change what it means to call a model "state of the art". The final section lists the research questions that we think deserve more attention.

Guruprakash J, Krithika L. B

SCOPE, VIT-AP University, Amaravathi, Andhra Pradesh, 522241, India · SCORE, VIT, Katpadi, Tamil Nadu, 632014, India

LLM EvaluationLLM AlignmentTransformer
AskPDF
Aug 27, 2026

Thomson: Continual Learning of Frontier Models for SovereignAI

The development of frontier models is commonly perceived to be the exclusive remit of a small number of heavily funded players, creating an information, economic and power asymmetry between developers and the diverse user base of modern AI. Recent public discourse acknowledges this concern, calling for SovereignAI (an organisation's capability to independently build, deploy and govern AI use), but offers little concrete advice on how this can be achieved in the short term under a diversity of funding settings. We argue that frontier performance is achievable by a wide range of institutions through Continual Learning on readily available open-weight models. Unlike limited approaches such as small-scale fine-tuning, prompt engineering, or tool-augmentation of a frozen model, our approach exploits a modern mid- & post-training stack while introducing safeguards that preserve both plasticity and stability at each stage, making the minimal number of high-impact interventions on the parameters. This yields gains comparable to those typically seen across multiple successive model generations, at compute and personnel budgets substantially lower than commonly thought, making ownership of large parts of the SovereignAI stack (model, tool infrastructure, values & data privacy) viable for far more actors. We demonstrate this with Thomson, a general-purpose frontier model trained with an enhanced focus on high-stakes professional work. Thomson performs competitively with recent frontier models across agentic tasks, safety, legal, tax & multilingualism, and large-scale Deep Research. Evaluations show a distinctive ππ-shaped pattern: distinct improvements across a wide range of capabilities, including those not explicitly targeted, while almost completely eliminating the forgetting problem common to narrow domain adaptation.

Shengzhuang Chen, Jerrod Parker, Yejin Bang +23

Thomson Reuters · DatologyAI · Lambda +1

Continual Learning for LLMsContinual LearningCatastrophic Forgetting
AskPDF