Apertus 1.5: From Open Text Models to Open Multimodal Reasoners

Project Apertus; :; Yixuan Xu; Yassine Turki; Vinko Sabolčec; Valentina Pyatkin; Tommy Chu; Sven Najem-Meyer

cs.CLOct 11, 2026

Apertus 1.5: From Open Text Models to Open Multimodal Reasoners

Authors: Project Apertus, :, Yixuan Xu, Yassine Turki, Vinko Sabolčec, Valentina Pyatkin, Tommy Chu, Sven Najem-Meyer, Sean F. Carroll, Robert Smith, Raphael Kreft, Oscar Alessandro Tazza, Oleg Lavrovsky, Nicola Irmiger, Matteo Santelmo, Marian Schneider, Juan García Giraldo, Hanna Yukhymenko, Eduard Ďurech, Davit Melikidze, Dani Tamayo, Bettina Messmer, Aryan Ahadinia, Anunay Yadav, Allen Huang, Ahmad Fraij, Alessandro Bombarda, Alex Padula, Alexander Hägele, Alexander Sternfeld, Anna Hedström, Anna Kosovskaia, Arthur Wuhrmann, Arundhati Balasubramaniam, Badr Al Mahouri, Barna Pasztor, Bartosz Szostakiewicz, Bich Ngoc Doan, Bryan Schmid, Burak Sezen, Camille Challier, Clément Charmillot, Dana Shayakhmetova, Daniel Perruchoud, David Brüggemann, David Isztl, Elia Palme, Emma Anastassova, Fabian Bösch, Faruk Zahiragić, Fawzi Roberto Mohamed, Fiona Panicali, Florian Kolly, Flynn Guerra, Francesco Monti, Frederike Lübeck, Gennaro Oliva, Haofeng Yang, Haoliang Shang, Helen El Khoury, Henrique Mendonça, Jannis Vamvas, Jérome Landolt, Jessica Lam, Jingchun Xiao, Jonathan Coles, Joshua Oyebanji, Khanh Nguyen, Konstantinos Pitas, Lasse B. Strand, Li Mi, Loïc Deslarzes, Lorenzo Baggi, Lorenzo Noci, Luca Baumann, Luca Sartori, Lukas Drescher, Marco De Negri, Marco Passerini, Mark Klein, Marko Stojanovic, Martin Majsec, Matin Ansaripour, Mattea Busato, Mauro Pellonara, Max Neuwinger, Michael Graber, Michal Tešnar, Michelle Wastl, Miguel Gila, Mikolaj Boronski, Morgane Magnin, Moritz Reihs, Mukhammadali Sayfiddinov, Nathan Ranchin, Nicolas Cuveillier, Oscar Wohlfahrt, Qianqian Zou, Rada Kamysheva, Rita Sevastjanova, Roman Machacek, Sebastian Keller, Simon Matrenok, Simon Pierre, Stefano Schuppli, Suei-Wen Chen, Szabina Horváth-Mikulas, Taliesin Perez, Thilo Spinner, Thomas Kleine Buening, Tim Robinson, Tymoteusz Kwiecinski, Vilém Zouhar, Vinzenz Thoma, Xiaozhe Yao, Yannick Metz, Yuhei Fukuhara, Yusif Askari, Zixuan Chen, Alexander Ilic, Ana Klimovic, Andreas Krause, Andrei Kucharavy, Elliott Ash, Joost VandeVondele, Martin Rajman, Mennatallah El-Assady, Thomas Schulthess, Timo Kehrer, Torsten Hoefler, Antoine Bosselut, Martin Jaggi, Imanol Schlag

Organizations: ETH Zurich · EPFL · HES-SO Valais-Wallis · FHNW ·

Abstract

Language models are becoming infrastructure, and infrastructure built in the open, by public institutions, on documented data, is the alternative to dependence on a few providers; Apertus shows this is viable at scale. Apertus 1.5 extends the fully open Apertus family from text to multimodal reasoning. Continued pretraining of the Apertus 1.0 models, on 4T additional tokens for the 8B and 2T for the 70B, adds image and audio understanding and extends the context window fourfold to 262,144 tokens; multimodality enters through the tokenizer, the language-model head remains restricted to text, and the architecture is unchanged. Supervised finetuning, long-context finetuning, reinforcement learning with verifiable rewards, and preference alignment complete the released models. Thinking is a flag of the chat template rather than a separate model, and lifts mathematical reasoning by up to 20 points on the 8B model. The released models improve over their predecessors on nearly every benchmark; among fully open models of its class, the 8B leads on knowledge, instruction following, and cultural benchmarks, and its instruction following surpasses all compared models, including open-weight ones. On image and audio understanding, a first for the family, the models are competitive with fully open multimodal models, strongest on document understanding and long-form and lower-resource speech recognition, while specialized vision and audio models remain ahead on perception and transcription of English and other high-resource languages. As in Apertus 1.0, we train only on openly available data, respect crawler opt-outs retroactively, and exclude restrictively licensed material. We release the model weights, data preparation and training recipes, and evaluation suites under permissive licenses; openness extends to values, with the Apertus charter guiding alignment stages published alongside the models.

Figures & tables

Panel of Figure 1 — Figure 1: The Apertus 1.5 training pipeline. Solid blue stages train on multimodal data (T $\,\cdot\,$ I $\,\cdot\,$ A, or T $\,\cdot\,$ I for RLVR), light-blue stages on text only (T); red outlines mark the released models, which the 70B reaches without SDPO. DPO uses online pairs for the 8B model and offline and online pairs for the 70B model.

Panel of Figure 2 — Figure 2: The Apertus 1.5 architecture.

Figure 3: Composition of the 8B vision samples. The inner segments show the relative dataset sizes (largest first). A dataset can span multiple categories.

Figure 4: Vision data composition by stage, with pool sizes in input tokens on the right.

Figure 5: Audio data composition by stage, with pool sizes in input tokens on the right.

Tokenizer	Tokens $\downarrow$	PSNR (dB) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Emu3	8,372	26.70	0.806	0.075
Emu3.5	2,080	24.04	0.740	0.099

Table 1: Reconstruction quality and token cost of the Emu3 and Emu3.5 tokenizers, averaged over 350 Stage 1 training images from six representative categories of Section 2.2.3 and preprocessed as in training. Tokens are the mean per image, arrows give the better direction, and bold marks the better tokenizer; Table E.1 in Appendix E lists the values per category.

Figure 6: Per-category reconstruction quality of the Emu3 and Emu3.5 tokenizers on the same 350 Stage 1 training images as Table 1 . Each marker is a category mean. Higher is better for PSNR and SSIM; lower is better for LPIPS. Table E.1 lists the values.

Stage	Data	Prediction Targets
1	Unpaired image, audio, and text data.	All modalities
2	Paired image-text, audio-transcript data, and text data	Text only
3	Higher-quality data, with some instruction-formatted data	Text only

Table 2: Stages 1 and 2 fall in the warmup and stable phases of the LR schedule, while Stage 3 is the decay phase. Stage 1 is trained to predict tokens of all modalities. Stages 2 and 3 are trained to predict only text tokens.

Stage	Vision	Audio	Text	Total
Apertus 1.5 8B
1	367B (45.6%)	110B (13.7%)	328B (40.7%)	806B
2	852B (30.4%)	89B (3.2%)	1,861B (66.4%)	2,802B
3	78B (19.4%)	95B (23.4%)	230B (57.2%)	403B
Apertus 1.5 70B
1	49B (12.2%)	52B (13.0%)	301B (74.8%)	403B

Table 3: Token budget per modality across the three curriculum stages of Apertus 1.5 8B and Apertus 1.5 70B, that is, the mix each run trains on. Percentages are the within-stage share of tokens. For Stages 2 and 3, the Vision and Audio counts include their paired text (image captions and audio transcripts); the Text column is the separate text-only data.

Name	Stage 1+2 (%)	Stage 3 (%)
HuggingFaceTB/dclm-edu	47.5	20.5
HuggingFaceFW/finepdfs-edu	2.8	3.2
nvidia/Nemotron-CC-v2.1	1.0	7.2
PleIAs/SYNTH	0.7	0.0
epfml/FineWeb2-HQ-Plus (top 10%)	20.1	14.4
HuggingFaceFW/finepdfs-edu (multilingual only)	8.5	3.6

Table 4: Text data mixture per training stage, as a percentage of each stage’s sampled text-only token budget.

Phase	English	Multilingual	Image	Audio	SFT	Total
Apertus 1.5 8B
32k	6.07B (25.6%)	4.05B (17.1%)	6.66B (28.1%)	1.30B (5.5%)	5.60B (23.6%)	23.69B
65k	5.63B (25.5%)	3.79B (17.2%)	6.25B (28.3%)	1.22B (5.5%)	5.17B (23.4%)	22.07B
131k	5.69B (25.9%)	3.80B (17.3%)	6.27B (28.6%)	1.23B (5.6%)	4.96B (22.6%)	21.95B
262k	5.69B (26.0%)	3.81B (17.4%)	6.29B (28.7%)	1.22B (5.6%)	4.86B (22.2%)	21.87B
Apertus 1.5 70B

Table 5: Token budget per class across the four long-context extension phases. Percentages are the within-phase share of tokens beyond the Cooldown subsample of Table 3 .

Panel of Figure 7 — Figure 7: Per-stage loss against consumed tokens. Top ( 7(a) , 7(b) ): text loss , computed over text tokens only in every stage and therefore comparable across stage boundaries. The text loss creeps upward through Stage 1 and then drops sharply over the Stage 3 cooldown. Bottom ( 7(c) , 7(d) ): combined loss over all of each stage’s prediction targets ( Table 2 ): image, audio, and text tokens in Stage 1, text only from Stage 2 onward. In Stage 1 the combined loss stays stable, so the text-loss creep visible in the top row is masked when text and multimodal targets are averaged together; the drop at the Stage 1 $\to$ 2 boundary reflects the change of targets, not a change in quality. Averages are computed within each stage.

Panel of Figure 8 — Figure 8: A conversation rendered by the inference chat template, with thinking enabled and one tool declared: the model reasons and calls the tool inside the thinking span, and answers after the tool output returns. The released models do not yet produce this combination ( Section 7.2 ). Line breaks are added for display.

Panel of Figure 9 — Figure 9: Capability filtering. We train an auxiliary nonthinking Apertus 1.5 8B checkpoint to evaluate verifiable text prompts. Prompts solved at least once in eight attempts enter the production text mixture without reasoning traces; those that fail all eight receive verified traces. Both production models use the resulting mixture.

Category	Dataset Source	Samples	Data Ratio
Foundation	OLMo2 OASST1 (Tulu-3) ( Köpf et al., 2023 )	7,028	6.14%
	Llama-Nemotron Chat ( Bercovich et al., 2025 )	20,585
	Llama-Nemotron Safety ( Bercovich et al., 2025 )	13,765
	Apertus FAQ ( Hernández-Cano et al., 2026 )	1,580
	IF-augmented (single- & multi-turn)	184,580
	Subtotal	227,538

Table 6: Text component of the final production SFT mixture by source and category. All datasets are decontaminated against evaluation benchmarks. Numbers indicate example count after filtering. Data Ratio is each category’s share of the text SFT pool by example count.

Dataset	Sequences	Tokens (M)	Processing
llava_ov_permissive ( An et al., 2025 )	10,019,044	24,859.2	Permissive subset
sensenova_si_8m ( Cai et al., 2026 )	5,205,020	21,859.4	Reasoning stripped
google_maptrace ( Panagopoulou et al., 2025 )	1,055,208	7,585.8	As released
lnqa_recap ( Pont-Tuset et al., 2020 ; Wiedmann et al., 2025 )	302,777	1,034.5	Answers regenerated
pixmo_cap_qa ( Deitke et al., 2025 )	163,359	768.3	As released
CoSyn_400k ( Yang et al., 2025b )	87,057	359.1	As released

Table 7: Image pool used to train the auxiliary checkpoint for capability filtering of text data: 20 datasets, 17.5M sequences, 58.1B tokens. The production SFT pool appears in Table B.5 .

Figure 10: Representative categorical answer distributions in image SFT sources with deterministic extraction. Left: RSRCC strongly favors the first two answer options among questions with four options, whereas BigEarthNet is nearly uniform. Right: RSRCC also strongly favors affirmative binary answers, unlike BigEarthNet, SpatialSense, and PathVQA. These source distributions can create answer priors but do not by themselves establish a causal effect on evaluation accuracy.

	8B		70B
Dataset	Tokens	Samples	Tokens	Samples
Text, 26–108K
SFT mixture subset ( Table 6 )	1,870,000,000	1,119,142	2,410,000,000	1,438,897
Tool (short-context), 119–16.4K
Toucan ( Xu et al., 2026 )	33,700,000	15,162	43,400,000	19,494
EnvScaler ( Song et al., 2026 )	9,300,000	763	11,900,000	981

Table 8: Long-context SFT data mixture. The mix each model’s run trains on, with tokens and samples per source; the runs draw subsets of the pools, so the totals stay below the pool sizes. Length range is the observed range from minimum to maximum sequence length within the class. The short-context text and audio rows are length-filtered slices of the production pools of Section 4.2 ; all three vision rows (short- and long-context) are length buckets of the same pool in Table B.6 . Those tables give the per-dataset composition.

Benchmark	SFT	+Offline DPO	+Online DPO	+RL	+RL +Online DPO
IFEval	80.2	80.0	86.1	85.8	89.6
Multi-IF	74.0	77.2	81.4	83.0	84.5

Table 9: Instruction following of Apertus 1.5 8B under different post-training paths from the same SFT checkpoint, thinking disabled. The SFT column is the shared starting point, the next two apply a single alignment stage to it, and the last two are the checkpoints of the release lineage ( Table 26 ). Best value per row in bold.

Figure 11: Validation accuracies ( mean@4 ) during the reinforcement learning run. Optimizer steps are shown on a log scale. Validation is performed with thinking mode disabled and 4,096 maximum output tokens, and relies on the verifiable answer for math domains. For faster train-time evaluation, benchmarks are randomly subsampled to 100 prompts each, so values may not exactly reflect the complete downstream evaluation.

Domain	Dataset	Samples
Mathematics	Big-Math-RL-Verified	50,000
	DeepMath-103K	50,000
Instruction following	In-house single-turn	60,000
	In-house hard	20,000
Code	TACO	13,000
	CodeContests	13,000

Table 10: Prompt composition of the Apertus 1.5 reinforcement-learning mixture. Counts refer to samples used in the production mixture.

Arm	Training Data	Traces	Conversations	Tokens	Updates
Text No-CoT	Text	No	3.48M	4.4B	535
Text CoT	Text	Yes	3.48M	10.2B	1,211
MM No-CoT	Text + multimodal	No	7.40M	19.6B	2,352
MM CoT	Text + multimodal	Yes	7.40M	27.5B	3,290

Table 11: Configuration of the SFT ablation. Conversations are reported per epoch and tokens over the full two-epoch run. CoT and No-CoT arms use the same conversations and differ only in whether reasoning traces are retained.

Panel of Figure 12 — Figure 12: SFT ablations of multimodal supervision and reasoning, one panel per question. Q1 compares text benchmark scores for MM No-CoT versus Text No-CoT and MM CoT versus Text CoT, evaluated under matched response modes. Q2 compares MM CoT with thinking on against MM No-CoT with thinking off. Q3 compares MM CoT against MM No-CoT, both with thinking off. Bars report mean score differences in percentage points; dots denote the three multimodal seeds. Comparisons use a single text-only reference seed in Q1 and paired seeds in Q2 and Q3. Scores are unweighted means over 16 text benchmarks, excluding MGSM ( Table H.1 gives the reason), and 14 vision benchmarks. Without thinking, decoding is greedy with a 16,384-token budget; thinking is sampled at temperature 0.6 and top-p 0.95 with a 32,768-token budget.

Prompt Set	Source	Prompts	Reward	Thinking
Mathematics	DAPO-Math-17k	15,984	Math-Verify	On
Reasoning	Reasoning Gym	14,912	Verifier, judge fallback	On
Instruction following	Nemotron-RL IF	46,390	All constraints met	Off

Table 12: Training configuration of the RL ablation, with one row per prompt set. The sets are drawn from DAPO-Math-17k ( Yu et al., 2025 ) , Reasoning Gym ( Stojanovski et al., 2025 ) , and nvidia/Nemotron-RL-instruction_following . Each run lasts 200 steps of GRPO ( Shao et al., 2024 ) , with 64 prompts and eight rollouts per step. We use binary rewards, a leave-one-out baseline, DAPO asymmetric clipping, no KL penalty, and a constant learning rate of

2\times 10^{-6}

. Rollouts are asynchronous, at most two policy versions stale, and capped at 14,336 new tokens with thinking on and 3,840 with thinking off. Mathematics is scored with Math-Verify ( Kydlíček, 2025 ) ; for reasoning tasks, Qwen3.6-35B-A3B is used as a fallback judge only when the task verifier rejects an answer.

Panel of Figure 13 — Figure 13: Held-out benchmarks during RL from three SFT checkpoints. Each group of panels comes from the runs on one prompt set, evaluated in the thinking mode used for training ( Table 12 ). Each curve is a single RL run; step 0 is the SFT checkpoint, and we evaluate every 50 steps. We report accuracy on MATH-500, mean@32 on AIME, zero-shot exact match on BBH, and prompt-level strict accuracy on IFEval, Multi-IF, and IFBench. A response without a final answer counts as wrong. AIME 2026 excludes problem 1, which appears in DAPO-Math-17k.

Panel of Figure 14 — Figure 14: Training dynamics of the runs in Figure 13 . The top row shows the mean reward of each step, and the bottom row the tokens per rollout, counting the reasoning trace when thinking is on and the whole response when it is off. Curves are 10-step trailing means from step 3, because the first two steps of asynchronous rollout collect the fastest and then the slowest rollouts, producing unrepresentative reward and length statistics.

	Fully open				Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	Marin	Qwen3.5	Llama 3.1
Parameters	8B	8B	7B	8B	9B	8B
HellaSwag	78.8	78.1	74.1	82.4	79.3	79.3
PIQA	80.2	80.3	78.7	81.9	80.5	81.2
CommonsenseQA	62.3	62.3	58.7	70.3	83.6	70.4
LAMBADA	70.1	68.8	65.1	68.8	70.5	72.3

Table 13: Pretraining Evaluation (8B-class models). Performance (%) on general language understanding tasks, higher is better. Bold = best overall; italic = best among fully open models.

	Fully open						Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	EuroLLM	Marin	K2-V2	Qwen3.5	Nemotron 3.5
Parameters	70B	70B	32B	22B	32B	70B	27B	30B-A3B
HellaSwag	83.0	82.2	81.5	79.0	85.0	83.7	83.4	63.7
PIQA	83.0	82.3	81.3	81.4	83.3	83.9	82.3	83.1
CommonsenseQA	53.8	66.8	71.1	60.5	75.5	73.3	87.4	77.7
LAMBADA	72.9	72.9	70.5	70.9	68.7	76.1	74.2	72.8

Table 14: Pretraining Evaluation (70B-class models). Performance (%) on general language understanding tasks, higher is better. Bold = best overall; italic = best among fully open models.

	Fully open				Open-weight
Benchmark	Apertus-8B	Apertus 1.5 8B	OLMo 3	Marin	Qwen3.5	Llama 3.1
Parameters	8B	8B	7B	8B	9B	8B
MMLU CoT	59.3	62.9	59.4	62.6	76.7	60.7
Global-MMLU	54.4	56.0	44.3	48.5	77.1	52.7
MMLU-Pro	29.6	38.5	33.0	30.9	58.4	36.1
AGIEval	36.1	37.2	35.0	37.3	58.0	35.5

Table 15: Pretraining Evaluation (8B-class models). Performance (%) on factual knowledge tasks, higher is better. Bold = best overall; italic = best among fully open models.

	Fully open						Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	EuroLLM	Marin	K2-V2	Qwen3.5	Nemotron 3.5
Parameters	70B	70B	32B	22B	32B	70B	27B	30B-A3B
MMLU CoT	61.8	67.0	71.4	62.9	71.6	70.5	84.9	75.4
Global-MMLU	56.3	59.3	49.1	54.9	54.2	56.1	78.6	69.4
MMLU-Pro	36.0	43.8	48.0	36.8	45.3	45.4	78.0	67.2
AGIEval	41.2	43.9	46.6	40.1	43.9	40.4	62.7	50.5

Table 16: Pretraining Evaluation (70B-class models). Performance (%) on factual knowledge tasks, higher is better. Bold = best overall; italic = best among fully open models.

	Fully open				Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	Marin	Qwen3.5	Llama 3.1
Parameters	8B	8B	7B	8B	9B	8B
Reasoning
BBH	48.8	60.2	63.9	55.8	84.4	65.2
ACP	22.1	28.1	31.8	32.0	52.7	35.1
Mathematics

Table 17: Pretraining Evaluation (8B-class models). Performance (%) on reasoning, mathematics, and code tasks, higher is better. GSM8K-P = GSM8K Platinum; Minerva = Minerva MATH; MGSM-En = MGSM English. Bold = best overall; italic = best among fully open models.

	Fully open						Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	EuroLLM	Marin	K2-V2	Qwen3.5	Nemotron 3.5
Parameters	70B	70B	32B	22B	32B	70B	27B	30B-A3B
Reasoning
BBH	58.5	68.2	78.8	59.4	71.4	69.3	91.7	77.9
ACP	32.8	32.0	41.7	34.4	40.6	41.6	77.0	47.1
Mathematics

Table 18: Pretraining Evaluation (70B-class models). Performance (%) on reasoning, mathematics, and code tasks, higher is better. GSM8K-P = GSM8K Platinum; Minerva = Minerva MATH; MGSM-En = MGSM English. Bold = best overall; italic = best among fully open models.

	Fully open				Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	Marin	Qwen3.5	Llama 3.1
Parameters	8B	8B	7B	8B	9B	8B
XNLI	44.1	45.3	39.6	42.4	39.8	45.6
XCOPA	65.6	64.9	55.1	57.0	64.1	62.5
ARC-ml	38.4	37.5	25.5	26.4	37.9	33.1
HellaSwag-ml	55.0	53.6	37.4	42.1	51.9	49.1

Table 19: Pretraining Evaluation (8B-class models). Performance (%) on multilingual tasks (“-ml” = multilingual variant), higher is better. Bold = best overall; italic = best among fully open models.

	Fully open						Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	EuroLLM	Marin	K2-V2	Qwen3.5	Nemotron 3.5
Parameters	70B	70B	32B	22B	32B	70B	27B	30B-A3B
XNLI	45.9	46.3	42.4	44.1	42.3	44.1	45.3	43.4
XCOPA	69.9	68.0	59.5	61.3	58.6	63.8	67.3	62.1
ARC-ml	39.6	40.2	34.0	35.9	29.1	34.7	43.6	39.4
HellaSwag-ml	57.6	57.1	47.2	52.9	45.2	52.4	57.3	41.8

Table 20: Pretraining Evaluation (70B-class models). Performance (%) on multilingual tasks (“-ml” = multilingual variant), higher is better. Bold = best overall; italic = best among fully open models.

	Fully open				Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	Marin	Qwen3.5	Llama 3.1
Parameters	8B	8B	7B	8B	9B	8B
BBQ	56.7	57.0	59.4	58.4	63.0	58.6
ToxiGen	52.2	42.7	58.8	48.3	59.8	43.0
WMDP	50.2	53.2	54.5	54.9	63.7	54.3
CulturalBench	69.7	69.9	70.6	72.8	78.1	67.2

Table 21: Pretraining Evaluation (8B-class models). Performance (%) on bias, safety, and cultural knowledge tasks, higher is better (higher ToxiGen/BBQ = less toxic/biased). Bold = best overall; italic = best among fully open models.

	Fully open						Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	EuroLLM	Marin	K2-V2	Qwen3.5	Nemotron 3.5
Parameters	70B	70B	32B	22B	32B	70B	27B	30B-A3B
BBQ	55.1	58.6	61.8	57.4	61.2	62.7	63.0	59.5
ToxiGen	44.1	47.2	61.9	39.7	50.3	43.2	58.6	45.5
WMDP	53.3	55.9	60.5	52.5	59.5	59.2	70.0	63.6
CulturalBench	69.9	72.6	74.7	59.5	77.2	75.5	83.7	64.3

Table 22: Pretraining Evaluation (70B-class models). Performance (%) on bias, safety, and cultural knowledge tasks, higher is better (higher ToxiGen/BBQ = less toxic/biased). Bold = best overall; italic = best among fully open models.

	Fully open					Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	Marin	EuroLLM	Qwen3.5	Llama 3.1
Parameters	8B	8B	7B	8B	9B	9B	8B
Knowledge
MMLU	60.6	70.5	67.1	63.5	62.0	83.7	72.4
MMLU-Pro	35.8	48.8	52.7	41.0	38.5	76.0	48.4
AGIEval	38.5	45.4	37.1	36.4	41.7	39.0	38.0

Table 23: Release evaluation of the 8B-class post-trained models, thinking disabled for all models including thinking-capable peers. All values in percent, higher is better, best per group in bold. Instruction following is scored by prompt-level strict accuracy and code by pass@1.

	Fully open					Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3.1	K2-V2	EuroLLM	Llama 3.3	Gemma 4	Qwen3.5	Nemotron 3.5
Parameters	70B	70B	32B	70B	22B	70B	31B	27B	30B-A3B
Knowledge
MMLU	70.1	76.5	80.1	88.1	69.0	87.6	90.5	89.5	85.0
MMLU-Pro	44.7	56.4	66.0	73.3	49.1	71.1	84.8	81.9	56.4
AGIEval	40.4	50.6	50.6	35.8	46.0	54.2	50.1	38.0	45.9

Table 24: Release evaluation of the 70B-class post-trained models, thinking disabled for all models including thinking-capable peers. All values in percent, higher is better, best per group in bold. Instruction following is scored by prompt-level strict accuracy and code by pass@1.

	Apertus 1.5 8B		Apertus 1.5 70B		Thinking-native peers
Benchmark	Off	On	Off	On	OLMo 3 7B	OLMo 3.1 32B	K2 Think V2 70B
Knowledge and Reasoning
MMLU	70.5	74.5	76.5	77.1	74.0	82.7	88.0
GPQA	27.2	32.1	32.1	30.1	27.2	39.7	59.6
Instruction Following
IFBench	35.3	44.3	25.7	32.7	20.0	23.0	36.7

Table 25: Thinking-mode comparison on reasoning-heavy benchmarks: the Apertus 1.5 models with thinking off and on, and thinking-native peers in their thinking mode. All values in percent, best per row in bold. We report mean@8 for AIME, and mean@4 for other math and coding benchmarks.

	Apertus 1.5 8B				Apertus 1.5 70B
Benchmark	SFT	+RL	+DPO	+SDPO	SFT	+RL	+DPO
Knowledge and Reasoning
MMLU	65.8	66.6	70.7	70.6	75.1	72.0	77.2
MMLU-Pro	40.1	43.0	49.4	49.2	46.3	44.4	56.9
BBH	67.9	69.1	71.1	68.2	72.4	75.2	66.7
ACPBench	26.1	31.2	37.9	37.1	38.3	49.7	63.6

Table 26: Post-training lineage of the Apertus 1.5 models on the development benchmark suite, thinking disabled. Each column adds one stage to the one before it. A representative subset of each skill area is shown; Table L.3 gives all 37 benchmarks, and the overall average is taken over all of them. Best value per row in bold, taken separately within each model.

	Apertus 1.5 8B				Apertus 1.5 70B
Benchmark	SFT	+RL	+DPO	+SDPO	SFT	+RL	+DPO
Knowledge and Reasoning
AGIEval	33.1	34.9	44.9	45.5	38.1	49.1	50.5
ARC-Challenge	78.4	84.0	84.9	84.8	85.1	86.9	87.3
GPQA	29.5	29.7	31.2	26.8	32.8	32.1	31.2
Math

Table 27: Held-out benchmarks, excluded from development, for both models, thinking disabled. All six held-out benchmarks are shown. Best value per row in bold, taken separately within each model.

	Apertus 1.5 8B				Apertus 1.5 70B
Benchmark	SFT	+RL	+DPO	+SDPO	SFT	+RL	+DPO
Knowledge and Reasoning
MMLU	69.9	70.7	74.2	73.2	73.4	76.6	78.6
MMLU-Pro	46.5	48.4	50.5	50.8	51.5	46.7	57.7
AGIEval †	33.2	35.4	42.9	44.4	34.0	48.7	47.0
BBH	67.9	68.1	70.4	71.0	72.0	75.4	65.1

Table 28: Post-training lineage with thinking enabled, on all 19 benchmarks evaluated in thinking mode. Benchmarks marked † are held out from development. The two averages cover the development and held-out benchmarks respectively, and neither is comparable with Table 26 , which spans a wider suite. Best value per row in bold, taken separately within each model.

	Fully open					Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3	Marin	EuroLLM	Qwen3.5	Llama 3.1
Parameters	8B	8B	7B	8B	9B	9B	8B
Multilingual Knowledge
Global-MMLU	55.5	59.4	46.4	48.7	55.0	74.6	56.9
INCLUDE-44	54.1	56.9	38.4	38.6	51.5	70.9	53.2
INCLUDE-45	38.9	39.5	32.1	34.6	34.4	44.5	33.8

Table 29: Multilingual and cultural evaluation of the 8B-class post-trained models, thinking disabled. All values in percent, higher is better, best per group in bold. Global-MMLU, Multi-IF, and SwitzerlandQA repeat from Table 23 for completeness. Multiple-choice benchmarks report accuracy, length-normalized for multilingual ARC.

	Fully open					Open-weight
Benchmark	Apertus 1.0	Apertus 1.5	OLMo 3.1	K2	EuroLLM	Llama 3.3	Gemma 4	Qwen3.5	Nemotron
Parameters	70B	70B	32B	70B	22B	70B	31B	27B	30B-A3B
Multilingual Knowledge
Global-MMLU	62.9	66.5	65.0	77.7	58.2	77.7	84.7	83.0	69.6
INCLUDE-44	58.3	61.6	51.6	66.6	55.3	71.7	78.9	77.1	56.7
INCLUDE-45	40.0	41.8	37.9	45.3	37.8	45.9	54.6	50.8	39.7

Table 30: Multilingual and cultural evaluation of the 70B-class post-trained models, thinking disabled. All values in percent, higher is better, best per group in bold. Multiple-choice benchmarks report accuracy, length-normalized for multilingual ARC.

Table 31: Automatic evaluation of the SwissGov translation tasks with COMETKiwi22.

Table 32: Automatic evaluation on the news domain of the WMT24, WMT24++, and WMT25 translation tasks with the ChrF metric.

Table 33: Human evaluation results for the translation tasks on a 0% to 100% cESA scale.

	Fully open					Open-weight
	Apertus 1.5			Molmo2	EuroVLM	Qwen3- VL	Qwen2.5- VL	Gemma 4	Pixtral	Emu3
Benchmark	8B	8Bth	70B	8B	9B	8B	7B	12B	12B	8B
General VQA and Perception
GQA	59.2	52.8	58.4	56.6	47.9	61.6	60.9	38.0	58.8	60.3
MMStar	45.2	63.1	47.0	64.5	45.5	63.9	62.4	67.9	55.8	44.4
RealWorldQA	60.0	63.5	64.3	71.1	49.9	68.8	68.9	64.7	63.4	54.0

Table 34: Image understanding of the released checkpoints against open vision-language models, on the 33 image benchmarks of the released model card plus MM-Safety. 8Bth is the 8B model with thinking enabled. All values in percent, higher is better, best per group in bold. Dashes mark cells with no valid run. * For MM-Safety we report the safety rate instead of the attack rate. † EuroVLM answers AI2D in prose rather than the requested option letter in 74.6% of responses, whereas every other model complies in essentially all of them; the harness credits only bare-letter replies, so this cell largely measures instruction following. A prose-tolerant extractor scores the same generations at 64.3.

Benchmark	Apertus 1.5 8B	Apertus 1.5 70B	Qwen2.5- Omni	Qwen2- Audio	Kimi- Audio
Parameters	8B	70B	7B	7B	7B
English Speech Recognition (WER, lower is better)
LibriSpeech test-clean	6.6	6.2	2.3	4.2	1.3
LibriSpeech test-other	19.9	19.3	4.4	8.2	2.5
VoxPopuli	11.2	10.3	11.9	14.0	8.2
SPGISpeech	18.4	12.6	5.5	6.0	3.5

Table 35: Audio evaluation of the released Apertus 8B and 70B checkpoints against open audio models. Word error rates (WER) are in percent, lower is better; accuracies are in percent and BLEU on a 0 to 100 scale, both higher is better. Best per row in bold. All models decode greedily, Qwen2-Audio with the repetition penalty ( Keskar et al., 2019 ) of 1.1 from its generation configuration, which its authors’ evaluation code ( QwenLM/Qwen2-Audio ) also applies, and use each task’s output cap; where a task sets none, all models share a cap of 4,096 tokens, which TED-LIUM long form also receives because its own cap of 256 tokens would cut off transcripts of up to 4,164 words; on that task, Qwen2.5-Omni uses PyTorch SDPA attention, whose fused kernels do not store the full attention matrix ( Dao et al., 2022 ; Rabe & Staats, 2021 ) , because with the default eager attention its longest talks exceed GPU memory.

	RULER						HELMET
Model	8k	16k	32k	64k	128k	256k	8k	16k	32k	64k	128k
Base models
Fully open
Apertus 1.0 8B	82.0	75.7	71.2	55.5	–	–	49.2	47.2	44.5	37.3	–
Apertus 1.0 70B	79.8	78.1	71.6	57.3	–	–	48.5	47.2	44.4	37.7	–
Apertus 1.5 8B	85.9	81.3	76.1	67.5	61.5	54.8	45.1	42.1	40.7	37.6	36.5

Table 36: Long-context evaluation on RULER and HELMET, in percent, higher is better. RULER cells average 13 subtasks; HELMET cells average seven task-group scores, HELMET’s own aggregation. Three HELMET settings use ROUGE in place of the reference LLM judge. Base and post-trained models are shown separately, fully open models above open-weight ones; bold marks the best value per column within a panel. “–” marks a length beyond the model’s position ceiling.

	Fully open						Open-weight
	Apertus
Category	1.0	1.0	1.5	1.5	OLMo 3.1	EuroLLM	Qwen3.5	Llama 3.1	Llama 3.3
Parameters	8B	70B	8B	70B	32B	9B	9B	8B	70B
Overall	56.4	58.0	61.2	70.1	73.8	31.1	69.1	64.5	71.6
Single Calls
Simple	90.3	91.8	78.8	93.3	85.3	79.8	87.0	94.3	94.3

Table 37: BFCL v3 function calling by call type, over the single-turn categories alone. All values in percent, higher is better, best per group in bold. The irrelevance categories reward declining to call a tool when none applies.

	Apertus
Benchmark	1.0 8B	1.5 8B	1.0 70B	1.5 70B	OLMo 3	Qwen3.5	Llama 3.3
Parameters	8B	8B	70B	70B	7B	9B	70B
Higher Is Better
BBQ	65.2	67.6	68.6	80.5	73.2	75.3	72.0
ToxiGen	70.4	81.8	62.5	86.8	83.6	74.8	87.3
Lower Is Better

Table 38: Safety evaluation of the released checkpoints against their predecessors and open-weight anchors. All values in percent; WMDP reports accuracy on hazardous-knowledge questions without a preferred direction, and is excluded from bolding.

	8B		70B
Metric	Apertus 1.0	Apertus 1.5	Apertus 1.0	Apertus 1.5
PreciseWiki (short-form factoid questions)
Hallucination $\downarrow$	71.9	71.5	54.0	59.6
Correct $\uparrow$	18.8	15.2	27.4	23.3
Refusal	9.4	13.3	18.7	17.1
Nonexistent-Entity Abstention

Table 39: HalluLens hallucination suite of the released checkpoints. All values in percent. The PreciseWiki rates partition all questions, so hallucination, correct, and refusal sum to roughly 100; the abstention rows reward declining to answer questions about nonexistent entities. The best value per row and model size is in bold on rows with a direction. The judge is a served Qwen3.5-27B, whereas the original benchmark uses a Llama-3.1-405B judge, so scores are not comparable with the official leaderboard.

Phase	Nodes	Tokens	Tokens/s/GPU	TFLOPs/GPU	Segments	Wall (h)
8B stage 1	256	0.81T	5,474	300	11	44
8B stage 2	256	2.80T	4,899	269	19	155
8B stage 3	256	0.40T	5,438	298	5	21
8B extension 32k	256	0.06T	3,510	260	1	6
8B extension 65k	256	0.06T	2,671	267	1	7
8B extension 131k	256	0.06T	1,449	220	2	12

Table 40: Measured performance of the production training phases, aggregated from the logged wandb run segments. Tokens are per phase, from the differences of the cumulative counter; throughput and TFLOPs per GPU are means over the longest segment of each phase; wall time sums the process time of all segments of the phase.

			Throughput (tokens/s)
Provider	Latency (s)	TTFT (ms)	Input	Output
P0 (ours)	1.90	103.6	179.7	67.9
P3	2.09	87.9	157.6	61.5
P5	2.18	171.2	150.9	58.9
P1	2.31	469.8	143.6	56.4
P4	3.63	81.6	92.8	35.3

Table 41: Serving performance of Apertus 1.5 70B on a simulated workload, by anonymized inference provider. Latency in seconds and TTFT (time to first token) in milliseconds, lower is better; throughput in tokens per second, higher is better.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Samples	Tokens	Dataset	Samples	Tokens
BigDocs-7.5M	2.2M	6.1B	MedTrinity-25M	4.4M	2.0B
BigEarthNet	24.0K	6.0M	MINT-1T ArXiv	4.6M	14.4B
BUSI	732	934.7K	MINT-1T HTML	125.0M	228.0B
COVID-US	9.7K	11.1M	MINT-1T PDF	58.8M	93.4B
DDTI	347	278.6K	MAmmoTH-VL	1.5M	3.1B
DFC2020	5.1K	1.5M	Open UFI	700	3.7M

Appendix

Table B.1: Vision pool of Stage 1.

Dataset	Samples	Tokens	Dataset	Samples	Tokens
Ultrasound Liver Images	735	1.8M	MedTrinity-25M	6.9M	5.1B
Art Museums PD-440K	10.6M	26.7B	Megalith-10M	8.9M	26.4B
BLIP3 Grounding 50M	33.7M	47.5B	MIT-10M	11.8M	30.5B
BLIP3o Long Caption	29.3M	36.9B	Molmo2-MultiImageQA	564.3K	2.6B
Brain Tumor MRI	7.2K	6.7M	MultiCare Case Images	96.7K	106.8M
Breast Ultrasound (BUSI)	773	1.3M	MultiCare Images	161.6K	132.9M

Appendix

Table B.2: Vision pool of Stage 2.

Dataset	Samples	Tokens	Dataset	Samples	Tokens
Art Museums PD-440K	10.6M	26.7B	MINT-1T ArXiv	4.4M	16.7B
ChartNet	4.3M	20.1B	NASA Images	23.0K	152.1M
Crello	23.1K	61.0M	OWID Articles	102	533.8K
DailyMed SPL	77.4K	328.0M	OWID Grapher	3.5K	8.4M
Data Insights	398	3.1M	PixMo Cap	706.5K	3.4B
DOCCI	9.6K	72.3M	Smithsonian Open Access	41.7K	314.8M

Appendix

Table B.3: Vision pool of the cooldown.

Dataset	Samples	Tokens	Dataset	Samples	Tokens
Argimi-Ardian Finance 10-K	138.7K	37.4B	Molmo2-MultiImageQA	466.2K	7.4B
ChartNet	764.6K	6.6B	NASA Images	225	1.9M
Data Insights	18	148.4K	OWID Articles	301	7.1M
HQ-50K	100	829.0K	PIN-200M	938.9K	11.3B
MedPix-2	41	507.2K	ShizhenGPT pretrain (book)	6.5K	110.0M
MINT-1T ArXiv	23.4K	193.4M	ShizhenGPT pretrain (web)	113.9K	1.9B

Appendix

Table B.4: Vision pool of the long-context extension; Appendix F gives the mix each run trains on.

Dataset	Samples	Tokens	Dataset	Samples	Tokens
BigEarthNet v2	160.4K	191.8M	MolmoPoint-GUISyn	25.3K	177.8M
ChineseMeme (English)	29.9K	51.1M	MultiHiertt	3.8K	16.5M
CoSyn-400K	143.8K	863.7M	Nemotron-Image-Training-3	4.1M	14.7B
Innovator-VL Instruct	24.8M	65.2B	PathVQA	13.8K	22.2M
LLaVA-OV 1.5 Instruct	7.2M	19.0B	PixMo Ask Model Anything	41.8K	145.9M
LLaVA-OV 2 (OSD)	546.7K	1.2B	PixMo Cap QA	117.1K	557.3M

Appendix

Table B.5: Vision pool of SFT, after splitting and packing for the Apertus 1.5 8B production run.

Dataset	Samples	Tokens	Dataset	Samples	Tokens
BigEarthNet v2	34.7K	41.5M	MolmoPoint-GUISyn	5.5K	38.7M
ChineseMeme (English)	6.5K	11.1M	MultiHiertt	825	3.6M
CoSyn-400K	31.1K	186.5M	Nemotron-Image-Training-3	939.1K	8.8B
GMAI-VL (permissive)	653.7K	341.3M	PathVQA	3.0K	4.8M
Innovator-VL Instruct	5.4M	14.2B	PixMo Ask Model Anything	9.0K	32.0M
LLaVA-OV 1.5 Instruct	1.6M	4.1B	PixMo Cap QA	25.3K	120.3M

Appendix

Table B.6: Vision pool of long-context SFT; Table 8 gives the mix each run trains on.

Dataset	Samples	Tokens	Dataset	Samples	Tokens
AudioSet (bal./unbal.)	1.7M	700.3M	Suno	235.4K	1.4B
Common Voice 24	2.0M	427.0M	People’s Speech (unsup.)	15.5M	51.2B
Gemeinderat Zurich	72.1K	172.7M	VoxPopuli	31.9M	56.7B
MTG-Jamendo	19.5K	107.6M

Appendix

Table C.1: Audio pool of Stage 1.

Dataset	Samples	Tokens	Dataset	Samples	Tokens
AISHELL-1/3/4	226.7K	48.8M	MediaSpeech	10.0K	6.3M
Aozora Hurigana	2.6M	703.6M	MultiMed	34.1K	15.2M
CC-Podcasts	238.5K	437.4M	OmniLingual ASR	183.9K	373.9M
Common Voice 24	11.7M	2.6B	ParlaMent Parla	241.7K	115.8M
Coral V3	295.2K	80.6M	ParlaSpeech-RS	159.7K	117.7M
EuroSpeech	2.1M	2.1B	People’s Speech (sup.)	2.1M	1.4B

Appendix

Table C.2: Audio pool of Stage 2 and the cooldown.

Dataset	Samples	Tokens	Dataset	Samples	Tokens
AISHELL-1/3/4	39	405.6K	ParlaSpeech-RS	286	2.9M
EuroSpeech	16.6K	206.2M	People’s Speech (sup.)	66.8K	1.8B
Granary (YODAS/YTC)	266.2K	3.3B	SPC-R	1.2K	16.0M
LegCo Speech	53.8K	1.7B	VoxPopuli	509.7K	30.2B
LibriHeavy	146.4K	4.1B	WenetSpeech	42.6K	841.1M
MLS	3.5K	556.9M	Zoengjyutgaai	838	16.3M

Appendix

Table C.3: Audio pool of the long-context extension.

Table 65

Figure D.1: Graphic axis of the taxonomy, subdivided into five coarse groups and then into 26 fine-grained groups.

Figure D.2: Photograph axis of the taxonomy, subdivided into seven coarse groups and then into 35 fine-grained groups.

Figure D.3: Breakdown of the main image datasets into Graphic and Photograph categories. We observe that most datasets are roughly evenly split between graphics and photographs. The main outlier is the MINT-1T (arXiv) dataset, which consists overwhelmingly of graphics.

Dataset	Graphic Coarse	Photograph Coarse	Graphic Fine	Photograph Fine
LLaVA-OneVision-1.5	10.38	12.87	4.29	3.15
BLIP3-Grounding-50M	9.25	19.83	4.10	4.24
Recap-DataComp-1B	9.68	18.97	3.97	4.11
MINT-1T (arXiv)	37.25	10.43	15.87	4.15
MINT-1T (PDF)	11.16	11.24	3.05	3.25
MINT-1T (HTML)	10.22	12.70	3.49	3.26

Appendix

Table D.1: Standard deviation (percentage points) of renormalized category shares across categories, per dataset, for each grouping. We observe significant variation across the different domain and type classes.

Figure D.6: Breakdown of the main image datasets by fine graphic type. We observe that the Brand Logo, Book or Magazine Cover, Scientific Plot, Scientific / Technical Imagery, and Quote or Text Overlay Composite types are overrepresented in the image distribution. By contrast, the Data Table, Geographic or Spatial Map, and Flowchart or Process Diagram classes are relatively underrepresented.

Figure D.7: Breakdown of the main image datasets by fine photograph type.

Figure D.8: Image samples from several prominent fine-grained classes. We observe that our embedding and classifier pipeline correctly captures the diverse classes in our taxonomy.

Figure D.9: Prevalence of broad overlapping task signals.

	Tokens $\downarrow$		PSNR (dB) $\uparrow$		SSIM $\uparrow$		LPIPS $\downarrow$
Category	Emu3	Emu3.5	Emu3	Emu3.5	Emu3	Emu3.5	Emu3	Emu3.5
General visual	8,748	2,184	26.55	24.00	0.784	0.691	0.081	0.115
Charts & diagrams	8,020	2,001	27.99	24.36	0.898	0.865	0.043	0.064
Illustration & synthetic	8,572	2,135	27.17	24.63	0.819	0.760	0.062	0.091
Documents & OCR	8,251	2,055	26.38	23.22	0.853	0.795	0.054	0.079
Medical	1,662	414	26.05	22.61	0.761	0.669	0.079	0.112

Appendix

Table E.1: Reconstruction quality and token cost of the Emu3 and Emu3.5 tokenizers on 350 Stage 1 training images, 50 per source category, grouped into six representative categories of Section 2.2.3 and preprocessed as in training. Tokens are the mean per image, arrows give the better direction, and bold marks the better tokenizer per row and metric.

Dataset	32k	65k	131k	262k
Cooldown
Cooldown Stage 3 Subsample	39,220,000,000	36,640,000,000	36,770,000,000	36,850,000,000
English
Biomed-Enriched	509,700,000	178,400,000	0	49,900,000
Dolma 3 OCR	2,780,000,000	2,480,000,000	211,500,000	0
FinePDFs-Edu (EN)	2,780,000,000	2,480,000,000	2,990,000,000	2,420,000,000

Appendix

Table F.1: Long-context data mixture (Apertus 1.5 8B). Per-dataset token counts for each long-context training step; columns denote the trained context length in tokens (32,768, 65,536, 131,072, and 262,144, respectively). We draw the Cooldown row and the short-context top-up share of Image tokens from documents under 16,384 tokens even though they count toward that step’s mixture. We sort rows within a class alphabetically.

Dataset	32k	65k	131k	262k
Cooldown
Cooldown Stage 3 Subsample	37,460,000,000	37,530,000,000	32,270,000,000	32,350,000,000
English
Biomed-Enriched	487,100,000	182,600,000	0	43,700,000
Dolma 3 OCR	2,660,000,000	2,540,000,000	185,600,000	0
FinePDFs-Edu (EN)	2,660,000,000	2,540,000,000	2,620,000,000	2,130,000,000

Appendix

Table F.2: Long-context data mixture (Apertus 1.5 70B). Per-dataset token counts for each long-context training step; columns denote the trained context length in tokens (32,768, 65,536, 131,072, and 262,144, respectively). We draw the Cooldown row and the short-context top-up share of Image tokens from documents under 16,384 tokens even though they count toward that step’s mixture. We sort rows within a class alphabetically.

	8B	70B
Nodes	256	512
Sequence length	8,192	8,192
Global batch size	1,024	2,048
Parallelism	TP2	TP4, PP8, VPP2
Peak learning rate	$1.1\times 10^{-4}$	$2\times 10^{-5}$
Minimum learning rate	$1.1\times 10^{-5}$	$2\times 10^{-6}$

Appendix

Table G.1: Continued-pretraining configuration of both models. The learning rate follows the warmup-stable-decay schedule of Section 3.2 : constant at the peak through stages 1 and 2, decaying to the minimum over stage 3 with the

1-\sqrt{x}

shape.

	Offline DPO (70B)	Online DPO (8B)	Online DPO (70B)
$\beta$	25 (length-normalized)	0.1	0.1
Learning rate	$1\times 10^{-6}$	$5\times 10^{-6}$	$2.5\times 10^{-6}$
Schedule	linear, warmup 0.1	linear, warmup 0.1	linear, warmup 0.1
Weight decay	0.0	0.01	0.01
Batch size	128 pairs	256 prompts	512 prompts
Rollouts per prompt		12	8

Appendix

Table G.2: Preference-alignment configurations. Offline DPO trains Apertus 1.5 70B only; online DPO trains both models. Online sequence lengths are given as prompt plus response.

	Q1: MM vs Text			Q2	Q3
Benchmark	No-CoT, Off	CoT, Off	CoT, On	CoT On vs No-CoT Off	CoT vs No-CoT, both Off
Mathematics
GSM8K	+0.5	+3.2	+2.2	+6.6	+0.8
MGSM *	$-$ 35.5	+0.7	+2.9	+50.4	+36.9
MATH	$-$ 2.6	+0.4	+0.5	+16.2	+3.8
Minerva MATH	$-$ 0.3	+0.4	+2.7	+18.3	+2.7

Appendix

Table H.1: Per-benchmark score changes behind Figure 12 , in percentage points. Q1 compares MM against Text at matched supervision and response mode; Q2 compares MM CoT with thinking on against MM No-CoT with thinking off; Q3 compares MM CoT against MM No-CoT, both with thinking off. Thinking is forced and sampled at temperature 0.6 and top-p 0.95; without thinking, decoding is greedy. Multimodal values average three seeds; the text-only reference is one seed. The average rows reproduce the bars of Figure 12 , and – marks comparisons the figure does not make. * MGSM is left out of the text average, as in the figure: its zero-shot prompt asks for a step-by-step answer, and the multimodal No-CoT arms mostly reply with the final number alone (median four words, against 137 for the text-only No-CoT arm), so their score reflects answer style and ranges from 27.2 to 39.2 across seeds. With reasoning traces in SFT, the multimodal arm writes out its work again and scores 69.9 even with thinking forced off, level with the text-only CoT arm (69.2).

Dataset	Original	Removed	Final	% Removed
OLMo2 OASST1	7,125	97	7,028	1.36%
Llama-Nemotron Chat	20,853	264	20,589	1.27%
Llama-Nemotron Safety	13,797	31	13,766	0.22%
EuroBlocks Aya	2,117	0	2,117	0.00%
EuroBlocks Refusal	7,648	0	7,648	0.00%
EuroBlocks Low-Resource	7,702	0	7,702	0.00%

Appendix

Table H.2: Decontamination statistics for the SFT data sources. For each dataset we report the example count before decontamination ( Original ), the number of examples removed by matching against evaluation benchmarks ( Removed ), the resulting count ( Final ), and the fraction removed. We compute removals per source prior to mixing.

Source	Category	Pairs	%
Dolci Instruct Precise IF	Precise IF	30,320	12.33
WildChat	Chat	28,203	11.47
OpenThoughts3 Science	Science	25,633	10.42
Dolci Instruct Python Algorithms	Code	23,616	9.60
UltraFeedback	General	21,226	8.63
FLAN	Other	14,845	6.04

Appendix

Table I.1: Composition of the preference prompt set by source dataset, after decontamination and length filtering. Categories follow the Dolci-Instruct-DPO grouping.

Model	Params (B)	Active (B)	Score	Chosen (%)	Rejected (%)
Qwen2.5-0.5B-Instruct	0.5	–	2.186	0.20	7.17
Qwen3-0.6B	0.6	–	2.624	0.33	3.33
Qwen2.5-1.5B-Instruct	1.5	–	2.942	0.41	1.42
Qwen3-1.7B	1.7	–	3.390	1.15	0.84
EuroLLM-1.7B-Instruct	1.7	–	1.598	0.07	37.51
Ministral-3-3B-Instruct-2512	3	–	2.541	0.97	16.82

Appendix

Table I.2: The 24 response models, ordered by parameter count. We give active parameters for the mixture-of-experts models. Score is the four-aspect judge mean over all prompts; the last two columns give how often each model supplies the chosen and the rejected side of a pair.

	Mean	Std	p5	Median	p95
Chosen	4.897	0.354	4.345	4.998	5.000
Rejected	1.132	0.385	1.000	1.003	1.738
Margin	3.766	0.503	2.804	3.974	4.000

Appendix

Table I.3: Summary of the four-aspect judge scores over the 245,898 preference pairs: the chosen score, the rejected score, and their margin.

Score Band	Chosen	%	Rejected	%
[1, 2)	527	0.21	238,016	96.73
[2, 3)	1,872	0.76	5,264	2.14
[3, 4)	5,425	2.21	1,586	0.64
[4, 4.5)	7,414	3.02	544	0.22
[4.5, 4.9)	17,868	7.27	297	0.12
[4.9, 5.0]	212,792	86.54	191	0.08

Appendix

Table I.4: Preference pairs by score band.

	Standard DPO										Baselines
$\beta$	0.05	0.1	0.1	0.01	0.1	0.05	0.1	0.05	0.05	0.05
Learning Rate ( $10^{-7}$ )	5	5	10	5	5	1	5	5	5	10
Batch Size	512	512	512	512	256	128	128	256	128	512	SFT	Instruct
Knowledge and Reasoning
MMLU	62.5	61.8	61.9	61.5	62.3	61.0	62.4	62.2	62.6	61.9	58.9	61.7
TruthfulQA	59.8	59.2	61.2	60.2	60.4	58.6	61.9	61.3	64.7	62.4	54.5	56.7

Appendix

Table J.1: Offline standard DPO from Apertus 1.0 8B SFT ( Hernández-Cano et al., 2026 ) : the ten best configurations of a 60-run grid over learning rate

\in\{10^{-6},5{\cdot}10^{-7},10^{-7},5{\cdot}10^{-8}\}

\beta\in\{0.01,0.05,0.1,0.5,1.0\}

, and batch size

\in\{128,256,512\}

, sorted by average; the learning-rate row is in units of

10^{-7}

. Scores are percentages (higher is better). The two rightmost columns are Apertus 1.0 8B SFT, from which all runs start, and Apertus 1.0 8B Instruct.

\Delta

is the difference in average from the SFT column, and – marks the SFT column itself. Bold marks the best DPO value per row.

	Length-Normalized DPO										Baselines
$\beta$	25.0	25.0	5.0	25.0	2.5	5.0	0.5	0.5	25.0	2.5
Learning Rate ( $10^{-7}$ )	10.0	5.0	1.0	1.0	1.0	5.0	1.0	0.1	0.1	0.1	SFT	Instruct
Knowledge and Reasoning
MMLU	62.0	62.4	62.2	61.5	63.3	61.9	60.0	58.1	59.1	59.3	58.9	61.7
TruthfulQA	61.1	60.1	59.2	57.8	59.5	63.7	58.0	54.7	54.5	54.7	54.5	56.7
HellaSwag	76.4	75.5	69.0	72.7	64.9	78.8	70.9	72.2	72.2	72.2	72.3	73.8

Appendix

Table J.2: Offline length-normalized DPO from Apertus 1.0 8B SFT with batch size 128: the ten best configurations of the grid

\beta\in\{0.5,2.5,5,25\}

\times

learning rate

\in\{10^{-6},5{\cdot}10^{-7},10^{-7},10^{-8}\}

, sorted by average; the learning-rate row is in units of

10^{-7}

. Scores are percentages (higher is better). The two rightmost columns are Apertus 1.0 8B SFT and Instruct;

\Delta

and – are as in Table J.1 . Bold marks the best DPO value per row.

	Length-Normalized DPO						Baselines
$\beta$	50.0	50.0	100.0	50.0	100.0	100.0
Learning Rate ( $10^{-7}$ )	10	5	10	1	5	1	SFT	Instruct
Knowledge and Reasoning
MMLU	61.2	60.7	60.7	61.2	62.4	59.9	58.9	61.7
TruthfulQA	59.0	58.6	57.8	56.9	57.2	56.1	54.5	56.7
HellaSwag	75.2	74.7	74.3	72.7	74.0	72.7	72.3	73.8

Appendix

Table J.3: Extension of the length-normalized DPO grid of Table J.2 to

\beta\in\{50,100\}

, from Apertus 1.0 8B SFT with batch size 128, sorted by average; the learning-rate row is in units of

10^{-7}

. Scores,

\Delta

, –, and the baseline columns are as in Table J.2 . Bold marks the best DPO value per row.

	Online DPO			Offline $\to$ Online
Learning Rate ( $10^{-7}$ )	50	10	100	50	10	5	Offline	SFT
Knowledge and Reasoning
MMLU	62.8	62.1	62.8	63.8	61.7	62.4	61.3	59.3
TruthfulQA	62.2	55.2	65.3	64.1	57.2	57.4	57.9	53.4
HellaSwag	70.8	68.0	72.4	71.9	69.3	70.8	71.6	70.8
BBH	54.2	55.9	51.1	52.5	55.8	55.2	55.1	54.6

Appendix

Table J.4: Learning-rate sweeps for online DPO and offline

\to

online DPO (

\beta{=}0.1

, 256 prompts per step) from an intermediate Apertus 1.5 8B SFT checkpoint, sorted by average within each group. Scores are percentages (higher is better). The learning-rate row is in units of

10^{-7}

. Offline is length-normalized offline DPO (

\beta{=}25

, learning rate

10^{-6}

) from the same checkpoint, which also initializes offline

\to

online; the rightmost column is the SFT checkpoint, and – marks it in the

\Delta

row. In these runs, rollouts come from the initial instead of the current policy because of an implementation issue fixed afterward; on-policy training then uses the selected learning rate. Bold marks the best DPO value per row.

Benchmark	Mixture	Offline $\to$ Online	Online	Offline	SFT
Knowledge and Reasoning
MMLU	67.7	68.1	68.3	68.3	63.8
MMLU-Pro	45.3	43.8	36.4	36.7	27.7
TruthfulQA	60.9	59.5	59.1	55.0	49.5
CommonsenseQA	75.3	74.7	74.9	74.4	74.5
SQuADv2	23.3	25.9	27.2	13.6	16.5

Appendix

Table J.5: DPO variants from another intermediate Apertus 1.5 8B SFT checkpoint (rightmost column) with thinking disabled, sorted by development average; scores are percentages (higher is better). Offline uses length-normalized DPO (

\beta{=}25

, learning rate

10^{-6}

, batch size 128); the others use

\beta{=}0.1

, learning rate

5{\cdot}10^{-6}

, and 256 prompts per step; the mixture adds 256 offline pairs to each step, and offline

\to

online starts from the offline checkpoint. Held-out groups contain the held-out benchmarks and the others the development benchmarks; each average is over all benchmarks of its split. Bold marks the best DPO value per row.

	Online DPO (Learning Rate / $\beta$ )
Skill Group	$2{\cdot}10^{-6}$ / 0.25	$2{\cdot}10^{-6}$ / 0.35	$5{\cdot}10^{-6}$ / 0.1	SFT
Knowledge	58.0	57.0	60.3	54.2
Reasoning	54.4	54.6	53.0	50.1
Instruction Following	76.8	75.7	79.7	68.9
Chat	10.7	9.2	15.7	6.4
Math	28.7	29.3	20.7	27.2

Appendix

Table J.6: Online DPO from Apertus 1.0 70B SFT with the learning rate /

\beta

given in the headers, sorted by average;

5{\cdot}10^{-6}

/ 0.1 is the Apertus 1.5 8B setting, and the rightmost column is the SFT checkpoint. Scores in percent (higher is better) with thinking disabled; for degeneration, lower is better. Bold marks the best DPO value per skill-group and average row. Each skill-group row is the mean over the benchmarks within that group, and the overall average is the mean over all individual benchmarks.

Skill Group	Offline	Mixture	Online	RL
Knowledge	56.4	61.7	61.7	59.4
Reasoning	62.2	63.5	62.4	60.9
Instruction Following	84.1	88.6	88.5	88.5
Chat	39.5	24.4	24.2	19.5
Math	34.2	35.6	36.0	35.7
Coding	75.7	75.7	74.5	74.1

Appendix

Table J.7: Offline DPO, mixture, and online DPO from the Apertus 1.5 70B RL checkpoint, sorted by average; the rightmost column is the RL checkpoint. Offline DPO uses the configuration of Section 4.6 ; online DPO and the mixture use

\beta{=}0.1

and a learning rate of

5{\cdot}10^{-6}

, and the mixture adds 256 offline pairs to each online step. Scores in percent (higher is better) with thinking disabled; for degeneration, lower is better. Bold marks the best DPO value per skill-group and average row. Each skill-group row is the mean over the benchmarks within that group, and the overall average is the mean over all individual benchmarks.

	Helpfulness	Honesty	Instr. Following	Truthfulness
Helpfulness	–	0.710	0.748	0.644
Honesty	0.710	–	0.680	0.745
Instruction Following	0.748	0.680	–	0.641
Truthfulness	0.644	0.745	0.641	–
Average	0.701	0.712	0.690	0.677

Appendix

Table J.8: Pairwise Pearson correlation between the four aspects, pooled across all response models, on scores with prompt and model effects removed via equation 1 .

	Helpfulness	Honesty	Instruction Following	Truthfulness
JudgeBench
MMLU-Pro	73.4	70.1	71.4	71.4
LiveBench Reasoning	69.4	67.3	71.4	69.4
LiveBench Math	83.9	85.7	83.9	82.1
LiveCodeBench	83.3	78.6	76.2	81.0
Macro Average	77.5	75.4	75.7	76.0

Appendix

Table J.9: Accuracy with Qwen3-235B-A22B-Instruct-2507 and different aspects.

	Qwen3-235B- A22B-Instruct-2507	Qwen3.5- 35B-A3B	Qwen3.6-27B
JudgeBench
MMLU-Pro	73.4	75.3	80.5
LiveBench Reasoning	69.4	71.4	75.5
LiveBench Math	83.9	83.9	89.3
LiveCodeBench	83.3	85.7	90.5
Macro Average	77.5	79.1	84.0

Appendix

Table J.10: Accuracy of the Helpfulness judge with different LLMs on three benchmarks.

Benchmark	Metric	Type	CoT	#Shots	Chat	Turns	#Langs
Knowledge and Reasoning
MMLU (zero-shot)	Exact Match	MCQ (Gen)	✓	0	✓	$\times$	1
MMLU (few-shot)	Exact Match	MCQ (Gen)	✓	4	$\times$	$\times$	1
MMLU-Pro	Exact Match	MCQ (Gen)	✓	5	✓	✓	1
AGIEval	Accuracy	MCQ (Gen/LLH)	$\times$	0	✓	$\times$	2
ARC Challenge Chat	Exact Match	MCQ (Gen)	$\times$	0	✓	$\times$	1

Appendix

Table L.1: Benchmark configurations used in the Apertus evaluation suites. Accuracy † denotes prompt-level loose accuracy.

Benchmark	Metric	Type	CoT	#Shots	Chat	Turns	#Langs
Multilingual and Cultural
Global-MMLU	Exact Match	MCQ (Gen)	$\times$	0	✓	$\times$	15
MGSM (English CoT)	Exact Match	Gen	✓	0	✓	$\times$	11
TruthfulQA Multilingual (MC2)	Accuracy	MCQ (LLH)	$\times$	0	✓	$\times$	31
INCLUDE-44 (generative)	Exact Match	MCQ (Gen)	$\times$	0	✓	$\times$	44
INCLUDE-44 (base)	Accuracy	MCQ (LLH)	$\times$	0	$\times$	$\times$	44

Appendix

Table L.2: Benchmark configurations used in the Apertus evaluation suites (continued). Accuracy ‡ denotes prompt-level strict accuracy.

	Apertus 1.5 8B				Apertus 1.5 70B
Benchmark	SFT	+RL	+DPO	+SDPO	SFT	+RL	+DPO
Knowledge and Reasoning
MMLU	65.8	66.6	70.7	70.6	75.1	72.0	77.2
MMLU-Pro	40.1	43.0	49.4	49.2	46.3	44.4	56.9
TruthfulQA	48.1	50.8	58.0	56.1	58.2	58.1	67.0
CommonsenseQA	50.8	27.6	75.5	75.3	64.8	73.9	77.0

Appendix

Table L.3: Full post-training lineage of the Apertus 1.5 models on the development benchmark suite, all 37 benchmarks, thinking disabled, with the behavioral measures of Section 5.1 . The subset shown in the main text is Table 26 . Best value per row in bold, taken separately within each model. Over-refusal is the refusal rate on the hard subset of OR-Bench; a share of its prompts is legitimately refusable, so only the trend across stages is meaningful ( Section 5.9.2 ).

NIST AI RMF 600.1	MLCommons Hazards v1.0	Apertus 1.5 Pre-Release Evaluation
CBRNE Information and Capabilities	CBRNE	Evaluated : Manually evaluated for usefulness in principle, given offline use potential, and then for the risk increase compared to a shallow web search output, in reference to CBRNE scenarios in a multi-turn and iterative multi-conversation setting, including abliterated release candidates. Early access for evaluation provided to Spiez Lab of the Federal Office for Civil Protection.
Dangerous, Violent, or Hateful Content	Hate	Evaluated : using the standardized RealToxicityPrompts , ToxiGen , and PolyglotToxicityPrompts benchmarks.
Harmful Bias and Homogenization		Evaluated : using the standardized BBQ and ToxiGen benchmarks.
Obscene, Degrading, and/or Abusive Content	Child Sexual Exploitation	Evaluated : No image generation supported; main concern is Child Sexual Abuse enablement. We apply the LLM under-alignment benchmark of Wuhrmann et al. (2026) , derived from cases reported in the Swiss Supreme Court rulings, as part of the manual evaluation.
	Sex-Related Crimes	Partially evaluated : Same evaluation as above. While the dataset of Wuhrmann et al. (2026) covers general sex-related crimes, it has fewer test cases.
	Violent Crimes	Evaluated : Same evaluation as above. Additional overlap for armed violence with manual dual-use capabilities evaluation; partially probed by HarmBench .

Appendix

Table N.1: NIST AI RMF 600.1 and MLCommons Hazards Taxonomy v1.0 risks evaluated for Apertus 1.5. Bold items indicate benchmark names. Color coding: fully evaluated , partially evaluated , not evaluated , not applicable .

MLLM Reasoning

See all 27 papers

Mar 4, 2026

Phi-4-reasoning-vision-15B Technical Report

We present Phi-4-reasoning-vision-15B, a compact open-weight multimodal reasoning model, and share the motivations, design choices, experiments, and learnings that informed its development. Our goal is to contribute practical insight to the research community on building smaller, efficient multimodal reasoning models and to share the result of these learnings as an open-weight model that is good at common vision and language tasks and excels at scientific and mathematical reasoning and understanding user interfaces. Our contributions include demonstrating that careful architecture choices and rigorous data curation enable smaller, open-weight multimodal models to achieve competitive performance with significantly less training and inference-time compute and tokens. The most substantial improvements come from systematic filtering, error correction, and synthetic augmentation -- reinforcing that data quality remains the primary lever for model performance. Systematic ablations show that high-resolution, dynamic-resolution encoders yield consistent improvements, as accurate perception is a prerequisite for high-quality reasoning. Finally, a hybrid mix of reasoning and non-reasoning data with explicit mode tokens allows a single model to deliver fast direct answers for simpler tasks and chain-of-thought reasoning for complex problems.

Jyoti Aneja, Michael Harrison, Neel Joshi +3

Microsoft Research

VLM Reasoning MLLM Reasoning Efficient Multimodal Inference Scientific Reasoning in Language Models

Ask PDF

Oct 1, 2026

MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning

Open multimodal reasoning models have benefited from large-scale reasoning supervision, yet reliable post-training remains challenging due to uneven data quality, inefficient supervision construction, imbalanced difficulty, and cross-domain interference. We introduce MMVistaReason (MVR), an open-data post-training recipe with three components: (1) broader capability coverage across complementary Analytical and Real-World reasoning groups, emphasizing structured reasoning versus visual perception and spatial grounding; (2) efficient SFT and RL data construction, standardizing heterogeneous open data through staged cleaning and annotation, combining difficulty-aware cascaded teacher distillation with answer-likelihood-based trajectory selection to construct MVR-SFT-528K, and applying scale-specific frontier filtering for MVR-RL-63K; and (3) specialize-then-integrate training, which trains complementary RL experts and consolidates their capabilities through multi-teacher on-policy distillation (MOPD). Our analyses reveal a capacity-dependent interaction between supervision difficulty, trajectory quality, and model capacity: smaller students benefit more from selected supervision, while larger students are robust to trajectory variation and mixed-domain interference. Mixed-domain RL introduces benchmark-level negative transfer, whereas MOPD provides consistent capability integration, with the preferred KL direction varying across model scales. Across 15 multimodal benchmarks, MVR-4B achieves an average score of 72.8, outperforming Qwen3.5-9B (Instruct) and MMFineReason-8B while using about 70% fewer samples than MMFineReason. Scaling to 9B improves the average to 74.4, surpassing Qwen3.5-35B-A3B (Instruct). Overall, MMVistaReason demonstrates that systematic open-data construction and capacity-aware post-training provide a practical and scalable path toward reliable multimodal reasoning.

Juekai Lin, Honglin Lin, Yuqian Yuan +7

Zhejiang University · Shanghai Artificial Intelligence Laboratory, OpenDataLab · Shanghai Jiao Tong University +1

Reinforcement Learning Multimodal Reasoning Multimodal Instruction Tuning

Ask PDF

Oct 19, 2026

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory

Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.

Yu Zhou, Cheng-Fu Yang, Rui Sun +8

University of California, Los Angeles · Apple

Tool Use in VLMs Multimodal Reasoning VLM Reasoning MLLM Reasoning Memory-Augmented VLMs

Ask PDF

Abstract

MLLM Reasoning

Explore similar work

Phi-4-reasoning-vision-15B Technical Report

MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning

OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory