Frontier models, general-purpose and domain-specific alike, now include a midtraining stage between pretraining and post-training, but little has been published about how this stage is built. We present an open midtraining recipe that primes base models for agentic post-training, and we show its effectiveness on two Nemotron models of different sizes. Pretraining typically runs at short context and sees almost no agent interactions, while SFT data is mostly synthetic, and very long SFT stages can hurt the model's later RL trainability. Our midtraining stage sits between the two. It trains at 256K context for hundreds of billions of tokens on a diverse mixture of natural software-engineering data like packed repositories, pull requests, commit histories, and build logs together with a small share of agentic traces. On Nemotron 3.5 Lightning, 500B midtraining tokens improve SWE-bench Verified from 60.7 to 64.5, SWE-bench Multilingual from 55.8 to 62.8, and SWE-bench Pro from 32.0 to 35.5 after identical full SFT. On the larger Nemotron 3.5 Super, 280B midtraining tokens improve DeepSWE, a harder benchmark of long-horizon tasks, from 11.7 to 15.7 and SWE-bench Verified from 72.9 to 74.3 after identical full SFT. Midtraining also improves robustness across agent harnesses on both models. We report the ablations behind the recipe and an evaluation protocol built on short fixed SFT runs, since base-model proxy metrics did not correlate reliably with post-training results.
Figures & tables
Figure 1: The midtraining blend, by share of training tokens. Replay is drawn from the base model’s pretraining data. Token counts and epochs per source are in Table 1 .
Source
Unique tokens
Share of blend
Tokens seen (500B run)
Epochs
Issue–PR
216.9B
40.8%
205.2B
0.95
Repositories
104.1B
20.0%
100.7B
0.97
Agentic traces
39.4B
10.0%
50.3B
1.28
Diff data
16.0B
3.0%
15.1B
0.95
Build and CI logs
6.5B
1.2%
6.1B
0.95
Pretraining replay
∼ 2.0T
20.0%
100.7B
0.05
Table 1: Sources of the midtraining blend in Figure 1 . Replay is drawn from the base model’s final-phase pretraining blend. Epochs are tokens seen divided by unique tokens.
SWE-V
SWE-Multi
TB 2.0
pass@1 / pass@3
pass@1 / pass@3
pass@1 / pass@8
Short SFT (33B tokens)
Base-8K
57.1 / 69.2
48.6 / 65.0
22.1 / 40.5
Midtrained, no traces
56.1 / 68.2
48.4 / 65.3
19.8 / 40.5
Midtrained
57.9 / 71.4
56.8 / 70.0
20.1 / 39.3
Full SFT (100B tokens)
Table 2: Early experiment on the N-3.5 Lightning-8K checkpoint, midtrained for 100B tokens with and without agentic traces and compared with the base model after identical short or full SFT (%).
Midtraining
SWE-V
SWE-Multi
TB 2.0
SWE-Pro
tokens (B)
pass@1 / pass@3
pass@1 / pass@3
pass@1 / pass@8
pass@1 / pass@3
0 (base)
60.7 / 72.2
55.8 / 72.3
22.6 / 43.8
32.0 / 41.5
100
64.1 / 74.0
57.4 / 71.7
22.5 / 44.9
34.9 / 45.1
200
64.3 / 73.4
59.7 / 76.7
21.4 / 39.3
35.8 / 45.3
300
63.5 / 73.6
62.0 / 76.7
24.9 / 48.3
36.3 / 46.4
500
64.5 / 75.0
62.8 / 77.0
23.6 / 46.1
35.5 / 45.4
Table 3: Midtraining scaling on N-3.5 Lightning-LC after identical full SFT. After 500B tokens, midtraining improves both pass@1 and pass@ k on every benchmark, so it raises what the model can solve rather than only its reliability.
LoCoDiff
DeepSWE
Repository
RepoBench
Exact (%) ↑
Distance ↓
BPB ↓
BPB ↓
↑
N-3.5 Lightning-LC
0.0
0.903
0.171
0.238
70.09
Midtrained, 100B
21.5
0.228
0.144
0.228
71.22
Midtrained, 500B
26.8
0.164
0.140
0.217
72.41
Table 4: Base-model post-trainability metrics for N-3.5 Lightning-LC and its midtrained checkpoints from the scaling run. All metrics improve as midtraining scales.
Midtraining
SWE-V
SWE-Multi
SWE-Pro
DeepSWE
BrowseComp
Short SFT (33B tokens)
0 (base)
71.3
65.9
52.1
0 6.4
64.2
65B
71.9
65.6
51.7
10.4
65.0
280B
72.1
67.4
52.2
12.4
67.2
Full SFT (100B tokens)
0 (base)
72.9
69.0
53.9
11.7
61.8
Table 6: N-3.5 Super after 65B and 280B midtraining tokens and identical short or full SFT (pass@1, %). Midtraining gives a large gain on DeepSWE, the hardest benchmark, and also improves SWE-bench Verified and BrowseComp.
N-3.5 Super, short SFT
N-3.5 Lightning-8K, full SFT
SWE-bench Verified
Terminal-Bench 2.1
SWE-bench Verified
Harness
Base
Mid 65B
Δ
Base
Mid 65B
Δ
Base
Mid 100B [-1pt](no traces)
Mid 100B
Δ
Pi
57.4
67.8
+10.4
30.3
35.2
+4.9
45.7
48.8
59.9
+14.2
OpenCode
54.4
65.8
+11.4
35.2
38.2
+3.0
52.0
50.1
63.1
+11.1
Hermes
59.8
63.2
+3.4
44.2
53.9
+9.7
43.3
43.3
50.0
+6.7
Codex
48.4
61.8
+13.4
27.0
39.7
+12.7
42.1
45.9
61.1
+19.0
Table 7: Midtraining improves pass@1 on every agent harness we evaluated, for both models, with the largest gain on the harness where the base model is weakest (%).
8K
16K
32K
64K
128K
256K
8K → long-context @1M
94.1
91.9
90.5
85.9
81.3
76.6
8K → midtraining @256K
92.3
91.4
87.8
81.9
76.1
62.2
8K → midtraining @256K → long-context @1M
92.1
91.0
89.8
86.6
83.8
79.7
Table 8: N-3.5 Lightning RULER scores up to 256K. Midtraining alone brings N-3.5 Lightning-8K close to N-3.5 Lightning-LC up to 128K, acting like a long-context stage.
SWE-V
pass@1
pass@3
Unmasked
27.5
41.6
Masked
37.8
53.0
Table 9: Tool-response masking. N-3.5 Lightning-8K midtrained for 100B tokens with and without tool responses in the loss, followed by identical SFT with an earlier version of our SFT recipe (SWE-bench Verified).
Table 10
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Document lengths in tokens for each source in the midtraining blend (log scale).
Language
Records
Tokens (B)
Natural share (%)
Blend weight (%)
Python
1,798,331
24.51
23.5
31.0
Java
945,513
18.71
18.0
17.0
TypeScript
1,273,349
13.20
12.7
12.0
C#
460,597
11.66
11.2
10.0
JavaScript
1,289,843
10.01
9.6
8.6
C++
400,540
8.27
7.9
7.6
Appendix
Table 12: Repository data by language. Tokens are measured with the model tokenizer.
Type
Examples
Document tokens
Trained tokens
Trained share
Multi-file edits
733,705
13.3B
6.8B
51%
Code evolution
377,618
3.5B
3.2B
91%
Total
1,111,323
16.8B
10.0B
60%
Appendix
Table 13: Diff data used in our experiments. Trained tokens exclude masked context. Counts use the tokenizer of our data pipeline. With the model tokenizer, the total is 16.0B (Table 1 ).
Scaffold label
Trajectories
Share (%)
codex
166,624
19.39
openhands
120,086
13.98
generic_code_cli
107,291
12.49
gemini_cli
104,717
12.19
terminus
99,915
11.63
taubench_react
50,399
5.87
Appendix
Table 14: Trajectories by scaffold label in the early snapshot. Shares are of trajectories, not tokens.
Tool name
Calls
Share (%)
bash
3,659,199
19.6
execute_bash
3,633,328
19.4
str_replace_editor
2,913,745
15.6
keystrokes
2,029,095
10.9
exec_command
736,655
3.9
write
524,113
2.8
Appendix
Table 15: The ten most frequent tool names in the early snapshot, as each scaffold records them, so similar tools such as bash and execute_bash are listed separately. Shares are of all tool calls.
Assistant turns
Trajectories
Share (%)
1
146,334
17.03
2
56,929
6.63
3
47,362
5.51
4–5
77,700
9.04
6–10
138,036
16.07
11–20
111,777
13.01
Appendix
Table 16: Trajectory length in assistant turns, in the early snapshot.
N-3.5 Lightning midtraining
N-3.5 Super midtraining
SFT (both models)
Parameters (total / active)
30B / 3B
120B / 12B
–
Sequence length (packed)
256K
256K
256K
Global batch size (sequences)
64
64
64
Peak learning rate
5×10−5
1×10−5
2×10−5
Warmup (sequences)
32K
30K
–
Schedule after warmup
constant
constant
cosine to 5×10−6
Appendix
Table 17: Training settings. TP, CP, and EP are tensor, context, and expert parallelism, and MTP is multi-token prediction.
SWE-V (pass@1 / pass@3)
SFT
RL step 10
Step 15
Step 20
Base-8K
61.2 / 71.8
60.4 / 70.8
61.4 / 72.2
60.4 / 71.7
Midtrained-100B
63.2 / 74.1
63.0 / 74.0
63.5 / 73.7
63.0 / 73.0
TB 2.1 (pass@1 / pass@8)
SFT
RL step 10
Step 15
Step 20
Base-8K
23.0 / 46.1
24.7 / 44.9
22.9 / 41.6
20.8 / 39.3
Midtrained-100B
24.2 / 44.9
23.5 / 44.9
24.4 / 44.9
23.2 / 46.1
Appendix
Table 18: RLVR from N-3.5 Lightning-8K and its midtrained checkpoint after full SFT. SFT-start scores are re-evaluated with the RL checkpoints. The midtrained model leads on SWE-bench, and its Terminal-Bench pass@8 holds.
Benchmark
Rollouts
N-3.5 Lightning-LC
Midtrained, 500B
SWE-bench Verified
1,500
40
26
SWE-bench Multilingual
900
25
14
SWE-bench Pro
2,193
91
13
Terminal-Bench
712
8
1
Appendix
Table 19: Rollouts that fail by filling the 256K context window, after identical full SFT.
Mid-training is increasingly recognized as a critical stage for shaping the capabilities of large language models. Recent work has shown that targeted mid-training can strengthen reasoning-intensive abilities such as math and science, and can also improve agentic capabilities in software-engineering settings. In this work, we study the parallel but less explored agentic capability: general tool use. We present MidTool, an open corpus construction pipeline for agentic tool-use mid-training that combines large-scale web, PDF, and code data with synthesized supervision from real-world tool APIs, MCP skills, and document-grounded workflows. MidTool is designed to teach models how to recognize tool affordances, ground arguments from context, compose tool call workflow, and recover from incomplete information. We mid-train Qwen3-4B-Base and Qwen3-8B-Base on MidTool-Mix, and then apply follow-up post-training with both supervised fine-tuning and reinforcement learning. Compared with baselines, MidTool-Mix consistently improves downstream performance under both SFT and RL on BFCL, tau2-Bench, and MCP Universe. These results suggest that general tool use, like other important LLM capabilities, benefits from dedicated mid-training rather than being left entirely to post-training.
Fengqing Jiang, Yite Wang, Boyi Liu +5
University of Washington · Snowflake · University of North Carolina at Chapel Hill
Agentic (tool-using) language models are mainly trained on tool-call traces and agent trajectories during post-training. These data provide direct behavioral supervision, but producing them requires task environments, execution, and verification, making broad tool and task coverage expensive. Publicly available skills offer another source of training data: they encode reusable tool semantics and workflows but are typically used only as inference-time context. We introduce Skill Pre-Training (SPT), a mid-training method that applies causal language modeling to SkillCorpus, a collection of public multi-file skill packages, optionally mixed with general data. To preserve relations among files within each package, we also introduce Reference Insert, a reference-aware assembly strategy that places supporting files near their mentions in the primary instruction. Experiments across multiple model scales and post-training recipes show that SPT consistently improves agentic performance over mid-training on general or trajectory data, while largely preserving general performance. Data mixture experiments show additional benefits from combining skill data with general annealing corpora. These results indicate that skill packages are a valuable data source for pre-training agentic language models.
Yufei Sun, Yudong Li, Yiming Cheng
Beijing University of Posts and Telecommunications · Tsinghua University
Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during pretraining and how execution patterns differ after post-training. Our results suggest that end-to-end agent evaluation can complement conventional pretraining benchmarks by providing a practical diagnostic of interactive model behavior.