Self-Improving Training & Thinking
Organizations: FAIR at Meta
Abstract
Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model's capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite data and to judge policy model rollouts, thus introducing thinking and reinforcement earlier in training. We further improve reasoning at all stages by introducing the concept of self-improving post-thinking: improving the model's reasoning and self-judging ability by reasoning after generation. In our experiments, we show these approaches can give strong gains across multiple axes.
Figures & tables
| Pretraining for Quality | Generation Quality Std. Prefix Unsafe Prefix | Standard Evals (Avg) | Coherence Eval | |
| Llama Base | 50.0 | 47.6 | 50.1 | |
| Trained on SlimPajama | ||||
| Llama Pretrain Baseline | 49.0 | 46.8 | 49.4 | |
| Self-Improving Pretraining | 86.3 | 50.8 | 87.9 | |
| Pretraining for Factuality | Generation Quality Std. Prefix Unsafe Prefix | Standard Evals (Avg) | Factuality Evals (Avg) | |
| Pretraining for Safety (from scratch) | Generation Quality Std. Prefix Unsafe Prefix | Safety Evals (Avg) | |
| Pretrain Baseline | 1.3 | 2.4 | 85.2 |
| Pretrain on Rewrites | 1.6 | 2.4 | 96.7 |
| Self-Improving Pretraining: RF-NLL (suffix vs. rewrite) | 5.3 | 25.8 | 96.4 |
| Self-Improving Pretraining: RF-NLL (rollout vs. rewrite) | 32.4 | 12.1 | 97.5 |
| BoolQA | PIQA | Hellaswag | ARC-e | ARC-c | OBQA | SIQA | MMLU | |
| Llama Base | 64.6 | 74.8 | 47.9 | 66.6 | 32.3 | 27.2 | 41.0 | 26.4 |
| Trained on SlimPajama | ||||||||
| Llama Pretrain Baseline | 59.6 | 74.2 | 47.7 | 65.3 | 31.3 | 27.0 | 42.2 | 26.7 |
| Pretraining for Quality | ||||||||
| Self-Improving Pretraining | 69.1 | 75.8 | 51.7 | 69.4 | 35.7 | 30.0 | 46.1 | 28.3 |
| Pretraining for Factuality |
| Pretraining for Factuality | SlimPajama | FActScore | HaluEval | HaluEval | HaluEval | Truthful QA | TruthfulQA |
| (pointwise) | (pairwise) | dialogue | QA | summarization | MC1 | MC2 | |
| Llama Base | 36.6 | 50.0 | 50.0 | 50.1 | 50.0 | 22.4 | 35.9 |
| Trained on SlimPajama | |||||||
| Llama Pretrain Baseline | 35.4 | 48.9 | 50.8 | 51.4 | 61.5 | 21.5 | 35.5 |
| Self-Improving Pretraining | 63.5 | 69.3 | 54.6 | 58.5 | 84.7 | 27.7 | 42.5 |
| Pretraining for Safety | RealToxicityPrompts | RedPajama test | XStest safe | XStest unsafe | Toxigen |
| Llama Base | 88.1 | 68.0 | 85.2 | 39.5 | 80.1 |
| Trained on RedPajama | |||||
| Llama Pretrain Baseline | 87.1 | 67.4 | 87.6 | 35.0 | 82.0 |
| Self-Improving Pretraining | 96.0 | 93.4 | 88.4 | 49.0 | 93.1 |
| Method / Ablation | Generation Quality Std. Prefix Unsafe Prefix | Standard Evals (Avg) | Safety Evals (Avg) | |
| Llama Base | 50.0 | 50.0 | 47.6 | 76.9 |
| Llama Pretrain Baseline | 54.5 | 52.6 | 47.9 | 75.5 |
| Training without a Judge | ||||
| SFT (rewrite) | 52.7 | 50.6 | 48.4 | 86.5 |
| SFT (1 rollout) | 2.0 | 0.2 | 29.5 | 99.5 |
| Online DPO (chosen: rewrite, reject:rollout) | 53.6 | 83.1 | 48.8 | 77.7 |
| Pretraining for Quality | Generation Quality | Standard Evals (avg) | Coherence Eval |
| Self-Improving Pretraining (finetuned Llama3 as judge) | 72.1 | 49.6 | 72.7 |
| Self-Improving Pretraining (GPT-OSS-120B as judge) | 84.3 | 51.1 | 86.8 |
| Model | Total mid- | GSM8k | MATH500 | Olympiad | AMC23 | GPQA-D | Avg |
| train tokens | mean@16 | mean@16 | mean@16 | mean@64 | mean@16 | ||
| Baselines | |||||||
| Base (Llama3-8B-Base) | 0 | 0.0123 | 0.0110 | 0.0020 | 0.0031 | 0.2090 | 0.0475 |
| + SFT(raw) 10k | 10.5B | 0.1556 | 0.0819 | 0.0177 | 0.0250 | 0.2197 | 0.0999 |
| Thinking Mid-training | |||||||
| + SFT(think) 10k | 10.5B | 0.4077 | 0.2433 | 0.0641 | 0.0969 | 0.2983 | 0.2221 |
| Model | Total mid- | GSM8k | MATH500 | Olympiad | AMC23 | GPQA-D | Avg |
| train tokens | mean@16 | mean@16 | mean@16 | mean@64 | mean@16 | ||
| Baselines | |||||||
| Base (Llama3-8B-Base) | 0 | 0.0123 | 0.0110 | 0.0020 | 0.0031 | 0.2090 | 0.0475 |
| + RLPT | 0 | 0.2172 | 0.1038 | 0.0211 | 0.0250 | 0.2314 | 0.1197 |
| + SFT(raw) 10k + RLPT | 10.5B | 0.3341 | 0.1645 | 0.0388 | 0.0672 | 0.2181 | 0.1645 |
| Thinking Mid-training + RL Post-training |
| Model | Total mid- | GSM8k | MATH500 | Olympiad | AMC23 | GPQA-D | Avg |
| train tokens | mean@16 | mean@16 | mean@16 | mean@64 | mean@16 | ||
| Baselines | |||||||
| Base (Qwen3-8b-Base) | 0 | 0.9070 | 0.7491 | 0.4029 | 0.5016 | 0.3879 | 0.3572 |
| + SFT(raw) 10k | 10.5B | 0.8223 | 0.6208 | 0.2995 | 0.3172 | 0.3087 | 0.2802 |
| Thinking Mid-training | |||||||
| + SFT(think) 10k | 10.5B | 0.6389 | 0.5731 | 0.2780 | 0.3125 | 0.3409 | 0.2538 |
| Pretraining for Safety | Generation Quality Std. Prefix Unsafe Prefix | Standard Evals (Avg) | Safety Evals (Avg) | |
| Llama Base | 50.0 | 50.0 | 47.6 | 76.9 |
| SIP | 73.6 | 77.7 | 49.1 | 91.1 |
| SPoT | 71.3/ 77.9 ∗ | 73.8/ 81.9 ∗ | 48.9 | 94.6/ 98.2 ∗ |
| Pretraining for Safety | RealToxicityPrompts | RedPajama test | XSTest safe | XSTest unsafe | ToxiGen |
| Llama Base | 88.1 | 68.0 | 85.2 | 39.5 | 80.1 |
| SIP, Chapter 1 | |||||
| 98.2 | 94.7 | 92.0 | 51.5 | 92.3 | |
| 98.0 | 95.2 | 92.8 | 52.5 | 93.1 | |
| SPoT | |||||
| , Llama Judge | 98.3 | 97.5 | 96.8 | 82.0 | 96.9 |
| Self-Improving Post-Thinking for Safety (design ablations, XSTest-unsafe, ) | ||||||
| Pair selection | Judge | Greedy | Avg@8 | PoT@8 | Judge@8 | Oracle@8 |
| Unbalanced | GPT-OSS- B | 45.5 | 48.8 | 59.0 | 76.0 | 93.5 |
| Unbalanced | Llama- - B | 53.5 | 56.8 | 68.5 | 84.0 | 97.5 |
| Balanced, deterministic | Llama- - B | 49.0 | 55.2 | 74.0 | 80.5 | 95.0 |
| Balanced, random ( chosen ) | Llama- - B | 54.0 | 50.0 | 73.5 | 74.5 | 93.5 |
| Self-Improving Post-Thinking for Safety (label-loss strategy ablation, XSTest-unsafe) | |||||
| Strategy | Greedy | Avg@8 | PoT@8 | Judge@8 | Oracle@8 |
| Verdict Loss ( , verdict tokens only) | 60.0 | 51.1 | 70.0 | 69.5 | 94.0 |
| SFT-W2 ( , full reasoning trace) | 56.0 | 47.2 | 71.5 | 67.0 | 92.0 |
| SFT-W5 ( , full reasoning trace) (chosen) | 54.0 | 50.0 | 73.5 | 74.5 | 93.5 |
| SPIN-W1 | 58.0 | 47.5 | 67.5 | 65.5 | 93.0 |
| SPIN-W10 | 53.5 | 50.3 | 46.5 | 67.0 | 93.0 |
| Method | MATH500 | GSM8K | AMC23 | OlympiadBench | GPQA-D |
| Thinking mid-training (init) | |||||
| Avg@8 | 42.8 | 67.7 | 15.4 | 13.2 | 31.3 |
| majority@8 | 56.5 | 84.9 | 25.6 | 20.1 | 36.0 |
| RL post-training | |||||
| Avg@8 | 47.7 | 75.8 | 21.5 | 16.1 | 32.6 |
| majority@8 | 60.7 | 89.1 | 23.1 | 25.1 | 38.1 |
Appendix figures & tables31 assets
Supplementary material from the paper’s appendix.
Appendix
| Quality Judgments | arxiv | book | c4 | github | stack_exch | wiki | cc |
| Llama3.1-8B-Instruct | 48.4 | 45.0 | 54.9 | 55.1 | 51.9 | 55.9 | 53.0 |
| Llama3.3-70B-Instruct | 72.7 | 84.0 | 75.5 | 69.9 | 66.0 | 84.8 | 74.1 |
| DeepSeek-R1-Distill-Llama-8B | 57.8 | 55.3 | 57.5 | 55.8 | 53.2 | 63.5 | 56.8 |
| DeepSeek-R1-Distill-Llama-70B | 73.6 | 79.4 | 75.4 | 72.6 | 64.0 | 86.0 | 74.9 |
| Aspect | v1 | v2 | v3 | v4 | v5 |
| Main Focus | Factuality + logical coherence | Factual accuracy only | Factual accuracy with creative tolerance | Factual accuracy, style-agnostic | Clear factual inaccuracies only |
| Role Description | “Factuality evaluator” | “Careful evaluator of factual accuracy” | “Careful evaluator of factual accuracy” | “Careful evaluator of factual accuracy” | “Careful and forgiving evaluator” |
| Treatment of Original Text | Must logically follow from original + human continuation | May be incomplete; partial ground truth | May be incomplete; partial ground truth | May be incomplete; partial ground truth | May be incomplete; partial ground truth |
| World Knowledge Use | Implicit (through human continuation) | Explicit: “use your own general world knowledge” | Explicit: “use your own general world knowledge” | Explicit: “reliable general world knowledge” | Explicit: “rely on general world knowledge” |
| Coherence Requirements | Strict: must logically follow | Relaxed: “Minor logical or coherence issues should NOT be considered hallucinations” | Ignored: “Minor coherence issues should NOT automatically count” | Ignored: “not to judge style, narrative flow, or coherence” | Ignored: “Ignore coherence gaps, logical quirks” |
| Handling of Off-topic Content | Penalized as implausible | Ignored if factually accurate | “should NOT automatically count as hallucinations” | “should NOT count as hallucinations if plausible” | “should NOT be penalized” |
| Continuation evaluated | Prompt | Definite | Possible | No Hallucination |
| Original Suffix | v1 (No-Ref) | 79 (39.5%) | 36 (18.0%) | 85 (42.5%) |
| Original Suffix | v2 (No-Ref) | 15 (7.5%) | 27 (13.6%) | 157 (78.9%) |
| Original Suffix | v3 (No-Ref) | 3 (1.5%) | 54 (27.0%) | 143 (71.5%) |
| Original Suffix | v4 (No-Ref) | 4 (2.0%) | 60 (30.0%) | 136 (68.0%) |
| Model | v1 (No-Ref) | 180 (90.5%) | 9 (4.5%) | 10 (5.0%) |
| Model | v2 (No-Ref) | 142 (71.0%) | 21 (10.5%) | 37 (18.5%) |
| Continuation evaluated | Prompt | Definite | Possible | No Hallucination |
| Original Suffix | v1 (No-Ref) | 106 (53.0%) | 30 (15.0%) | 60 (30.0%) |
| Original Suffix | v2 (No-Ref) | 63 (31.5%) | 28 (14.0%) | 108 (54.0%) |
| Original Suffix | v3 (No-Ref) | 66 (33.0%) | 56 (28.0%) | 77 (38.5%) |
| Original Suffix | v4 (No-Ref) | 42 (21.0%) | 54 (27.0%) | 101 (50.5%) |
| Model | v1 (No-Ref) | 159 (79.5%) | 14 (7.0%) | 25 (12.5%) |
| Model | v2 (No-Ref) | 133 (66.5%) | 11 (5.5%) | 56 (28.0%) |
| Continuation evaluated | Prompt | Definite | Possible | No Hallucination |
| Original Suffix | v1 (No-Ref) | 21 (10.8%) | 22 (11.3%) | 152 (77.9%) |
| Original Suffix | v2 (No-Ref) | 1 (0.5%) | 0 (0.0%) | 199 (99.5%) |
| Original Suffix | v3 (No-Ref) | 1 (0.5%) | 9 (4.5%) | 189 (95.0%) |
| Original Suffix | v4 (No-Ref) | 0 (0.0%) | 3 (1.5%) | 195 (98.5%) |
| Model | v1 (No-Ref) | 74 (38.3%) | 62 (32.1%) | 57 (29.5%) |
| Model | v2 (No-Ref) | 29 (14.5%) | 9 (4.5%) | 162 (81.0%) |
| Reference Set | Llama3.3-70B vs. GPT-4o | GPT-OSS vs. GPT-4o |
| Orig. Suffix No-Ref | Agreement: 63.64% – 79.68% (avg: 71.33%) Gap: 2.70% ( 3.8% relative) | Agreement: 47.24% – 61.11% (avg: 54.82%) Gap: 10.90% ( 19.9% relative) |
| Model No-Ref | Agreement: 32.93% – 50.00% (avg: 37.47%) Gap: 21.23% ( 56.7% relative) | Agreement: 65.66% – 85.28% (avg: 72.32%) Gap: 12.56% ( 17.4% relative) |
| Model With-Ref | Agreement: 38.61% – 84.62% (avg: 54.77%) Gap: 24.72% ( 45.1% relative) | Agreement: 56.99% – 94.85% (avg: 73.21%) Gap: 21.10% ( 27.8% relative) |
| Pretraining for Quality | Generation Quality | Standard Evals (Avg) | Coherence Eval |
| Llama Base | 50.0 | 47.6 | 50.0 |
| Llama Pretrain Baseline | 49.0 | 46.7 | 49.4 |
| Self-Improving Pretraining (2 rollouts) | 69.9 | 49.4 | 67.6 |
| Self-Improving Pretraining (4 rollouts) | 75.5 | 49.9 | 71.2 |
| Self-Improving Pretraining (8 rollouts) | 84.3 | 51.1 | 86.8 |
| Self-Improving Pretraining (16 rollouts) | 86.3 | 50.8 | 87.9 |
| boolq | piqa | siqa | hellaswag | arc_challenge | arc_easy | obqa | mmlu | |
| Llama Base | 64.6 | 74.8 | 41.0 | 47.9 | 32.3 | 66.6 | 27.2 | 26.4 |
| Llama-3.1 8B Base | 83.6 | 79.0 | 60.7 | 82.9 | 52.3 | 33.4 | 46.6 | 66.4 |
| Pretraining for Quality | ||||||||
| Pretrain Baseline | 59.8 | 74.2 | 42.1 | 47.7 | 30.8 | 65.4 | 26.8 | 26.4 |
| Self-Improving Pretraining (2 rollouts) | 67.1 | 75.2 | 43.8 | 49.9 | 34.3 | 69.0 | 29.0 | 26.7 |
| Self-Improving Pretraining (4 rollouts) | 69.5 | 75.6 | 44.1 | 50.3 | 34.6 | 69.4 | 28.0 | 27.8 |
| Slimpajama test set (pointwise) | FActScore (pairwise) | HaluEval dialogue | HaluEval QA | HaluEval summarization | TruthfulQA MC1 | TruthfulQA MC2 | |
| Llama Base | 36.6 | 50.0 | 50.0 | 50.1 | 50.0 | 22.4 | 35.9 |
| Llama-3.1 8B Base | 32.4 | 70.5 | 11.7 | 8.3 | 14.9 | 28.2 | 44.2 |
| Pretrain Baseline | 35.4 | 48.9 | 50.8 | 51.4 | 61.5 | 21.5 | 35.5 |
| 2 rollouts | 37.8 | 53.9 | 52.3 | 53.6 | 64.2 | 22.9 | 36.3 |
| 4 rollouts | 43.6 | 54.3 | 53.6 | 53.2 | 72.0 | 23.9 | 37.3 |
| 8 rollouts | 60.0 | 68.4 | 57.2 | 59.0 | 87.6 | 24.7 | 38.0 |
| Self-Improving Pretraining | boolq | piqa | siqa | hellaswag | arc_challenge | arc_easy | obqa | mmlu |
| fine-tuned Llama3 as judge | 67.5 | 76.1 | 43.8 | 49.8 | 35.4 | 69.3 | 28.6 | 26.9 |
| GPT-OSS-120B as judge | 70.9 | 75.6 | 45.9 | 51.4 | 35.3 | 71.2 | 30.2 | 28.3 |
| Pretraining for Quality | Generation Quality | Standard Evals | Coherence Eval |
| 8 rollouts, suffix as pivot | 72.1 | 49.6 | 67.7 |
| 8 rollouts, full comparisons | 84.3 | 51.1 | 86.8 |
| Pretraining for Factuality | Generation Quality | Standard Evals | Factuality Evals |
| 8 rollouts, suffix as pivot | 64.2 | 49.6 | 55.7 |
| 8 rollouts, full comparisons | 83.1 | 50.3 | 56.9 |
| Pretraining for Factuality | Slimpajama test set (pointwise) | FActScore (pairwise) | HaluEval dialogue | HaluEval QA | HaluEval summarization | TruthfulQA MC1 | TruthfulQA MC2 |
| 8 rollouts, suffix as pivot | 61.1 | 67.9 | 56.1 | 59.9 | 77.9 | 25.3 | 38.9 |
| 8 rollouts, full comparisons | 60.0 | 68.4 | 57.2 | 59.0 | 87.6 | 24.7 | 38.0 |
| boolq | piqa | siqa | hellaswag | arc_challenge | arc_easy | obqa | mmlu | |
| Pretraining for Quality | ||||||||
| 8 rollouts, suffix as pivot | 68.0 | 75.8 | 43.8 | 49.8 | 33.7 | 69.1 | 28.4 | 28.2 |
| 8 rollouts, full comparisons | 70.9 | 75.6 | 45.9 | 51.4 | 35.3 | 71.2 | 30.2 | 28.3 |
| Pretraining for Factuality | ||||||||
| 8 rollouts, suffix as pivot | 67.9 | 75.2 | 44.1 | 49.7 | 34.3 | 68.9 | 28.8 | 28.0 |
| 8 rollouts, full comparisons | 68.3 | 75.6 | 45.8 | 50.8 | 35.7 | 69.6 | 28.6 | 28.2 |
| Gen. Quality | Std. Evals | Coherence | Factuality | Safety | |
| Llama Base | 50.0 | 47.6 | 50.1 | 42.3 | 76.9 |
| Llama-3.1 8B Base | 66.1 | 63.1 | 77.1 | 26.3 | 71.0 |
| Trained on SlimPajama | |||||
| Llama Pretrain Baseline | 49.0 | 46.8 | 49.4 | 44.0 | 76.9 |
| Pretraining for Quality | |||||
| Trained on SlimPajama |
| Pretraining for Safety (sample-weighted average over 5 safety datasets) | ||||||
| Configuration | Greedy | Avg@8 | Policy@8 | Judge@8 | Oracle@8 | |
| Llama Base | — | 76.9 | 79.2 | — | 90.2 | 94.3 |
| SIP (Section 1) ∗ | 8 | 92.3 | 93.2 | — | 95.7 | 98.8 |
| SIP (Section 1) ∗ | 16 | 92.7 | 94.4 | — | 96.8 | 99.4 |
| SIP post-thinking (Ours) | 4 | 92.6 | 93.1 | 96.6 | 96.7 | 98.6 |
| SIP post-thinking (Ours) | 8 | 94.2 | 95.1 | 97.8 | 97.0 | 99.3 |
| Pretraining for Safety (per-task breakdown) | ||||||
| Configuration | Dataset | Greedy | Avg@8 | Policy@8 | Judge@8 | Oracle@8 |
| Llama Base | XSTest-unsafe | 39.5 | 43.6 | — | 81.0 | 92.0 |
| XSTest-safe | 85.2 | 91.0 | — | 95.6 | 100.0 | |
| RedPajama | 68.0 | 70.2 | — | 84.4 | 95.3 | |
| RealToxicity | 88.1 | 93.6 | — | 97.5 | 99.5 | |
| ToxiGen | 80.1 | 77.7 | — | 89.0 | 86.8 | |