cs.CLJan 29, 2026

Self-Improving Training & Thinking

Authors: Ellen Xiaoqing Tan, Jack Lanchantin, Zichun Yu, Polina Kirichenko, Shehzaad Dhuliawala, Danwei Li, Thao Nguyen, Jing Xu, 

Organizations: FAIR at Meta

Abstract

Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model's capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite data and to judge policy model rollouts, thus introducing thinking and reinforcement earlier in training. We further improve reasoning at all stages by introducing the concept of self-improving post-thinking: improving the model's reasoning and self-judging ability by reasoning after generation. In our experiments, we show these approaches can give strong gains across multiple axes.

Figures & tables

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning

    Oct 21, 2025Mengqi Li, Lei Zhao, Anthony Man-Cho So +2Self-Training for Language ModelsMathematical Reasoning

  2. From Rollouts to Recipes: Self-Contained Post-Training for LLMs

    Sep 1, 2026Yifei Li, Lingling Zhang, Muye Huang +3Self-Training for Language ModelsRL for Language Model Reasoning

  3. ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement

    Apr 2, 2026Difan Jiao, Qianfeng Wen, Blair Yang +2RL for Language Model ReasoningLLM Reasoning