cs.LGNov 14, 2026

Understanding the Role of Reasoning Data in Pretraining

Authors: Jianhao Huang, Allan Zhang, Ali Payani, Baharan Mirzasoleiman

Organizations: University of California, Los Angeles · Cisco Research

Abstract

Reasoning data expose models to multi-step reasoning, but reasoning traces may cover only a subset of the available facts. How should pretraining balance atomic facts with reasoning traces? We study this question by constructing a setting in which every fact provides an atomic training signal, while only a subset appears in reasoning supervision, enabling us to measure generalization to transitions observed atomically but never used in training traces. For a linear transformer, we derive the weight assigned to atomic training signals that minimizes held-transition squared error and show that its leading-order optimum increases with reasoning-chain length and decreases with the fraction of facts held out from reasoning traces. These trends persist in a full transformer layer with an MLP, softmax attention, and cross-entropy loss. We further validate the role of reasoning complexity and data coverage in two natural-language settings: controlled multi-hop reasoning and mathematical pretraining. Together, our results provide a principled view of how reasoning-chain structure and data coverage should shape the allocation of training data during pretraining.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. A Primer in Post-Training Reasoning Data: What We Know About How It Works

    Jun 1, 2026Yaoming Li, Guangxiang Zhao, Qilong Shi +3RL Post-TrainingLLM Post-Training

  2. Understanding Reasoning from Pretraining to Post-Training

    Jul 17, 2026Jingyan Shen, Ang Li, Salman Rahman +4Language Model PretrainingRL for Language Model Reasoning

  3. Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models

    Mar 1, 2026Adel Javanmard, Baharan Mirzasoleiman, Vahab MirrokniData Selection for Language Model Fine-TuningLLM Fine-Tuning