Existing studies on Long-Context Continual Pre-training (LCCP) mainly focus on small-scale models and limited data regimes (tens of billions of tokens). We argue that directly migrating these small-scale settings to industrial-grade models risks insufficient adaptation and premature training termination. Furthermore, current evaluation methods rely heavily on downstream benchmarks (e.g., Needle-in-a-Haystack), which often fail to reflect the intrinsic convergence state and can lead to "deceptive saturation". In this paper, we present the first systematic investigation of LCCP learning dynamics using the industrial-grade Hunyuan-A13B (80B total parameters), tracking its evolution across a 200B-token training trajectory. Specifically, we propose a hierarchical framework to analyze LCCP dynamics across behavioral (supervised fine-tuning probing), probabilistic (perplexity), and mechanistic (attention patterns) levels. Our findings reveal: (1) Necessity of Massive Data Scaling: Training regimes of dozens of billions of tokens are insufficient for industrial-grade LLMs' LCCP (e.g., Hunyuan-A13B reaches saturation after training over 150B tokens). (2) Deceptive Saturation vs. Intrinsic Saturation: Traditional NIAH scores report "fake saturation" early, while our PPL-based analysis reveals continuous intrinsic improvements and correlates more strongly with downstream performance. (3) Mechanistic Monitoring for Training Stability: Retrieval heads act as efficient, low-resource training monitors, as their evolving attention scores reliably track LCCP progress and exhibit high correlation with SFT results. This work provides a comprehensive monitoring framework, evaluation system, and mechanistic interpretation for the LCCP of industrial-grade LLM.
Figures & tables
Hyper-parameter
Value
Total Parameters
80B
Active Parameters
13B
Layers
32
Hidden Size
4096
FFN Hidden Size
3072
Heads (Q / KV)
32 / 8
Table 1: Key hyper-parameter of Hunyuan-A13B.
Source
Ratio (%)
Common Crawl (CC)
36.3
Books
28.6
arXiv
24.0
Code
10.8
Wikipedia
0.3
Total
100.0
Table 2: Distribution of data sources for long-context documents.
Figure 1: The performance on the RULER, MRCR, and LongBio benchmarks of different LCCP checkpoints of the Hunyuan-A13B model after lightweight SFT.
Figure 2: The PPL distribution for NIAH tasks across different context lengths and depths for Hunyuan-A13B base model at different stages of LCCP. Each cell represents the average PPL of the answer tokens.
Figure 3: Average NIAH score and PPL across Hunyuan-A13B base model at different stages of LCCP.
Metric
NIAH Score
NIAH PPL
RULER
0.8552
-0.9115
MRCR
0.8927
-0.9231
LongBio
0.4977
-0.6283
Average
0.7486
-0.8210
Table 3: The Pearson correlation coefficients between NIAH metrics and performance on downstream SFT probe benchmarks.
Figure 4: The retrieval scores of all attention heads across all layers of the Hunyuan-A13B base model at different stages of LCCP.
Figure 5: The number of retrieval heads and the average retrieval score of the initial retrieval heads of the Hunyuan-A13B base model at different stages of LCCP.
Metric
# of Retrieval Heads
Avg. Retrieval Score
RULER
0.8243
0.8956
MRCR
0.8326
0.8469
LongBio
0.5715
0.6210
Average
0.7428
0.7878
Table 4: The Pearson correlation coefficients between retrieval head metrics and performance on downstream SFT probe benchmarks.
Figure 6: The PPL curve as a function of the number of training tokens of Hunyuan-A13B base model. The left panel displays the standard scale, while the right panel presents the logarithmic scale.
Figure 7: Average PPL across different document depths for Hunyuan-A13B base model at different stages of LCCP.
Figure 8: The PPL distribution for modified NIAH tasks across different context lengths and depths for Hunyuan-A13B base model at different stages of long-context continual pre-training. Each cell represents the average PPL of the answer tokens. Absolute Distance is the length of [Context After Needle Text] and Interference Context Length is the length of [Context Before Needle Text] .
Figure 9: Answer PPL under increasing interference context length at a fixed absolute distance of 16K.
Training Stage
Overlap Heads
Spearman
0B → 20B
24/30
0.7316
20B → 50B
29/30
0.9238
50B → 100B
28/30
0.9164
100B → 150B
28/30
0.8808
150B → 200B
28/30
0.8921
Table 5: Overlap of top 30 retrieval heads and Spearman correlation between consecutive training stages.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
NIAH Score
NIAH PPL
RULER
0.0299
0.0114
MRCR
0.0166
0.0087
LongBio
0.3151
0.1816
Appendix
Table 6: The p-value of Pearson correlation coefficients between NIAH metrics and performance on downstream SFT probe benchmarks.
Metric
# of Retrieval Heads
Avg. Retrieval Score
RULER
0.0436
0.0158
MRCR
0.0397
0.0334
LongBio
0.2361
0.1882
Appendix
Table 7: The p-vaue of Pearson correlation coefficients between retrieval head metrics and performance on downstream SFT probe benchmarks.
Long-context continual pre-training (LCCP) is widely used to improve long-context capability, but how its benefits scale with the number of tokens used for LCCP remains unclear. We investigate this question through modeling, monitoring, and validating the benefits of scaling LCCP. First, we establish an empirical additive power law relating long-context performance to model size and the respective token budgets of short-context pre-training and LCCP. Experiments across five model scales and more than one hundred short- and long-context budget configurations show that the fitted law reproduces the observed trends and extrapolates well to held-out model scales. Second, we introduce Constant-Last4, a training and observation protocol that averages the four most recent consecutive checkpoints from a constant-learning-rate run to enable stable monitoring along a single extensible training trajectory. Under this protocol, the same scaling-law form fits well and accurately predicts the LCCP performance trajectory of a 1.22B-parameter model trained on approximately 100B LCCP tokens. Finally, downstream evaluations along this trajectory show continued improvements in long-context performance, with no clear sign of saturation within the evaluated range. These gains persist after both 128K context extension and supervised fine-tuning, with benefits extending to long-context understanding and agentic tasks. Together, these results argue for continuing to scale LCCP rather than stopping early: its gains are stably observable along an extensible trajectory, quantitatively predictable, and transferable to downstream tasks.
Ziqing Qiao, Xiaoyue Xu, Sikui Zhang +4
Department of Computer Science and Technology, Tsinghua University, Beijing, China · OpenBMB
Continual learning (CL) has emerged as a pivotal paradigm to enable large language models (LLMs) to dynamically adapt to evolving knowledge and sequential tasks while mitigating catastrophic forgetting, a critical limitation of the static pre-training paradigm inherent to modern LLMs. This paper presents a comprehensive overview of CL methodologies tailored for LLMs, structured around three core training stages: continual pre-training, continual fine-tuning, and continual alignment. Beyond the canonical taxonomy of rehearsal-, regularization-, and architecture-based methods, we further subdivide each category by its distinct forgetting mitigation mechanisms and conduct a rigorous comparative analysis of the adaptability and critical improvements of traditional CL methods for LLMs. In doing so, we explicitly highlight core distinctions between LLM CL and traditional machine learning, particularly with respect to scale, parameter efficiency, and emergent capabilities. Our analysis covers essential evaluation metrics, including forgetting rates and knowledge transfer efficiency, along with emerging benchmarks for assessing CL performance. While current methods show promising results, fundamental challenges persist in achieving seamless knowledge integration, and we identify key open problems and future directions for the field.
The efficacy of continued pre-training for Large Language Models (LLMs) hinges upon hyperparameter configurations, such as learning rate and batch size. However, current practices often rely on heuristics or grid searches, leading to training instability and excessive costs. In this work, we first empirically discover that optimal hyperparameters follow stable and predictable scaling laws throughout the continued pre-training process. Leveraging these insights, we propose a novel framework to establish quantitative relationships between compute budget and optimal hyperparameters for a given checkpoint. Our approach has two stages: (1) Empirical Law Discovery, where we train small-scale proxy models to derive functions mapping compute budget to optimal hyperparameters via standard loss-compute scaling laws; and (2) State-Aware Hyperparameter Prediction, where we evaluate an initial checkpoint's validation loss and use the inverse scaling law to estimate its equivalent pre-training compute -- the compute needed to achieve the same loss from scratch. Combining this with the planned compute budget, we predict optimal hyperparameters for the target run. Empirical results demonstrate that our method reduces the hyperparameter search overhead by up to 90% while achieving comparable or superior performance relative to baselines. This model-agnostic framework generalizes across architectures, providing a principled and efficient methodology for diverse continued pre-training scenarios starting from any given point.
Yongwei Zhou, Juncheng Diao, Junlin Shang +2
MeiTuan · University of Chinese Academy of Sciences · Harbin Institute of Technology