Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, their immense number of parameters and complex transformer-based architectures result in significant resource demands and computational complexity during training, making it challenging to optimize them efficiently on large datasets. To reduce training costs while preserving performance, researchers have investigated coreset selection techniques, which aim to identify small, representative subsets of the entire training dataset to accelerate LLM training. However, existing coreset selection methods fail to adapt to the dynamic nature of LLM training and often struggle with scalability for models of this size. To address these limitations, we propose a graph-guided adaptive and dynamic coreset selection framework for LLMs, namely GRACE. GRACE dynamically constructs and updates coresets by combining representation diversity with gradient-based importance metrics, ensuring both informativeness and efficiency. To mitigate the computational cost of frequent updates, GRACE leverages a k-NN graph-based propagation mechanism and selectively updates scores and embeddings, adapting to evolving training dynamics. Extensive experiments on three benchmarks demonstrate that GRACE significantly improves training efficiency and downstream performance across diverse LLMs and tasks.
Figures & tables
Notation
Description
X,xi,T
Input sequence, i -th token, sequence length
X,x1
Initial hidden states, i -th token’s embedding
Hi(l)
Hidden states of Xi after l -th layer
Hˉi(l)
Mean hidden states of Xi along sequence
Z(l),O^
MHSA output at l -th layer, predicted logits
P^θ(X)
Predicted probabilities of sample X
Table 1. Important notations.
Figure 1. Overview of GRACE. Stage 1: Hidden states and importance scores of all training samples are extracted from a warm-up trained model. A k -NN graph is constructed from these indicators. Stage 2: Sample a subset of samples from the graph and select the coreset, then train the model with the coreset. Stage 3: At checking steps, if significant changes in importance scores are detected, the graph will be updated. A subset is selected for recomputation and updated scores and embeddings are propagated through the graph structure. The k -NN graph is also updated and then reused in the next selection.
Figure 2. Overview of the graph update process. From left to right: we begin with a k -NN graph constructed from initial representations and EL2N scores (blue nodes). When adaptive update checking is triggered, a subset of recently trained samples (green) is loaded and we use their history scores as new scores. Another small number of samples (purple) are then selected for recomputation. Combining the history scores and recomputed scores as updated scores, they are then propagated to their neighbors through the graph structure (blue to green), allowing approximate updates without full recomputation.
Type
Methods
Phi-2
Llama2-7B
Qwen2.5-7B
In-domain
Out-domain
In-domain
Out-domain
In-domain
Out-domain
-
Base (Pretrained Model)
19.66
22.96
3.39
2.56
47.73
58.39
Static
Random
48.35
50.79
22.66
17.57
60.66
63.39
Middle Perplexity
47.46
47.55
22.90
18.33
61.02
62.07
Facility Location
46.16
48.89
21.01
15.54
60.66
65.26
DivIR
47.58
52.16
23.33
21.90
61.34
64.23
Table 2. MathInstruct results with 10% training budget. We report in-domain (GSM8K, MATH, NumGlue) and out-of-domain (SVAMP, DeepMind, SimuLeq) average accuracies (%).
Figure 3. Time Comparison
Figure 4. Experiment results under different budgets on MathInstruct
Figure 7
Model
Phi-2
Llama-2
In
Out
In
Out
G ⋅ \I
48.49
52.98
22.78
18.40
G ⋅ \R
50.23
54.82
21.67
21.22
G ⋅ \Update
48.79
51.54
23.42
23.62
G ⋅ \AdaUpdate
50.08
53.03
23.19
22.57
GRACE
50.54
55.27
23.89
24.05
Table 3. Ablation Experiments to test the Effect of Scores. In means in-domain tasks, and Out means out-of-domain tasks.
Full-parameter fine-tuning of large language models is constrained by substantial GPU memory requirements. Low-rank adaptation methods mitigate this challenge by updating only a subset of parameters. However, these approaches often limit model expressiveness and yield lower performance than full-parameter fine-tuning. Layer-wise fine-tuning methods have emerged as an alternative, enabling memory-efficient training through static layer importance sampling strategies. However, these methods overlook variations in layer importance across tasks and training stages, resulting in suboptimal performance on downstream tasks. To address these limitations, we propose GRASS, a gradient-based adaptive layer-wise importance sampling framework. GRASS utilizes mean gradient norms as a task-aware and training-stage-aware metric for estimating layer importance. Furthermore, GRASS adaptively adjusts layer sampling probabilities through an adaptive training strategy. We also introduce a layer-wise optimizer state offloading mechanism that overlaps computation and communication to further reduce memory usage while maintaining comparable training throughput. Extensive experiments across multiple models and benchmarks demonstrate that GRASS consistently outperforms state-of-the-art methods, achieving an average accuracy improvement of up to 4.38 points and reducing memory usage by up to 19.97%.
Kaiyuan Tian, Yu Tang, Gongqingjian Jiang +5
National University of Defense Technology · Information Support Force Engineering University
As Large Language Model (LLM) datasets scale to trillions of tokens, data selection has emerged as a critical frontier to filter out uninformative noise and construct adaptive learning trajectories. Beyond static heuristic filtering, advanced data selection methods for LLM training largely follow two paradigms, each with fundamental limitations. Influence-based methods provide principled bi-level objectives but require intractable inverse-Hessian computations, while excess-loss methods are computationally efficient but rely on a static reference model that becomes misaligned with the evolving proxy model during training. We propose BLADE (Bi-Level Adaptive Data sElection), a Hessian-free framework for data selection. BLADE reformulates the bi-level optimization problem underlying influence-based methods as a penalized single-level objective via Lagrange multipliers, avoiding inverse-Hessian computation while revealing a principled connection to excess-loss based data selection. The resulting objective recovers an excess-loss form but replaces the static reference model with a dynamic one that stays synchronized with training. Theoretically, we prove that this penalized formulation guarantees first-order convergence. For efficient online batch selection, we instantiate BLADE as a memoryless randomized block-coordinate Frank-Wolfe algorithm. Extensive experiments show that BLADE consistently outperforms state-of-the-art data selection baselines, providing a practical recipe for LLM training.
Training large language models (LLMs) at scale incurs substantial communication overhead, while static gradient compression cannot adapt to gradient evolution and may degrade model quality. We propose EDGC, an entropy-driven dynamic gradient compression framework that adapts compression ranks to gradient entropy during training. EDGC combines efficient entropy estimation through gradient sampling, a theoretical model relating entropy to compression rank under a bounded-error constraint, and window-based rank adjustment across pipeline stages. Experiments on 32-V100 and 64-H100 GPU clusters training GPT2 models with 2.5B and 12.1B parameters show that EDGC reduces communication latency by up to 46.45% and end-to-end training time by 16.13%, while maintaining model quality.
Qingao Yi, Jiaang Duan, Jun Zhang +5
University of Shanghai for Science and Technology, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Zhejiang University, Hangzhou, China