cs.LGOct 6, 2026
SaveWhich and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning
Organizations: Arizona State University · Clemson University · Meta AI · NEC Laboratories America
Abstract
LoRA fine-tuning adapts small language models (SLMs) to heterogeneous instruction data within a low-rank update subspace, making it vulnerable to three structural problems: conflicting gradients that cancel, static data selection that cannot track evolving learning dynamics, and subspace saturation that causes later updates to overwrite useful directions. We argue that effective adaptation therefore requires controlling which data-induced gradients enter the LoRA subspace and when. We propose GRADE (GRadient-Aligned Data-centric rEcipe), a data-centric framework combining two mechanisms: a state-aware selector that continually admits samples aligned with the evolving multi-task gradient field, and a self-calibrating step-level gate that rejects updates likely to cause destructive overwrite near saturation. Across three current-generation backbones and a heterogeneous seven-dataset instruction pool, GRADE outperforms strong data-selection and PEFT-stabilization baselines in accuracy and robustness. It is the only method to improve consistently over standard LoRA on every architecture, while producing more coherent gradient trajectories and less destructive overwrite. These results show that successful SLM adaptation depends not only on which data are selected, but also on which gradients are allowed to enter and persist in the constrained update subspace.
Figures & tables
Figure 1: Per-sample LoRA gradient cosine on Gemma-2-9B: training-pool datasets have inconsistent gradient directions (intra-dataset , inter-dataset ). Setup in App. O .
Figure 2: Overview of GRADE. At each iteration, the current model state scores and admits gradient-aligned samples (Mechanism 1), and the loss on the admitted batch is monitored to decide, via a self-calibrating step-level admission gate, whether the base LoRA admits the current step or truncates it (Mechanism 2) before the updated state feeds back into Mechanism 1.
| Llama-3.1-8B (3t) | Qwen3-8B (3t) | Gemma-2-9B (3t) | Cross-arch | |||||
|---|---|---|---|---|---|---|---|---|
| Method | Avg ±std | Avg ±std | Avg ±std | Worst | Mean | |||
| LoRA-MGPO | +0.85 | 0.28 | +1.84 | 0.28 | +0.80 | |||
| Sensitivity-LoRA | +0.29 | 0.29 | +0.14 | 0.29 | +0.05 | |||
| GRAD-MATCH | +0.19 | 0.28 | +0.73 | 0.28 | +0.21 | |||
| PCGrad | +0.46 | 0.12 | +1.33 | 0.12 | +0.56 | |||
| CAGrad | +0.57 | 0.07 | +1.48 | 0.07 | +0.66 | |||
Table 1: Main results on the heterogeneous pool ( ): eight methods on three base backbones, ranked on the 3-task held-out average (ARC-C + HellaSwag + Winogrande). in pp vs. LoRA; Worst- is the minimum across architectures, Mean- the unweighted mean. Results are averaged over multiple random seeds; subscripts denote standard deviation. Per-architecture and cross-architecture leaders bolded. Per-backbone learning rates and the ClusterUCB † reproduction note are in App. T .
Figure 3: Selection mechanism. (a) coherence ratio across epochs (log- , LLaMA-2-7B mechanism diagnostic): selection-equipped runs ( adaptive, frozen) reach – , while the no-selection LoRA baseline ( ) collapses to as rises on the full heterogeneous pool. (b) Per-dataset selection rate under loss-keyed (red) vs. alignment-keyed (blue) top-half cuts on the noisy heterogeneous pool. (c) Per-dataset selection-rate trajectory across epochs on Gemma-2-9B under alignment-keyed selection; dolly15k and xsum highlighted as the anti-correlated pair, the rest dimmed: selection is non-stationary and tracks the evolving model state.
Figure 4: Evidence for the GRADE admission gate. (a) Step-level dynamics on Llama-3.1-8B: 100-step-smoothed probe-loss EMA (blue, left axis) and cumulative skip fraction (red, right axis); gray dotted line marks the stationary-noise prediction. (b) Cross-architecture gate ablation: 3-task average accuracy for gate-OFF (gray) vs gate-ON (blue) on the three backbones. (c) Per-task signed heatmap. Gate-ON wins the 3-task average on all three backbones.
Figure 5: Admission Necessity Matrix: vs (on, on) GRADE (pp) across the four corners of the sample, step admission lattice at matched lr e 5.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone | ( ) | ( ) | |
|---|---|---|---|
| Llama-3.1-8B | |||
| Qwen3-8B | |||
| Gemma-2-9B |
Table 2: Measured at on the heterogeneous pool ( datasets, auto_frac_n_samples ), and the resulting closed-form . All three rows produce , so the safeguards in _r_star are inactive; the floor / hard-fallback / clamp serve as numerical guards rather than as load-bearing components of the recipe.
Figure 6: Operating regime of Eq. ( 2 ) on . Colored markers: Apr-2026 main-table backbones (Tab. 2 ); gray markers: alternate-LR / smoke-test runs. Diagonal dotted line is the aligned-dataset limit (where ); horizontal dashed line is the orthogonal-dataset limit (where ); the shaded band is where the floor would activate. Light gray contours are iso- curves at . All eleven measurements lie in the well-posed positive- region, so the safeguards in _r_star are never load-bearing in the reported numbers.
Algorithm 1 GRADE Training
Figure 7: Selection–state coupled dynamics (LLaMA-2-7B mechanism diagnostic). (a) Admitted-population retention (Jaccard) across consecutive epochs for adaptive (top) vs. frozen (bottom) selection. (b) Selection-mechanism dynamics on a normalized axis: criterion sharpness (admitted/rejected score gap, blue) and population churn (Jaccard, red), both normalized to their epoch value.
Figure 8: Routing-statistic monotonicity vs. noise on the four instrumented Phase-1.5 runs. The desirable region is the lower-right corner (monotone w.r.t. step, quiet within an epoch); is the unique statistic in that region. and its variant occupy the upper-left.
Figure 9: Per-dataset admission-gate gain vs. residual (left) and the same scatter with ARC-C excluded (right). The OLS-through-origin slope drops from to but stays positive; Dolly remains the high-norm / no-trigger diagnostic deviation in both panels.
Figure 10: Probe vs. exact per-sample LoRA gradient, end-to-end. (a) Cost–loss Pareto: held-out loss vs. wall-clock cost for the forward probe and the exact vmapped gradient (mean , 3 seeds). (b) Seed-paired held-out loss for the two estimators across 3 seeds.
Figure 11: Per-sample alignment vs. loss on the Phase-1 noisy heterogeneous pool (Llama-3.1-8B, end of epoch , ). Anchors gsm8k arc in blue, the instruction-style datasets in gray; the dashed lines are the loss-median (red, vertical) and alignment-median (blue, horizontal). The geometric companion to Figure 3 (b) of the main text.
Figure 12: Pool-fit improvement vs. the no-selection reference for three selection rules on the noisy heterogeneous pool (left) and the matched clean pool (right). Loss-based selection is the only rule whose noisy-pool improvement is negative; the gap collapses on the clean pool, identifying the failure as a noise-specific pathology. The per-sample geometric signature of this pathology is in Figure 3 (b).
Figure 13: Per-epoch gap, on a log- axis, between the mean alignment score of selected vs. rejected samples (blue, probe units) and the absolute mean CE-loss gap of the same two populations (red, nats); dashed line at separates the two decades. Result analysis appears in App. S .
| Llama-3.1-8B | Qwen3-8B | Gemma-2-9B | ||||
|---|---|---|---|---|---|---|
| GRADE configuration | Avg | Avg | Avg | |||
| GRADE, lr | 0.6568 | +0.46 | 0.6589 | +0.18 | 0.6828 | +0.64 |
| GRADE, lr | 0.6602 | +0.80 | 0.6550 | 0.21 | 0.6964 | +2.00 |
Table 3: GRADE under two learning-rate operating points on each backbone of Table 1 . The main-table GRADE row uses, per column, the configuration with the higher Avg — lr on Llama-3.1-8B and Gemma-2-9B; lr on Qwen3-8B. The Qwen3-8B and Gemma-2-9B preferences point in opposite LR directions, which is why a single shared learning rate would underperform a per-backbone choice on at least one backbone. Per-column Avg leaders are bolded.