Selecting small, effective instruction-tuning datasets requires identifying which aspects of the original data to retain. We introduce Follow the Features (FtF), a data-selection method that uses sparse autoencoder (SAE) features to characterize training data and guide subset construction. FtF ranks features using dataset-level activation statistics, scores samples by their expression of highly ranked features, and incorporates feature resampling to encourage diversity. The LLM-SAE pair used for selection is independent of the model being fine-tuned, enabling reuse across target models. We evaluate FtF on two model-dataset settings at a training budget of 100 samples with a fixed sequence length of 256 tokens. FtF achieves performance competitive with GRAPE and S2L, with relative rankings varying across settings. On Qwen 3 8B trained on Alpaca, FtF exceeds full-dataset training by 1.59 percentage points in the harmonic mean of an AlpacaEval-based win rate and Open LLM Leaderboard performance, although full-dataset training retains a higher Leaderboard score. These results support SAE feature statistics as a useful signal for data-efficient instruction tuning.
Figures & tables
Figure 1: The generalized Follow the Features (FtF) pipeline. FtF encompasses top- k feature selection and top- N sample selection to produce a refined data subset of N samples, which is then used for instruction-tuning via Supervised Fine-Tuning (SFT). The set of top- k features, the pruned dataset, and the trained model are the deliverable items produced by our pipeline. We evaluate the quality of FtF on the basis of the task performance of the trained model.
Figure 2: Illustration of Algorithm 2 ’s pipeline for assigning sample importance scores
Model
AlpacaEval
Leaderboard
HMean
Ref ( All )
0.31
26.85
0.61
Ref (100)
0.00
24.13
0.00
GRAPE
0.28
23.85
0.55
S2L
0.55
23.99
1.07
FtF
0.95
23.70
1.83
Table 1: Task performance results for Llama 3.1 8B trained on UltraChat 200k. Bold indicates the best result in each column.
Model
AlpacaEval
Leaderboard
HMean
Ref ( All )
12.16
42.58
18.92
Ref (100)
12.00
40.80
18.55
GRAPE
15.60
41.10
22.61
S2L
14.77
41.07
21.73
FtF
13.66
41.17
20.51
Table 2: Task performance results for Qwen 3 8B (Base) trained on Alpaca (cleaned). Bold indicates the best result in each column.
Rank
Index
Description
1
1198
possessive ’s
2
480
numbers after quotes
3
329
not available
4
1067
quotation marks followed by numbers
Table 3: Top- k features identified by FtF for both Llama/UltraChat and Qwen/Alpaca sets of experiments.
Model
AlpacaEval
Leaderboard
HMean
Ref (constant)
1.92
30.60
3.62
Constant
2.55
32.28
4.74
Ref (cosine)
1.45
31.50
2.77
Cosine
2.24
31.50
4.18
Table 4: Comparison of constant and cosine learning rate schedulers for FtF with k=5 and N=100 . Results are percentages. HMean is the harmonic mean of AlpacaEval and Open LLM Leaderboard scores. Ref randomly selects an N -sample dataset and uses the indicated learning rate scheduler. Bold indicates the highest score in each column.
Model
AlpacaEval
Leaderboard
HMean
Reference
1.45
31.50
2.77
270M PT
2.09
30.53
3.90
12B IT
2.24
31.50
4.18
Table 5: Superfiltering analysis summary. AlpacaEval, Open LLM Leaderboard, and harmonic mean results comparing 270M PT and 12B IT in FtF with k=5 and N=100 . Reference is trained on a randomly selected N=100 sample dataset. Results are percentages. Bold indicates the highest score in each column.
Model
AlpacaEval
Leaderboard
HMean
Reference
1.45
31.50
2.77
Mean
1.96
31.46
3.68
LT
2.24
31.50
4.18
Table 6: Sequence aggregation analysis summary. Comparison of last-token pooling (LT) and mean pooling (Mean) for FtF with k=5 and N=100 . Results are percentages. Reference is trained on a randomly selected N=100 sample dataset. Bold indicates the highest score in each column, including ties at the displayed precision.
Model
AlpacaEval
Leaderboard
HMean
Reference
1.45
31.50
2.77
No Div
1.96
31.44
3.68
Div
2.24
31.50
4.18
Table 7: Diversity resampling ablation summary. Comparison of FtF with and without feature diversity resampling. Results are percentages. Reference is trained on a randomly selected N=100 sample dataset. Bold indicates the highest score in each column, including ties at the displayed precision.
Model
AlpacaEval
Leaderboard
HMean
Reference
1.45
31.50
2.77
No Ent
1.96
31.46
3.68
Ent
2.24
31.50
4.18
Table 8: Feature entropy ablation summary. Comparison of FtF with and without entropy included in our definition of feature importance. Reference is trained on a randomly selected N=100 sample dataset. Bold indicates the highest score in each column, including ties at the displayed precision.
k
Spearman’s ρ
Union size
Fraction shared
5
1.0000
5
100.0%
10
1.0000
10
100.0%
50
0.6258
67
66.0%
100
0.2103
155
45.0%
200
−0.1861
347
26.5%
500
−0.4372
793
41.4%
Table 9: Feature-ranking correlations between UltraChat 200K and Alpaca (Cleaned). Correlations are computed over the union of the two top- k feature sets. A feature absent from one dataset’s top- k subset is assigned rank k+1 . Fraction shared is the intersection size divided by k , (2k−∣U∣)/k , where U is the union.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Model
AlpacaEval
Leaderboard
HMean
Baseline
0.79
28.88
1.55
Ref (All)
2.45
32.29
4.55
Dataset size: 100
Reference
1.45
31.50
2.77
k=1
2.93
31.31
5.36
k=5
2.93
31.31
5.36
Appendix
Table 10: Results for the sweep across dataset sizes and k values for FtF Phase 1. Results are percentages. HMean is the harmonic mean of AlpacaEval and Open LLM Leaderboard scores. Baseline is the starting checkpoint (i.e., before training) and Reference trains on N randomly selected data samples, with All indicating a random ordering of the full dataset. Bold indicates the highest score within each dataset size, including ties at the displayed precision. Italics indicate the best observed value at any dataset size.
Model
BBH
GPQA
IFEval
MATH-Hard
MMLU-Pro
MuSR
Baseline
42.27
24.92
26.14
16.16
31.37
32.41
Ref (All)
45.39
27.29
41.41
12.99
29.86
36.82
Dataset size: 100
Reference
43.03
27.01
35.01
15.11
31.28
37.57
k=1
43.13
28.52
35.01
15.26
31.40
34.52
k=5
43.13
28.52
35.01
15.26
31.40
34.52
Appendix
Table 11: Detailed Open LLM Leaderboard results by task for the FtF Phase 1 sweep, grouped by dataset size. Results are percentages. Bold indicates the highest score within each dataset size, including ties at the displayed precision. Italics indicate the best observed value at any dataset size.
Model
AlpacaEval
Leaderboard
HMean
Baseline
0.79
28.88
1.55
Ref (All)
2.45
32.29
4.55
Dataset size: 100
Reference
1.45
31.50
2.77
k=1
2.23
31.50
4.17
k=5
2.24
31.50
4.18
Appendix
Table 12: Results for the sweep across dataset sizes and k values for FtF Phase 2. Results are percentages. HMean is the harmonic mean of AlpacaEval and Open LLM Leaderboard scores. Baseline is the starting checkpoint (i.e., before training) and Reference trains on N randomly selected data samples, with All indicating a random ordering of the full dataset. Bold indicates the highest score within each dataset size, including ties at the displayed precision. Italics indicate the best observed value at any dataset size.
Model
BBH
GPQA
IFEval
MATH-Hard
MMLU-Pro
MuSR
Baseline
42.27
24.92
26.14
16.16
31.37
32.41
Ref (All)
45.39
27.29
41.41
12.99
29.86
36.82
Dataset size: 100
Reference
43.03
27.01
35.01
15.11
31.28
37.57
k=1
43.29
28.33
35.77
15.06
31.42
35.14
k=5
43.39
28.41
35.73
14.95
31.44
35.10
Appendix
Table 13: Detailed Open LLM Leaderboard results by task for the FtF Phase 2 sweep, grouped by dataset size. Results are percentages. Bold indicates the highest score within each dataset size, including ties at the displayed precision. Italics indicate the best observed value at any dataset size.
Parameter
Value
Epochs
1
Sequence length
256
Per-device batch size
32
Optimizer
adamw_torch_fused
Learning rate
2×10−4
Scheduler
Cosine
Appendix
Table 14: Supervised fine-tuning configuration.
Parameter
Value
Rank r
16
LoRA α
32
LoRA dropout
0.05
Bias
none
Target modules
q_proj, k_proj ,
v_proj, o_proj ,
Appendix
Table 15: LoRA configuration.
Parameter
Value
Judge model
Qwen 3.8 27B FP8
Dataset
alpaca_eval_gpt4_baseline
Dataset source
tatsu-lab/alpaca_eval
AlpacaEval samples
300
Leaderboard samples
all
Batch size
4 (1 for Full FT)
Appendix
Table 16: Evaluation configuration.
Figure 3: Illustration of the SAE decoder step. A specific activation zi in the latent space scales its corresponding column vector wi in the decoder’s weight matrix. The final reconstruction x^ is the sum of all such scaled basis vectors.
Model
BBH
GPQA
IFEval
MATH-Hard
MMLU-Pro
MuSR
Ref ( All )
39.36
27.63
29.62
3.70
23.85
36.95
Ref (100)
37.85
25.06
26.30
6.29
13.46
35.85
GRAPE
37.51
25.03
26.54
6.02
12.84
35.14
S2L
37.57
24.97
26.30
5.87
13.82
35.41
FtF
37.10
25.06
26.10
5.54
12.75
35.67
Appendix
Table 17: Detailed Open LLM Leaderboard results for Llama 3.1 8B trained on UltraChat 200k. Bold indicates the best result in each column.
Model
BBH
GPQA
IFEval
MATH-Hard
MMLU-Pro
MuSR
Ref ( All )
58.77
35.23
47.16
24.90
47.02
42.42
Ref (100)
56.55
29.53
49.00
24.30
47.39
38.05
GRAPE
56.56
29.64
49.56
25.13
47.49
38.18
S2L
56.78
29.81
49.00
24.80
47.55
38.49
FtF
56.51
29.53
49.52
24.82
47.70
38.93
Appendix
Table 18: Detailed Open LLM Leaderboard results for Qwen 3 8B (Base) trained on Alpaca (cleaned). Bold indicates the best result in each column.
Model
BBH
GPQA
IFEval
MATH-Hard
MMLU-Pro
MuSR
Reference
43.03
27.01
35.01
15.11
31.28
37.57
Mean
43.23
28.33
35.49
14.90
31.41
35.36
LT
43.39
28.41
35.73
14.95
31.44
35.10
Appendix
Table 19: Mean sequence aggregation analysis details. Open LLM Leaderboard results by task comparing last-token pooling (LT) with mean pooling (Mean) for FtF with k=5 and N=100 . Results are percentages. Bold indicates the highest score in each column, including ties at the displayed precision.
Model
BBH
GPQA
IFEval
MATH-Hard
MMLU-Pro
MuSR
Reference
43.03
27.01
35.01
15.11
31.28
37.57
270M PT
42.83
25.50
33.33
14.12
31.48
35.93
12B IT
43.39
28.41
35.73
14.95
31.44
35.10
Appendix
Table 20: Superfiltering analysis. Detailed Open LLM Leaderboard results by task comparing 270M PT and 12B IT for FtF with k=5 and N=100 . Results are percentages. Bold indicates the highest score in each column.
Model
BBH
GPQA
IFEval
MATH-Hard
MMLU-Pro
MuSR
Reference
43.03
27.01
35.01
15.11
31.28
37.57
No Div
43.23
28.33
35.41
14.90
31.41
35.36
Div
43.39
28.41
35.73
14.95
31.44
35.10
Appendix
Table 21: Diversity resampling ablation details. Open LLM Leaderboard results by task comparing FtF with (Div) and without (No Div) resampling for feature diversity. Results are percentages. Bold indicates the highest score in each column, including ties at the displayed precision.
Model
BBH
GPQA
IFEval
MATH-Hard
MMLU-Pro
MuSR
Reference
43.03
27.01
35.01
15.11
31.28
37.57
No Ent
43.23
28.33
35.53
14.90
31.41
35.36
Ent
43.39
28.41
35.73
14.95
31.44
35.10
Appendix
Table 22: Feature entropy ablation details. Open LLM Leaderboard results by task comparing FtF with (Ent) and without (No Ent) entropy in the feature importance definition. Bold indicates the highest score in each column, including ties at the displayed precision.
Instruction tuning has optimized the specialized capabilities of large language models (LLMs), but it often requires extensive datasets and prolonged training times. The challenge lies in developing specific capabilities by identifying useful data and efficiently fine-tuning. High-quality and diverse pruned data can help models achieve lossless performance at a lower cost. In this paper, we propose SLAP, a novel batch-aware data selection framework that evaluates the learnability of entire batch compositions rather than individual. SLAP ensures comprehensive data distribution coverage through distribution-aware stratified sampling while maximizing intra-batch diversity through relative distance optimization. By leveraging Hessian-approximated gradient information for dynamic batch selection, SLAP significantly outperforms existing state-of-the-art methods across multiple model architectures (LLaMA, ChatGLM) and diverse downstream tasks including multi-turn dialogue, multilingual translation, and question answering. Most notably, SLAP achieves superior performance with 20-40% less training data compared to full dataset training, substantially reducing computational costs while maintaining or improving model capabilities. These results establish SLAP as a powerful approach for efficient and effective instruction tuning of large language models.
Run Zou, Jianhang Ding, Yifan Ding +3
Alibaba International Digital Commerce Group, HangZhou, China · Hangzhou Lingju Intelligence AI Lab, Hangzhou, China · Hangzhou Dianzi University, HangZhou, China
Recent research demonstrates that pretrained large language models (LLMs) require minimal but carefully curated data for instruction-tuning. Building on LLMs, multi-modal models further contend with far larger data volumes, heterogeneous sources, and cross-modal alignment, amplifying both the necessity and the complexity of data selection. However, existing methods rely on deeply extracted embeddings, gradients or narrow quality scores that are time-consuming to compute and limited in expressiveness, frequently failing to surpass a simple random-sampling baseline. To overcome these limitations, we step back from complex embedding spaces and target the fundamental objective of instruction tuning: how and what knowledge should be elicited from the pretrained model and exposed to users. We leverage interaction Styles as a proxy for structural diversity, and decompose data quality into capability-specific Scores, yielding an explicit yet Rich (SSR) profile for each sample. We further cast the selection as submodular coverage maximization over the capability--style grid, providing an approximation guarantee. SSR consistently delivers superior performance across two distinct architecture stacks, LLaVA-1.5 (Vicuna + CLIP) and LLaVA-OneVision (Qwen2 + SigLIP), two large-scale data pools, six data budget settings, and 14 multi-modal benchmarks. To our knowledge, SSR is the first data selection method validated at million-scale, achieving 99.1% of full-data performance using only 30% of a 2.6M pool. Furthermore, the SSR profile models and selected subsets transfer training-free across data pools and model architectures with capability extensibility.
Mengyao Lyu, Yan Li, Huasong Zhong +4
Tsinghua University · BNRist · Bytedance (work as an intern)
Instruction fine-tuning is employed to enhance the instruction-following ability of large language models (LLMs). As the amount of instruction fine-tuning data increases, selecting the optimal core set becomes particularly important. However, ensuring the diversity of the core set remains a significant challenge. Existing methods predominantly distinguish different training data based on the text features themselves, decoupled from LLMs' own understanding and representation of the data. To address this issue, we propose a Model-Aware Diverse Core Set Selection method, which distinguishes data features based on the neural activation states during LLM inference. This approach serves as an efficient instantiation of coverage-based selection using model-intrinsic activation features to ensure the diversity in the core set. We extensively evaluate our method on six benchmarks that cover five distinct tasks. In our method, the core set selected by the 3B-parameter LLM performs effectively when utilized to fine-tune larger models with 7B, 8B, and 13B parameters. Experimental results on the Alpaca-GPT4 dataset, which comprises 52K instruction-response pairs, show that the core set, sized at 15% of the original dataset and selected by Llama-3.2-3B-Instruct, achieves an average improvement of 2.5% when fine-tuning four larger base models compared with training on the full dataset. The experimental results demonstrate that our method enhances model performance on multiple downstream tasks while reducing data requirements.
Yi Bai, Wenhao Zhang, Yao Chen +3
Shandong University, Qingdao, China · Inspurcloud, Jinan, China