Follow the Features: SAE Feature-Guided Data-Efficient Instruction-Tuning of Large Language Models
Organizations: Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, CA, USA.
Abstract
Selecting small, effective instruction-tuning datasets requires identifying which aspects of the original data to retain. We introduce Follow the Features (FtF), a data-selection method that uses sparse autoencoder (SAE) features to characterize training data and guide subset construction. FtF ranks features using dataset-level activation statistics, scores samples by their expression of highly ranked features, and incorporates feature resampling to encourage diversity. The LLM-SAE pair used for selection is independent of the model being fine-tuned, enabling reuse across target models. We evaluate FtF on two model-dataset settings at a training budget of 100 samples with a fixed sequence length of 256 tokens. FtF achieves performance competitive with GRAPE and S2L, with relative rankings varying across settings. On Qwen 3 8B trained on Alpaca, FtF exceeds full-dataset training by 1.59 percentage points in the harmonic mean of an AlpacaEval-based win rate and Open LLM Leaderboard performance, although full-dataset training retains a higher Leaderboard score. These results support SAE feature statistics as a useful signal for data-efficient instruction tuning.
Figures & tables
| Model | AlpacaEval | Leaderboard | HMean |
|---|---|---|---|
| Ref ( All ) | 0.31 | 26.85 | 0.61 |
| Ref (100) | 0.00 | 24.13 | 0.00 |
| GRAPE | 0.28 | 23.85 | 0.55 |
| S2L | 0.55 | 23.99 | 1.07 |
| FtF | 0.95 | 23.70 | 1.83 |
| Model | AlpacaEval | Leaderboard | HMean |
|---|---|---|---|
| Ref ( All ) | 12.16 | 42.58 | 18.92 |
| Ref (100) | 12.00 | 40.80 | 18.55 |
| GRAPE | 15.60 | 41.10 | 22.61 |
| S2L | 14.77 | 41.07 | 21.73 |
| FtF | 13.66 | 41.17 | 20.51 |
| Rank | Index | Description |
|---|---|---|
| 1 | 1198 | possessive ’s |
| 2 | 480 | numbers after quotes |
| 3 | 329 | not available |
| 4 | 1067 | quotation marks followed by numbers |
| Model | AlpacaEval | Leaderboard | HMean |
|---|---|---|---|
| Ref (constant) | 1.92 | 30.60 | 3.62 |
| Constant | 2.55 | 32.28 | 4.74 |
| Ref (cosine) | 1.45 | 31.50 | 2.77 |
| Cosine | 2.24 | 31.50 | 4.18 |
| Model | AlpacaEval | Leaderboard | HMean |
|---|---|---|---|
| Reference | 1.45 | 31.50 | 2.77 |
| 270M PT | 2.09 | 30.53 | 3.90 |
| 12B IT | 2.24 | 31.50 | 4.18 |
| Model | AlpacaEval | Leaderboard | HMean |
|---|---|---|---|
| Reference | 1.45 | 31.50 | 2.77 |
| Mean | 1.96 | 31.46 | 3.68 |
| LT | 2.24 | 31.50 | 4.18 |
| Model | AlpacaEval | Leaderboard | HMean |
|---|---|---|---|
| Reference | 1.45 | 31.50 | 2.77 |
| No Div | 1.96 | 31.44 | 3.68 |
| Div | 2.24 | 31.50 | 4.18 |
| Model | AlpacaEval | Leaderboard | HMean |
|---|---|---|---|
| Reference | 1.45 | 31.50 | 2.77 |
| No Ent | 1.96 | 31.46 | 3.68 |
| Ent | 2.24 | 31.50 | 4.18 |
| Spearman’s | Union size | Fraction shared | |
|---|---|---|---|
| 5 | 5 | 100.0% | |
| 10 | 10 | 100.0% | |
| 50 | 67 | 66.0% | |
| 100 | 155 | 45.0% | |
| 200 | 347 | 26.5% | |
| 500 | 793 | 41.4% |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | AlpacaEval | Leaderboard | HMean |
|---|---|---|---|
| Baseline | 0.79 | 28.88 | 1.55 |
| Ref (All) | 2.45 | 32.29 | 4.55 |
| Dataset size: 100 | |||
| Reference | 1.45 | 31.50 | 2.77 |
| 2.93 | 31.31 | 5.36 | |
| 2.93 | 31.31 | 5.36 | |
| Model | BBH | GPQA | IFEval | MATH-Hard | MMLU-Pro | MuSR |
|---|---|---|---|---|---|---|
| Baseline | 42.27 | 24.92 | 26.14 | 16.16 | 31.37 | 32.41 |
| Ref (All) | 45.39 | 27.29 | 41.41 | 12.99 | 29.86 | 36.82 |
| Dataset size: 100 | ||||||
| Reference | 43.03 | 27.01 | 35.01 | 15.11 | 31.28 | 37.57 |
| 43.13 | 28.52 | 35.01 | 15.26 | 31.40 | 34.52 | |
| 43.13 | 28.52 | 35.01 | 15.26 | 31.40 | 34.52 | |
| Model | AlpacaEval | Leaderboard | HMean |
|---|---|---|---|
| Baseline | 0.79 | 28.88 | 1.55 |
| Ref (All) | 2.45 | 32.29 | 4.55 |
| Dataset size: 100 | |||
| Reference | 1.45 | 31.50 | 2.77 |
| 2.23 | 31.50 | 4.17 | |
| 2.24 | 31.50 | 4.18 | |
| Model | BBH | GPQA | IFEval | MATH-Hard | MMLU-Pro | MuSR |
|---|---|---|---|---|---|---|
| Baseline | 42.27 | 24.92 | 26.14 | 16.16 | 31.37 | 32.41 |
| Ref (All) | 45.39 | 27.29 | 41.41 | 12.99 | 29.86 | 36.82 |
| Dataset size: 100 | ||||||
| Reference | 43.03 | 27.01 | 35.01 | 15.11 | 31.28 | 37.57 |
| 43.29 | 28.33 | 35.77 | 15.06 | 31.42 | 35.14 | |
| 43.39 | 28.41 | 35.73 | 14.95 | 31.44 | 35.10 | |
| Parameter | Value |
|---|---|
| Epochs | 1 |
| Sequence length | 256 |
| Per-device batch size | 32 |
| Optimizer | adamw_torch_fused |
| Learning rate | |
| Scheduler | Cosine |
| Parameter | Value |
|---|---|
| Rank | 16 |
| LoRA | 32 |
| LoRA dropout | 0.05 |
| Bias | none |
| Target modules | q_proj, k_proj , |
| v_proj, o_proj , |
| Parameter | Value |
|---|---|
| Judge model | Qwen 3.8 27B FP8 |
| Dataset | alpaca_eval_gpt4_baseline |
| Dataset source | tatsu-lab/alpaca_eval |
| AlpacaEval samples | 300 |
| Leaderboard samples | all |
| Batch size | 4 (1 for Full FT) |
| Model | BBH | GPQA | IFEval | MATH-Hard | MMLU-Pro | MuSR |
|---|---|---|---|---|---|---|
| Ref ( All ) | 39.36 | 27.63 | 29.62 | 3.70 | 23.85 | 36.95 |
| Ref (100) | 37.85 | 25.06 | 26.30 | 6.29 | 13.46 | 35.85 |
| GRAPE | 37.51 | 25.03 | 26.54 | 6.02 | 12.84 | 35.14 |
| S2L | 37.57 | 24.97 | 26.30 | 5.87 | 13.82 | 35.41 |
| FtF | 37.10 | 25.06 | 26.10 | 5.54 | 12.75 | 35.67 |
| Model | BBH | GPQA | IFEval | MATH-Hard | MMLU-Pro | MuSR |
|---|---|---|---|---|---|---|
| Ref ( All ) | 58.77 | 35.23 | 47.16 | 24.90 | 47.02 | 42.42 |
| Ref (100) | 56.55 | 29.53 | 49.00 | 24.30 | 47.39 | 38.05 |
| GRAPE | 56.56 | 29.64 | 49.56 | 25.13 | 47.49 | 38.18 |
| S2L | 56.78 | 29.81 | 49.00 | 24.80 | 47.55 | 38.49 |
| FtF | 56.51 | 29.53 | 49.52 | 24.82 | 47.70 | 38.93 |
| Model | BBH | GPQA | IFEval | MATH-Hard | MMLU-Pro | MuSR |
|---|---|---|---|---|---|---|
| Reference | 43.03 | 27.01 | 35.01 | 15.11 | 31.28 | 37.57 |
| Mean | 43.23 | 28.33 | 35.49 | 14.90 | 31.41 | 35.36 |
| LT | 43.39 | 28.41 | 35.73 | 14.95 | 31.44 | 35.10 |
| Model | BBH | GPQA | IFEval | MATH-Hard | MMLU-Pro | MuSR |
|---|---|---|---|---|---|---|
| Reference | 43.03 | 27.01 | 35.01 | 15.11 | 31.28 | 37.57 |
| 270M PT | 42.83 | 25.50 | 33.33 | 14.12 | 31.48 | 35.93 |
| 12B IT | 43.39 | 28.41 | 35.73 | 14.95 | 31.44 | 35.10 |
| Model | BBH | GPQA | IFEval | MATH-Hard | MMLU-Pro | MuSR |
|---|---|---|---|---|---|---|
| Reference | 43.03 | 27.01 | 35.01 | 15.11 | 31.28 | 37.57 |
| No Div | 43.23 | 28.33 | 35.41 | 14.90 | 31.41 | 35.36 |
| Div | 43.39 | 28.41 | 35.73 | 14.95 | 31.44 | 35.10 |
| Model | BBH | GPQA | IFEval | MATH-Hard | MMLU-Pro | MuSR |
|---|---|---|---|---|---|---|
| Reference | 43.03 | 27.01 | 35.01 | 15.11 | 31.28 | 37.57 |
| No Ent | 43.23 | 28.33 | 35.53 | 14.90 | 31.41 | 35.36 |
| Ent | 43.39 | 28.41 | 35.73 | 14.95 | 31.44 | 35.10 |