cs.LGNov 2, 2026

Follow the Features: SAE Feature-Guided Data-Efficient Instruction-Tuning of Large Language Models

Authors: Ryan Swift, Konstantinos Psounis

Organizations: Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, CA, USA.

Abstract

Selecting small, effective instruction-tuning datasets requires identifying which aspects of the original data to retain. We introduce Follow the Features (FtF), a data-selection method that uses sparse autoencoder (SAE) features to characterize training data and guide subset construction. FtF ranks features using dataset-level activation statistics, scores samples by their expression of highly ranked features, and incorporates feature resampling to encourage diversity. The LLM-SAE pair used for selection is independent of the model being fine-tuned, enabling reuse across target models. We evaluate FtF on two model-dataset settings at a training budget of 100 samples with a fixed sequence length of 256 tokens. FtF achieves performance competitive with GRAPE and S2L, with relative rankings varying across settings. On Qwen 3 8B trained on Alpaca, FtF exceeds full-dataset training by 1.59 percentage points in the harmonic mean of an AlpacaEval-based win rate and Open LLM Leaderboard performance, although full-dataset training retains a higher Leaderboard score. These results support SAE feature statistics as a useful signal for data-efficient instruction tuning.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SLAP: Stratified Loss-based Pruning for On-Policy Data-Efficient Instruction Tuning

    May 13, 2026Run Zou, Jianhang Ding, Yifan Ding +3LLM Fine-TuningInstruction-Tuning Data Selection

  2. An Embarrassingly Simple Data Selection Strategy for Million-Scale Multi-Modal Instruction-Tuning

    Mar 17, 2025Mengyao Lyu, Yan Li, Huasong Zhong +4Instruction-Tuning Data SelectionMultimodal Instruction Tuning

  3. MADS: Model-Aware Diverse Core Set Selection for Instruction Tuning

    May 29, 2026Yi Bai, Wenhao Zhang, Yao Chen +3Instruction-Tuning Data SelectionTraining Data Selection