GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry

Guanghui Min; Tianhao Huang; Ke Wan; Chen Chen

cs.LGFeb 20, 2026

GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry

Authors: Guanghui Min, Tianhao Huang, Ke Wan, Chen Chen

Organizations: Department of Computer Science, University of Virginia, Charlottesville, USA.

Abstract

Targeted data selection has emerged as a crucial paradigm for efficient instruction tuning, aiming to identify a small yet influential subset of training examples for a specific target task. In practice, influence is often measured through the effect of an example on parameter updates. To make selection scalable, many approaches leverage optimizer statistics (e.g., Adam states) as an axis-aligned surrogate for update geometry (i.e., diagonal precondition), implicitly treating parameters as coordinate-wise independent. We show that this assumption breaks down in parameter-efficient fine-tuning (PEFT) methods such as LoRA. In this setting, the induced optimization geometry exhibits strong cross-parameter coupling with non-trivial off-diagonal interactions, while the task-relevant update directions are confined to a low-dimensional subspace. Motivated by this mismatch, we propose GIST (Gradient Isometric Subspace Transformation), a simple yet principled alternative that replaces axis-aligned scaling with robust subspace alignment. GIST recovers a task-specific subspace from validation gradients via singular value decomposition (SVD), projects training gradients into this coupled subspace, and scores examples by their alignment with target directions. Extensive experiments have demonstrated that GIST matches or outperforms the state-of-the-art baseline with only 0.29% of the storage and 25% of the computational time under the same selection budget.

Figures & tables

Panel of Figure 1 — Figure 1: Spectral analysis of the MMLU validation gradient $\mathbf{G}_{\text{val}}$ on Llama2-7B. We decompose the gradient matrix via SVD to obtain singular values $\sigma_{i}$ . (a) Cumulative explained variance. A steeper curve indicates that a smaller principal subspace dimension is sufficient to capture the majority of the variance (e.g., Rank $150$ captures $95\%$ ), confirming high directional information density. (b) The singular values ( $\sigma_{k}$ ) exhibit precipitous decay, further verifying the intrinsic low-rank structure.

Figure 2: Overview of GIST . Step 1: Lightweight warmup performs a short LoRA warmup on a sampled subset and computes validation gradients. Step 2: Spectral filtering applies an SVD on the validation gradient matrix to construct a low-rank target subspace (Target projector). Step 3: Geometric scoring projects candidate gradients onto the target subspace and selects Top- $k$ samples.

Dataset	Shots	Tasks	Val.	Test	Output Format
			Data Size
MMLU	5	57	285	18,721	Multiple Choice
TyDiQA	1	9	9	1,713	Extractive Span
BBH	3	23	81	920	Generation (CoT)

Table 1: Statistics of evaluation datasets. We select tasks covering diverse output formats, including multiple-choice, extractive spans, and generative reasoning.

Model	Dataset	Base (0%)	Full (100%)	Rand.	Length	PPL.	Embed.	RDS+	LESS	GIST
Llama2-7B	MMLU	45.6	51.6	46.5 $\pm$ 0.5	50.5 $\pm$ 0.3	38.9 $\pm$ 0.9	47.3 $\pm$ 0.3	46.1 $\pm$ 0.6	50.2 $\pm$ 0.5	51.2 $\pm$ 0.7
	TydiQA	46.4	54	52.7 $\pm$ 0.4	51.1 $\pm$ 0.3	32.9 $\pm$ 0.4	49.8 $\pm$ 0.8	51.0 $\pm$ 0.3	56.2 $\pm$ 0.7	55.8 $\pm$ 0.6
	BBH	38.3	43.2	38.9 $\pm$ 0.5	39.8 $\pm$ 0.7	34.5 $\pm$ 0.2	38.6 $\pm$ 0.6	39.0 $\pm$ 0.7	41.5 $\pm$ 0.6	41.8 $\pm$ 0.8
	Avg. $\Delta$	–	+ 6.2	+2.6	+3.7	-8.0	+1.8	+1.9	+5.9	+6.2
Llama3.2-3B	MMLU	53.9	55.2	53.2 $\pm$ 0.6	57.6 $\pm$ 0.9	51.8 $\pm$ 0.3	53.9 $\pm$ 0.6	51.9 $\pm$ 0.2	56.5 $\pm$ 0.6	56.1 $\pm$ 0.4
	TydiQA	60.4	66.6	64.1 $\pm$ 0.4	63.9 $\pm$ 1.3	57.1 $\pm$ 0.7	62.8 $\pm$ 0.4	62.6 $\pm$ 0.5	67.1 $\pm$ 0.8	69.2 $\pm$ 0.3

Table 2: Accuracy across datasets and models. Base denotes 0% (no selection) and Full denotes 100% (full dataset). All other methods select 5% of the data under the same finetuning budget. Gray

\pm

values report standard deviations. Bold numbers denote the best selected subset in each row, and underlined numbers denote the second best selected subset. Avg.

\Delta

reports the average absolute improvement over Base over MMLU , TydiQA and BBH .

Panel of Figure 3 — Figure 3: Impact of Checkpoint Selection. (a) Using single-epoch gradients shows a clear performance drop in later epochs. (b) Aggregating multiple checkpoints (weighted) does not outperform the early-stop strategy, confirming that early gradients contain the essential task optimization directions.

	MMLU	TydiQA	BBH	Avg.
LESS	56.5 $\pm$ 0.6	67.1 $\pm$ 0.8	46.1 $\pm$ 0.7	56.6
$N=4$	56.3 $\pm$ 0.3	67.3 $\pm$ 0.1	46.8 $\pm$ 0.4	56.8
$N=1$ (default)	56.1 $\pm$ 0.4	69.2 $\pm$ 0.3	48.0 $\pm$ 0.2	57.8

Table 3: Number of checkpoints (

N

) used for select data with GIST for Llama3.2-3B.

Figure 4: Accuracy as a function of the projection rank used by GIST , compared with LESS at the same selection budget.

Method	Metric	Warmup	Target SVD	Grad. Feats.
LESS	Time	$6.0$ h	–	$48.0$ h
LESS	Compl.	$\mathcal{O}(\|\mathcal{D}^{\prime}\|\cdot N)$	–	$\mathcal{O}(\|\mathcal{D}\|\cdot N)$
GIST	Time	1.5 h	$<$ 1 m	12.0 h
GIST	Compl.	$\mathcal{O}(\|\mathcal{D}^{\prime}\|)$	$\mathcal{O}(\|\mathcal{D}_{\text{val}}\|^{2}d)$	$\mathcal{O}(\|\mathcal{D}\|)$

Table 4: Efficiency comparison between GIST (Ours) and LESS. Runtime is measured in single A100 GPU hours . GIST significantly reduces the overhead in Warmup and Feature Extraction stages (approx.

4\times

speedup) and requires negligible time for SVD.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Datasets and Models
$\mathcal{D}$	Large-scale instruction tuning candidate pool, $\mathcal{D}=\{\boldsymbol{z}_{i}\}_{i=1}^{\|\mathcal{D}\|}$
$\boldsymbol{z}$	A data instance $\boldsymbol{z}=(\boldsymbol{x},\boldsymbol{y})$
$\mathcal{D}_{\mathrm{val}}$	Target validation set defining the target distribution/task
$\mathcal{D}_{\mathrm{test}}$	Held-out test set for evaluation
$S$	Selected subset from the candidate pool, $S\subseteq\mathcal{D}$

Appendix

Table 5: Summary of notations used in this paper.

Panel of Figure 5 — Figure 5: Toy 2D optimization dynamics with the same initialization $\boldsymbol{\theta}_{0}=(-2.5,0)$ . Newton (full-matrix) follows the direct descent direction, while Adam (diagonal) cannot express the rotation induced by coupling, leading to a “zig-zag” trajectory on the coupled landscape.

Dataset	# Instance	Sourced from	# Rounds	Prompt Len.	Completion Len.
flan v2	100,000	NLP datasets and human-written instructions	1	355.7	31.2
cot	100,000	NLP datasets and human-written CoTs	1	266	53.2
dolly	15,011	Human-written from scratch	1	118.1	91.3
open assistant 1	55,668	Human-written from scratch	1.6	34.8	212.5

Appendix

Table 6: Datails of training dataset from Wang et al. (2023) . Len. is short for token length.

Panel of Figure 6 — Figure 6: Spectral Analysis of Gradient Subspaces across Models and Epochs. Top Row: Singular value spectra (log scale) of the gradient covariance matrix. Early epochs (blue solid lines) show slower decay, indicating a higher-dimensional optimization landscape. Bottom Row: Cumulative explained variance. Later epochs (dotted red lines) reach 95% variance with fewer components, signaling dimensional shrinkage. Notably, larger models (e.g., Llama2-7B) exhibit a slower spectral decay compared to smaller counterparts, reflecting a higher intrinsic dimensionality for task adaptation.

Panel of Figure 7 — Figure 7: Impact of Dataset Scale and Task Type on Gradient Geometry. We analyze the spectral properties of Llama3.2-3B gradients across three datasets with varying sizes. Top Row: Singular value spectra show that intrinsic dimension scales with data size. MMLU exhibits a smooth, heavy-tailed decay, whereas TydiQA suffers from extreme spectral sparsity due to data scarcity. Bottom Row: Cumulative explained variance. Note that across all scales, Epoch 1 (blue solid lines) consistently preserves a higher effective dimension (slower variance saturation) compared to later epochs (red dotted lines). This confirms that early-stage gradients are essential for preventing subspace collapse, especially in low-resource regimes like TydiQA .

Panel of Figure 8 — Figure 8: Rapid loss convergence in the first epoch creates a stable geometric basin. We observe consistent training dynamics across varying data scales (13.5k vs. 270k) and model architectures, where the loss drops precipitously within the initial phase ( $<0.5$ epoch) before entering a bounded oscillatory regime.

	Llama2-7B				Llama3.2-3B
	Base	GIST	GIST-T		Base	GIST	GIST-T
MMLU	45.6	51.2 $\pm$ 0.7	47.0 $\pm$ 0.4	MMLU	53.9	56.1 $\pm$ 0.4	54.5 $\pm$ 0.1
TydiQA	46.4	55.8 $\pm$ 0.6	52.7 $\pm$ 0.7	TydiQA	60.4	69.2 $\pm$ 0.3	65.0 $\pm$ 1.4
BBH	38.3	41.8 $\pm$ 0.8	41.2 $\pm$ 0.3	BBH	45.5	48.0 $\pm$ 0.5	46.8 $\pm$ 0.6

Appendix

Table 7: Main results with GIST and transfer GIST-T . GIST selects 5% of training data using the target model itself. GIST-T selects 5% data once using Qwen2.5-1.5B and then reuses the same selected subset to fine-tune other models.

Task	Base	GIST w/o warmup	GIST
MMLU	53.9	53.0 $\pm$ 0.2	56.1 $\pm$ 0.4
TydiQA	60.4	64.5 $\pm$ 1.7	69.2 $\pm$ 0.3
BBH	45.5	46.2 $\pm$ 0.4	48.0 $\pm$ 0.5

Appendix

Table 8: Warmup ablation on Llama3.2-3B under the same 5% selection budget. Removing warmup consistently degrades performance across tasks.

Dataset	Base	Random	Default Setting		Low-rank Ablation
Dataset	Base	Random	(LoRA $r{=}128$ )		(LoRA $r{=}8$ )
			LESS	GIST	LESS	GIST
MMLU	53.9	53.2 $\pm$ 0.6	56.5 $\pm$ 0.6	56.1 $\pm$ 0.4	56.4 $\pm$ 0.2	55.2 $\pm$ 0.3
TydiQA	60.4	64.1 $\pm$ 0.4	67.1 $\pm$ 0.8	69.2 $\pm$ 0.3	65.0 $\pm$ 0.9	67.5 $\pm$ 0.6
BBH	45.5	45.1 $\pm$ 0.2	46.1 $\pm$ 0.7	48.0 $\pm$ 0.5	46.1 $\pm$ 0.6	48.4 $\pm$ 0.3
Average	53.3	54.1	56.6	57.8	55.8	57.0

Appendix

Table 9: LoRA rank ablation on Llama3.2-3B (5% selection budget). We compare the robustness of GIST against LESS under different LoRA rank constraints. Our method ( GIST ) maintains significant gains even in the low-rank setting (

r{=}8

). Gray scripts denote standard deviations across 3 runs.

Task	Base	Random	GIST	GIST-tail
MMLU	53.9	53.2 $\pm$ 0.6	56.1 $\pm$ 0.4	56.4 $\pm$ 0.2
TydiQA	60.4	64.1 $\pm$ 0.4	69.2 $\pm$ 0.3	63.1 $\pm$ 1.5
BBH	45.5	45.1 $\pm$ 0.2	48.0 $\pm$ 0.5	38.5 $\pm$ 1.8

Appendix

Table 10: Tail ablation on Llama3.2-3B under the same 5% selection budget. While GIST consistently improves over Base and Random , selecting only the smallest principal direction ( GIST-tail ) can be unstable and may hurt task performance.

A Telugu QA Example in Validation Set
Multilingual QA in Telugu
PC1	PC2	PC3
PC1: Review Sentiment Classification User: Title: It’s a cute movie!! Review: It Takes Two is a cute movie that is loosely based on The Parent Trap but also seems loosely based on The Prince and the Pauper . Mary-Kate and Ashley are very good and so are Kirstie Alley and Steve Guttenberg. This movie is a lot of fun to watch and worth buying or renting. Question: Is the review positive or negative? Assistant: Positive	PC2: Multi-step Reasoning (MCQ) User: Given a multiple-choice question, reason step by step and output the selected option. Example: Question: What describes something that requires activation energy to get started? Options: (A) igniting fuel and oxidiser …(G) chemical equations …(H) Firecrackers Let’s think first: All chemical reactions require activation energy to get started. Chemical equations describe chemical reactions. … So the answer is [(G)]. … (additional questions omitted) Assistant: … So the answer is [(H)] .	PC3: Common-sense Contradiction User: See question below:Choose the sentence that goes against common sense.Options:- Sentence A: “she’s asleep, we should keep noisy”- Sentence B: “she’s asleep, we should keep quiet” Reason slowly and give your answer. Assistant: If you are loud when someone is asleep you will wake them up.The final answer: Sentence A.
PC4	PC5	PC6
PC4: Word Game (Scrabble) User: I’m playing Scrabble with my family. My seven letters are: A P N W Q V E. What words can I spell with those? Assistant: APE, AWE, ANEW, PAW, PEW, PAN, PEN, PEA, PAVE, PAWN, PANE, PEAN, NAP, NEW, NAVE, NAPE, VAN, VAPE, EW	PC5: Multilingual Mixed Task (Spanish) User: ¿Que sabes sobre el integrador de Mulesoft? ¿Es mejor plataforma de integración que SAP PI? Assistant: MuleSoft is an integration software company that provides the Anypoint platform to connect applications, data, and devices. SAP PI is SAP’s tool for centralizing information exchange between SAP and non-SAP systems. They have similar goals but different focuses; the choice depends on the client’s needs and what they want to develop.	PC6: Multilingual Reasoning User: Given the following reasoning and answer, what was the question? Reasoning: Tom served on Monday to $8\times 10=80$ clients. So Tom earned $20/100\times 80=16$ bonus points. Answer: 16 Assistant: In a company, an employee earns bonus points equal to 20% of the number of customers served. Tom served 10 customers per hour for 8 hours on Monday. How many bonus points did Tom earn?
PC7	PC8	PC9

Appendix

Table 11: We use one Telugu TydiQA validation instance to retrieve the most relevant training examples under GIST . Target Projector (nine principal directions) is precomputed via SVD on the original target set and reused here. For each direction, candidate examples are scored by their projection-based alignment, and the highest-scoring examples are retrieved. We then aggregate the nine directional scores to obtain a single overall score per example, and report the Top-5 examples with the largest aggregated scores.

Instruction-Tuning Data Selection

Instruction-tuning data selection chooses and organizes training examples to improve model behavior while reducing fine-tuning cost.

See all 45 papers

GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection
Mar 10, 2026Kai Yao, Zhenghan Song, Kaixin Wu +5Sparse Fine-Tuning Selective Fine-Tuning
1Ant Group · 2Cornell University · University of Liverpool +2
A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Feb 16, 2026Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund +2Instruction-Tuning Data Selection Training Data Selection
Harvard University · MIT · Kempner Institute
Let the Target Select for Itself: Data Selection via Target-Aligned Paths
May 10, 2026Huitao Yang, Hengzhi He, Tung Sum Thomas Kwok +1Data Selection Training Data Selection
University of California, Los Angeles

Mar 10, 2026J/K move · Enter open · S save

GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection

Kai Yao, Zhenghan Song, Kaixin Wu, Mingjie Zhong+4

1Ant Group · 2Cornell University · University of Liverpool+2

Sparse Fine-Tuning Selective Fine-Tuning Parameter-Efficient Fine-Tuning Training Data Selection

Open paper AskPDF arXiv

Parameter-Efficient Fine-Tuning (PEFT) has become a key strategy for adapting large language models, with recent advances in sparse tuning reducing overhead by selectively updating key parameters or subsets of data. Existing approaches generally focus on two distinct paradigms: layer-selective methods aiming to fine-tune critical layers to minimize computational load, and data-selective methods aiming to select effective training subsets to boost training. However, current methods typically overlook the fact that different data points contribute varying degrees to distinct model layers, and they often discard potentially valuable information from data perceived as of low quality. To address these limitations, we propose Gradient-aligned Sparse Tuning (GAST), an innovative method that simultaneously performs selective fine-tuning at both data and layer dimensions as integral components of a unified optimization strategy. GAST specifically targets redundancy in information by employing a layer-sparse strategy that adaptively selects the most impactful data points for each layer, providing a more comprehensive and sophisticated solution than approaches restricted to a single dimension. Experiments demonstrate that GAST consistently outperforms baseline methods, establishing a promising direction for future research in PEFT strategies.

Abstract

Instruction-Tuning Data Selection

Explore similar work

GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

Let the Target Select for Itself: Data Selection via Target-Aligned Paths