Hidden Cartographer: Hidden State Guided Data Selection for Dual Budgeted Instruction Tuning

Zixuan Qin; Qi Qin; Erbo Li; Ting Wei; Yifan Sun; Zhaoxin Fan

cs.CLOct 18, 2026

Hidden Cartographer: Hidden State Guided Data Selection for Dual Budgeted Instruction Tuning

Authors: Zixuan Qin, Qi Qin, Erbo Li, Ting Wei, Yifan Sun, Zhaoxin Fan

Organizations: Center for Applied Statistics, School of Statistics, Renmin University of China · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing

Abstract

Given the high cost of expert annotation and limited fine-tuning resources, effective data selection from large unlabeled pools under strict budgets has become a crucial step in adapting LLMs to specialized domains. Existing label-dependent methods risk distorted selection when using pseudo-labels, while traditional active learning methods do not explicitly characterize instruction-tuning dynamics. To address these limitations, we propose Hidden Cartographer (HC), a two-stage framework that uses hidden states to guide annotation and exact training dynamics to select the final training subset. The first stage is Progressive Candidate Expansion, which utilizes the proxy to iteratively select top unlabeled candidates for expert annotation, merging them to recompute training dynamics and update the proxy until the annotation budget is exhausted; the second stage is Exact Cartography Pruning, which ranks these annotated candidates by their exact dynamics scores to isolate a compact subset under the final training budget. Experiments across the medical, legal, and financial domains demonstrate that HC achieves the highest average accuracy among all budgeted baselines, requiring annotation for only 20--30% of the original pool and fine-tuning on just 10--15%.

Figures & tables

Figure 1: Training-data value and selection distributions in the medical domain. All panels use the same Dataset Cartography space, with confidence on the x-axis and variability on the y-axis. Top left: Four regional subsets are constructed by sampling 15% of the full training pool from each region. Darker blue indicates better performance on medical benchmarks after fine-tuning on the corresponding subset. Top middle: The best-performing region among the four, identified as the region containing high-value samples. Other panels: Distributions of samples selected by HC, Random, SPICE, and Coreset at the same 15% selection ratio.

Figure 2: Overview of the Hidden Cartographer framework.

Benchmark	SPICE	Fisher	LESS	LEAD
MedQA	33.8 / 35.8	32.9 / 35.1	28.6 / 36.4	18.4 / 36.7
MedQA-4-opt	39.2 / 41.3	38.1 / 40.5	29.4 / 40.8	29.3 / 42.4
PubMedQA	74.8 / 75.0	75.8 / 75.2	60.2 / 74.4	76.4 / 73.8
MedMCQA	38.2 / 41.8	38.2 / 42.0	35.6 / 43.8	25.6 / 44.8
MMLU-Med	53.9 / 53.7	54.2 / 53.2	44.6 / 51.6	20.5 / 53.8
Avg.	48.0 / 49.5	47.8 / 49.2	39.7 / 49.4	34.0 / 50.3

Table 1: Pseudo-label vs. oracle-label selection in the medical domain. For each method, we select 15% of the training pool and fine-tune the model under the same setting. Pseudo uses model-generated labels to compute selection signals, while Oracle uses ground-truth labels. Each cell reports Pseudo / Oracle accuracy, and the underline indicates the better result within each pair.

Method	Medical Domain (30%-15%)						Legal Domain (20%-10%)				Financial Domain (30%-15%)
Method	MQA	MQA4	PMQA	MMCQA	MMLU	Avg.	Exam	LED	LCC	Avg.	FPB	FIQA	NWGI	Avg.
Base	22.07	29.85	55.20	30.31	37.78	35.04	25.60	40.80	31.20	32.53	19.69	54.81	35.89	36.80
Full (100%)	39.51	44.93	78.00	46.45	56.61	53.10	24.20	44.40	34.00	34.20	38.13	83.68	54.11	58.64
Random	35.21	39.71	75.08	41.87	51.30	48.63	21.72	42.80	41.76	35.43	78.46	80.67	64.56	74.56
SPICE	34.17	40.30	75.60	40.86	55.03	49.19	22.40	42.40	40.80	35.20	65.76	83.26	61.35	70.12
Fisher	34.49	39.75	75.40	41.19	55.24	49.21	22.20	42.20	41.00	35.13	67.41	81.59	32.31	60.44

Table 2: Main experimental results (Acc %) across medical, legal, and financial domains. Full denotes training on the entire available dataset. For budgeted selection methods, the notation

(B\%-K\%)

denotes first constructing a candidate pool under a

B\%

annotation budget and then selecting a

K\%

final training subset. The best method in each column is highlighted in bold , and the second best is underlined . For stochastic methods, we report the mean over random seeds 42, 142, 242, 342, and 442. The

t

-tests were conducted on the Average results, where ‡ indicates

p<0.01

against the Random selection baseline.

Figure 3: Runtime–performance comparison on the medical domain. Each point corresponds to one data-selection method. The $x$ -axis reports the time cost over the methods shown in this figure, $x=(t-t_{\min})/(t_{\max}-t_{\min})$ . The $y$ -axis reports the average accuracy over five medical benchmarks. The dashed horizontal line denotes the accuracy of random. Methods whose average score is below $47$ are clipped onto the $x$ -axis rather than plotted at their true $y$ -values.

Evaluator	Med.		Law		Fin.
Evaluator	O@10	J@10	O@10	J@10	O@10	J@10
Human Expert	0.70	0.54	0.60	0.43	0.80	0.67
GPT-5.6	0.40	0.25	0.70	0.54	0.50	0.33
Gemini 3.6	0.50	0.33	0.70	0.54	0.60	0.43
Grok 4.5	0.60	0.43	0.30	0.18	0.90	0.82
Avg.	0.55	0.39	0.58	0.42	0.70	0.56

Table 3: Human and LLM judge study on high-value sample selection. For each domain, 100 candidate training samples are randomly sampled, and each evaluator selects the 10 samples considered most valuable for instruction tuning. Med., Law, and Fin. denote the medical, legal, and financial domains. O@10 denotes overlap with HC’s top-10 selection, computed as

|\mathcal{S}_{\mathrm{HC}}\cap\mathcal{S}_{\mathrm{Eval}}|/10

, and J@10 denotes Jaccard similarity, computed as

|\mathcal{S}_{\mathrm{HC}}\cap\mathcal{S}_{\mathrm{Eval}}|/|\mathcal{S}_{\mathrm{HC}}\cup\mathcal{S}_{\mathrm{Eval}}|

Variant	MedQA	MQA4	PQA	MMCQA	MMLU	Avg.
R + E	37.94	42.97	74.80	42.86	56.51	51.02
S + E	35.04	39.36	75.00	38.08	50.05	47.51
C + E	37.55	43.05	76.80	42.51	55.66	51.11
D + E	38.10	42.42	76.20	42.77	55.34	50.97
P (15%)	37.08	41.60	74.20	42.15	54.78	49.96
P (30%)	38.26	43.99	75.20	44.32	56.19	51.59

Table 4: Component ablation of HC in the medical domain. P denotes PCE, E denotes ECP, R denotes Random, S denotes SPICE, C denotes Coreset, and D denotes DPP. PCE (15%) directly selects a 15% subset in Phase 1 using proxy-predicted scores, while PCE (30%) trains on the full 30% Phase-1 candidate pool without final pruning. Other variants replace either the candidate construction or the final pruning module.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Pearson correlation between true and predicted dynamics as a function of the labeled seed size. Each panel is one domain; the two curves are confidence (

C

) and variability (

V

Method	Medical Domain (30%-15%)						Legal Domain (20%-10%)				Financial Domain (30%-15%)
Method	MQA	MQA4	PMQA	MMCQA	MMLU	Avg.	Exam	LED	LCC	Avg.	FPB	FIQA	NWGI	Avg.
Base	49.41	55.46	37.00	53.67	71.53	53.41	49.40	41.80	28.60	39.93	76.92	64.85	64.93	68.90
Full (100%)	54.67	59.86	78.80	58.62	72.70	64.93	47.60	43.20	71.60	54.13	86.44	87.45	65.97	79.95
Random	54.12	56.23	76.20	55.52	73.12	63.04	48.00	42.80	69.60	53.47	81.47	82.01	65.01	76.16
SPICE	52.95	50.21	75.20	49.92	71.85	60.03	48.60	43.00	48.40	46.67	64.68	76.57	62.12	67.79
Fisher	53.26	53.52	75.40	48.52	71.75	60.49	48.00	43.00	47.40	46.13	62.78	77.41	61.87	67.35

Appendix

Table 5: Qwen2-7B results (Accuracy %) across medical, legal, and financial domains. Full denotes training on the entire available dataset. For budgeted methods,

(B\%-K\%)

denotes constructing a

B\%

candidate pool and selecting a

K\%

final training subset. The best method in each column is highlighted in bold , and the second best is underlined . The

t

-tests were conducted on the Average results, where ‡ indicates

p<0.01

against Random.

(a) Replace hidden-state features
Method	Cov. $\uparrow$	Dens. $\uparrow$	Mean $\uparrow$
Text + Metadata	56.03	28.02	0.4223
Sentence Embedding	55.26	27.63	0.4213
(b) Vary active expansion rounds
1 Round (Direct Proxy)	61.58	30.79	0.4280
10 Rounds	68.26	34.13	0.4321

Appendix

Table 6: Ablation of feature sources and expansion rounds on the medical domain. Block (a) replaces HC’s hidden-state features, while Block (b) varies the number of active rounds under the same 30% candidate budget.

Variant	FPB	FiQA	NWGI	Avg.
Random + ECP	81.37	79.74	65.28	75.46
SPICE + ECP	82.80	80.75	60.10	74.55
Coreset + ECP	83.46	80.75	66.05	76.75
DPP + ECP	83.54	78.52	64.32	75.46
PCE (15%)	83.37	83.26	65.75	77.46
PCE (30%)	84.70	86.61	65.51	78.94

Appendix

Table 7: Component ablation of HC in the financial domain. PCE (15%) directly selects a 15% subset in Phase 1 using proxy-predicted scores, while PCE (30%) trains on the full 30% Phase-1 candidate pool without final pruning. Other variants replace either the candidate construction or the final pruning module.

Variant	Exam	LED	LCC	Avg.
Random + ECP	21.40	43.20	65.40	43.33
SPICE + ECP	23.00	42.80	36.00	33.93
Coreset + ECP	23.40	40.40	55.80	39.87
DPP + ECP	21.20	43.40	56.60	40.40
PCE (10%)	25.40	42.80	62.20	43.47
PCE (20%)	23.40	43.20	57.20	41.27

Appendix

Table 8: Component ablation of HC in the legal domain. PCE (10%) directly selects a 10% subset in Phase 1 using proxy-predicted scores, while PCE (20%) trains on the full 20% Phase-1 candidate pool without final pruning. Other variants replace either the candidate construction or the final pruning module.

Figure 5: Sensitivity analysis of HC scoring parameters on the medical domain. The top row shows the selected regions under different scoring parameters in the confidence-variability space. Gray points denote the full training pool, and colored points denote the selected subset. The bottom row reports the corresponding average downstream performance.

Figure 6: Performance under different top-score training ratios in medicine, law, and finance. The x-axis shows the percentage of the full training pool retained for fine-tuning. Dashed vertical lines mark the final training ratios used in the main experiments: 15% for medicine and finance, and 10% for law.

Training Data Selection

Training data selection studies how to choose, filter, weight, or schedule examples to improve learning while reducing data and compute costs.

See all 186 papers

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Feb 16, 2026Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund +2Instruction-Tuning Data Selection Training Data Selection
Harvard University · MIT · Kempner Institute
HARP: Efficient Data Selection for Finetuning Large Language Models
Jun 5, 2026Ning Wang, Zhengxin Zhang, Maosen Tang +3LLM Fine-Tuning Active Learning
Cornell University · The Hong Kong University of Science and Technology
The Long-Term Effects of Data Selection in LLM Fine-Tuning
May 28, 2026Yuxin Yang, Aoxiong Zeng, Xiangquan YangContinual Learning for LLMs LLM Fine-Tuning
Shanghai University · East China Normal University

Feb 16, 2026J/K move · Enter open · S save

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund, Sara Beery+1

Harvard University · MIT · Kempner Institute

Instruction-Tuning Data Selection Training Data Selection Data Selection for Language Model Fine-Tuning LLM Fine-Tuning

Open paper AskPDF arXiv

Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque: methods vary widely in selection budgets, often omit zero-shot baselines, and frequently entangle the contributions of key components. As a result, practitioners lack actionable guidance on selecting instructions for their target tasks. In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms. Our framework enables controlled comparisons across models, tasks, and budgets. We find that only gradient-based data representations choose subsets whose similarity to the query consistently predicts performance across datasets, models, and candidate pools. While no single method dominates, gradient-based representations paired with greedy round-robin selection often perform best on average at low budgets, but these gains diminish at larger budgets. Finally, we unify several existing selection algorithms as forms of approximate distance minimization between the selected subset and the query set, and support this view with new generalization bounds. More broadly, our findings provide critical insights and a foundation for more principled data selection in LLM fine-tuning. The code is available at https://github.com/dcml-lab/targeted-instruction-selection.

Abstract

Training Data Selection

Explore similar work

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

HARP: Efficient Data Selection for Finetuning Large Language Models

The Long-Term Effects of Data Selection in LLM Fine-Tuning