Instruction-Tuning Data Selection

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 20

All topics

A helps B while B hurts A: directed transfer in instruction-tuning mixture
Sep 30, 2026Nima H. Siboni, Vahid RostamiLLM Fine-Tuning Instruction-Tuning Data Selection
Juna.ai, Kastanienallee 32, 10435 Berlin, Germany · Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Germany
How code helps different tasks? A decompositional lens on LLM post-training
Sep 27, 2026Zheng Yu, Yiwei Li, Yishen Chen +4LLM Fine-Tuning Instruction-Tuning Data Selection
Sun Yat-sen University · Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen +1
Form Over Content In Gradient-Based Data Attribution Methods
Sep 17, 2026Sunwoo Kim, Seokwon Jung, Sohyung Kim +2Gradient-Based Attribution Instruction-Tuning Data Selection
KAIST
Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuning
Sep 14, 2026Hyunjin Kim, Youngeun Nam, Jaemin Han +2LLM Fine-Tuning Instruction-Tuning Data Selection
KAIST
DICS: Exploring Data Intrinsic Consistency for Visual Instruction Selection
Aug 31, 2026Yuyang Hong, Jinhui Guo, Jiaqi Gu +6Instruction-Tuning Data Selection Training Data Selection
School of Artificial Intelligence, University of Chinese Academy of Sciences · 2MAIS, Institute of Automation, Chinese Academy of Sciences · 3Alibaba Token Hub, Alibaba Group
Agentic Instruction Data Selection: Let DataMaster Interpret Your Intent
Aug 11, 2026Fanqi Zhou, Qiaosheng Chen, Zixian Huang +1Training Data Curation Instruction-Tuning Data Selection
State Key Laboratory of Novel Software Technology, Nanjing University · Shanghai Artificial Intelligence Laboratory
Online Data Selection Is Implicit Alignment
Jul 8, 2026Aoxiong Zeng, Yuxin Yang, Xiangquan YangSupervised Fine-Tuning LLM Alignment
East China Normal University · Shanghai University
Online Data Selection for Instruction Tuning via Gaussian Processes
Jun 29, 2026Jun Wang, Quoc Phong Nguyen, Julien Monteil +1Instruction-Tuning Data Selection Instruction Tuning
Amazon
DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuning
May 29, 2026Junbo Zhang, Qianli Zhou, Xinyang Deng +3LLM Fine-Tuning Instruction-Tuning Data Selection
MADS: Model-Aware Diverse Core Set Selection for Instruction Tuning
May 29, 2026Yi Bai, Wenhao Zhang, Yao Chen +3Instruction-Tuning Data Selection Training Data Selection
Shandong University, Qingdao, China · Inspurcloud, Jinan, China
OnceSelect: Reusable Data Selection for Efficient Multimodal Instruction Tuning
May 26, 2026Mingkang Dong, Muxin Pu, Hongyi Cai +5Instruction-Tuning Data Selection Multimodal Large Language Models
Faculty of Computer Science Universiti Malaya Kuala Lumpur, Malaysia · School of Information Technology Monash University Malaysia Subang Jaya, Selangor, Malaysia · Faculty of Computer Science University of Science and Technology Beijing Beijing, China
SEED: Targeted Data Selection by Weighted Independent Set
May 15, 2026Yuan Zhang, Lifeng Guo, Junwen Pan +5Training Data Curation Instruction-Tuning Data Selection
School of Computer Science, Peking University · Beijing University of Posts and Telecommunications · Tianjin University +2
Interaction-Aware Influence Functions for Group Attribution
May 15, 2026Jaeseung Heo, Kyeongheung Yun, Youngbin Choi +3Instruction-Tuning Data Selection Influence Functions
GSAI, POSTECH · CSE, POSTECH
SLAP: Stratified Loss-based Pruning for On-Policy Data-Efficient Instruction Tuning
May 13, 2026Run Zou, Jianhang Ding, Yifan Ding +3LLM Fine-Tuning Instruction-Tuning Data Selection
Alibaba International Digital Commerce Group, HangZhou, China · Hangzhou Lingju Intelligence AI Lab, Hangzhou, China · Hangzhou Dianzi University, HangZhou, China
From Instance Selection to Fixed-Pool Data Recipe Search for Supervised Fine-Tuning
May 13, 2026Haodong Wu, Jiahao Zhang, Lijie Hu +1Supervised Fine-Tuning Training Data Curation
The Hong Kong University of Science and Technology (Guangzhou) · Mohamed bin Zayed University of Artificial Intelligence
Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxies
May 10, 2026Jingze Song, Zihao Chen, Wenqing Chen +1LLM Fine-Tuning Instruction-Tuning Data Selection
School of Software Engineering, Sun Yat-sen University
What Makes Good Instruction-Tuning Data? An In-Context Learning Perspective
Apr 28, 2026Guangzeng Han, Xiaolei HuangInstruction-Tuning Data Selection Training Data Selection
University of Memphis
Aligning the Query Space: Greedy Information Projection for Language Model Data Selection
Mar 14, 2026Victor Ye Dong, Kuan-Yun Lee, Jiamei Shuai +3LLM Fine-Tuning Instruction-Tuning Data Selection
Microsoft
Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning
Mar 12, 2026Rujie Wu, Haozhe Zhao, Hai Ci +1Video Understanding Instruction-Tuning Data Selection
School of Computer Science, Peking University, Beijing, China · Institute for Artificial Intelligence, Peking University, Beijing, China · State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, China +2
QAQ: Bidirectional Semantic Coherence for Selecting High-Quality Synthetic Code Instructions
Mar 12, 2026Jiayin Lei, Ming Ma, Yunxi Duan +2Instruction-Tuning Data Selection Code Generation
Beijing University of Technology, Beijing, 100124, China · Institute of Neuroscience, State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, 200031, China · University of Chinese Academy of Sciences, Beijing, 100049, China +1

Sep 30, 2026, cs.AIJ/K move · Enter open · S save

A helps B while B hurts A: directed transfer in instruction-tuning mixture

Nima H. Siboni, Vahid Rostami

Juna.ai, Kastanienallee 32, 10435 Berlin, Germany · Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Germany

LLM Fine-Tuning Instruction-Tuning Data Selection Transfer Learning Instruction Tuning Negative Transfer

Open paper AskPDF arXiv

Adapting a language model to a specialized corpus means choosing which instruction-tuning tasks to train on under a fixed budget, and testing one choice costs a fine-tuning run. Common heuristics add more source tasks or pick sources similar to the target. The first assumes transfer is never negative; the second, that it is symmetric. We show that both assumptions fail: task

A

can help task

B

while

B

hurts

A

, so helpfulness is a signed property of ordered source--target pairs. We introduce the transfer map, a signed estimate of how much each source helps or hurts each held-out target. We fit the map in hundreds of fine-tuning runs on Qwen3 and Mistral models from 0.6B to 32B parameters, with all sources drawn from one corpus and no training examples from the target. The map predicts a held-out target's accuracy on unseen mixtures: recorded before those runs, its predictions have less than half the error of a mixture-agnostic baseline. The map is specific to its target and corpus but transfers across model scale: a mixture selected in advance at one size beats training on all source tasks at every other size we tested. Transfer is thus a property of the data. The map selects the tasks that help and drops the one that interferes: accuracy on the reasoning targets (causal explanation, multi-hop questions and methodological critique) rises by up to 14 percentage points over training on all source tasks.

Momentum

Related topics

Latest papers 20