CLIVO: Differentiable Cluster-Based Instruction Mixing with Learned Validation Objectives

Yuhan Wang; Rui Hu; Batian Di; Tan Jiang; Dongsheng Ma; Leichao Dong; Yuanzi Li; Yalin Feng

cs.AIOct 12, 2026

CLIVO: Differentiable Cluster-Based Instruction Mixing with Learned Validation Objectives

Authors: Yuhan Wang, Rui Hu, Batian Di, Tan Jiang, Dongsheng Ma, Leichao Dong, Yuanzi Li, Yalin Feng

Organizations: Peking University · Renmin University of China · Zhejiang University · Xi’an Jiaotong University · Nanyang Technological University

Abstract

Target-aware multimodal instruction selection requires useful examples and an appropriate balance of supervision. Sample-level rankings can overrepresent similar content without optimizing this balance; group-level methods face coarse units, predefined set-level validation signals, and costly or inaccurate mixture search. We propose CLIVO, a target-aware framework connecting fine-grained semantic allocation, automatically learned validation objectives, and differentiable mixture search. Specifically, semantic clusters provide independently adjustable shares at the hundred-cluster scale. Correlations between per-example merged-proxy losses and real probe performance identify target-relevant validation examples; learned nonnegative weights combine their losses into an objective. A single backward pass through the merged proxy supplies gradients for all shares, avoiding an additional mixture-response regression model. Selecting 100K instructions from an 8M pool, CLIVO attains 97.2% of the full-data four-target average score with only 1.25% of the data. It improves on the strongest evaluated selection baseline by 1.16 points using random within-cluster sampling; combining learned quotas with sample ranking increases this advantage to 1.55 points. Analyses show that finer units benefit most targets, short-trained experts preserve mixture responses, sample weighting retains useful validation information, and direct search consistently reduces the learned objective. CLIVO also supports interpretable allocation, modular selection, and reuse across targets and granularities.

Figures & tables

Method	O	G	P	K	Average
Full (8M)	71.01	69.23	80.87	63.81	71.23
Random	66.61	63.57	77.73	60.05	66.99
PPL	64.26	60.78	75.65	58.09	64.70
CLIP-Score	66.16	62.23	79.42	58.80	66.65
MM-Selector	66.46	64.05	76.96	59.59	66.77
PRISM	67.27	65.73	76.97	59.86	67.46

Table 1: Selection quality at a 100K budget. O/G/P/K: four targets; Average: their mean. Random, Acc + Reg., Source + Reg., and CLIVO average three runs. Bold/underline: best/second-best at 100K.

Target	C–E	C–W	S–E	S–W
General	0.621	0.684	0.619	0.714
Parsing	0.397	0.430	0.379	0.461
Knowledge	0.586	0.594	0.589	0.596
Overall	0.469	0.569	0.470	0.591

Table 2: Validation aggregation. Held-out Spearman over 20 folds, with 64 matched examples. C/S: cluster/sample; E/W: equal/weighted. Maximum correlations are bold.

Target	CV- $\rho$	$t^{\star}$	$D_{J}$	$B$	$\bar{F}$	$\Delta F$
Gen.	0.714	100	3.724	66.600	67.28	$+0.68$
Pars.	0.461	120	0.293	79.310	79.64	$+0.33$
Know.	0.596	40	0.056	61.465	61.76	$+0.30$
Ovrl.	0.591	100	2.040	67.919	68.13	$+0.21$

Table 3: From feedback to final training.

K=100

. CV-

\rho

: held-out Spearman;

t^{\star}

: best step;

D_{J}

: loss decrease (%);

B

: best probe;

\bar{F}

: three-run mean;

\Delta F=\bar{F}-B

. Protocols: Section B.9 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset or branch	Link	Upstream size	Processed rows
ECD-10K Images	HF	$\sim$ 321.5K	321,544
Honey-Data-1M	HF	1,072,111	1,070,304
Llama-Nemotron-VLM v1, VQA-9	HF	46,745	46,745
MGrounding-630K	HF	$\sim$ 630K	623,979
MM-IFInstruct-23K	HF	22,554	22,555
MM-IFInstruct-23K, second branch	HF	22,554	22,551

Appendix

Table 4: Instruction-pool sources. Public release sizes and processed source-branch rows before 20% sampling. Derived branches list parent-release sizes.

Stage	Updates / checkpoint	Peak learning rate	Token budget
Alignment	4,500	$10^{-3}$	16,384
Mid-training	35,999 of 40,000	$10^{-5}$	8,192
Cluster experts	1 / 5 / 10	$10^{-5}$	16,384
Real probes	50	$10^{-5}$	16,384
Final instruction tuning	100	$10^{-5}$	16,384

Appendix

Table 5: Training schedules. Expert checkpoint indices are defined in Section B.8 .

Benchmark	Domain target	Domain weight	Overall weight
MMBench-EN	General	$1/3$	$1/7$
MMStar	General	$1/3$	$1/7$
GQA	General	$1/3$	$1/7$
OCRBench	Parsing	$1/2$	$1/7$
ChartQA	Parsing	$1/2$	$1/7$
MMLU	Knowledge	$1/2$	$1/7$

Appendix

Table 6: Benchmark-to-target mapping. Domain targets average their constituent benchmark scores equally. Overall averages all seven scores, not the three domain means.

Target	Mean	Standard deviation	Sample variance
General	67.28	0.53	0.28
Parsing	79.64	0.59	0.34
Knowledge	61.76	0.11	0.01
Overall	68.13	0.13	0.02

Appendix

Table 7: CLIVO final-training repeats. Three runs at fixed searched shares with uniform within-cluster sampling. Variance uses the

n-1

denominator.

Method	Runs	Overall	General	Parsing	Knowledge
Random	3	$66.61\pm 0.23$	$63.57\pm 0.28$	$77.73\pm 0.09$	$60.05\pm 0.44$
Source + Reg.	3	$67.00\pm 0.17$	$62.91\pm 0.39$	$79.12\pm 0.26$	$59.80\pm 0.16$
Acc + Reg.	3	$67.53\pm 0.05$	$65.44\pm 0.38$	$77.80\pm 0.03$	$61.40\pm 0.26$

Appendix

Table 8: Baseline repeats. Three-seed mean

\pm

population standard deviation.

Target	Benchmark	CLIVO	CLIVO + ICONS
General	MMBench-EN	84.87	85.22
	MMStar	55.98	58.27
	GQA	60.99	61.32
Parsing	OCRBench	81.23	81.00
Parsing	ChartQA	78.04	78.92
Knowledge	MMLU	72.23	72.25

Appendix

Table 9: Benchmark scores by target (%). CLIVO: three runs with random sampling; CLIVO + ICONS: one run, same quotas.

Target	$K$	$J^{\star}$	$t^{\star}$	$C^{\star}$	$F$
General	20	0.32639	120	0.662	67.197
	50	0.32801	80	$\geq 0.343$	66.833
	100	0.32145	100	0.808	67.017
Parsing	20	0.85952	0	0.00	79.810
	50	0.85965	100	0.833	80.120
	100	0.85866	120	0.993	80.260

Appendix

Table 10: Allocation freedom.

J^{\star}

: minimum proxy loss;

t^{\star}

: best step;

C^{\star}

: 64-H100-equivalent hours;

F

: single-seed score. Bold: best

F

. Accounting: Section C.4 .

Bank	Mean loss	Fidelity $\rho$	G $\rho$	P $\rho$	K $\rho$	O $\rho$
Step 1	1.283	0.485	0.383	0.435	0.523	0.367
Step 5	1.007	0.647	0.409	0.462	0.524	0.387
Step 10	0.972	0.730	0.518	0.448	0.527	0.460
Full schedule	0.940	0.644	0.577	0.382	0.440	0.457

Appendix

Table 11: Expert budget and two kinds of agreement. Fidelity compares proxy with real-model loss over 128 mixtures. G/P/K/O are held-out correlations with real benchmark scores after budget-specific validation-cluster selection. Bold: highest in each correlation column.

Target	Start recipe	$t^{\star}$	$J_{0}$	$J^{\star}$
General	101	100	0.333885	0.321450
Parsing	119	120	0.861185	0.858662
Knowledge	27	40	0.687988	0.687602
Overall	101	100	0.376730	0.369045

Appendix

Table 12: Search starts and checkpoints. Full BF16 evaluation selects

t^{\star}

; remerged step-zero loss

J_{0}

defines the decrease

D_{J}

Sample annotation. The interaction’s text and visual inputs follow this prompt.

⬇ You are an expert Data Annotation Specialist . Your task is to analyze User - Assistant interaction samples and classify them into 4 dimensions using STRICT standardized labels . The data comes from diverse multi - modal sources including VQA , Document Reasoning , Mathematical Logic , Web Agents , and Grounding tasks . You must output a SINGLE valid JSON object . ### 1. Task Dimension ( Hierarchical ) Classify the user ’ s primary intent . - " task_category ": Choose one of [" Perception ", " Reasoning ", " Generation ", " Knowledge ", " Chat "] - " task_type ": Choose the specific sub - type based on the category : - If Perception : [" OCR ", " Object Detection ", " Image Captioning ", " Visual Grounding ", " Video Tracking "] - If Reasoning : [" Math Solving ", " Chart / Data Analysis ", " Document Understanding ", " Logic / Puzzle ", " Web / UI Navigation ", " Scientific Analysis "] - If Generation : [" Creative Writing ", " Translation ", " Summarization ", " Code Implementation ", " Long - form Content "] - If Knowledge : [" Fact Lookup ", " Concept Explanation ", " Trivia ", " How - to Guide "] - If Chat : [" Roleplay ", " General Chat ", " Multi - turn Dialogue "] ### 2. Content Dimension ( Visual & Semantic split ) Analyze the domain and visual inputs . - " visual_content ": Choose one : [" No Image ", " Natural Scene ", " Human / Face ", " Document / Text ", " Chart / Graph / Plot ", " UI / Webpage ", " Art / Illustration ", " Abstract / Pattern "] - " text_domain ": Choose one : [" General / Daily Life ", " STEM ", " Humanities / Art ", " Medical / Health ", " Legal / Finance ", " Code / Programming ", " Web / Internet "] ### 3. Format Dimension ( Input & Output Structure ) Analyze the structural form . - " input_composition ": Choose one : [" Text - Only ", " Single - Image ", " Multi - Image ", " Video ", " Interleaved - Text - Image "] - " response_style ": Choose one : [" Multiple Choice ", " Yes / No ", " Short Answer ", " Long Explanation / CoT ", " Code Block ", " Structured Format ( JSON / Table )", " Bounding Box / Coordinates "] ### 4. Difficulty Dimension ( Cognitive Load ) - " difficulty_level ": Choose one : [" Low ", " Medium ", " High "] ** CRITICAL :** Output raw JSON only . No markdown . No explanations .

Cluster summary. Inputs follow under Cluster Aggregated Statistics (label histograms) and Sampled Annotations (up to 100 records).

⬇ You are an expert Data Analyst and Dataset Curator . I will provide you with the aggregated annotation statistics and a sample of individual interactions from a specific data cluster . These individual samples were previously annotated across 4 strict dimensions : Task , Content , Format , and Difficulty . Your objective is to analyze this aggregated data and the samples to construct a comprehensive , multi - dimensional profile of this specific cluster . Please output a valid JSON object strictly matching the following schema . Use the predefined label vocabularies from the sample annotations when identifying dominant categories . { " task_dimension_summary ": { " dominant_task_category ": " The most frequent task category ( e . g ., Perception , Reasoning , etc .)", " top_task_types ": [" List the top 1-3 specific task_types present in this cluster "], " task_diversity_note ": " A 1- sentence analysis of whether this cluster is highly focused on one task or mixes several ." }, " content_dimension_summary ": { " primary_visual_content ": [" List the top 1-2 most frequent visual_content types "], " dominant_text_domain ": " The most frequent text_domain ", " content_fusion_analysis ": " A 1- sentence description of how the visual and textual domains interact in this cluster ." }, " format_dimension_summary ": { " dominant_input_composition ": " The most frequent input_composition ( e . g ., Interleaved - Text - Image )", " typical_response_style ": [" List the top 1-2 expected response_styles "] }, " difficulty_dimension_summary ": { " overall_difficulty_level ": " Low / Medium / High / Mixed ( based on majority distribution )", " cognitive_load_description ": " A 1- sentence explanation of why the data requires this level of cognitive load ." }, " comprehensive_cluster_profile ": { " short_definition ": " A precise , 1- sentence title or definition for this cluster ( e . g ., ’ Complex Scientific Document Reasoning and Formatting ’).", " detailed_description ": " A comprehensive 3-4 sentence paragraph synthesizing all 4 dimensions . Describe the typical user intent , the nature of the multi - modal inputs , the expected output structure , and the overall technical or cognitive demands of the data within this cluster ." } } ** CRITICAL INSTRUCTIONS :** 1. Output raw JSON ONLY . 2. Do NOT wrap the JSON in markdown code blocks ( e . g ., no ‘‘‘ json ... ‘‘‘). 3. Do NOT include any introductory or concluding text . 4. Ensure all dimensional analyses strictly reflect the provided sample data and statistics .

Appendix

Table 13: GPT annotation prompts. Sample labels describe task, content, format, and difficulty; cluster summaries aggregate these annotations.

Coarse capability	Primary task types
Visual QA & Captioning	Image Captioning
Grounding & Detection	Object Detection; Visual Grounding; Video Tracking
OCR & Documents	OCR; Document Understanding
Charts & Data	Chart/Data Analysis
Math & Logic	Math Solving; Logic/Puzzle
Knowledge & Science	Scientific Analysis; Concept Explanation; Fact Lookup

Appendix

Table 14: Coarse capability labels for visualization. Each cluster’s primary annotated task determines its coarse label. The original task descriptions are retained for finer interpretation.

Target pair	TV	JSD (bits)
General / Parsing	0.9783	0.9481
General / Knowledge	0.9999	0.9995
General / Overall	0.2610	0.2126
Parsing / Knowledge	0.9895	0.9789
Parsing / Overall	0.8823	0.8338
Knowledge / Overall	0.9765	0.9235

Appendix

Table 15: Target-dependent mixtures. TV and JSD compare final cluster shares.

Workload	Compute budget
Gradient: General	0.960
Gradient: Parsing	0.993
Gradient: Knowledge	1.123
Gradient: Overall	0.971
Gradient: four-target total	4.046
Regression: 4,096 shared labels	5.509

Appendix

Table 16: Incremental search compute. Gradient costs cover all 120 updates. Regression cost covers constructing the shared 4,096 proxy-loss labels. Units: 64-H100-equivalent hours.

Target	GD 20	GD 50	GD 100	Regression
General	67.20	66.83	67.02	67.09
Parsing	79.81	80.12	80.26	80.10
Knowledge	61.42	61.57	61.83	62.17
Overall	67.99	67.85	68.02	68.22

Appendix

Table 17: Search endpoints. Single-seed 100K scores. GD: gradient search; regression: 4,096 labeled mixtures.

Explore similar work

May 26, 2026

OnceSelect: Reusable Data Selection for Efficient Multimodal Instruction Tuning

Multimodal instruction tuning is widely used to adapt multimodal large language models (MLLMs), yet the large-scale image-text datasets it relies on are often highly redundant. Existing data selection methods are commonly tied to specific datasets, target models, or training states, requiring retraining or recomputation when transferred to new settings. We ask whether a selection signal can instead be learned once and reused across datasets and models. We propose OnceSelect, a reusable data selector trained once and directly applied to unseen datasets. OnceSelect encodes image-instruction pairs in a frozen joint multimodal space, clusters them into pseudo-labels capturing coarse semantic structure, and trains a lightweight selector using a fixed validation macro-accuracy criterion. Low-confidence samples under the learned partition are treated as informative candidates. As the selector is independent of the downstream MLLM, it can be transferred across datasets without retraining or model-dependent scoring, while selected subsets can be reused across model architectures. Using only 15% of LLaVA-625K, OnceSelect retains 99.2% of full-data aggregate performance across nine evaluation metrics. Without retraining, it achieves 103.4% and 102.5% relative performance on unseen Vision-Flan-186K and LRV-Sub-180K. The same 93.7K LLaVA subset further achieves 99.9-106.7% relative performance across Qwen3-VL and InternVL3 models without model-specific scoring or reselection. These results demonstrate efficient and reusable multimodal data selection across datasets and model architectures. Code is available at https://github.com/DMK041218/OnceSelect

Mingkang Dong, Muxin Pu, Hongyi Cai +5

Faculty of Computer Science Universiti Malaya Kuala Lumpur, Malaysia · School of Information Technology Monash University Malaysia Subang Jaya, Selangor, Malaysia · Faculty of Computer Science University of Science and Technology Beijing Beijing, China

Instruction-Tuning Data Selection Multimodal Large Language Models Uncertainty Sampling Multimodal Learning Multimodal Instruction Tuning

Ask PDF

Feb 16, 2026

A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque: methods vary widely in selection budgets, often omit zero-shot baselines, and frequently entangle the contributions of key components. As a result, practitioners lack actionable guidance on selecting instructions for their target tasks. In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms. Our framework enables controlled comparisons across models, tasks, and budgets. We find that only gradient-based data representations choose subsets whose similarity to the query consistently predicts performance across datasets, models, and candidate pools. While no single method dominates, gradient-based representations paired with greedy round-robin selection often perform best on average at low budgets, but these gains diminish at larger budgets. Finally, we unify several existing selection algorithms as forms of approximate distance minimization between the selected subset and the query set, and support this view with new generalization bounds. More broadly, our findings provide critical insights and a foundation for more principled data selection in LLM fine-tuning. The code is available at https://github.com/dcml-lab/targeted-instruction-selection.

Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund +2

Harvard University · MIT · Kempner Institute

Instruction-Tuning Data Selection Training Data Selection LLM Fine-Tuning

Ask PDF

Apr 12, 2026

Instruction Data Selection via Answer Divergence

Instruction tuning relies on large instruction-response corpora whose quality and composition strongly affect downstream performance. We propose Answer Divergence-Guided Selection (ADG), which selects instruction data based on the geometric structure of multi-sample outputs. ADG draws several high-temperature generations per instruction, maps responses into an embedding space, and computes an output divergence score that jointly encodes dispersion magnitude and shape anisotropy. High scores correspond to instructions whose answers are both far apart and multi-modal, rather than clustered paraphrases along a single direction. Across two backbones and three public instruction pools, fine-tuning on only 10K ADG-selected examples consistently outperforms strong selectors on six benchmarks spanning reasoning, knowledge, and coding. Analyses further show that both dispersion magnitude and shape anisotropy are necessary, supporting answer divergence as a practical signal for instruction data selection. Code and appendix are included in the supplementary materials.

Bo Li, Mingda Wang, Shikun Zhang +1

National Engineering Research Center for Software Engineering, Peking University · School of Computer Science, Peking University · PKU-CMCC(Hubei) Joint Research Lab for LLM Industrial Applications +1

Instruction-Tuning Data Selection Data Selection Instruction Tuning

Ask PDF