Token Utility Is Selection-Conditioned: Coupled Selection of Prompt Context and Response Supervision for Instruction Tuning

Can Wu; Xinrui Chen; Ou Wu; Yi Du

cs.LGSep 19, 2026

Token Utility Is Selection-Conditioned: Coupled Selection of Prompt Context and Response Supervision for Instruction Tuning

Authors: Can Wu, Xinrui Chen, Ou Wu, Yi Du

Organizations: Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou, China

Abstract

Token selection for instruction tuning is coupled: supervision utility depends on retained context, and vice versa. Existing methods largely optimize prompt compression or response supervision separately. Combining selectors can value supervision using removed context, or context using excluded supervision, creating selection-state mismatch. BRIDGE (Budgeted Response-Prompt Interaction via Directional Gradient-guided Efficient Token Selection) models selection-conditioned token utility through a shared validation-directed surrogate objective. BRIDGE routes response utility to supporting prompt tokens through attention, then alternates budgeted selection to revalue each side under the other's retained subset. Updates reuse precomputed interactions without repeated model scoring, while structure-aware projection promotes coherent supervision spans. Across three model families and three task domains, BRIDGE performs best overall among evaluated selection methods. Its MATH advantage over independent selection grows with compression, reaching 1.66 percentage points on Llama-3.2-3B.

Figures & tables

Figure 1: Token utility depends on retained context and supervision across prompt–response selection states. Response selection controls loss positions; the complete response remains available as causal history.

Figure 2: Evidence for coupled selection: (a) allocation, (b) ranking shifts, and (c) gains under compression.

Figure 3: Overview of BRIDGE: Coupled Prompt-Response Valuation for Instruction Tuning.

Model	Method	General Capability (%)					Target (%)	Overall
Model	Method	ARC-C	HellaSwag	MMLU	HumanEval	Avg.	MATH	Avg.
Llama-3.2-3B	Pretrained	43.17	54.20	56.64	28.05	45.52	0.62	23.07
	Full SFT	$40.73_{\pm 0.36}$	$52.18_{\pm 0.17}$	$51.84_{\pm 0.29}$	$18.29_{\pm 1.06}$	$40.76_{\pm 0.24}$	$16.66_{\pm 0.52}$	$28.71_{\pm 0.29}$
	Random	$41.64_{\pm 0.52}$	$54.83_{\pm 0.36}$	$53.80_{\pm 0.26}$	$30.89_{\pm 0.70}$	$45.29_{\pm 0.21}$	$11.27_{\pm 0.87}$	$28.28_{\pm 0.33}$
	RHO-1 \venuebox NeurIPS’24	$41.27_{\pm 0.27}$	$53.92_{\pm 0.23}$	$55.38_{\pm 0.17}$	$29.07_{\pm 1.27}$	$44.91_{\pm 0.25}$	$14.67_{\pm 0.45}$	$29.79_{\pm 0.35}$
	LLMLingua-2 \venuebox ACL’24	$42.12_{\pm 0.40}$	$54.56_{\pm 0.19}$	$54.49_{\pm 0.23}$	$29.07_{\pm 0.35}$	$45.06_{\pm 0.26}$	$14.96_{\pm 0.84}$	$30.01_{\pm 0.43}$

Table 1: Results of token selection methods on downstream benchmarks, reported in accuracy (%). Excluding Pretrained and Full SFT , the best and second-best results in each column are highlighted in red and blue , respectively.

Variant	MATH	General Avg.	Overall Avg.
BRIDGE	15.96	46.92	31.44
w/o warmup for scoring	13.96	46.02	29.99
SFT initialized from warmup	15.42	45.98	30.70
w/o L2 anchor	15.30	46.09	30.69
w/o response-history term	15.22	46.24	30.73
w/o structure projection	15.20	46.53	30.87

Table 2: Single-run design comparisons on Llama-3.2-3B under fixed token budgets.

Figure 4: Single-run parameter sensitivity on Llama-3.2-3B.

Side	Proxy	Spearman $\uparrow$	Top- $b$ Overlap $\uparrow$	Jaccard $\uparrow$	Norm. Regret $\downarrow$
Prompt	Attention	$-0.0528$	$0.7094$	$0.5509$	$0.5905$
Prompt	Attention $\times a$	$\mathbf{0.9123}$	$\mathbf{0.9462}$	$\mathbf{0.9192}$	$\mathbf{0.0347}$
Response	Attention	$-0.0409$	$0.7639$	$0.6187$	$0.5182$
Response	Attention $\times a$	$\mathbf{0.9463}$	$\mathbf{0.9667}$	$\mathbf{0.9362}$	$\mathbf{0.0251}$

Table 3: Selection-level fidelity to reference scores.

Figure 5: Efficiency and performance comparison.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Size	Ratio	Role
$D_{\mathrm{w}}$	2,004	5.01%	Warmup checkpoint
$D_{\mathrm{val}}$	1,963	4.91%	Validation direction
$D_{\mathrm{sft}}$	36,006	90.08%	Scoring, selection, and final SFT

Appendix

Table 4: MetaMathQA-40K data partition used throughout the main experiments. Ratios are computed over the 39,973-example pool and rounded to two decimal places.

Configuration	Value
Model	Llama-3.2-3B / Gemma-3-4B / Qwen3.5-9B
Optimizer	AdamW
Learning Rate	$2\times 10^{-5}$ , cosine
Batch Size	64 (effective)
Gradient Accumulation	8
Training Epochs	3

Appendix

Table 5: Training configurations used for BRIDGE.

Method	Llama-3.2-3B	Gemma-3-4B	Qwen3.5-9B
Full SFT	$2\times 10^{-5}\,/\,3$	$2\times 10^{-5}\,/\,3$	$2\times 10^{-5}\,/\,3$
Random	$2\times 10^{-5}\,/\,3$	$2\times 10^{-5}\,/\,3$	$1\times 10^{-5}\,/\,2$
RHO-1	$2\times 10^{-5}\,/\,3$	$2\times 10^{-5}\,/\,3$	$2\times 10^{-5}\,/\,3$
LLMLingua-2	$2\times 10^{-5}\,/\,3$	$2\times 10^{-5}\,/\,3$	$1\times 10^{-5}\,/\,2$
ssToken	$2\times 10^{-5}\,/\,3$	$2\times 10^{-5}\,/\,3$	$1\times 10^{-4}\,/\,1$
TokenSeek	$2\times 10^{-5}\,/\,3$	$2\times 10^{-5}\,/\,3$	$2\times 10^{-5}\,/\,3$

Appendix

Table 6: Selected optimization hyperparameters. Each cell reports learning rate / training epochs. The same configuration is used on MetaMathQA-40K, Magicoder, and Alpaca for each method–backbone pair.

Model	Prompt	Response loss	Executed / full
Llama	75.00%	75.00%	7.893 / 8.552 M
Gemma	75.01%	75.00%	8.602 / 9.285 M
Qwen	75.00%	75.00%	8.680 / 9.378 M

Appendix

Table 7: Retention rates for eligible positions and unpadded executed token counts per pass through the SFT split in the saved runs. Closure uses

97\%

of its eligible response budget in these runs.

Configuration	Value
Gradient Direction	Validation-split direction
Gradient Scope	Final four Transformer layers
Retention Anchor	L2 parameter-space anchor $\lambda=0.2$
Attention Layer	Llama: final Transformer layer Gemma: last global-attention layer (29) Qwen: last full-attention layer (31)
Interaction Surrogate	$A_{k,j,i},a_{k,j}$
Warmup Ratio	$\approx 5\%$

Appendix

Table 8: Default BRIDGE scoring and selection settings. The L2 anchor affects scoring only and is not used in final SFT.

Method	Prompt Selector	Response Selector	Cross-Side Dependency	MATH	General Avg.	Overall Avg.
LLM+RHO-1 (Ind.)	LLMLingua-2	RHO-1	Independent	12.1	45.81	28.96
LLM+ssToken (Ind.)	LLMLingua-2	ssToken	Independent	12.34	46.02	29.18
LLM+TC (Ind.)	LLMLingua-2	Token Cleaning	Independent	13.32	45.57	29.44
LLM $\rightarrow$ TC (Seq.)	LLMLingua-2	Token Cleaning	Prompt $\rightarrow$ Response	13.52	46.06	29.79
BRIDGE \venuebox Ours	BRIDGE	BRIDGE	Mutual, alternating	15.96	46.92	31.44

Appendix

Table 9: Independent, sequential, and alternating prompt–response selection on Llama-3.2-3B with matched retention ratios and final SFT settings.

Model	Method	General Capability (%)					Target (%)	Overall
Model	Method	ARC-C	HellaSwag	MMLU	GSM8K	Avg.	HumanEval	Avg.
Llama-3.2-3B	Pretrained	43.17	54.20	56.64	28.28	45.57	28.05	36.81
	Full SFT	$39.96_{\pm 0.26}$	$52.45_{\pm 0.27}$	$48.23_{\pm 0.19}$	$29.11_{\pm 0.57}$	$42.44_{\pm 0.24}$	$44.72_{\pm 0.93}$	$43.58_{\pm 0.39}$
	Random	$40.73_{\pm 0.69}$	$53.36_{\pm 0.46}$	$53.65_{\pm 0.41}$	$25.45_{\pm 0.44}$	$43.30_{\pm 0.36}$	$35.37_{\pm 1.06}$	$39.34_{\pm 0.39}$
	RHO-1 \venuebox NeurIPS’24	$41.33_{\pm 0.26}$	$54.07_{\pm 0.11}$	$55.93_{\pm 0.36}$	$24.92_{\pm 0.29}$	$44.06_{\pm 0.19}$	$35.37_{\pm 0.61}$	$39.72_{\pm 0.21}$
	LLMLingua-2 \venuebox ACL’24	$41.98_{\pm 0.52}$	$52.80_{\pm 0.29}$	$53.32_{\pm 0.32}$	$25.45_{\pm 0.44}$	$43.39_{\pm 0.28}$	$41.87_{\pm 1.27}$	$42.63_{\pm 0.51}$

Appendix

Table 10: Code-generation accuracy (%) across model backbones. Red and blue indicate the best and second-best results among selection methods, excluding Pretrained and Full SFT .

Model	Method	General Capability (%)				Gen.	IFEval (%)		Overall
Model	Method	MMLU	ARC-C	HellaSwag	GSM8K	Avg.	Prompt Level	Instr. Level	Avg.
Llama-3.2-3B	Pretrained	56.64	43.17	54.20	28.28	45.57	7.39	12.58	27.78
	Full SFT	$51.21_{\pm 0.26}$	$41.64_{\pm 0.35}$	$54.63_{\pm 0.20}$	$12.13_{\pm 0.65}$	$39.90_{\pm 0.25}$	$27.35_{\pm 1.15}$	$41.96_{\pm 1.25}$	$37.28_{\pm 0.42}$
	Random	$56.17_{\pm 0.30}$	$41.78_{\pm 0.42}$	$52.78_{\pm 0.17}$	$28.00_{\pm 0.84}$	$44.68_{\pm 0.35}$	$25.94_{\pm 1.59}$	$38.43_{\pm 1.71}$	$38.43_{\pm 0.39}$
	RHO-1 \venuebox NeurIPS’24	$57.20_{\pm 0.25}$	$42.32_{\pm 0.39}$	$54.19_{\pm 0.33}$	$29.92_{\pm 0.93}$	$45.91_{\pm 0.32}$	$27.26_{\pm 0.74}$	$39.89_{\pm 1.95}$	$39.74_{\pm 0.38}$
	LLMLingua-2 \venuebox ACL’24	$56.36_{\pm 0.22}$	$43.54_{\pm 0.58}$	$53.41_{\pm 0.38}$	$30.12_{\pm 0.66}$	$45.86_{\pm 0.38}$	$27.15_{\pm 0.63}$	$39.75_{\pm 1.28}$	$39.66_{\pm 0.53}$

Appendix

Table 11: Strict prompt-level and instruction-level accuracy (%) on IFEval across model backbones. General-capability metrics follow the protocol used in the main results table. Selection methods and Full SFT report means

\pm

standard deviations over three seeds; Pretrained reports results from a single seed. Red and blue indicate the best and second-best results among selection methods.

Method	ARC-C	HellaSwag	MMLU	HumanEval	General Avg.	$\boldsymbol{\Delta_{\mathrm{Gen}}}$	MATH	Overall Avg.
Pretrained	43.17	54.20	56.64	28.05	45.52	–	0.62	23.07
Full SFT	$40.73_{\pm 0.36}$	$52.18_{\pm 0.17}$	$51.84_{\pm 0.29}$	$18.29_{\pm 1.06}$	$40.76_{\pm 0.24}$	$-4.76_{\pm 0.24}$	$16.66_{\pm 0.52}$	$28.71_{\pm 0.29}$
FLOW \venuebox ICML’25	$43.77_{\pm 0.46}$	$56.32_{\pm 0.21}$	$56.34_{\pm 0.19}$	$31.30_{\pm 0.76}$	$46.93_{\pm 0.23}$	$1.41_{\pm 0.23}$	$14.84_{\pm 0.66}$	$30.88_{\pm 0.30}$
SAM \venuebox EMNLP’24	$43.00_{\pm 0.35}$	$57.34_{\pm 0.26}$	$55.21_{\pm 0.24}$	$31.50_{\pm 1.15}$	$46.76_{\pm 0.25}$	$1.24_{\pm 0.25}$	$14.52_{\pm 0.74}$	$30.64_{\pm 0.33}$
NanoAdam \venuebox NeurIPS’25	$43.23_{\pm 0.52}$	$55.12_{\pm 0.17}$	$55.06_{\pm 0.31}$	$28.46_{\pm 0.57}$	$45.47_{\pm 0.21}$	$-0.05_{\pm 0.21}$	$13.72_{\pm 0.55}$	$29.59_{\pm 0.26}$
TALR \venuebox ICLR’26	$42.35_{\pm 0.28}$	$54.93_{\pm 0.23}$	$55.61_{\pm 0.16}$	$29.88_{\pm 0.86}$	$45.69_{\pm 0.18}$	$0.17_{\pm 0.18}$	$13.94_{\pm 0.47}$	$29.81_{\pm 0.24}$

Appendix

Table 12: Capability retention on Llama-3.2-3B, reported as accuracy (%) with three-seed means

\pm

standard deviations. Pretrained denotes the original pretrained checkpoint;

\Delta_{\mathrm{Gen}}

measures the change in General Avg. relative to this checkpoint. Methods use their respective token budgets and training configurations.

Scoring Checkpoint	Training Init.	MATH	General Avg.	Overall Avg.
Old	Old	13.96	46.02	29.99
Warmup 5%	Old	15.96	46.92	31.44
Warmup 10%	Old	16.06	46.30	31.18
Warmup 5%	Warmup 5%	15.42	45.98	30.70

Appendix

Table 13: Scoring-checkpoint and final SFT initialization ablation. “Old” denotes the pretrained parameters

\theta_{\mathrm{old}}

. All other settings are fixed.

Checkpoint Pair	$\boldsymbol{\rho_{p}}$	$\boldsymbol{\rho_{r}}$	$\boldsymbol{O^{\mathrm{corr}}_{p}}$	$\boldsymbol{O^{\mathrm{corr}}_{r}}$
Warmup 5% vs. Old	0.170	-0.139	-0.007	-0.102
Warmup 10% vs. Old	0.098	0.154	-0.021	0.148
Warmup 10% vs. 5%	0.435	0.051	0.311	0.154

Appendix

Table 14: Ranking agreement on

36{,}006

examples: mean per-example Spearman correlations for prompt and response tokens (

\rho_{p},\rho_{r}

) and chance-corrected Top-

75\%

overlap.

Checkpoint Pair	Side	$q=0.05$	$q=0.10$	$q=0.25$	$q=0.50$	$q=0.75$
Warmup 5% vs. Old	Prompt	0.312	0.247	0.204	0.109	-0.007
	Response	0.021	0.053	0.077	-0.116	-0.102
Warmup 10% vs. Old	Prompt	0.225	0.164	0.120	0.053	-0.021
	Response	0.013	0.033	0.136	0.207	0.148
Warmup 10% vs. Warmup 5%	Prompt	0.425	0.402	0.373	0.341	0.311
	Response	0.362	0.381	0.318	-0.117	0.154

Appendix

Table 15: Chance-corrected Top-

q

overlap across scoring checkpoints. Positive values indicate agreement above chance.

Variant	$\boldsymbol{\lambda}$	MATH	General Avg.	Overall Avg.
No L2 Anchor	0.0	15.30	46.09	30.69
Weak Anchor	0.1	15.78	46.75	31.27
BRIDGE	0.2	15.96	46.92	31.44
Strong Anchor	0.4	15.26	47.00	31.13

Appendix

Table 16: Absolute scores corresponding to Figure 4 (b) for the scoring-only L2 anchor sweep on Llama-3.2-3B.

Setting	MATH	General Avg.	Overall Avg.
Attention layers
1 (default)	15.96	46.92	31.44
2	15.28	46.99	31.14
4	15.76	46.89	31.33
Gradient-scoring layers
1	15.74	47.06	31.40

Appendix

Table 17: Sensitivity to attention layer count, gradient-scoring scope, and validation set size on Llama-3.2-3B.

$\mathbf{T}$	Spearman $\uparrow$	Top- $b$ $\uparrow$	Utility Ret. $\uparrow$	Norm. Regret $\downarrow$
1	0.9446 [0.9391, 0.9502]	0.9686 [0.9661, 0.9710]	0.9925 [0.9890, 0.9954]	0.0075
2	0.9477 [0.9426, 0.9529]	0.9690 [0.9664, 0.9716]	0.9948 [0.9921, 0.9968]	0.0052
4	0.9477 [0.9422, 0.9527]	0.9690 [0.9662, 0.9715]	0.9948 [0.9922, 0.9969]	0.0052

Appendix

Table 18: Agreement between fixed-surrogate response scores and fresh scores recomputed after physical prompt compaction. Brackets denote 95% bootstrap confidence intervals over examples.

Figure 6: Finite-difference additivity across joint perturbation sizes. Curves show means over 200 MetaMathQA examples and three deterministic perturbation subsets per example; shaded bands denote 95% example-level cluster-bootstrap confidence intervals. At

|M|=32

, 198 examples have sufficient eligible prompt positions (

n=198

). Lower symmetric error and higher agreement indicate greater fidelity.

Prompt Retention	Sym. Err. $\downarrow$	Cosine $\uparrow$	Spearman $\uparrow$	Top- $k$ $\uparrow$	Norm. Top- $k$ $\uparrow$	Weighted Sign $\uparrow$
90%	0.392 (0.379–0.405)	0.741 (0.718–0.763)	0.645 (0.629–0.661)	0.720 (0.709–0.730)	0.626 (0.612–0.640)	0.891 (0.882–0.900)
70%	0.548 (0.537–0.559)	0.590 (0.566–0.614)	0.522 (0.506–0.538)	0.637 (0.627–0.647)	0.516 (0.502–0.530)	0.822 (0.810–0.834)
50%	0.617 (0.607–0.627)	0.531 (0.507–0.554)	0.490 (0.473–0.507)	0.606 (0.594–0.617)	0.474 (0.458–0.489)	0.808 (0.798–0.819)
30%	0.674 (0.665–0.682)	0.493 (0.468–0.519)	0.459 (0.441–0.477)	0.579 (0.567–0.591)	0.439 (0.423–0.455)	0.786 (0.774–0.798)
10%	0.711 (0.703–0.720)	0.458 (0.431–0.484)	0.432 (0.414–0.450)	0.550 (0.538–0.561)	0.399 (0.384–0.415)	0.770 (0.757–0.782)

Appendix

Table 19: Finite-difference additivity across prompt retention ratios. Lower retention corresponds to a larger jointly perturbed subset. Parentheses denote 95% example-level cluster-bootstrap confidence intervals.

Rule	Util. $\uparrow$	Budget $\uparrow$	Ans. $\uparrow$	MATH	General Avg.
Token Random	0.75	1.00	0.75	11.48	46.08
Span Random	0.77	0.98	0.37	12.63	46.05
Score-only	1.00	1.00	0.95	15.20	46.53
Score+Closure	0.83	0.97	1.00	15.96	46.92

Appendix

Table 20: Structural-projection ablation on Llama-3.2-3B with a shared response-supervision budget cap.

Figure 7: Effect of structure-aware projection.

Method	Throughput (tok/s)	Pre-SFT Overhead (GPU-h)	Final SFT (GPU-h)	Total (GPU-h)
Full SFT	3.10k	–	2.29	2.29
Random	3.31k	–	2.15	2.15
RHO-1	3.04k	1.00	2.34	3.34
LLMLingua-2	3.02k	0.91	2.35	3.26
ssToken	2.76k	0.28	2.57	2.85
TokenSeek	2.77k	1.55	2.56	4.11

Appendix

Table 21: Throughput and GPU-hours on Llama-3.2-3B, normalized to one A100-80GB GPU. Throughput counts original training tokens per second of final SFT. Pre-SFT overhead includes warmup, preprocessing, scoring, and selection; online selection costs are included in final SFT.

Explore similar work

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding
Apr 15, 2026Tatsuya Ichinose, Youmi Ma, Masanari Oi +2Instruction Tuning Synthetic Data Generation for Language Models
Department of Computer Science, School of Computing, Institute of Science Tokyo · National Institute of Advanced Industrial Science and Technology · Research and Development Center for Large Language Models, NII
Instruction Data Selection via Answer Divergence
Apr 12, 2026Bo Li, Mingda Wang, Shikun Zhang +1Instruction-Tuning Data Selection Data Selection
National Engineering Research Center for Software Engineering, Peking University · School of Computer Science, Peking University · PKU-CMCC(Hubei) Joint Research Lab for LLM Industrial Applications +1
One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness
Apr 14, 2026Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu +1Language Model Robustness Instruction Tuning
University of Southern California, Los Angeles, USA · Intel AI, USA

Apr 15, 2026J/K move · Enter open · S save

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike+1

Department of Computer Science, School of Computing, Institute of Science Tokyo · National Institute of Advanced Industrial Science and Technology · Research and Development Center for Large Language Models, NII

Instruction Tuning Synthetic Data Generation for Language Models Contrastive Decoding Language Model Distillation

Open paper AskPDF arXiv

Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures.

Abstract

Explore similar work

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Instruction Data Selection via Answer Divergence

One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness