When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Use

Anjie Xu; Zhiyu Zhang; Ruiqing Ding; Fengli Xu; Leye Wang

cs.AISep 26, 2026

When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Use

Authors: Anjie Xu, Zhiyu Zhang, Ruiqing Ding, Fengli Xu, Leye Wang

Organizations: Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China · School of Computer Science, Peking University, Beijing, China · Zhongguancun Academy, Beijing, China · Yuanpei College, Peking University, Beijing, China ·

Abstract

Agent skills are expected to improve task performance. Yet we find that they often provide no benefit, and can even hurt performance while incurring additional token costs. Can we predict whether a skill will help before the agent acts? We introduce SkillDelta, a framework for predicting task-conditional skill gains from paired executions of the same agent with and without the skill. A local predictor transfers these historical gains to new tasks without retraining the agent. Under explicit transfer assumptions, our analysis links support coverage, representation mismatch, and execution noise to prediction error and decision regret. Across five benchmarks and three target agents, paired history improves observed-gain ranking over skill-assisted outcomes alone in 12 of 15 settings. At matched expected skill-use rates, SkillDelta improves success over random activation in all 15 settings, with an average absolute gain of 4.3%. Most of this advantage comes from allocation across task groups. Evidence for additional within-group selection value is strongest on ToolQA and weaker elsewhere. Code is available at https://github.com/TankTechnology/skilldelta.

Figures & tables

Figure 1: SkillDelta predicts skill gains before execution. Paired outcomes provide observed gains, which are transferred through nearby task embeddings to predict gain on a new task. Thresholding this prediction determines whether to use the skill. Only task text is encoded; the geometry is schematic.

Benchmark	Tasks	Skills
ToolQA	1,430	14
MedCalc-Bench	1,100	55
BigCodeBench	1,136	139
LogicBench	760	19
SpreadsheetBench	399	1 ‡

Table 1: Task and skill inventory.

Figure 2: Paired history improves observed-gain ranking in most settings.

		Task success (%)				Token savings (%)
Model	Benchmark	Off	On	Self-judge	SkillDelta	Self-judge	SkillDelta
	ToolQA	28.6	44.7	44.5	44.5	-5.3	10.4
	MedCalc-Bench	43.2	71.7	70.5	70.6	-65.8	14.8
Qwen-Turbo	BigCodeBench	43.7	51.4	51.4	47.4	-98.2	49.4
	LogicBench	75.5	87.2	87.4	85.2	-78.7	29.1
	SpreadsheetBench	15.5	19.8	18.3	17.8	-72.3	36.2

Table 2: Task success and token change under the unified protocol.

Figure 3: Success–token trade-offs as the decision threshold changes.

Figure 4: Support, neighborhood size, and repetition affect prediction differently.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: How the theory fits together. The main chain turns historical evidence into prediction and decision guarantees. Policy evaluation uses a separate action–outcome independence argument.

	Relevance gate		Self-judge
Benchmark	Success (%)	Saved tokens (%)	Use (%)	Router tokens/task	BA gain (%) [95% CI]
Qwen-Turbo
ToolQA	36.99	3.07	94.2	1164	+30.41 [27.99, 32.75]
MedCalc-Bench	60.00	25.31	98.1	1583	+19.65 [17.11, 22.21]
BigCodeBench	50.18	46.76	99.5	4976	+1.77 [-2.49, 6.09]
LogicBench	83.03	35.69	98.3	1314	+8.55 [4.78, 12.27]

Appendix

Table 3: No-history baseline controls. BA gain is the absolute balanced-accuracy advantage of SkillDelta over self-judge.

Benchmark	Estimator	AUROC	Success (%)	Tokens saved (%)
ToolQA	SkillDelta	0.879	44.55	10.37
	TF–IDF kNN	0.877	44.27	10.28
	Family mean	0.786	45.38	7.55
MedCalc-Bench	SkillDelta	0.781	70.64	14.82
	TF–IDF kNN	0.789	70.55	13.33
	Family mean	0.785	71.55	7.71

Appendix

Table 4: Paired-evidence estimator controls on the complete Qwen-Turbo inventories.

Stack	Benchmark	AUROC: 95% CI	Success gain (%): 95% CI
Qwen-Turbo	ToolQA	[0.849, 0.907]	[13.9, 18.0]
Qwen-Turbo	MedCalc-Bench	[0.751, 0.810]	[24.5, 30.4]
Qwen-Turbo	BigCodeBench	[0.478, 0.577]	[1.8, 5.5]
Qwen-Turbo	LogicBench	[0.574, 0.667]	[7.2, 12.2]
Qwen-Turbo	SpreadsheetBench	[0.478, 0.681]	[-0.8, 5.5]
GLM-5.3-Flash	ToolQA	[0.792, 0.875]	[7.5, 10.7]

Appendix

Table 5: Task-bootstrap intervals for the main results. Gains are absolute success-rate differences from Always-off. Point estimates appear in Figures 2 and 2 . Scores and actions are fixed across 10,000 task resamples.

Figure 6: Selection value at the same expected skill-use rate. Points show the absolute success-rate advantage over random activation. Bars are 95% intervals from 5,000 task or whole-family resamples, keeping actions fixed and recomputing the matching rate. All estimates are positive; some intervals include zero.

Figure 7: Where selection value comes from. Within-group selection and between-group allocation sum to the advantage over global matched-rate random activation. Bars show conditional 95% intervals from task resampling within the recorded groups.

			Single Ridge		Dual Ridge		Dual–On
Benchmark	$N$	On Acc.	Acc.	Save	Acc.	Save	$\Delta$ (%) [95% CI]
Qwen-Turbo
ToolQA	286	44.76	43.71	9.09	43.71	11.27	$-1.05$ [-3.51,+1.40]
MedCalc-Bench	220	75.91	72.27	19.17	73.64	19.55	$-2.27$ [-5.07,+0.46]
BigCodeBench	228	50.88	50.88	4.21	50.44	27.90	$-0.44$ [-3.07,+2.19]
LogicBench	152	89.04	87.94	22.83	87.94	22.83	$-1.10$ [-4.25,+2.03]

Appendix

Table 6: Ridge policy controls on identical fixed test tasks across three agents. Acc. and Save are percentages; saving is relative to Always-on.

\Delta

is the absolute accuracy difference. Single Ridge uses the same validation search as Dual Ridge with

\lambda=0

Transfer axis	Known	Target	AUROC	Balanced accuracy	Success gain	Tokens saved
ToolQA, unseen tasks within 8 families	96	160	.753	.689	+20.00%	3.62%
MedCalc, new tasks across 55 families	440	660	.719	.667	+16.21%	32.35%
BigCode, scale follow-up (20/family)	200	150	.716	.663	+2.00%	64.43%
BigCode, natural remaining distribution	350	200	.502	.572	+7.00%	32.88%

Appendix

Table 7: Prospective generalization stress tests. Success gain is the absolute difference between SkillDelta and Always-off; tokens saved are relative to Always-on. The panels differ in target construction.

Task Transferability Estimation

Task transferability estimation predicts which models, datasets, tasks, or skills will help on a new target before costly adaptation.

See all 38 papers

Towards Strategy-Level RSI for Skill-Augmented Agents: Learning When to Reuse Skills from Execution Feedback
Sep 22, 2026Shuang Guo, Xiaobo Sui, Fei Jiang +4LLM Agent Skill Retrieval LLM Agents
Central China Normal University · Beijing Kuafu Technology Co., Ltd. · Monash University
Demystifying Agent Skills: Why They Work-Until They Don't
Aug 14, 2026Zhiyuan Jiang, Fangrui Huang, Hanwen Xing +6LLM Agent Skill Evaluation LLM Agents
Princeton University · Stanford University · University of Southern California +2
Skill Coverage: A Test Adequacy Metric for Agent Skills
Jun 9, 2026Boyin Tan, Xiaowei Huang, Youcheng SunLLM Agent Evaluation AI Agent Benchmarks
1Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates · University of Liverpool Liverpool, United Kingdom

Sep 22, 2026J/K move · Enter open · S save

Towards Strategy-Level RSI for Skill-Augmented Agents: Learning When to Reuse Skills from Execution Feedback

Shuang Guo, Xiaobo Sui, Fei Jiang, Ruixin Li+3

Central China Normal University · Beijing Kuafu Technology Co., Ltd. · Monash University

LLM Agent Skill Retrieval LLM Agents Agent Skill Learning

Open paper AskPDF arXiv

Long-running agents accumulate reusable Skills, but a Skill that is semantically relevant to a task is not necessarily worth loading in the current state. We study the applicability question that arises once a candidate Skill is known: should it be loaded in the current state? We propose SkillApt, which uses matched WITH/WITHOUT Skill executions on the same task state as persistent evidence, estimates the conditional marginal utility of the Skill, and chooses LOAD or ABSTAIN accordingly. The base model, agent architecture, and Skill contents stay fixed; only the external deployment policy changes. We call this constrained setting strategy-level recursive self-improvement (Strategy-Level RSI). On 20 Skills and 160 held-out states, as paired evidence accumulates, SkillApt's task success rises from 81.9% under a cold start to 91.3%, matching a strong zero-shot LLM controller; yet SkillApt activates Skills on only 26.3% of states, versus 98.8% for the zero-shot controller. A hard-candidate study shows that non-optimal Skills mostly leave correctness unchanged while raising execution cost, and occasionally cause correctness harm. An ablation shows that a history recording only WITH success makes the policy load almost everywhere, whereas paired evidence substantially improves selectivity. These results indicate that relevance is not applicability: the main effect of execution evidence is not to make the model stronger but to change how existing Skills are deployed, moving the system from near-always loading to selective reuse.

Abstract

Task Transferability Estimation

Explore similar work

Towards Strategy-Level RSI for Skill-Augmented Agents: Learning When to Reuse Skills from Execution Feedback

Demystifying Agent Skills: Why They Work-Until They Don't

Skill Coverage: A Test Adequacy Metric for Agent Skills