Does Your Generalist Skill Optimizer Really Incentivize Agentic Tasks? Rethinking Skill Optimization via Bipartite Attribution

Jiawei Zhou; Bo Peng; Yuxuan Huang; Ming Zhang; Youbang Sun; Xiang Wang; Chaochao Lu

cs.AIOct 27, 2026

Does Your Generalist Skill Optimizer Really Incentivize Agentic Tasks? Rethinking Skill Optimization via Bipartite Attribution

Authors: Jiawei Zhou, Bo Peng, Yuxuan Huang, Ming Zhang, Youbang Sun, Xiang Wang, Chaochao Lu

Organizations: Shanghai AI Laboratory · Fudan University · Shanghai Jiao Tong University · Tsinghua University ·

Abstract

Skill optimizers adapt an agent to long-horizon tasks, such as repairing a terminal environment, by rewriting its skill, a natural-language procedure in its context, and keeping the edits that raise success on a small dev set. On such tasks success is noisy, so a gain on a small dev set does not show that an edit helps, and a success score does not show what the edit costs. We introduce BiPA, which replays an optimizer's acceptance rule on candidates identical to its seed to measure what selection alone reports, and records each episode as a bipartite graph of turns and the text they read to trace extra input tokens to the behavior behind them. Across three optimizers, three agentic benchmarks and two models, the learned skills succeed about as often as their hand-written seed, 0.1 points higher on average. Their dev gains vanish on test: keeping the best of several candidates on at most 30 tasks reports gains this large even when none changed. They read about 16% more input tokens: the optimizers almost always add rules, half to re-check work and some to act one call at a time, and the agent follows them as extra turns, each re-sending the whole conversation. Four principles follow: accept an edit only if its gain exceeds what selection alone reports; charge it for the turns it adds, not for its length; let the editor delete and merge rules; show it what earlier rules cost. Code is available at https://github.com/valleysprings/BiPA.git.

Figures & tables

Figure 1: On one AutomationBench task, SkillOpt’s skill passes like the seed but reads 1.42 times its input tokens, through extra turns. (a) Seed, (b) learned skill; box numbers: tool calls. (c) The ratio as a product of four factors.

Figure 2: BiPA splits a skill’s extra input tokens by content and by behavior. (a) Each turn re-sends the prefix and the history. (b) Turns linked to the context components they read or call. (c) The two splits against the seed.

Skill	TB2				$\tau^{3}$ -bench				AutoBench
Skill	$Q$	$\rho$	$\rho^{\rm turns}$	$\rho^{\rm calls}$	$Q$	$\rho$	$\rho^{\rm turns}$	$\rho^{\rm calls}$	$Q$	$\rho$	$\rho^{\rm turns}$	$\rho^{\rm calls}$
DeepSeek-V4.1-Flash
No skill	72.2 $-3.7$	0.94	1.00	1.00	73.8 $-2.7$	0.98	1.03	1.01	36.4 $-8.6$	0.88	0.92	0.94
GEPA	74.1 $-1.9$	1.34	1.18	1.21	75.6 $-0.9$	1.15	1.07	1.06	44.2 $-0.8$	1.21	1.09	1.00
SkillOpt	64.8 $-11.1$	1.43	1.22	1.23	79.1 $+2.7$	1.22	1.04	1.09	41.7 $-3.3$	1.25	1.11	1.04
SkillAdam	68.5 $-7.4$	1.37	1.22	1.27	75.1 $-1.3$	1.09	1.03	1.07	44.7 $-0.3$	1.08	1.03	0.97

Table 1: Test results (avg@3).

Q

: success (%), small: change from the seed in points;

\rho

\rho^{\rm turns}

\rho^{\rm calls}

: input tokens, turns and tool calls relative to the seed (Eq. ( 1 )), greener: lower. Bold, underline: best, second-best

Q

Figure 3: Reported dev gains are what selection alone reports, and test does not follow them. DeepSeek-V4.1-Flash. (a) Each run’s reported dev gain against replays with all candidates unchanged. (b) Test difference to the seed against the reported dev gain, nine returned skills (Table 1 ). SkillAdam’s dev gain sums its accepted edits’ margins on small batches, so it has its own axis in (a). Replays: Appendix C.3 .

Figure 4: SkillOpt reads 20 to 32% more input per task than the seed (DeepSeek-V4.1-Flash). Stacked by context component, mean over tasks. Dashed: the seed’s total on the same tasks; label: SkillOpt’s total over the seed’s, a ratio of totals, not

\rho

Figure 5: Input tokens grow faster than turns (DeepSeek-V4.1-Flash). Dots: test episodes, one per skill and task; solid: within-task fit, dashed: slope 1.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Most tasks are solved almost always or almost never, about as the fitted prior expects. Bars: share of dev and test tasks the seed solves in none, some or all of its runs, DeepSeek-V4.1-Flash; diamonds: the share the fitted prior expects; titles: its

(a,b)

Figure 7: Most learned skills differ from the seed in success by less than the seed’s own spread from run to run, but under DeepSeek-V4.1-Flash almost every one of them reads more input tokens than any run of the seed. One point per test run, DeepSeek-V4.1-Flash above and GPT-6-Luna below; a mean of a skill’s runs inside a success row’s grey band differs from the seed by at most the seed’s spread. A run’s input tokens: per task, the ratio to the geometric mean of the seed’s three runs, then the geometric mean over tasks; token-row means are geometric.

Figure 8: On dev the returned skill rarely loses a task the seed solved; on test it gains about as many tasks as it loses, except GPT-6-Luna on AutoBench, where it gains more. Per optimizer run, the returned skill against the seed, both scored once inside that run on the whole dev set (solid bars) and on the whole test set (hatched bars); up: share of tasks the seed fails and the skill solves; down: the reverse; triangles: the share of tasks one run of the seed solves and another fails on the same set, averaged over pairs of its runs, so the same up and down; GPT-6-Luna’s dev bars have none.

Figure 9: Along both

\tau^{3}

-bench paths, from the first kept skill on, the reported dev gain climbs 6.7 points while the test difference stays within 2.7 points of its value there. DeepSeek; the test runs here, of the skills and of the seed rerun, are on an updated release, so only differences within this figure compare. Columns: the skills each run kept in turn as its best on dev; round: the optimizer iteration that wrote the skill; filled dots: reported dev gain over the starting score; open circles and italic grey labels: test difference to the seed rerun, whiskers

\pm 1

standard error paired by task; gained/lost: test tasks the skill solves and the seed rerun fails, and the reverse. The two series have different baselines, so compare how each changes along the path, not their levels.

Figure 10: One panel per design principle of Section 6 , in order. DeepSeek-V4.1-Flash. (a) Simulated acceptance rules (1) to (4) of Appendix C.4 ; grey: an unchanged candidate, blue: one that truly solves two more dev tasks; bar: mean over the three benchmarks, whiskers: lowest and highest benchmark. (b) Share of the variance of an edit’s change in log input tokens that its change in turns, tool calls or skill length explains, each on its own. (c) Mean

\log\rho

of a benchmark’s three learned skills, or of all nine; grey: as recorded, blue: recomputed with each skill’s calls packed into turns as densely as the seed’s. (d) One dot per edit; short bar: mean, dashed line: zero.

Figure 11: One rewrite of the three-word seed reaches the runs from the hand-written seed on AutoBench and adds nothing on

\tau^{3}

-bench ; rewriting it again does not help. (a, b) Test success of GPT-6-Luna; the number at right is the mean of the three test runs, or the single test runs. (c, d) Every candidate’s dev score against its length; the dashed line is the three-word seed’s dev score; marks that share a dev score are set slightly apart in height.

Figure 12: Over the nine learned skills, extra turns carry most of the extra input tokens, and half of the added rules ask the agent to verify (DeepSeek-V4.1-Flash). (a) Each skill’s change in log input tokens against the seed (total, mean over tasks), split into the four terms of Eq. ( 6 ), which add up to it (rounded): more tool calls (more actions), the same calls spread over more turns (turn-splitting), more text per turn (content per turn) and a longer skill (skill text); the first two, the turn terms, sum to the cost of extra turns; red: more tokens than the seed, blue: fewer. (b) Rules added to the seed, by category (Appendix E.1 ); verify: re-check a result; serialize: act one call at a time; ask or stop: ask the user, hand off or stop; domain facts: about the domain and its tools; format: how to word the answer; other: the rest, parallelize included. Right: how many added rules verify, serialize or limit reading, and the skill’s length as a multiple of the seed’s; limit reading is a column only, as none of the nine skills adds such a rule.

Figure 13: Tool outputs are the largest part of the extra input tokens for seven of the nine DeepSeek skills and for the three Luna skills on AutoBench. One row per model and benchmark; curves average over test runs, then over tasks, and an ended episode keeps its total, so a gap can fall where the seed’s longer episodes keep reading. Left: the seed, stacked by context component; other columns: learned skill minus seed, black line the total gap, one vertical scale for a row’s three gap panels. Bold, top left: the seed’s total, or the skill’s total gap and that gap as a percentage of the seed’s total, a ratio of totals, not

\rho

Figure 14: On AutoBench the learned skills issue writes one per turn while reads stay grouped. (a) The task of Figure 1 : one column per turn, one square per tool call issued in it; a write (read) shares a turn if another write (read) is issued in the same turn. (b) The same shares, pooled over the writes (reads) of all AutoBench test episodes with complete logs (outcome, tool calls and input tokens all recorded); bottom row: the agent without a skill; a cross marks the share in that row’s episode of (a).

Figure 15: Transferred skills mostly score below the seed, as they do in the settings they leave or reach, and keep most of their extra input tokens. Row: the model and benchmark a skill runs on; lower block: learned

\rightarrow

run, the unchanged part written once; (1 run): one test run per skill, three in all other rows; marker: one optimizer’s skill; dashed line: the seed in the same setting.

Figure 16: The seed covers new tasks; the learned skills make episodes agree more on tasks whose success rose. (a) Tasks covered (solved in at least one of three test runs) by one side only, learned skills summed over optimizers;

p

: sign test. (b) Share of tasks whose episodes agree more than under the reference, by change in success; in bars: such tasks / all tasks; dotted line: half the tasks.

Figure 17: The route does not change a loaded skill’s cost; without an instruction, the description largely decides loading. DeepSeek-V4.1-Flash, updated release. (a)

\rho

of SkillOpt’s skill against the seed by route (left), and of each skill on demand against always resident (right). (b) Share of episodes that load the skill; label: loaded/total. Hollow: a skill learned on the other benchmark.

Agent Skill Learning

Agent skill learning develops reusable procedures and behaviors that help AI agents solve new tasks and adapt from experience.

See all 174 papers

SkillOpt: Executive Strategy for Self-Evolving Agent Skills
May 22, 2026Yifan Yang, Ziyang Gong, Weiquan Huang +12Large Language Model-Guided Optimization LLM Agent Self-Improvement
Microsoft · Shanghai Jiao Tong University · Tongji University +1
Skill $^{\infty}$ : Recursive Self-Improvement for Agent Skill Optimization
Oct 26, 2026Wangcheng Tao, Zhanglu Yan, Han Wu +1LLM Agent Self-Improvement LLM Agent Skill Learning
National University of Singapore · City University of Hong Kong
Write, Execute, Refine: From Skill Follower to Skill Optimizer via Reinforcement Learning from Execution Feedback
Aug 18, 2026Kang Peng, Zhiwei Zhang, Yichen Zhang +7LLM Agent Skill Learning RL for Tool Use
Harbin Institute of Technology, Shenzhen, China · The Chinese University of Hong Kong · MoE Key Laboratory of High Confidence Software Technologies +2

May 22, 2026J/K move · Enter open · S save

SkillOpt: Executive Strategy for Self-Evolving Agent Skills

Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang+11

Microsoft · Shanghai Jiao Tong University · Tongji University+1

Large Language Model-Guided Optimization LLM Agent Self-Improvement Agent Skill Learning

Open paper AskPDF arXiv

Agent skills today are hand-crafted, generated one-shot, or evolved through loosely controlled self-revision, none of which behaves like a deep-learning optimizer for the skill, and none of which reliably improves over its starting point under feedback. We argue the skill should instead be trained as the external state of a frozen agent, with the same discipline that makes weight-space optimization reproducible. SkillOpt is, to our knowledge, the first systematic controllable text-space optimizer for agent skills: a separate optimizer model turns scored rollouts into bounded add/delete/replace edits on a single skill document, and an edit is accepted only when it strictly improves a held-out validation score. A textual learning-rate budget, rejected-edit buffer, and epoch-wise slow/meta update make skill training stable while adding zero inference-time model calls at deployment. Across six benchmarks, seven target models, and three execution harnesses (direct chat, Codex, Claude Code), SkillOpt is best or tied on all 52 evaluated (model, benchmark, harness) cells and beats every per-cell competitor among human, one-shot LLM, Trace2Skill, TextGrad, GEPA, and EvoSkill skills. On GPT-5.5 it lifts the average no-skill accuracy by +23.5 points in direct chat, by +24.8 inside the Codex agentic loop, and by +19.1 inside Claude Code. Transfer experiments further show that optimized skill artifacts retain value when moved across model scales, between Codex and Claude Code execution environments, and to a nearby math benchmark without further optimization. Code: https://aka.ms/skillopt

Abstract

Agent Skill Learning

Explore similar work

SkillOpt: Executive Strategy for Self-Evolving Agent Skills

Skill∞^{\infty}∞: Recursive Self-Improvement for Agent Skill Optimization

Write, Execute, Refine: From Skill Follower to Skill Optimizer via Reinforcement Learning from Execution Feedback

Skill $^{\infty}$ : Recursive Self-Improvement for Agent Skill Optimization