Agent skills, structured artifacts distilled from interaction trajectories and dynamically reused from skill banks, have become a central mechanism for enabling large language model (LLM)-based self-evolving agents to learn from past experience. Yet existing work has largely focused on the operational aspects of skills, such as acquisition, evolution, and retrieval, while leaving a more fundamental reliability question unresolved: Do accumulated skills in an agent's skill bank actually make positive mechanistic contributions? We investigate this question through systematic skill-bank audits across alternative bank compositions and deployment domains, measuring the resulting changes in agent behavior. These audits reveal two recurring reliability failures: coalition pollution, where bank-level gains conceal negative coalition-level skill contributions, and cross-domain utility reversal, where source-beneficial skills reverse their effects after transfer. These findings motivate two reliability interventions: coalition-aware skill selection during skill accumulation and label-free skill masking after transfer. Coalition-Aware Skill Selection (CASS) selects more reliable candidate skills for the current bank using sampled Shapley marginals. Unsupervised Skill-Masked Coalition Optimizer (u-SMCO) masks transferred skills whose exclusion improves retrieval quality on unlabeled target-domain data. Agentic experiments on LoCoMo, LongMemEval, HotpotQA, and ALFWorld show that CASS and u-SMCO consistently improve task performance and cross-domain generalization over strong skill-based self-evolving agent baselines. Beyond accuracy, coalition-conditioned reliability modeling reduces sensitivity to noisy outcome-reward fluctuations during reinforcement learning and exposes the limits of isolation-based skill evaluation.
Figures & tables
Figure 1: Motivation. (a) Existing self-evolving agents overlook coalition pollution during skill evolution, admitting unreliable skills with negative coalition-level contributions. (b) During cross-domain transfer, prior methods ignore cross-domain utility reversal, allowing skills that become unreliable in the target domain to influence subsequent decisions. These two mechanistic failure modes motivate our work.
Figure 2: Skill reliability is not intrinsic. We audit the same skills across bank compositions and deployment domains (skills relabeled Skill_XX ). (a) Acceptance is not contribution: the gate admits Skill_AP (purple star), yet it contributes ≈0 while the incumbent Skill_IN carries the bank. (b) Contribution depends on partners: the interaction matrix reveals synergies and antagonisms, strongest between Skill_IN and Skill_CR (boxed). (c) Same bank, different verdict: Skill_IN helps at the source (LoCoMo) and reverses at the target (HotpotQA-50).
Figure 3: Reliability failures revealed by SMRA. (a) Decisions at the noise floor: accept/reject margins are commensurate with the ±1 SE error bars; the shaded band marks the ±0.04 single-skill audit resolution, an order of magnitude wider. (b) Partner removed, marginal drops: insert falls from +0.084 to +0.018 once capture_contextual_relationships is removed. (c) Source benefit, target reversal: insert helps on the source (LoCoMo), reverses on the target (HP-50).
Skill
Δs(B3)
Δs(B4)
Full name
Skill_IN
+0.061∗
+0.084∗
insert
Skill_DE
+0.035
−0.011
delete
Skill_UP
+0.013
+0.019
update
Skill_NP
−0.002
+0.011
noop
Skill_TD
−0.010
−0.002
capture_temporal_details
Skill_CR
+0.014
+0.029
capture_contextual_relationships
Table 1: Per-skill SMRA contributions before ( B3 ) and after ( B4 ) the gate’s acceptance (LoCoMo). ∗ : 95% CI excludes zero; † : newly accepted.
Figure 4: One CASS gate decision. Each sampled coalition is knocked out and scored against the full bank on one fixed probe, so every margin is a paired difference from V(B) ; green marks removal that costs value, red that gains it. Algorithm A1 gives the full procedure.
Conversational Benchmarks
Embodied Interactive Tasks
LoCoMo
LongMemEval
Avg.
ALF-Seen
ALF-Unseen
Avg.
Method
F1
L-J
F1
L-J
F1
L-J
SR
#Stps ↓
SR
#Stps ↓
SR
CoN
30.86
41.72
30.78
56.44
30.82
49.08
75.00
19.15
80.60
17.38
77.80
ReadAgent
28.63
38.25
24.48
42.62
26.56
40.44
62.86
26.14
71.64
22.88
67.25
MemoryBank
36.80
44.43
30.56
41.96
33.68
43.20
60.71
28.23
66.42
24.64
63.57
A-MEM
39.39
49.71
25.83
38.04
32.61
43.88
62.86
27.53
70.15
23.79
66.51
Table 2: Main results (%). L-J is the LLM-as-Judge score; #Stps is the mean number of environment interactions (lower is better). † our reproduction from MemSkill ’s released project.
Figure 5: Gate decisions at the four outer epochs, and the resulting bank size. CASS accepts 1/12 , MemSkill 6/12 .
Benchmark
MemSkill e12
MemSkill final
Δ
LoCoMo
0.5653
0.5303
−3.50
LME
0.5000
0.5248
+2.48
HP-50
0.3008
0.2695
−3.13
Avg.
0.4554
0.4415
−1.38
Table 3: K -controlled ablation of the gate’s additions.
Figure 6: Case study. A coalition-polluting skill accepted by the baseline gate, a coalition-aware skill accepted by CASS , and a utility-reversing skill masked by u -SMCO .
Bank
K
Masked skills
Δ HP-50 (pp)
CASS s1
10 → 9
AP
2.13
CASS s2
11 → 9
IN , UP
4.09
CASS s3
10 → 8
ES , IN
4.55
MemSkill s1
13 → 9
ES , IN , TD , AP
5.01
MemSkill s2
12 → 10
IN , ES
8.58
MemSkill s3
11 → 9
TD , IN
7.92
Table 4: u -SMCO on the six trained banks with an unlabeled 20-query target probe; Δ HP-50 is the LLM-Judge change from masking. Skill codes follow Table 1 .
Large language model (LLM) agents can continually improve without parameter updates by converting historical experience into reusable procedural knowledge. However, existing methods often consolidate experience based on semantic similarity or LLM judgments, which may merge superficially related but behaviorally incompatible strategies and thereby degrade performance. To address the issue, we propose SkillCommit, an online skill evolution framework that continuously transforms experience into a hierarchical library of reusable skills. Rather than generalizing from individual experiences immediately, SkillCommit first preserves each experience as an instance-specific patch, retaining its locally validated behavior. As patches accumulate, SkillCommit abstracts those sharing a common behavioral mechanism into higher-level skills. Specifically, for each incoming skill, embedding-based retrieval first identifies candidate related skills. Cross-instance replay and an LLM-based mechanism check determine whether these skills transfer across cases and share a common underlying mechanism. Candidates that pass both checks are abstracted into a higher-level skill and committed only if it preserves the validated behavior of all constituent skills. Experiments on RuleArena, OpenExempt and KOR-Bench demonstrate that SkillCommit consistently improves agent performance across diverse domains. Moreover, the learned skills transfer across model scales and families, enabling cross-model experience transfer.
Yu He, Weikai Yang
Nanjing University · The Hong Kong University of Science and Technology (Guangzhou)
Self-evolving agents accumulate capability by distilling reusable skills from their execution trajectories, but we find this process is not monotonic: past a critical pool size, newly added skills degrade performance instead of improving it. We formalize this capability-contamination phase transition and trace it to a structural cause: once a defective skill enters the decision context, it becomes reference material for distilling later skills, forming cross-round contamination chains. We further show the contamination is structurally irreversible: removing a source skill after the fact cannot erase the flawed reasoning its descendants have already inherited, so post-hoc rollback recovers only a small fraction of the lost performance. This makes skill admission a pre-commit necessity rather than a post-hoc fix, and motivates Verifier-as-Gatekeeper (VaG): a progressive trust hierarchy whose three heterogeneous critics - structural validity, behavioral harmlessness, and semantic consistency - filter each skill individually, coupled with a marginal-gain subset selection that removes combinatorial contamination at the top tier before skills reach the runtime context. On Terminal-Bench 2, unconditional accumulation rises to a peak and then degrades, giving back most of its gains as the pool keeps growing, and post-hoc removal of the culprit skills recovers only a small part of the drop - the empirical signature of irreversibility. In contrast, VaG improves every round, reaching 72% pass@1 with a pool roughly 5x smaller, and its frozen skill pool transfers positively to four other backbones and a second benchmark without re-evolution. Ablations confirm the three critics are complementary and mutually non-substitutable, each intercepting a largely disjoint class of harmful skills.
Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: When do skills help, why do they work, and where do they fail? Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7% of skill cases, versus 4.5% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6% to 3.3%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.
Zhiyuan Jiang, Fangrui Huang, Hanwen Xing +6
Princeton University · Stanford University · University of Southern California +2