SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents

Bo-Yu Chen; Zhi Zhou; Yu-Feng Li

cs.AIOct 24, 2026

SkillShift-Bench: Benchmarking Skill Learning under Environment Evolution for Web Agents

Authors: Bo-Yu Chen, Zhi Zhou, Yu-Feng Li

Organizations: School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University

Abstract

Skill learning enables web agents to reuse prior experience for long-horizon interactive tasks. However, learned skills are typically evaluated in closed environments, overlooking the fact that real-world web deployment involves continuous shifts in invocation context, perceptual grounding, and execution feasibility. To this end, we introduce SkillShift-Bench, the first benchmark for evaluating skill learning with controlled open-environment shifts. SkillShift-Bench simulates Contextual Shift, Perceptual Shift, and Execution Shift across three self-hosted web systems, and evaluates agents with diagnostics covering task success, interaction cost, skill use, and skill failure. Comprehensive experiments reveal systematic robustness degradation under controlled shifts, characterized by lower success rates, higher interaction costs, and more frequent skill failures, with Execution Shift exerting the strongest pressure. We further formulate shifted skill execution as a nonstationary local executability problem and provide both a dynamic-regret analysis and a practical test-time adaptation method, demonstrating the feasibility of improving learned-skill execution without relearning the skill library. Code, tasks, environments, documentation, and the leaderboard are available at https://skillshift-bench.github.io/.

Figures & tables

Figure 1: Three controlled shift modes in SkillShift-Bench. Aligned variants preserve task intent, task-critical entities, required functionality, and success criteria, while Contextual, Perceptual, and Execution Shifts respectively target skill invocation, grounding evidence, and runtime feasibility.

Figure 2: Execution flow of MPCR within the web-agent loop.

Metric	Method	Website			Environment						O/A
Metric	Method	Git- Lab	Ma- gento	Word- Press	Src	Src- Perc	Src- Exec	Tgt	Tgt- Perc	Tgt- Exec	O/A
SR (%)	AWM	17.71	25.49	62.66	50.07	43.35	22.45	41.28	37.65	16.90	35.29
	ASI	20.81	27.78	60.27	53.65	48.80	24.43	41.84	33.19	15.79	36.29
	SkillWeaver	13.78	14.37	32.29	35.13	28.75	15.63	17.80	13.85	9.71	20.15
	WALT	21.07	27.95	55.38	46.82	43.53	25.46	41.17	32.74	19.09	34.80
AS	AWM	8.03	8.22	4.76	6.04	6.32	8.31	6.34	6.59	8.44	7.01

Table 1: Transfer diagnostics across websites and environment settings. O/A denotes the overall average across the corresponding settings. Bold values mark the highest SR and the lowest AS / SFR entries. ASC is reported as a non-directional diagnostic. “–” indicates metrics not applicable to AWM because it uses prompted textual workflows rather than executable skill invocations Wang et al. (2025b) .

Figure 3: Robustness profiles under controlled environment evolution.

Figure 4: Cumulative task-failure advantage of MPCR over ASI.

Figure 5: Success and trajectory cost for fixed-library executor adaptation. Points denote method centers across environment condition groups, and shaded hulls show configuration-level variation.

Environ- ment	MPCR		ASI
Environ- ment	SR (%)	ASC	SR (%)	ASC
Src	56.67 (+7.35%)	0.22	52.79	0.05
Src-Perc	57.09 (+7.68%)	0.20	53.02	0.13
Src-Exec	33.23 (+24.55%)	0.19	26.68	0.07
Tgt	43.32 (+1.14%)	0.12	42.83	0.04
Tgt-Perc	37.24 (+8.51%)	0.14	34.32	0.05

Table 2: MPCR versus ASI across transfer conditions. Relative SR gains are computed against ASI.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Variation Diversity	Variation Controllability	Evaluation Reproducibility	Website Fidelity	Diagnostic Granularity	Experience Transferability	Failure Attributability
Mind2Web	✗	✗	✓	$\triangle$	$\triangle$	✗	✗
WebArena	✗	✗	✓	✓	$\triangle$	✗	✗
Visual WebArena	✗	✗	✓	✓	$\triangle$	✗	✗
WorkArena / WorkArena++	✗	✗	$\triangle$	✓	$\triangle$	✗	✗
WebCanvas	$\triangle$	✗	$\triangle$	✓	✓	✗	$\triangle$
Online- Mind2Web	✗	✗	$\triangle$	✓	$\triangle$	✗	✗

Appendix

Table 3: Comparison of benchmarks across evaluation criteria. ✓ = explicitly supported as a central benchmark objective under the criterion definitions above,

\triangle

= partially supported or indirectly covered, and ✗ = not a primary target of the original benchmark design or not publicly verifiable.

Panel of Figure 6 — Figure 6: GitLab source and target variants.

Panel of Figure 7 — Figure 7: Magento source and target variants.

Panel of Figure 8 — Figure 8: WordPress source and target variants.

Regime	Behavior	Mechanism
once	Injected once per episode; not reinstated after dismissal	Server-side episode flag plus client-side storage guard
multiple	Reinjected on every page navigation	Persistent guards removed; injection predicate always true
random	Injected with fixed probability at a randomized in-page delay	Per-load Bernoulli draw plus client-side delay timer

Appendix

Table 4: Injection regimes available for Execution Shift.

Regression set	n	Steps	Trunc.
Contextual	13	5.4 $\to$ 6.2	6/13
Perceptual	10	5.9 $\to$ 9.7	9/10
Execution	29	5.4 $\to$ 9.5	26/29

Appendix

Table 5: Regression sets used for failure-mode analysis, on Magento with GPT-5 mini. “Steps” reports mean trajectory length in the reference condition and in the shifted condition; “Trunc.” counts shifted runs that exhaust the step budget.

\widetilde{Q}_{h}^{k}(\xi,\omega)=\min\left\{H-h+1,\hat{r}_{h}^{k}(\xi,\omega)+b_{h}^{k}(\xi,\omega)+\hat{P}_{h}^{k}(\cdot\mid\xi,\omega)^{\top}\widetilde{V}_{h+1}^{k}\right\}.

Appendix

Algorithm 1 SWOP: Sliding Window Optimistic Prior-Guided Planning

SWOP (analysis)	MPCR (executor)
Reference prior $\pi_{\mathrm{ref}}$	Fixed skill-library and base-executor preference
Sliding-window recent evidence	Current-page feasibility verification
Optimism for unresolved uncertainty	Soft scoring that downweights rather than filters
KL regularization toward the prior	Reranking constrained to the proposed shortlist

Appendix

Table 6: Componentwise correspondence between the SWOP analysis model and the MPCR executor.

Variant	Ver.	Sco.	Rer.	SR (%)	ASC
Propose only	✗	✗	✗	32.73	0.60
Verify + Score	✓	✓	✗	27.93	0.15
Rerank only (blind)	✗	✗	✓	37.04	0.68
Verify + Score + Random	✓	✓	rand.	0 8.09	0.31
MPCR (full)	✓	✓	✓	39.18	0.24

Appendix

Table 7: Component ablation of MPCR on Magento, averaged over the six evaluation conditions. “Ver.”, “Sco.”, and “Rer.” denote the verification, scoring, and reranking stages. “Rerank only” applies the pairwise comparator to the raw proposal order without feasibility evidence, and therefore issues the same number of reranking calls as full MPCR. “Verify + Score + Random” keeps the verified shortlist but replaces the comparator with a fixed-seed random choice, and therefore issues no reranking call at all.

	DeepSeek-V3.2			GPT-5 mini			Claude Haiku 4.5
Method	GitLab	Magento	WordPress	GitLab	Magento	WordPress	GitLab	Magento	WordPress
AWM	35	23	5	27	15	6	10	6	3
ASI	14	19	12	9	13	4	11	13	10
SkillWeaver	28	18	23	9	26	15	12	17	14
WALT	8	8	6	6	7	5	5	6	3

Appendix

Table 10: Skill library size after learning. Counts are descriptive rather than directly comparable across methods because the retained units have different semantics.

Metric	LLM	Method				Environment						O/A
Metric	LLM	AWM	ASI	Skill- Weaver	WALT	Src	Src- Perc	Src- Exec	Tgt	Tgt- Perc	Tgt- Exec	O/A
SR (%)	DS	35.30	38.32	17.12	36.44	47.89	37.32	23.98	35.81	29.76	16.01	31.80
	GPT	37.56	37.90	24.99	40.46	46.30	45.41	27.14	38.72	33.58	20.23	35.23
	Cld	33.00	32.64	18.32	27.50	45.07	40.60	14.87	32.03	24.73	9.89	27.86
AS	DS	7.35	6.94	17.53	15.90	8.65	10.45	14.35	11.25	12.03	14.84	11.93
	GPT	6.20	6.21	12.80	9.87	7.25	7.50	11.31	7.51	8.56	10.49	8.77

Appendix

Table 11: Expanded transfer-oriented diagnostic results across methods and environment settings. O/A denotes the overall average across the corresponding settings. LLM abbreviations are as follows: DS = DeepSeek-V3.2, GPT = GPT-5 mini, and Cld = Claude Haiku 4.5 DeepSeek-AI (2025) ; OpenAI (2025) ; Anthropic (2025) . Bold values mark the best SR, AS, and SFR entries. ASC is reported as a non-directional diagnostic. “–” denotes non-applicable metrics: AWM uses textual workflows, so executable-skill metrics are undefined.

Condition	AWM	AWM + MPCR	Rel.
Src	32.46	41.23	+27.0%
Src-Perc	31.00	40.35	+30.2%
Src-Exec	13.27	20.35	+53.4%
Tgt	49.11	57.02	+16.1%
Tgt-Perc	32.46	39.47	+21.6%
Tgt-Exec	10.00	18.42	+84.2%

Appendix

Table 12: Success rate (%) of AWM with and without MPCR on Magento. “Rel.” is the relative gain over the AWM baseline.

Condition	MPCR	ASI	Rel.
Src	49.12 $\pm$ 2.32	43.27 $\pm$ 1.34	+13.5%
Src-Perc	53.80 $\pm$ 2.03	38.30 $\pm$ 1.34	+40.5%
Src-Exec	29.24 $\pm$ 1.34	17.54 $\pm$ 0.88	+66.7%
Tgt	41.52 $\pm$ 1.34	40.06 $\pm$ 0.51	0 +3.6%
Tgt-Perc	36.84 $\pm$ 0.88	36.55 $\pm$ 1.34	0 +0.8%
Tgt-Exec	19.01 $\pm$ 0.51	17.25 $\pm$ 1.01	+10.2%

Appendix

Table 13: Success rate (%), mean

\pm

standard deviation over three seeds on Magento. “Rel.” is the relative gain of MPCR over the ASI baseline.

Regime	Context	MPCR	ASI	Rel.
once	Src	30.70	16.96	+81.0%
once	Tgt	18.42	16.67	+10.5%
multiple	Src	20.35	17.95	+13.4%
multiple	Tgt	0 8.77	0 7.86	+11.6%
random	Src	30.97	26.95	+14.9%
random	Tgt	20.18	18.18	+11.0%

Appendix

Table 14: Success rate (%) under the three Execution Shift injection regimes on Magento. “Rel.” is the relative gain of MPCR over the ASI baseline.

Method	Tgt-Perc	Tgt-Exec
ASI (fixed Src library)	36.84	16.67
Re-learned in Tgt	35.09	14.91
MPCR (same fixed library)	37.72	18.42

Appendix

Table 15: Success rate (%) of target-environment re-learning against fixed-library transfer under the two shifted target conditions on Magento.

Agent Skill Learning

Agent skill learning develops reusable procedures and behaviors that help AI agents solve new tasks and adapt from experience.

See all 172 papers

SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents
Apr 19, 2026Ziao Zhang, Kou Shi, Shiting Huang +13Lifelong Learning Agents Long-Horizon Agent Evaluation
University of Science and Technology of China · University of Toronto · University of Sydney
SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
Apr 22, 2026Shanshan Zhong, Yi Lu, Jingjie Ning +7Continual Learning for LLM Agents LLM Agent Evaluation
Carnegie Mellon University · Amazon AGI
SkillEvolver: Skill Learning as a Meta-Skill
May 11, 2026Genrui Zhang, Erle Zhu, Jinfeng Zhou +2LLM Agent Self-Improvement LLM Agent Skill Learning
2Beijing Jiaotong University · 1Tsinghua University

Apr 19, 2026J/K move · Enter open · S save

SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents

Ziao Zhang, Kou Shi, Shiting Huang, Avery Nie+12

University of Science and Technology of China · University of Toronto · University of Sydney

Lifelong Learning Agents Long-Horizon Agent Evaluation AI Agent Benchmarks Agent Skill Learning

Open paper AskPDF arXiv

As the capability frontier of autonomous agents continues to expand, they are increasingly able to complete specialized tasks through plug-and-play external skills. Yet current benchmarks mostly test whether models can use provided skills, leaving open whether they can discover skills from experience, repair them after failure, and maintain a coherent library over time. We introduce SkillFlow, a benchmark of 166 tasks across 20 families in which task construction within each family follows a Domain-Agnostic Execution Flow (DAEF) that defines an agent workflow framework, allowing these tasks to share a consistent workflow. Agents are evaluated under an Agentic Lifelong Learning protocol in which they begin without skills, solve tasks sequentially within each family, externalize lessons through trajectory- and rubric-driven skill patches, and carry the updated library forward. Experiments reveal a substantial capability gap. For Claude Opus 4.6, lifelong skill evolution improves task success from 62.65% to 71.08% (+8.43 points). However, high skill usage does not necessarily imply high utility: Kimi K2.5 gains only +0.60 points despite 66.87% skill usage, while Qwen-Coder-Next reaches only a 44.58% task completion rate and still regresses relative to the vanilla setting. SkillFlow contributes a structured testbed for this direction and an in-depth empirical analysis of skill discovery, patching, transfer, and their failure modes under lifelong evaluation.

Abstract

Agent Skill Learning

Explore similar work

SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents

SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks

SkillEvolver: Skill Learning as a Meta-Skill