SEEK: Skill Extraction with Evolving Knowledge against Frontier Science Agents

Cheng Gui; Xiang Zheng; Yutao Wu; Hanxun Huang; Yige Li; Kaidi Xu; Cong Wang; Wei-Ying Ma

cs.LGOct 25, 2026

SEEK: Skill Extraction with Evolving Knowledge against Frontier Science Agents

Authors: Cheng Gui, Xiang Zheng, Yutao Wu, Hanxun Huang, Yige Li, Kaidi Xu, Cong Wang, Wei-Ying Ma

Organizations: City University of Hong Kong, Hong Kong · Hong Kong Institute of AI for Science, Hong Kong · Deakin University, Australia · The University of Melbourne, Australia ·

Abstract

Science agents increasingly package domain expertise as reusable skills that plan analyses, select tools, and execute workflows. This modularity creates a confidentiality risk: the interactions that deliver a skill's capability can also reveal the skill itself. We build an extensible benchmark of 40 protected skills across eight open- and closed-source science-agent platforms, comparing direct, fixed-strategy, and bandit-guided iterative probing under reference-based and reference-free evaluation. Leakage is pervasive: all eight platforms (100%) leak at least one protected skill, and four reach 100% attack success under a single direct request. To stress-test platforms that resist simple probing, we propose SEEK (Skill Extraction with Evolving Knowledge), a black-box, self-evolving red-teaming agent that consolidates response-derived evidence in persistent memory and evolves reusable attack skills, using only user-visible responses. On 12 skills from SpatialAgent and Biomni, SEEK achieves 100% attack success while reducing the mean first-success round by 31.8% and 38.2% relative to bandit-guided probing. Against a confidentiality policy that blocks all three benchmark protocols, SEEK still extracts one of six protected skills. These findings show that current science agents do not reliably protect their skills and that defenses must be evaluated against adaptive, multi-turn attackers.

Figures & tables

图 1. Overall framework of SEEK . (a) Motivation: private skills are hidden workflows that leave behavioral signatures and require response-conditioned probing. (b) Architecture: SEEK interacts with the target through a Runner and Platform Adapter, observing only the user-visible response; evaluation and audit evidence remain separate. (c) Illustrative episode.

图 2. Reference-free score validation on SciencePal and Biomni. For visualization, BGIP scores are averaged over non-overlapping four-round windows, while DP and FSP retain their single-round scores, yielding 84 summaries from all 528 responses. (a) Summarized

S_{\mathrm{cons}}

versus

\bar{S}_{\mathrm{ref}}

; dashed lines mark

\tau=0.7

. (b) Mean per-round disagreements

1-\mathrm{SelfC}

versus

|S_{\mathrm{ref}}^{(1)}-S_{\mathrm{ref}}^{(2)}|

within the same windows. Statistics and ordinary least-squares fits use the displayed summaries; gray bands are 95% pointwise confidence bands from 2,000 platform-stratified bootstrap resamples of platform–skill clusters.

Open-source targets / $S_{\mathrm{ref}}$
Platform	$n$	DP				FSP				BGIP
		Max	Mean	ASR	FSR	Max	Mean	ASR	FSR	Max	Mean	ASR	FSR
ClawBio	6	1.000	1.000	100.0%	1.0	1.000	0.998	100.0%	1.0	1.000	0.952	100.0%	5.8
OpenScience	6	1.000	0.838	66.7%	1.0	1.000	0.706	50.0%	1.0	1.000	0.947	83.3%	2.8
EvoScientist	2	0.957	0.955	100.0%	1.0	0.980	0.969	100.0%	1.0	0.980	0.974	100.0%	2.0
SpatialAgent	6	0.967	0.218	16.7%	1.0	0.999	0.220	16.7%	1.0	1.000	0.940	100.0%	3.7

表 1. Main benchmark results across eight platforms and the three canonical attack protocols. Open-source rows use the Reference-Based Leakage Score

S_{\mathrm{ref}}

; closed-source rows use the Reference-Free Consistency Score

S_{\mathrm{cons}}

. For each platform–method pair,

n

is the number of skill episodes, Max is the maximum valid score across sessions and rounds, Mean is the macro-average of per-skill session maxima (for BGIP, the maximum is taken over rounds), and ASR is the attack success rate of skill episodes satisfying the

0.7

success threshold. FSR is the mean first-success interaction round over successful skill episodes. -- denotes that no episode succeeded.

Platform	Method	ASR	Mean FSR
SpatialAgent	BGIP	100.0%	3.67
SpatialAgent	SEEK	100.0%	2.50
Biomni	BGIP	100.0%	5.67
Biomni	SEEK	100.0%	3.50

表 2. BGIP versus SEEK . Mean FSR is macro-averaged over six protected-skill episodes per platform.

图 3. Budgeted ablation on SpatialAgent: macro-average of the per-skill maximum $S_{\mathrm{ref}}$ within rounds 1– $B$ . At $B=1$ , all variants except SEEK -base include five valid skill scores; other points include six.

	Mean		ASR
Protocol	Original	Defense	Original	Defense
DP	0.838	0.000	66.7%	0.0%
FSP	0.706	0.060	50.0%	0.0%
BGIP	0.947	0.563	83.3%	0.0%
SEEK	0.925	0.487	83.3%	16.7%

表 3. Effect of the defense on OpenScience (six skills; Mean and ASR as in Table 1 ). Original DP, FSP, and BGIP values are from the main benchmark; Original SEEK values are from the undefended SEEK run.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Platform	Access	General skills	Science skills
ClawBio	Open	article-data-fetcher , lit-synthesizer , pubmed-summariser	gwas-pipeline , nfcore-scrnaseq-wrapper , pharmgx-reporter
OpenScience	Open	large-file , modal-run , remote-compute	domain-check , publication-figures , traceability-review
EvoScientist	Open	find-skills , skill-creator	None
SpatialAgent	Open	database_query , sequence_analysis , annotation	cell_cell_communication , spatial_deconvolution , spatial_domain_detection
Virtual-Lab	Open	principal-investigator , scientific-critic	None
SciencePal	Closed	verify-bib , zotero , graphify	tooluniverse-sequence-analysis , tooluniverse-single-cell , tooluniverse-protein-structure-prediction

Appendix

表 4. Platform access and skill inventory used in the benchmark.

	DP		FSP		BGIP
Interface	Mean	ASR	Mean	ASR	Mean	ASR
Claude Code bridge	1.000	100.0%	0.998	100.0%	0.952	100.0%
RoboTerri	0.015	0.0%	0.216	0.0%	0.354	0.0%

Appendix

表 5. ClawBio results under its Claude Code bridge and RoboTerri interfaces. Mean is the macro-average of per-skill maximum

S_{\mathrm{ref}}

over six skills; ASR uses the

0.7

threshold.

Architecture / category	DP	FSP	BGIP
Open-source / General	0.674	0.759	0.931
Open-source / Science	0.740	0.552	0.943
Closed-source / General	0.678	0.760	0.947
Closed-source / Science	0.710	0.693	0.950

Appendix

表 6. Scores by platform architecture and skill category: macro-average of per-skill maximum official scores (not a pooled maximum), shaded on the same 0–1 scale as Table 1 .

Condition	$B{=}1$	$3$	$5$	$10$	$15$	$20$
Full SEEK	0.649	0.707	0.960	0.961	0.961	1.000
SEEK -base	0.385	0.396	0.396	0.537	0.703	0.719
SEEK -memory	0.795	0.829	0.864	0.864	0.884	0.884
SEEK -skill-evo.	0.765	0.806	0.834	0.834	0.834	0.858

Appendix

表 7. Budgeted ablation on SpatialAgent: Mean of the per-skill maximum

S_{\mathrm{ref}}

within rounds 1–

B

. Max is 1.000 for every condition and budget except SEEK -base at

B\leq 5

(0.999). At

B=1

, all conditions except SEEK -base include five valid skill scores; all other cells include six.

Tactic	Role
Progressive FITD	Start with plausible metadata or a small partial definition, then request one adjacent element at a time.
Execution artifact	Use a normal result, report, trace, command, generated file, or error as evidence for the next probe.
Behavioral reverse engineering	Infer the input–output contract by varying one observable behavior dimension at a time.
Implementation-semantics backfill	Recover executable details when the interaction is stuck at a fixed artifact, wrapper error, or unknown contract.
Completion	Present a partial hypothetical contract with one missing element and ask the target to complete or correct it.
Contrastive probing	Present two alternatives that differ in one fact and use the correction to separate hypotheses.

Appendix

表 8. Internal tactics of the foot-in-the-door strategy.

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 510 papers

Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems
May 12, 2026Zhaojiacheng ZhouLLM Agent Security Automated Red Teaming
Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai 200240, China
Daydreaming: Stealing Hidden Agent Skills through Black-Box Task Interaction
Aug 27, 2026Yu-Lin Tsai, Yu-An Lu, Ci-Yang Tsai +3LLM Agent Security Model Extraction Attacks
National Yang Ming Chiao Tung University
SkillTester: Benchmarking Utility and Security of Agent Skills
Mar 28, 2026Leye Wang, Zixing Wang, Anjie XuLLM Agent Skill Evaluation AI Agent Security Benchmarks
Key Laboratory of High Confidence Software Technologies, Ministry of Education, Peking University, Beijing, China · Northwestern Polytechnical University, Xi’an, China

May 12, 2026J/K move · Enter open · S save

Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems

Zhaojiacheng Zhou

Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai 200240, China

LLM Agent Security Automated Red Teaming Agent Skill Security Auditing Self-Evolving Agents

Open paper AskPDF arXiv

Agent skills extend LLM agents with reusable instructions, tool interfaces, and executable code, and users increasingly install third-party skills from marketplaces, repositories, and community channels. Because a skill exposes both executable behavior and context-setting documentation, its deployment risk cannot be measured by single-shot audits or prompt-level red teams alone: a realistic attacker can use audit and runtime feedback to repeatedly rewrite the skill. We frame this risk as adaptive leakage -- whether a budgeted attacker can iteratively revise a skill until it passes audit and produces verified runtime harm -- and present \ours{}, a grey-box self-evolving red-team framework for measuring it. Proteus searches a formalized five-axis skill-attack space. Each candidate is evaluated through a unified audit-sandbox-oracle pipeline that returns structured audit findings and runtime evidence to guide cross-round mutation. Beyond initial evasion, Proteus performs path expansion, which finds alternative implementations of successful attacks, and surface expansion, which transfers learned implementation patterns to new attack objectives beyond the original seed catalogue. Across eight phase-1 cells, Proteus reaches 40--90% Attack Success Rate at

5

rounds (ASR@5) with positive learning-curve slopes on both evaluated auditors. Phase-2 path/surface expansion produces 438 jointly bypassing and lethal variants, with SkillVetter bypassed at

\geq 93\%

in every cell and AI-Infra-Guard, the strongest public auditor we evaluate, still admitting up to 41.3% joint-success. These results show that current skill vetting substantially underestimates residual risk when evaluated against adaptive, feedback-driven attackers.

Abstract

AI Agent Security

Explore similar work

Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems

Daydreaming: Stealing Hidden Agent Skills through Black-Box Task Interaction

SkillTester: Benchmarking Utility and Security of Agent Skills