Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale

Hao Li; Chunjiang Mu; Jianhao Chen; Siyue Ren; Zhiyao Cui; Yiqun Zhang; Lei Bai; Shuyue Hu

cs.CLMar 2, 2026

Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale

Authors: Hao Li, Chunjiang Mu, Jianhao Chen, Siyue Ren, Zhiyao Cui, Yiqun Zhang, Lei Bai, Shuyue Hu

Organizations: Shanghai Artificial Intelligence Laboratory

Abstract

The rapid proliferation of Claude agent skills has raised the central question of how to effectively leverage, manage, and scale the agent skill ecosystem. In this paper, we propose AgentSkillOS, the first principled framework for skill selection, orchestration, and ecosystem-level management. AgentSkillOS comprises two stages: (i) Manage Skills, which organizes skills into a capability tree via node-level recursive categorization for efficient discovery; and (ii) Solve Tasks, which retrieves, orchestrates, and executes multiple skills through DAG-based pipelines. To evaluate the agent's ability to invoke skills, we construct a benchmark of 30 artifact-rich tasks across five categories: data computation, document creation, motion video, visual design, and web interaction. We assess the quality of task outputs using LLM-based pairwise evaluation, and the results are aggregated via a Bradley-Terry model to produce unified quality scores. Experiments across three skill ecosystem scales (200 to 200K skills) show that tree-based retrieval effectively approximates oracle skill selection, and that DAG-based orchestration substantially outperforms native flat invocation even when given the identical skill set. Our findings confirm that structured composition is the key to unlocking skill potential. Our GitHub repository is available at:https://github.com/ynulihao/AgentSkillOS.

Figures & tables

Figure 1: AgentSkillOS is a principled framework for efficient skill retrieval, orchestration, and ecosystem-level management to solve user-specified tasks.

Figure 2: Overview of our benchmark framework. The benchmark consists of a human-crafted dataset of 30 tasks spanning five categories: Data Computation, Document Creation, Motion Video, Visual Design, and Web Interaction. For evaluation, artifacts produced by agents in diverse raw formats (csv, video, json, web, pdf, gif) are first converted into LLM-evaluable formats (text and image) via scripted pipelines. An LLM judge then performs pairwise comparisons between systems with position swapping to mitigate order bias, and the results are aggregated into a global win matrix. Finally, a Bradley–Terry model is fitted via maximum likelihood to obtain strength parameters $\beta_{i}$ , which are rescaled to produce continuous ranking scores $S_{i}$ for fine-grained differentiation of agent performance.

Figure 3: Overview of benchmark tasks. (a) All 30 tasks organized into five categories: Data Computation, Document Creation, Motion Video, Visual Design, and Web Interaction, with six tasks per category. Each task is listed with its abbreviated name. (b) Task complexity distributions across categories, measured by three dimensions: the number of skills required to complete the task, the number of output files the task expects, and the number of distinct output formats involved.

$\|\mathcal{S}\|$	Method	Data	Document	Video	Visual	Web	BT
$200$	Quality-First	18/1/5	12/6/6	11/7/6	17/7/0	20/3/1	100.0
	Efficiency-First	13/2/9	7/10/7	6/9/9	11/8/5	14/3/7	58.5
	Simplicity-First	10/4/10	7/9/8	8/7/9	12/6/6	13/3/8	53.6
	w/ Full Pool.	5/4/15	11/8/5	13/3/8	3/4/17	2/4/18	24.3
	Vanilla	6/5/13	4/5/15	5/8/11	2/5/17	2/5/17	0.0
$1\text{K}$	Quality-First	14/4/6	11/7/6	13/2/9	9/10/5	20/2/2	100.0

Table 1: Per-category W / T / L (wins / ties / losses) counts from LLM-judge pairwise comparisons against all other methods. Category abbreviations: Data = Data Computation, Document = Document Creation, Video = Motion Video, Visual = visual Design, Web = Web Interaction. Each pair is judged twice with reversed presentation order and merged into a single debiased decision. The last column reports the Bradley–Terry score rescaled to

[0,100]

, where

100

corresponds to the top-ranked method and

0

to the bottom-ranked method within each ecosystem size. Bold indicates the best result per column within each ecosystem size.

Figure 4: Per-category and overall Bradley-Terry scores (rescaled to

[0,100]

) derived from the pairwise comparisons in Table 1 , shown for three skill ecosystem sizes (

|\mathcal{S}|{=}200

1\text{K}

200\text{K}

). Larger polygon area indicates stronger performance across categories. Three AgentSkillOS variants achieve the broadest coverage in all settings, consistent with the top Bradley–Terry scores reported in Table 1 .

Figure 5: Ablation study of AgentSkillOS components. Each panel shows pairwise W / T / L counts (green = win, gray = tie, orange = lose) of Quality-First against four ablation variants for

|\mathcal{S}|{=}200

1\text{K}

, and

200\text{K}

. Removing DAG orchestration (w/ Oracle Skills, w/ Retrieval) or both retrieval and orchestration (w/ Full Pool) consistently degrades performance, confirming that both components are essential. Quality-First closely approaches the oracle upper bound (Quality-First (Oracle)), validating the effectiveness of tree-based skill retrieval.

Figure 6: Structural properties of DAG-based skill orchestration plans across three strategies, aggregated over all 30 benchmark tasks and three skill ecosystem sizes (

|\mathcal{S}|{=}200

1\text{K}

200\text{K}

). Each panel reports a different graph-level metric: (a) node count, (b) edge count, (c) max width, and (d) max depth. Boxes span the interquartile range with individual task results overlaid as jittered points. Quality-First produces the largest and deepest DAGs, reflecting its strategy of adding preparation and refinement stages. Efficiency-First produces graphs of comparable size but with notably greater width and shallower depth, reflecting its preference for maximizing parallel execution over sequential refinement. Simplicity-First yields the most compact graphs, consistent with its minimal-node design objective.

Figure 7: Qualitative comparison between the vanilla Claude Code baseline (left) and the Quality-First variant of AgentSkillOS (right) on three representative tasks from the $|\mathcal{S}|{=}200$ setting. Case 1: an immersive scrolling web page on space exploration history with illustrations. Case 2: a calculus derivatives teaching package comprising an animation video and a companion PDF handout. Case 3: a quantum computing fundamentals presentation with generated illustrations. In all cases, AgentSkillOS produces substantially higher-quality artifacts by effectively selecting and composing relevant skills, whereas the vanilla baseline, lacking structured skill discovery and orchestration, yields visually plain or text-heavy outputs.

LLM Agent Orchestration

Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.

See all 458 papers

OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents
May 22, 2026Jiahao Ying, Boxian Ai, Wei Tang +2LLM Agent Skill Evaluation LLM Agent Evaluation
Singapore Management University · Institute of Trustworthy Embodied AI, Fudan University · Joy Future Academy, JD
SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents
Jul 17, 2026Yanze Wang, Pengfei Yao, Tianyi Sun +7LLM Agent Evaluation LLM Agent Skill Retrieval
1EverMind · 2Shanda Group · 3Peking University
SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale
Sep 30, 2026Guanqun Yang, Wenlong Zhang, Tian Shi +1LLM Agent Skill Retrieval Information Retrieval
Stevens Institute of Technology, Hoboken, NJ, USA · Independent Researcher

May 22, 2026J/K move · Enter open · S save

OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents

Jiahao Ying, Boxian Ai, Wei Tang, Siyuan Liu+1

Singapore Management University · Institute of Trustworthy Embodied AI, Fudan University · Joy Future Academy, JD

LLM Agent Skill Evaluation LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Skills, i.e., structured workflow instructions distilled for large language models (LLMs), are becoming an increasingly important mechanism for improving agent performance on real-world downstream tasks. However, as the open-source skill ecosystem rapidly expands, it remains unclear how different models and agent frameworks interact with skills, how to evaluate skill quality, and how users should select skills under practical cost-performance trade-offs. In this paper, we present OpenSkillEval, an automatic evaluation framework for both skill-augmented agent systems and the skills themselves. Instead of relying on static benchmarks, OpenSkillEval automatically constructs realistic task instances from evolving real-world artifacts across five categories of downstream applications: presentation generation, front-end web design, poster generation, data visualization, and report generation. It further collects and organizes community-contributed skills for controlled comparison under unified task settings. Using more than 600 dynamically generated task instances and 30 open-source skills, we conduct a systematic evaluation of state-of-the-art models and agent frameworks. Our results show that skill availability does not guarantee effective skill usage, that the benefit of skill augmentation depends strongly on both the underlying model and the agent framework, and that many publicly popular skills do not consistently outperform base agents without skills. These findings highlight the need for dynamic, task-grounded evaluation and provide practical insights into the design, selection, and deployment of skills for LLM agents. Additional cases and benchmark resources are available on the project website: https://yingjiahao14.github.io/OpenSkillEval-Web/.

Abstract

LLM Agent Orchestration

Explore similar work

OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents

SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents

SkillSeek: Revisiting Agent Skill Retrieval at Marketplace Scale