cs.AIOct 16, 2026

MS-Eval: Benchmarking Sub-Agent Delegation and Orchestration in MCP-Intensive Working Scenarios

Authors: Bowen Ding, Qixin Wang, Lexing Ruan, Weiyi Lu, Qiwei Ge, Ruishan Fang, Tao Lin

Organizations: Zhejiang University · School of Engineering, Westlake University · Independent Researcher · Institute of Advanced Technology, Westlake Institute for Advanced Study

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM agents are increasingly integrated into real-world works, yet they must handle diverse domain-specific tools and complex multi-sub-task problems that sequential tool invocation resolves inefficiently and with severe context bloat. The Main-Agent Delegation and Sub-Agent Execution (MAD-SAE) paradigm addresses these issues by delegating sub-tasks to specialized sub-agents for parallel execution in isolated contexts, but its delegation and orchestration quality remains unquantified. We introduce MS-Eval, a unified evaluation kit comprising an MCP-intensive benchmark, a controlled execution harness, tailored delegation and orchestration metrics, and the MSQ index for comprehensive assessment. Evaluating four MAD-SAE variants and eight frontier models reveals that (1) sub-agent specialization (Spec) is the best practical practice; (2) open-source models are largely competitive with few exceptions; and (3) MAD-SAE generally raises cost with contingent payoff, benefiting most open-source models in both performance and efficiency while leaving closed-source models with higher cost and degraded performance. Our case study further traces these gains to better single-agent solutions, effective context offloading, and genuine sub-task parallelism, whereas unnecessary or redundant delegation turns the same cost into pure overhead. Codes: https://github.com/LINs-lab/MS-Eval

Figures & tables

Figure 1: The leaderboard of the MSQ Index. The index measures the comprehensive quality of Main-Agent Delegation and Sub-Agent Execution paradigm (detailed in § 4.3 ), which is dominated by open-source models. Oracle, highlighted in yellow, probes the upper bound of orchestration utilities.
Figure 2: The Overview of MS-Eval Kit. Benchmark exemplifies arxiv-survey-presentation with its Task Description and Delegation Priors with 3 waves and 9 sub-agents annotation (§ 4.1 ). Harness defines principles and tool permissions governing agent behavior across execution paradigms (§ 4.2 ). Eval Metrics visualizes MAD-SAE metric computation (§ 4.3 ).
SettingParam.Delegation Appropriateness (DA)Orchestration Utilities (OU)MSQ ↑\uparrow
UDR ↓\downarrowODR ↓\downarrow∣Ddel∣|\mathcal{D}_{\mathrm{del}}|Phit↑\mathcal{P_{\text{hit}}}\uparrowRhit↑\mathcal{R_{\text{hit}}}\uparrowF1 hit↑\mathcal{F}_{\text{1 hit}}\uparrowUp↑U_{p}\uparrowUe↑U_{e}\uparrowUc↑U_{c}\uparrowAvg. ↑\uparrow
(a) MAD-SAE Variants Comparison on Deepseek-V4-Flash-0731
Gen284B (A13B)52.8%9.0%4948.0%18.8%27.1%48.7%48.9%31.2%42.9%11.6%
Spec16.8%57.1%11841.0%56.2%47.4%50.4%48.0%33.1%43.8%20.8%
Plan-1st15.7%61.0%12238.4%51.7%44.1%48.0%42.1%29.4%39.9%17.6%
Oracle2.2%1.3%8888.8%96.6%92.5%49.6%53.5%23.6%42.2%39.1%
Table 1: The Comparison of MAD-SAE variants and frontier models. (a) In the variants comparison , shaded area distinguishes Oracle from practical variants; bold marks the best practical result. (b)In the model comparison , bold and underline denote the best and second-best results, respectively.
Figure 3: Performance-cost ( left ) and performance-efficiency ( right ) trade-offs of MAD-SAE relative to SA across the full benchmark D\mathcal{D} . Each point represents a model-variant configuration, with changes computed as MAD-SAE minus SA. All tags are not explicitly stated are in the Spec mode. Upper-left regions indicate higher success rates with lower cost or fewer steps.
Figure 4: The distribution of beneficial and adverse categories, alongside with their underlying reasons.
Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

DomainTask NumberRead-OnlyWritableRead & Write
NumberWaveSub- AgentNumberWaveSub- AgentNumberWaveSub- Agent
Learning Management System25111/1/1.03/22/14.351/1/1.01/3/2.292/3/2.14/25/11.2
Enterprise Data Warehouse150--101/2/1.11/4/2.452/2/2.04/5/4.6
E-Commerce1541/1/1.02/2/2.021/1/1.02/2/2.092/3/2.32/7/4.1
Stock Market511/1/1.02/2/2.00--42/3/2.36/9/7.3
Video Platform40--21/1/1.01/3/2.022/2/2.04/4/4.0
Appendix
Table 2: Distribution of the 89 tasks in Dneed\mathcal{D}_{\text{need}} by domain and sub-agent type. Tasks are grouped into Read-Only, Writable, and Read & Write. For each group, we report the number of tasks and the min/max/mean of delegation waves and sub-agents.
ModelReasoning EffortTemperature (Top-p)Input ($)Cached Input ($)Output ($)
GPT-5.6-Solmedium-4 / 80.4 / 0.820 / 30
Claude-Sonnet-5adaptive-20.210
Qwen-3.8-Maxenable1.0 (1.0)1.790.225.37
Kimi-K3max1.0 (0.95)2.980.3014.91
GLM-5.3max1.0 (0.95)1.40.264.4
MiniMax-M3adaptive1.0 (0.95)0.31 / 0.630.06 / 0.131.25 / 2.5
Appendix
Table 3: Inference configuration and official API pricing for each model. Prices are in USD per 1M tokens. For tiered models, we report low / high prices, where tiers are triggered when input tokens exceed 272K for GPT-5.6-Sol and 512K for MiniMax-M3.
ModeDescriptionSub-Agent TypesRelevant Harness
Single-AgentThe main-agent resolves the task independently without delegation.–Kimi-Code, Codex, Claude Code, Cursor, TRAE
General (Gen)The main-agent autonomously decides when and how to delegate to generally defined sub-agents.Planner , Explorer , CoderKimi-Code, Claude Code, TRAE
Specialized (Spec)The main-agent autonomously decides when and how to delegate to domain-specialized sub-agents.Planner , ALR , EDA , EWO , FMA , ORB , WDRCursor
Plan-1stThe main-agent is forced to call the planner for task decomposition and delegation planning at first, then follows the plan.Planner , Explorer , CoderKimi-Code, Codex, Claude Code, Cursor, TRAE
OracleThe main-agent views annotated delegation priors and is required to follow them.Planner , Explorer , CoderKimi-Code, Codex, Claude Code, Cursor, TRAE
Appendix
Table 4: The outline of common MAD-SAE variants. Orange denotes read-only sub-agents; blue denotes writable sub-agents.
PartitionCategorySuccess SituationEfficiency BenefitsCost Benefits
Beneficial Cases (625)Failure-to-Success (F2S, 130)MAD-SAE success, SA failure✓\checkmark✓\checkmark
✓\checkmark×\times
×\times✓\checkmark
×\times×\times
Cheaper-or-Efficient Success (CES, 495)MAD-SAE success, SA success✓\checkmark×\times
✓\checkmark✓\checkmark
Appendix
Table 5: The taxonomy of MAD-SAE (Spec) cases. Across both partitions, ✓\checkmark denotes strictly fewer critical steps (Efficiency) or strictly lower cost (Cost), and ×\times denotes the absence of that gain (equality or an increase); == denotes equality. CIS additionally requires strictly higher cost, and CIF requires a strict increase in critical steps or cost.
ModelCase Composition (full D\mathcal{D} )ΔSR\Delta\mathrm{SR}Delegation AppropriatenessOrchestration Utilities
B/AB/AMAO ( B/AB/A )Non-MAO ( B/AB/A )F2S/S2FUDR ↓\downarrowODR ↓\downarrowUp↑U_{p}\uparrowUe↑U_{e}\uparrowUc↑U_{c}\uparrow
GLM-5.389 /7725 /2164 /5616 /9+4.2+4.28.9%64.9%50.8%52.7%40.5%
GLM-5.3-Flash85 /8132/ 4153 /407 /5+1.2+1.226.9%50.6%51.3%54.0%32.0%
Kimi-K3130 /3689 /2241 /1426 /9+10.2+10.250.5%16.8%56.4%50.3%50.9%
Qwen-3.8-Max80/ 8632 /2248/ 6420 /3+10.2+10.215.7%59.7%54.4%49.9%38.4%
DeepSeek-V4-Flash-073175/ 9126 /1949/ 7211/110.00.016.8%57.1%50.4%48.0%33.1%
Appendix
Table 6: Per-model case composition and delegation-quality metrics under MAD-SAE (Spec); UDR, ODR, and UpU_{p} / UeU_{e} / UcU_{c} are reproduced from Table 1 . B/AB/A : beneficial/adverse case counts, with the larger side of each pair bolded (ties unbolded). ΔSR\Delta\mathrm{SR} : success-rate change on D\mathcal{D} from converting SA to MAD-SAE (Spec). Bold/underline mark best/second-best for UDR and ODR.
Figure 5: The distribution of beneficial and adverse categories, alongside with their underlying reasons of GLM-5.3.
Figure 6: The distribution of beneficial and adverse categories, alongside with their underlying reasons of GLM-5.3-Flash.
Figure 7: The distribution of beneficial and adverse categories, alongside with their underlying reasons of Kimi-K3.
Figure 8: The distribution of beneficial and adverse categories, alongside with their underlying reasons of Qwen-3.8-Max.
Figure 9: The distribution of beneficial and adverse categories, alongside with their underlying reasons of DeepSeek-V4-Flash-0731.
Figure 10: The distribution of beneficial and adverse categories, alongside with their underlying reasons of Claude-Sonnet-5.
Figure 11: The distribution of beneficial and adverse categories, alongside with their underlying reasons of GPT-5.6-Sol.
Figure 12: The distribution of beneficial and adverse categories, alongside with their underlying reasons of MiniMax-M3.
Figure 13: The Illustration of the Context Offloading benefit on GLM-5.3.
Figure 14: The Illustration of the Delegated Parallelism benefit on GLM-5.3.
Figure 15: The Illustration of the Delegated Parallelism benefit on Kimi-K3.
Figure 16: The Illustration of the Better Main-Agent-Only solution on Minimax-M3.
Figure 17: The Illustration of the Unnecessary Delegation drawback on DeepSeek-V4-Flash-0731.
Figure 18: The Illustration of the Delegated Falure Propagation drawback on GPT-5.6-Sol.
Figure 19: The Illustration of the Unrealized Delegated Parallelism drawback on GPT-5.6-Sol.
Figure 20: The Illustration of the Execution Bloat drawback on GPT-5.6-Sol.
Task Description
I am a PhD student preparing a survey presentation on recent advances in natural language processing, specifically focusing on large language models and their reasoning capabilities. I need to gather information from multiple sources, organize it systematically, and create both a data spreadsheet and a presentation. First, visit the survey methodology guide at http://localhost:30231 to learn about how to structure a literature survey. The main page explains the overall approach including comparison criteria and taxonomy development. There is also a guide page at http://localhost:30231/guide.html with tips on presenting survey results effectively. Take notes on the recommended structure and presentation tips. Next, search the arxiv paper database for recent papers related to NLP, language models, and reasoning. You should find around five relevant papers on these topics. For each paper, look up its full details including the abstract and read its LaTeX source to extract the key methodology, the datasets used in experiments, and the main findings. The LaTeX source of each paper (its section headings and content) is available through the arxiv-latex tools (list_paper_sections, get_paper_section, get_paper_prompt). In the content returned by these tools, each section heading appears on its own line starting with ‘#’, so you can count the sections of a paper by counting those heading lines (or by reading the sections list returned by list_paper_sections). Count the LaTeX sections of each paper, because the survey needs the average number of sections across papers. After gathering all the paper information, create an Excel workbook called Survey_Data.xlsx in the workspace with three sheets. The first sheet should be named “Paper Comparison” with columns Paper_ID, Title, Year, Method, Dataset, and Key_Finding. The Paper_ID column must contain each paper’s arXiv identifier in the standard form YYMM.NNNNN (for example 2404.00001), with no prefix such as “arXiv:” and no version suffix such as “v1”. The Method column should describe the core approach in a few words, the Dataset column should list the primary evaluation dataset, and Key_Finding should summarize the main result. The second sheet should be named “Taxonomy” with columns Category and Papers. Group the papers into meaningful categories based on their approaches such as prompting methods, training methods, and architectural innovations. The third sheet should be named “Summary Statistics” with columns Metric and Value, including Total_Papers, Most_Common_Category, Year_Range, and Average_Sections (average number of LaTeX sections across papers). All values in the Value column must be written as literal text or numbers in the cell itself – do NOT write Excel formulas (for example, do not enter “=COUNTA(…)” or “=AVERAGE(…)”; instead write the plain number, e.g. 5 or 4.0). Then create a PowerPoint presentation called NLP_Survey.pptx in the workspace. The presentation should have at least six slides. The first slide should be a title slide with a title containing the words “NLP Survey” or “Language Model Survey” or similar. The second slide should present the survey methodology and structure based on what you learned from the methodology guide. Then include one slide for each of at least three key papers, showing the paper title, its method, dataset, and key finding. End with a summary slide that provides a taxonomy overview and identifies key trends across the surveyed papers. Please review the survey outline and presentation style guide files in the workspace for additional context about the expected structure and formatting guidelines. Important coordination note: this is a multi-agent task and all agents share the same workspace. Each of the two final files (Survey_Data.xlsx and NLP_Survey.pptx) must be written to disk by a single sub-agent. Never have more than one agent write the same file at the same time – concurrent writes to the same file overwrite each other. Coordinate your work so that, after the research information has been gathered, one designated sub-agent is responsible for writing Survey_Data.xlsx and one (or the same agent) is responsible for writing NLP_Survey.pptx.
Appendix
Table 7: The full task description of arxiv-survey-presentation
Delegation Prior
Task-Solving Rationable with Sub-task Decomposition: To solve this task efficiently, follow this exact orchestration plan. A wave must finish and return all stated handoffs before the next wave begins. Within a wave, dispatch exactly one sub-agent for each numbered sub-task and run those sub-agents concurrently. Before Wave 1, the main agent must list and read all task-visible survey-outline and presentation-style reference files in read-only mode, without assuming their filenames, and freeze the survey comparison schema, taxonomy/statistics rules, at-least-six-slide contract, and single-writer ownership of Survey_Data.xlsx and NLP_Survey.pptx. It must resolve the task-visible workspace directory from the runtime allowed-directory listing (or literal accessible-workspace directory already supplied by the harness), freeze ‘<workspace_root>’, and reject bare ‘/workspace’ and dumps/run-specific paths. It must then inspect the granted runtime tool catalog and verify that the task-required LaTeX section operations can actually read section lists and contents for discovered paper IDs. If those operations are unavailable or no authorized source can return the required paper bodies, the main agent must stop before delegation, report the exact missing-capability blocker, and must not fabricate Method, Dataset, Key_Finding, section counts, or either dependent deliverable. The following waves apply only when the required source capability is present; all paper facts must come from task-visible runtime sources. 1. Wave 1 – dispatch exactly 2 sub-agents in parallel, one for each numbered sub-task: (1). Explorer 0 ( Web-Domain-Researcher 0 ): Use one explore sub-agent to inspect both task-provided survey-methodology pages in read-only mode and combine their source-backed guidance with the frozen local-reference packet supplied by the main agent. Use only the granted non-interactive browser navigation, snapshot, search, and wait operations; do not click, type, evaluate scripts, select options, invoke terminal, or reread workspace files. Return a compact source-referenced methodology packet covering comparison criteria, taxonomy guidance, recommended presentation structure, slide tips, and per-page/reference coverage. Do not write deliverables or alter the pages. (2). Explorer 1 ( Academic-Literature-Researcher 0 ): Use one explore sub-agent to scout the task-relevant local arXiv candidate space read-only for recent NLP, language-model, and reasoning papers. Use complementary queries at the supported per-call result bound, union and deduplicate by stable ID because no search cursor is exposed, validate relevance from title/abstract, and return a candidate manifest with authors/date, abstract, available arxiv-latex status, per-query/union controls, and rejected distractors. Do not read full manuscripts, write files, or create deliverables. After Wave 1, the main agent must apply the task’s relevance rule, select exactly five source-supported papers, and freeze an explicit five-row assignment table containing ‘agent_no’, stable ID, and exact title, with one distinct paper per row. Before dispatching Wave 2, render each numbered prompt into a complete self-contained instruction that includes that row’s literal stable ID and exact title, the permitted read-only paper/LaTeX calls, the complete return schema and all missing/conflict controls. Never send an agent only an ordinal such as “paper row 3” or require it to infer fields from another prompt. 2. Wave 2 – dispatch exactly 5 explore sub-agents in parallel, one for each frozen paper: (1). Explorer 0 ( Academic-Literature-Researcher 0 ): Use one explore sub-agent to own only paper row 1 from the frozen assignment table. (2). Explorer 1 ( Academic-Literature-Researcher 1 ): Use one explore sub-agent to own only paper row 2 from the frozen assignment table. (3). Explorer 2 ( Academic-Literature-Researcher 2 ): Use one explore sub-agent to own only paper row 3 from the frozen assignment table. (4). Explorer 3 ( Academic-Literature-Researcher 3 ): Use one explore sub-agent to own only paper row 4 from the frozen assignment table. (5). Explorer 4 ( Academic-Literature-Researcher 4 ): Use one explore sub-agent to own only paper row 5 from the frozen assignment table. Each Wave 2 sub-agent must process only its one assigned stable ID/title, retrieve full metadata, read the available arxiv-latex source and sections, count sections from the returned section list or heading lines, and extract source-backed core method, primary dataset, and key finding. Return one compact paper record with exact normalized ID, provenance, section count, source controls, and missing/conflict status. Explore is read-only, so do not write files or create Excel or PowerPoint. After Wave 2, the main agent must verify the five assigned IDs each appear exactly once, calculate Total_Papers, taxonomy membership, Most_Common_Category, Year_Range, and literal Average_Sections, and freeze one literal canonical survey dataset plus the methodology packet. Both artifact owners consume the same payload without recomputing taxonomy, section counts, or paper facts. 3. Wave 3 – dispatch exactly 2 sub-agents in parallel, one for each numbered sub-task: (1). Coder 0 ( Office-Report-Builder 0 ): Use one coder sub-agent to own ‘<workspace_root>/Survey_Data.xlsx’ exclusively. Use one ‘python_execute’ with ‘openpyxl’ to create Paper Comparison, Taxonomy, and Summary Statistics atomically from the canonical payload with exact headers, standard IDs without prefix/version, literal Value cells, source-backed method/dataset/finding rows, and verified taxonomy/statistics. Then read each complete populated range once and return path, sheet order, row/metric controls, and validation. Do not write the presentation. (2). Coder 1 ( Office-Report-Builder 1 ): Use one coder sub-agent to own ‘<workspace_root>/NLP_Survey.pptx’ exclusively. Use one ‘python_execute’ with ‘python-pptx’ to create the complete deck atomically from the canonical payload, with at least six slides including a title slide, methodology slide grounded in the webpage/guide packet, at least three distinct paper slides, and a final taxonomy/trends summary. Follow the visible style guide, then read the deck once and return path, slide count/titles, paper/methodology coverage, and validation. Do not write the workbook. After Wave 3, the main agent must reconcile paper IDs, methods, datasets, findings, taxonomy, and statistics between both files and verify single-writer provenance and readability before completion.
Appendix
Table 8: The full delegation prior of arxiv-survey-presentation . It includes the annotations of sub-agent types under Gen and Spec (Refer to Appx. B.4 ). Read-only sub-agents are highlighted in orange, while writable sub-agents are highlighted in blue.
Basic Principles & Tools
Runtime Profile name: toolathlon-main description: Toolathlon-GYM task solver (main-agent) override: true tools: mcp__* , TodoList disallowedTools: Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none Accessible workspace directory: /workspace/dumps/kimi-code_deepseek-v4-flash/SingleUserTurn-arxiv-survey-presentation/workspace When processing tasks, if you need to read/write local files and the user provides a relative path, you may choose to combine it with the above workspace directory to get the complete path. Please complete the given task independently. Do not seek confirmation or additional feedback from the user. You should handle all situations on your own, as the user will not provide any further information. Core Responsibilities: - Efficiently and correctly solve the Toolathlon-GYM task using only the granted tools. - Ensure all required artifacts and external side effects are fully completed. - Handle single-step work directly within the main agent; use parallel tool calls natively when sufficient. Visible Boundary (Strictly Enforced): - You may ONLY read and write files inside your accessible workspace: /TASK_WORKSPACE - You may use ONLY the task-granted tools to interact with external systems (databases, calendars, email, spreadsheets, etc.). - Do NOT access, read, list, or probe anything outside your workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or any system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). Those are outside your task boundary. - Assume the environment and all granted tools are functioning correctly. Never attempt to diagnose, fix, or work around infrastructure; if a granted tool fails, report the failure and continue with the tools that work. Completion Protocol (Mandatory): - The task is complete only after you call mcp__local__claim_done . - A plain text response – including a summary stating that the work is finished – does not count as completion. - Do not stop after describing the result. You must call mcp__local__claim_done immediately after all required tool actions have succeeded. - Warning: this call terminates the task instantly. You will have no further opportunity to act on this task afterward.
Appendix
Table 9: The basic principles and runtime tool profile of the Single-Agent harness. The main-agent resolves the task independently with only the granted MCP tools; all built-in shell, file, and web tools are disallowed.
Main-Agent Principles & Tools (Gen)
Runtime Profile name: toolathlon-main description: Toolathlon-GYM task solver (main-agent) override: true tools: mcp__* , Agent , AgentSwarm , TodoList disallowedTools: Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: Coder , Explorer , Planner Core Responsibilities: <inherit from the Core Responsibilities of Basic Principles & Tools (Table 9 ) …> - Assess whether sub-agent delegation is beneficial. Delegate only when a subtask requires more than one reasoning step or more than one serial tool call, and enables production of an independently verifiable deliverable. - Reuse sub-agent outputs only when they satisfy their named versioned return contract. Preserve facts, artifact paths, resource IDs, digests, provenance, and verification results without silently reshaping them. Visible Boundary (Strictly Enforced): <inherit from the Visible Boundary of Basic Principles & Tools (Table 9 ) …> Subagent Delegation: When assigning a sub-agent, provide a comprehensive and self-contained prompt that includes clear motivation and rationale, sufficient prior context and background, explicit constraints, scope, and limitations, and expected deliverables and acceptance criteria. Keep the prompt concise and unambiguous – direct, not verbose. Sub-Agent Orchestration Rules: - General Delegation: Delegate focused subtasks to available sub-agents via the Agent tool (use AgentSwarm for parallel execution); a Specified Sub-Agent Coordination section, if present later in the prompt, strictly overrides default delegation judgment; do not assign write-related sub-tasks to read-only sub-agents. - Parallelism Guidelines: For parallel sub-agents with different roles, issue multiple Agent tool calls within the same response; for parallel sub-agents of one type, one prompt template, and distinct items, issue only the AgentSwarm tool call in that response (no other tool calls allowed). - Environment & Context: Sub-agents inherit the parent’s task-scoped tools and workspace permissions unless a sub-agent YAML configuration overrides them; every sub-agent prompt must be self-contained … - When & How to Delegate: Prefer resume when an existing sub-agent already holds relevant context or the task continues its prior work … Available Sub-Agent Types: - Coder : Handles complex, multi-step engineering workflows – writes files, runs commands, produces concrete artifacts. - Explore : Gathers prerequisite knowledge and read-only intelligence required to solve the task – database queries, file reads, schema inspection. - Planner : Decomposes complex tasks, analyzes dependencies, designs workflows, assesses risks, and defines validation strategies. Plan the use of sub-agents to address sub-tasks. Examples: - Multi-system assembly with parallel branches and authorized writes →\rightarrow first launch Planner to decompose, then assign to Explorer and Coder in parallel or sequence … - Read-only research/comparison →\rightarrow launch an Explorer agent via the Agent tool … - Three independent monthly summaries, one document each →\rightarrow launch three Coder agents in parallel with distinct ownership … - Homogeneous read-only check of five pages →\rightarrow use AgentSwarm with one Explorer prompt template over the five URLs … Completion Protocol (Mandatory): <inherit from the Completion Protocol (Mandatory) of Basic Principles & Tools (Table 9 ) …>
Appendix
Table 10: The main-agent principles and runtime tool profile of Gen. The main-agent autonomously decides when and how to delegate to Coder , Explorer , and Planner .
Main-Agent Principles & Tools (Spec)
Runtime Profile name: toolathlon-main description: Toolathlon-GYM task solver (main-agent) override: true tools: mcp__* , Agent , AgentSwarm , TodoList disallowedTools: Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: Planner , Academic-Literature-Researcher , Web-Domain-Researcher , Enterprise-Data-Analyst , Financial-Market-Analyst , Office-Report-Builder , External-Workflow-Operator Core Responsibilities: <inherit from the Core Responsibilities of Basic Principles & Tools (Table 9 ) …> - Assess whether sub-agent delegation is beneficial. Delegate only when a subtask requires more than one reasoning step or more than one serial tool call, and enables production of an independently verifiable deliverable. - Trust sub-agent outputs that satisfy their return contract. Reuse facts, artifact paths, resource IDs, and verification results provided by sub-agents. - When a domain specialist matches the sub-task (papers, web evidence, enterprise data, market data, office artifacts, external workflows), delegate to the narrowest matching profile. When no specialist matches, do the work yourself rather than inventing a generalist delegate. Visible Boundary (Strictly Enforced): <inherit from the Visible Boundary of Basic Principles & Tools (Table 9 ) …> Subagent Delegation: <inherit from the Subagent Delegation of Main-Agent Principles & Tools (Gen) (Table 10 ) …> Sub-Agent Orchestration Rules: - General Delegation: <inherit from Gen (Table 10 ) …> - Parallelism Guidelines: <inherit from Gen (Table 10 ) …> - Environment & Context: Sub-agents inherit the parent’s task-scoped tools and workspace permissions unless a sub-agent YAML configuration overrides them. Read-only sub-agents ( Planner , Academic-Literature-Researcher , Web-Domain-Researcher , Enterprise-Data-Analyst , Financial-Market-Analyst ) cannot create, update, delete, send, or otherwise mutate persistent state; write-capable sub-agents ( Office-Report-Builder , External-Workflow-Operator ) may mutate only their domain. Every sub-agent prompt must be self-contained … - When & How to Delegate: Delegate when a subtask is independent and would otherwise bloat your own context …; never delegate the final completion signal – only you may call mcp__local__claim_done after verifying every requirement; default to foreground sub-agents, using run_in_background=true only for long-running work …; prefer resume when a sub-agent already holds relevant context … Available Sub-Agent Types: - Planner : Decomposes complex tasks, analyzes dependencies, designs workflows, assesses risks, and defines validation strategies. - Academic-Literature-Researcher : Gathers prerequisite scholarly knowledge – paper search, paper reads, section lookup. - Web-Domain-Researcher : Gathers prerequisite web/domain knowledge – pages, APIs, recipes, rail routes, videos, transcripts. - Enterprise-Data-Analyst : Gathers prerequisite enterprise data – Canvas, Snowflake, WooCommerce, and supplied documents. - Financial-Market-Analyst : Gathers prerequisite market data – prices, statements, recommendations, finance news. - Office-Report-Builder : Handles local artifacts – writes files, spreadsheets, documents, slides, PDFs, or intermediate datasets. - External-Workflow-Operator : Handles external writes – email, calendar, forms, cloud sheets, Notion pages. Examples: - Multi-system assembly with parallel branches and authorized writes →\rightarrow first launch Planner to decompose, then assign to the matching specialists in parallel or sequence … - Read five papers and summarize each →\rightarrow launch Academic-Literature-Researcher via the Agent tool … - Homogeneous read-only check of five pages →\rightarrow AgentSwarm with one Web-Domain-Researcher prompt template over the five URLs … - Roster-count cross-check against a registrar snapshot →\rightarrow launch Enterprise-Data-Analyst … - Ten-day market comparison of prices, statements, and headlines →\rightarrow launch Financial-Market-Analyst … - Three independent monthly summaries, one document each →\rightarrow launch three Office-Report-Builder agents in parallel with distinct ownership … - One approved calendar hold plus a notification email →\rightarrow launch External-Workflow-Operator … Completion Protocol (Mandatory): <inherit from the Completion Protocol (Mandatory) of Basic Principles & Tools (Table 9 ) …>
Appendix
Table 11: The main-agent principles and runtime tool profile of Spec. The main-agent autonomously decides when and how to delegate to the domain-specialized sub-agents.
Main-Agent Principles & Tools (Plan-1st)
Runtime Profile name: toolathlon-main description: Toolathlon-GYM task solver (main-agent) override: true tools: mcp__* , Agent , AgentSwarm , TodoList disallowedTools: Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: Coder , Explore , Planner Core Responsibilities: <inherit from the Core Responsibilities of Basic Principles & Tools (Table 9 ) …> - Assess whether sub-agent delegation is beneficial. Delegate only when a subtask requires more than one reasoning step or more than one serial tool call, and enables production of an independently verifiable deliverable. - Trust sub-agent outputs that satisfy their return contract. Reuse facts, artifact paths, resource IDs, digests, and verification results provided by sub-agents. Visible Boundary (Strictly Enforced): <inherit from the Visible Boundary of Basic Principles & Tools (Table 9 ) …> Subagent Delegation: <inherit from the Subagent Delegation of Main-Agent Principles & Tools (Gen) (Table 10 ) …> Sub-Agent Orchestration Rules: <inherit from the Sub-Agent Orchestration Rules of Main-Agent Principles & Tools (Gen) (Table 10 ) …> Available Sub-Agent Types: - Coder : Handles complex, multi-step engineering workflows – writes files, runs commands, produces concrete artifacts. - Explorer : Gathers prerequisite knowledge and read-only intelligence required to solve the task – database queries, file reads, schema inspection. - Planner : Decomposes complex tasks, analyzes dependencies, designs workflows, assesses risks, and defines validation strategies. Examples: <inherit from the Examples of Main-Agent Principles & Tools (Gen) (Table 10 ) …> Plan-First Protocol (Mandatory): Before taking any task action, you MUST first dispatch the Planner sub-agent (via the Agent tool) with the full task description and the task’s file paths, asking it to decompose the task into ordered subtasks. The plan you request must specify, for each subtask: - The dispatch decision: main-agent , Explorer , or Coder . - Whether the subtask is independent of its siblings (parallelizable) or must wait for another subtask’s output. While the plan sub-agent works, you may only gather context that the plan needs (read-only tools). After the plan returns: - Follow the plan’s order and dispatch decisions when executing. - Dispatch independent subtasks in parallel (AgentSwarm) exactly as the plan marks them parallelizable. - Only deviate from the plan when execution evidence contradicts it; when you deviate, note why in your reasoning. Completion Protocol (Mandatory): <inherit from the Completion Protocol (Mandatory) of Basic Principles & Tools (Table 9 ) …>
Appendix
Table 12: The main-agent principles and runtime tool profile of Plan-1st. The main-agent is forced to call the Planner sub-agent for task decomposition and delegation planning before any other action, then follows the plan.
Planner Principles & Tools
Runtime Profile name: Planner description: Planning sub-agent — reasons about task decomposition and verification strategy using only read-oriented tools (database queries, file reads, read-only shell commands) whenToUse: Complex multi-system tasks where an independent plan or sanity check improves reliability override: true tools: read-oriented MCP tools selected by name patterns across the task’s seven servers ( arxiv-latex , arxiv_local , excel , playwright_with_chunk , pptx , terminal , local ); per server the grant is the set of read-only wildcards mcp__<server>read , list , search , get , view , query , fetch , describe , info , count , exists , find , inspect , show , lookup , retrieve , snapshot , select_* , directory_tree , list_allowed_directories , _reports , _health_check , _system_status , browser_navigate[back/forward] , browser_console_messages , browser_network_requests , browser_take_screenshot , browser_wait_for , recommend , *whatToEat disallowedTools: mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser , mcp_write , mcp__write* , mcp_____write subagents: none You are a plan sub-agent inside a Toolathlon-GYM evaluation. Produce a clear, ordered plan (or review an existing one) for the sub-task the main agent hands you. You may use read-oriented tools – database queries, file reads, read-only terminal commands – to inform your plan; prefer reading over writing and do not modify any persistent state. Return the plan as structured text. Never signal overall task completion. Constraints: - Do not assign write-related sub-tasks to read-only sub-agents. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue.
Appendix
Table 13: The principles and runtime tool profile of the Planner sub-agent. A read-only planner that decomposes the assigned sub-task into an ordered, dispatch-annotated plan and never signals task completion.
Explorer Principles & Tools
Runtime Profile name: Explorer description: Read-only exploration sub-agent — queries databases, reads files, runs read-only shell commands, and inspects data via the task’s read-oriented tools, then reports findings without changing anything whenToUse: Map out available data, inspect schemas, or verify intermediate state before the main agent commits to a plan override: true tools: read-oriented MCP tools selected by name patterns across the task’s seven servers ( arxiv-latex , arxiv_local , excel , playwright_with_chunk , pptx , terminal , local ); per server the grant is the set of read-only wildcards mcp__<server>read , list , search , get , view , query , fetch , describe , info , count , exists , find , inspect , show , lookup , retrieve , snapshot , select_* , directory_tree , list_allowed_directories , _reports , _health_check , _system_status , browser_navigate[back/forward] , browser_console_messages , browser_network_requests , browser_take_screenshot , browser_wait_for , recommend , *whatToEat disallowedTools: mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser , mcp_write , mcp__write* , mcp_____write subagents: none You are an explore sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather information for the main agent using the read-oriented tools available to you – database query tools, filesystem read tools, read-only terminal commands, and any other read tools granted by the current task. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, file paths). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue.
Appendix
Table 14: The principles and runtime tool profile of the Explorer sub-agent. A read-only investigator that gathers and reports findings for the main agent and never signals task completion.
Coder Principles & Tools
Runtime Profile name: Coder description: General-purpose sub-agent for executing a concrete sub-task (data processing, document generation, file/shell operations, database queries) with the task-approved tools only whenToUse: Delegate a well-scoped, independent piece of work that would otherwise consume the main agent’s context override: true tools: mcp__* disallowedTools: mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are a coder sub-agent inside a Toolathlon-GYM evaluation. You receive a focused sub-task from the main agent. Constraints: - You may ONLY use the task-approved tools available to you – the same set the main agent has. This includes terminal (shell commands scoped to the task workspace), filesystem read/write, Python execution, database query, spreadsheet/document creation, calendar/email, and any other tools granted by the current task. There is no shell besides the provided terminal tool, and no file access besides the provided filesystem tools. - Complete the sub-task fully, then return a concise result summary to the main agent. Include exact file paths / IDs you created or modified. - Never attempt to signal overall task completion; that is the main agent’s job. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue.
Appendix
Table 15: The principles and runtime tool profile of the Coder sub-agent. A writable executor that carries out a focused sub-task with the task-approved tools and returns a concise result summary, never signaling task completion.
Academic-Literature-Researcher Principles & Tools
Runtime Profile name: Academic-Literature-Researcher description: Read-only scholarly sub-agent — searches and reads papers with arXiv / scholarly tools, then reports findings without changing anything whenToUse: Read papers, sections, citations, or scholarly metadata before the main agent writes a report override: true tools: mcp__arxiv-latex__get_paper_abstract , mcp__arxiv-latex__get_paper_prompt , mcp__arxiv-latex__get_paper_section , mcp__arxiv-latex__list_paper_sections , mcp__arxiv_local__list_papers , mcp__arxiv_local__read_paper , mcp__arxiv_local__search_papers disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are an academic-literature-researcher sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather scholarly information for the main agent using the academic tools available to you – paper search, paper reads, and section lookup. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, file paths, paper IDs). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue.
Appendix
Table 16: The principles and runtime tool profile of the Academic-Literature-Researcher sub-agent, which searches and reads papers and reports findings.
Enterprise-Data-Analyst Principles & Tools
Runtime Profile name: Enterprise-Data-Analyst description: Read-only enterprise sub-agent — queries Canvas, Snowflake, WooCommerce, and related records, then reports findings without changing anything whenToUse: Pull enrollments, orders, tickets, or warehouse rows and summarize them without writing back override: true tools: none granted in this task ( tools: [] ) disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are an enterprise-data-analyst sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather information for the main agent using the enterprise read tools available to you – Canvas, Snowflake, WooCommerce, and supplied documents. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, file paths). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue.
Appendix
Table 17: The principles and runtime tool profile of the Enterprise-Data-Analyst sub-agent, which queries enterprise records and reports findings.
Financial-Market-Analyst Principles & Tools
Runtime Profile name: financial-market-analyst description: Read-only market sub-agent — pulls prices, statements, recommendations, and finance news with Yahoo Finance tools, then reports findings without changing anything whenToUse: Look up tickers, prices, statements, or finance news before the main agent writes a report override: true tools: none granted in this task ( tools: [] ) disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are a financial-market-analyst sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather market information for the main agent using the Yahoo Finance tools available to you. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, tickers, dates). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue.
Appendix
Table 18: The principles and runtime tool profile of the financial-market-analyst sub-agent, which pulls market data and reports findings.
Web-Domain-Researcher Principles & Tools
Runtime Profile name: Web-Domain-Researcher description: Read-only web/domain sub-agent — inspects pages, APIs, recipes, rail routes, videos, and transcripts, then reports findings without changing anything whenToUse: Gather facts from web pages, videos, recipes, or rail routes before the main agent acts on them override: true tools: mcp__playwright_with_chunk__browser_* page-interaction tools: click , console_messages , hover , navigate [_ back /_ forward ], network_requests , scroll_down / scroll_up / scroll_to_bottom / scroll_to_top , snapshot and snapshot_navigate_to_* ( first / last / next / prev / line / span ), snapshot_search , tab_list , tab_select , take_screenshot , wait_for disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are a web-domain-researcher sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather information for the main agent using the read-oriented tools available to you – browser/page tools, fetch, recipes, rail tickets, videos, and transcripts. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, URLs). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue.
Appendix
Table 19: The principles and runtime tool profile of the Web-Domain-Researcher sub-agent, which inspects web/domain sources and reports findings.
External-Workflow-Operator Principles & Tools
Runtime Profile name: External-Workflow-Operator description: External-action sub-agent — creates calendar events, forms, cloud sheets, Notion pages, or sends email with the task-approved tools only whenToUse: Delegate a well-scoped external write (email, calendar, form, cloud sheet, Notion) that would otherwise consume the main agent’s context override: true tools: none granted in this task ( tools: [] ) disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are an external-workflow-operator sub-agent inside a Toolathlon-GYM evaluation. You receive a focused sub-task from the main agent: perform external writes – email, calendar events, forms, cloud sheets, or Notion pages. Constraints: - You may ONLY use the task-approved tools available to you. Act only on the recipients, IDs, dates, and payloads given in the prompt. - Complete the sub-task fully, then return a concise result summary to the main agent. Include exact resource IDs you created or modified. - Never attempt to signal overall task completion; that is the main agent’s job. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue.
Appendix
Table 20: The principles and runtime tool profile of the External-Workflow-Operator sub-agent, which performs external writes (email, calendar, forms, cloud sheets, Notion).
Office-Report-Builder Principles & Tools
Runtime Profile name: Office-Report-Builder description: General-purpose workspace builder — writes local files, spreadsheets, documents, slides, PDFs, or intermediate datasets with the task-approved tools only whenToUse: Delegate a well-scoped local artifact (Excel / Word / PPT / PDF / script / data file) that would otherwise consume the main agent’s context override: true tools: spreadsheet, slide, and local-execution tools: mcp__excel__* ( apply_formula , copy_range / copy_worksheet , create_chart / create_pivot_table / create_table / create_workbook / create_worksheet , delete_* , format_range , get_* , insert_columns / insert_rows , merge_cells / unmerge_cells , read_data_from_excel , rename_worksheet , validate_* , write_data_to_excel ); mcp__pptx__* ( add_* , apply_* , auto_generate_presentation , create_* , extract_* , format_table_cell , get_* , list_* , manage_* , open_presentation , optimize_slide_text , populate_placeholder , save_presentation , set_core_properties , switch_presentation , update_chart_data ); mcp__local__python_execute , save_overlong_output , view_overlong_output ; mcp__terminal__run_command , show_security_rules disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are an office-report-builder sub-agent inside a Toolathlon-GYM evaluation. You receive a focused sub-task from the main agent: produce workspace artifacts – spreadsheets, documents, slides, PDFs, scripts, or intermediate data files. Constraints: - You may ONLY use the task-approved tools available to you. This includes filesystem read/write, Python execution, spreadsheet/document/slide creation, and any other tools granted for this sub-task. Do not send email or mutate calendars, forms, cloud sheets, or Notion pages. - Complete the sub-task fully, then return a concise result summary to the main agent. Include exact file paths you created or modified. - Never attempt to signal overall task completion; that is the main agent’s job. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue.
Appendix
Table 21: The principles and runtime tool profile of the Office-Report-Builder sub-agent, which produces local workspace artifacts (spreadsheets, documents, slides, PDFs, scripts).
More in this topic

LLM Agent Orchestration

Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.

See all 463 papers

Explore similar work

CardsList
Dec 31, 2025

MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use

Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (MCP) is considered a future trend. Current MCP evaluation sets suffer from issues such as reliance on external MCP services and a lack of difficulty awareness. To address these limitations, we propose MCPAgentBench, a benchmark based on real-world MCP definitions designed to evaluate the tool-use capabilities of agents. We construct a dataset containing authentic tasks and simulated MCP tools. The evaluation employs a dynamic sandbox environment that presents agents with candidate tool lists containing distractors, thereby testing their tool selection and discrimination abilities. Furthermore, we introduce comprehensive metrics to measure both task completion rates and execution efficiency. Experiments conducted on state-of-the-art LLMs reveal significant performance differences in handling complex, multi-step tool invocations. All code is open-source at https://github.com/Brunestuder/MCPAgentBench.

Zixiang Liu, Wenrui Liu, Elsie Dai +5

Peking University · ZTE

LLM Agent EvaluationAI Agent BenchmarksLLM Tool UseModel Context Protocol
AskPDF
Jun 30, 2026

ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents

Production large language-model (LLM) agents are increasingly deployed not as lone problem-solvers but as managers: a main model creates specialized subagents, delegates work, and orchestrates their parallel, asynchronous returns through dynamic workflows. Whether one model can actually run such a team is largely unmeasured: existing benchmarks score a policy's own task-solving or a fixed multi-agent system's emergent behavior, but none isolate the management ability of the single LLM acting as leader. We introduce ClawArena-Team, a benchmark of 41 multi-turn, multimodal, multi-directory scenarios spanning 258 evaluation rounds and 72 staged updates that measures this management ability. The main agent is deliberately constrained: it natively perceives only text and directly accesses only part of the workspace. It commands a fixed, locally served subagent pool, so score differences reflect management skill, not raw capability. All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor. Across twelve proprietary, community-hosted, and self-hosted models, experiments show that the management bottleneck is privilege granting rather than perception (no model exceeds 50% workspace-permission precision); that cost and management quality are decoupled (API cost spans over 100 times while the overall score spans under 4 times, with the cheapest open models on the Pareto frontier); and that most leaderboard scores cluster within a 9.9-point band while orchestration behaviors diverge by more than an order of magnitude. Code is available at https://github.com/aiming-lab/ClawArena.

Kaiwen Xiong, Haonian Ji, Shi Qiu +4

1UNC-Chapel Hill · University of California, Berkeley · University of California, Santa Cruz

LLM Agent OrchestrationLLM Agent EvaluationAI Agent BenchmarksTool Access Control for LLM Agents
AskPDF
Jul 10, 2026

DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers

Large language model (LLM) agents are increasingly deployed over Model Context Protocol (MCP) servers, yet the benchmarks used to evaluate them score the final answer or a fixed "ground-truth" list of tools, both of which are fragile once the underlying data is live and stateful. We present DynamicMCPBench, a reusable framework rather than a fixed dataset. A practitioner can run it on their own MCP servers to test models on their own tasks, or let it collect servers automatically to measure a model's general ability to solve agentic tasks. Given the servers and any set of models, it generates realistic goals, pursues each one live to record a successful trajectory, distills that trajectory into path-agnostic effect checkpoints, and scores an agent on whether it reproduces those effects, never on the final answer. To show what the framework reveals, we run it at scale: 24 models over 121 servers and 750 tasks spread evenly over 15 task categories (50 each), where each category targets a distinct tool-use challenge of the generated questions. Each task is scored by pass^3: it counts as solved only if all three independent attempts succeed. Even the strongest agents solve only about half of the tasks, 31% of tasks are solved by no model at all, and accuracy collapses as the required tool chain grows longer (from 39% on the shortest chains to 13% on the longest). A human validation study confirms the automatic scoring is reliable (chance-corrected agreement of 0.76). DynamicMCPBench thus turns benchmark construction into something practitioners can rerun on their own servers and models, while exposing a consistent inability of current agents to handle long, multi-step agentic tasks.

Jerzy Kamiński, Ilya Galyukshev, Artem Kuznetsov +4

ITMO University · Central University

Computer-Use Agent BenchmarksLLM Agent EvaluationLong-Horizon Agent EvaluationAI Agent BenchmarksModel Context ProtocolTool-Using Language Agents+1
AskPDF