MS-Eval: Benchmarking Sub-Agent Delegation and Orchestration in MCP-Intensive Working Scenarios
Organizations: Zhejiang University · School of Engineering, Westlake University · Independent Researcher · Institute of Advanced Technology, Westlake Institute for Advanced Study
Abstract
LLM agents are increasingly integrated into real-world works, yet they must handle diverse domain-specific tools and complex multi-sub-task problems that sequential tool invocation resolves inefficiently and with severe context bloat. The Main-Agent Delegation and Sub-Agent Execution (MAD-SAE) paradigm addresses these issues by delegating sub-tasks to specialized sub-agents for parallel execution in isolated contexts, but its delegation and orchestration quality remains unquantified. We introduce MS-Eval, a unified evaluation kit comprising an MCP-intensive benchmark, a controlled execution harness, tailored delegation and orchestration metrics, and the MSQ index for comprehensive assessment. Evaluating four MAD-SAE variants and eight frontier models reveals that (1) sub-agent specialization (Spec) is the best practical practice; (2) open-source models are largely competitive with few exceptions; and (3) MAD-SAE generally raises cost with contingent payoff, benefiting most open-source models in both performance and efficiency while leaving closed-source models with higher cost and degraded performance. Our case study further traces these gains to better single-agent solutions, effective context offloading, and genuine sub-task parallelism, whereas unnecessary or redundant delegation turns the same cost into pure overhead. Codes: https://github.com/LINs-lab/MS-Eval
Figures & tables
| Setting | Param. | Delegation Appropriateness (DA) | Orchestration Utilities (OU) | MSQ | ||||||||
| UDR | ODR | Avg. | ||||||||||
| (a) MAD-SAE Variants Comparison on Deepseek-V4-Flash-0731 | ||||||||||||
| Gen | 284B (A13B) | 52.8% | 9.0% | 49 | 48.0% | 18.8% | 27.1% | 48.7% | 48.9% | 31.2% | 42.9% | 11.6% |
| Spec | 16.8% | 57.1% | 118 | 41.0% | 56.2% | 47.4% | 50.4% | 48.0% | 33.1% | 43.8% | 20.8% | |
| Plan-1st | 15.7% | 61.0% | 122 | 38.4% | 51.7% | 44.1% | 48.0% | 42.1% | 29.4% | 39.9% | 17.6% | |
| Oracle | 2.2% | 1.3% | 88 | 88.8% | 96.6% | 92.5% | 49.6% | 53.5% | 23.6% | 42.2% | 39.1% | |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Task Number | Read-Only | Writable | Read & Write | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Number | Wave | Sub- Agent | Number | Wave | Sub- Agent | Number | Wave | Sub- Agent | ||
| Learning Management System | 25 | 11 | 1/1/1.0 | 3/22/14.3 | 5 | 1/1/1.0 | 1/3/2.2 | 9 | 2/3/2.1 | 4/25/11.2 |
| Enterprise Data Warehouse | 15 | 0 | - | - | 10 | 1/2/1.1 | 1/4/2.4 | 5 | 2/2/2.0 | 4/5/4.6 |
| E-Commerce | 15 | 4 | 1/1/1.0 | 2/2/2.0 | 2 | 1/1/1.0 | 2/2/2.0 | 9 | 2/3/2.3 | 2/7/4.1 |
| Stock Market | 5 | 1 | 1/1/1.0 | 2/2/2.0 | 0 | - | - | 4 | 2/3/2.3 | 6/9/7.3 |
| Video Platform | 4 | 0 | - | - | 2 | 1/1/1.0 | 1/3/2.0 | 2 | 2/2/2.0 | 4/4/4.0 |
| Model | Reasoning Effort | Temperature (Top-p) | Input ($) | Cached Input ($) | Output ($) |
|---|---|---|---|---|---|
| GPT-5.6-Sol | medium | - | 4 / 8 | 0.4 / 0.8 | 20 / 30 |
| Claude-Sonnet-5 | adaptive | - | 2 | 0.2 | 10 |
| Qwen-3.8-Max | enable | 1.0 (1.0) | 1.79 | 0.22 | 5.37 |
| Kimi-K3 | max | 1.0 (0.95) | 2.98 | 0.30 | 14.91 |
| GLM-5.3 | max | 1.0 (0.95) | 1.4 | 0.26 | 4.4 |
| MiniMax-M3 | adaptive | 1.0 (0.95) | 0.31 / 0.63 | 0.06 / 0.13 | 1.25 / 2.5 |
| Mode | Description | Sub-Agent Types | Relevant Harness |
|---|---|---|---|
| Single-Agent | The main-agent resolves the task independently without delegation. | – | Kimi-Code, Codex, Claude Code, Cursor, TRAE |
| General (Gen) | The main-agent autonomously decides when and how to delegate to generally defined sub-agents. | Planner , Explorer , Coder | Kimi-Code, Claude Code, TRAE |
| Specialized (Spec) | The main-agent autonomously decides when and how to delegate to domain-specialized sub-agents. | Planner , ALR , EDA , EWO , FMA , ORB , WDR | Cursor |
| Plan-1st | The main-agent is forced to call the planner for task decomposition and delegation planning at first, then follows the plan. | Planner , Explorer , Coder | Kimi-Code, Codex, Claude Code, Cursor, TRAE |
| Oracle | The main-agent views annotated delegation priors and is required to follow them. | Planner , Explorer , Coder | Kimi-Code, Codex, Claude Code, Cursor, TRAE |
| Partition | Category | Success Situation | Efficiency Benefits | Cost Benefits |
| Beneficial Cases (625) | Failure-to-Success (F2S, 130) | MAD-SAE success, SA failure | ||
| Cheaper-or-Efficient Success (CES, 495) | MAD-SAE success, SA success | |||
| Model | Case Composition (full ) | Delegation Appropriateness | Orchestration Utilities | |||||||
| MAO ( ) | Non-MAO ( ) | F2S/S2F | UDR | ODR | ||||||
| GLM-5.3 | 89 /77 | 25 /21 | 64 /56 | 16 /9 | 8.9% | 64.9% | 50.8% | 52.7% | 40.5% | |
| GLM-5.3-Flash | 85 /81 | 32/ 41 | 53 /40 | 7 /5 | 26.9% | 50.6% | 51.3% | 54.0% | 32.0% | |
| Kimi-K3 | 130 /36 | 89 /22 | 41 /14 | 26 /9 | 50.5% | 16.8% | 56.4% | 50.3% | 50.9% | |
| Qwen-3.8-Max | 80/ 86 | 32 /22 | 48/ 64 | 20 /3 | 15.7% | 59.7% | 54.4% | 49.9% | 38.4% | |
| DeepSeek-V4-Flash-0731 | 75/ 91 | 26 /19 | 49/ 72 | 11/11 | 16.8% | 57.1% | 50.4% | 48.0% | 33.1% | |
| Task Description |
| I am a PhD student preparing a survey presentation on recent advances in natural language processing, specifically focusing on large language models and their reasoning capabilities. I need to gather information from multiple sources, organize it systematically, and create both a data spreadsheet and a presentation. First, visit the survey methodology guide at http://localhost:30231 to learn about how to structure a literature survey. The main page explains the overall approach including comparison criteria and taxonomy development. There is also a guide page at http://localhost:30231/guide.html with tips on presenting survey results effectively. Take notes on the recommended structure and presentation tips. Next, search the arxiv paper database for recent papers related to NLP, language models, and reasoning. You should find around five relevant papers on these topics. For each paper, look up its full details including the abstract and read its LaTeX source to extract the key methodology, the datasets used in experiments, and the main findings. The LaTeX source of each paper (its section headings and content) is available through the arxiv-latex tools (list_paper_sections, get_paper_section, get_paper_prompt). In the content returned by these tools, each section heading appears on its own line starting with ‘#’, so you can count the sections of a paper by counting those heading lines (or by reading the sections list returned by list_paper_sections). Count the LaTeX sections of each paper, because the survey needs the average number of sections across papers. After gathering all the paper information, create an Excel workbook called Survey_Data.xlsx in the workspace with three sheets. The first sheet should be named “Paper Comparison” with columns Paper_ID, Title, Year, Method, Dataset, and Key_Finding. The Paper_ID column must contain each paper’s arXiv identifier in the standard form YYMM.NNNNN (for example 2404.00001), with no prefix such as “arXiv:” and no version suffix such as “v1”. The Method column should describe the core approach in a few words, the Dataset column should list the primary evaluation dataset, and Key_Finding should summarize the main result. The second sheet should be named “Taxonomy” with columns Category and Papers. Group the papers into meaningful categories based on their approaches such as prompting methods, training methods, and architectural innovations. The third sheet should be named “Summary Statistics” with columns Metric and Value, including Total_Papers, Most_Common_Category, Year_Range, and Average_Sections (average number of LaTeX sections across papers). All values in the Value column must be written as literal text or numbers in the cell itself – do NOT write Excel formulas (for example, do not enter “=COUNTA(…)” or “=AVERAGE(…)”; instead write the plain number, e.g. 5 or 4.0). Then create a PowerPoint presentation called NLP_Survey.pptx in the workspace. The presentation should have at least six slides. The first slide should be a title slide with a title containing the words “NLP Survey” or “Language Model Survey” or similar. The second slide should present the survey methodology and structure based on what you learned from the methodology guide. Then include one slide for each of at least three key papers, showing the paper title, its method, dataset, and key finding. End with a summary slide that provides a taxonomy overview and identifies key trends across the surveyed papers. Please review the survey outline and presentation style guide files in the workspace for additional context about the expected structure and formatting guidelines. Important coordination note: this is a multi-agent task and all agents share the same workspace. Each of the two final files (Survey_Data.xlsx and NLP_Survey.pptx) must be written to disk by a single sub-agent. Never have more than one agent write the same file at the same time – concurrent writes to the same file overwrite each other. Coordinate your work so that, after the research information has been gathered, one designated sub-agent is responsible for writing Survey_Data.xlsx and one (or the same agent) is responsible for writing NLP_Survey.pptx. |
| Delegation Prior |
| Task-Solving Rationable with Sub-task Decomposition: To solve this task efficiently, follow this exact orchestration plan. A wave must finish and return all stated handoffs before the next wave begins. Within a wave, dispatch exactly one sub-agent for each numbered sub-task and run those sub-agents concurrently. Before Wave 1, the main agent must list and read all task-visible survey-outline and presentation-style reference files in read-only mode, without assuming their filenames, and freeze the survey comparison schema, taxonomy/statistics rules, at-least-six-slide contract, and single-writer ownership of Survey_Data.xlsx and NLP_Survey.pptx. It must resolve the task-visible workspace directory from the runtime allowed-directory listing (or literal accessible-workspace directory already supplied by the harness), freeze ‘<workspace_root>’, and reject bare ‘/workspace’ and dumps/run-specific paths. It must then inspect the granted runtime tool catalog and verify that the task-required LaTeX section operations can actually read section lists and contents for discovered paper IDs. If those operations are unavailable or no authorized source can return the required paper bodies, the main agent must stop before delegation, report the exact missing-capability blocker, and must not fabricate Method, Dataset, Key_Finding, section counts, or either dependent deliverable. The following waves apply only when the required source capability is present; all paper facts must come from task-visible runtime sources. 1. Wave 1 – dispatch exactly 2 sub-agents in parallel, one for each numbered sub-task: (1). Explorer 0 ( Web-Domain-Researcher 0 ): Use one explore sub-agent to inspect both task-provided survey-methodology pages in read-only mode and combine their source-backed guidance with the frozen local-reference packet supplied by the main agent. Use only the granted non-interactive browser navigation, snapshot, search, and wait operations; do not click, type, evaluate scripts, select options, invoke terminal, or reread workspace files. Return a compact source-referenced methodology packet covering comparison criteria, taxonomy guidance, recommended presentation structure, slide tips, and per-page/reference coverage. Do not write deliverables or alter the pages. (2). Explorer 1 ( Academic-Literature-Researcher 0 ): Use one explore sub-agent to scout the task-relevant local arXiv candidate space read-only for recent NLP, language-model, and reasoning papers. Use complementary queries at the supported per-call result bound, union and deduplicate by stable ID because no search cursor is exposed, validate relevance from title/abstract, and return a candidate manifest with authors/date, abstract, available arxiv-latex status, per-query/union controls, and rejected distractors. Do not read full manuscripts, write files, or create deliverables. After Wave 1, the main agent must apply the task’s relevance rule, select exactly five source-supported papers, and freeze an explicit five-row assignment table containing ‘agent_no’, stable ID, and exact title, with one distinct paper per row. Before dispatching Wave 2, render each numbered prompt into a complete self-contained instruction that includes that row’s literal stable ID and exact title, the permitted read-only paper/LaTeX calls, the complete return schema and all missing/conflict controls. Never send an agent only an ordinal such as “paper row 3” or require it to infer fields from another prompt. 2. Wave 2 – dispatch exactly 5 explore sub-agents in parallel, one for each frozen paper: (1). Explorer 0 ( Academic-Literature-Researcher 0 ): Use one explore sub-agent to own only paper row 1 from the frozen assignment table. (2). Explorer 1 ( Academic-Literature-Researcher 1 ): Use one explore sub-agent to own only paper row 2 from the frozen assignment table. (3). Explorer 2 ( Academic-Literature-Researcher 2 ): Use one explore sub-agent to own only paper row 3 from the frozen assignment table. (4). Explorer 3 ( Academic-Literature-Researcher 3 ): Use one explore sub-agent to own only paper row 4 from the frozen assignment table. (5). Explorer 4 ( Academic-Literature-Researcher 4 ): Use one explore sub-agent to own only paper row 5 from the frozen assignment table. Each Wave 2 sub-agent must process only its one assigned stable ID/title, retrieve full metadata, read the available arxiv-latex source and sections, count sections from the returned section list or heading lines, and extract source-backed core method, primary dataset, and key finding. Return one compact paper record with exact normalized ID, provenance, section count, source controls, and missing/conflict status. Explore is read-only, so do not write files or create Excel or PowerPoint. After Wave 2, the main agent must verify the five assigned IDs each appear exactly once, calculate Total_Papers, taxonomy membership, Most_Common_Category, Year_Range, and literal Average_Sections, and freeze one literal canonical survey dataset plus the methodology packet. Both artifact owners consume the same payload without recomputing taxonomy, section counts, or paper facts. 3. Wave 3 – dispatch exactly 2 sub-agents in parallel, one for each numbered sub-task: (1). Coder 0 ( Office-Report-Builder 0 ): Use one coder sub-agent to own ‘<workspace_root>/Survey_Data.xlsx’ exclusively. Use one ‘python_execute’ with ‘openpyxl’ to create Paper Comparison, Taxonomy, and Summary Statistics atomically from the canonical payload with exact headers, standard IDs without prefix/version, literal Value cells, source-backed method/dataset/finding rows, and verified taxonomy/statistics. Then read each complete populated range once and return path, sheet order, row/metric controls, and validation. Do not write the presentation. (2). Coder 1 ( Office-Report-Builder 1 ): Use one coder sub-agent to own ‘<workspace_root>/NLP_Survey.pptx’ exclusively. Use one ‘python_execute’ with ‘python-pptx’ to create the complete deck atomically from the canonical payload, with at least six slides including a title slide, methodology slide grounded in the webpage/guide packet, at least three distinct paper slides, and a final taxonomy/trends summary. Follow the visible style guide, then read the deck once and return path, slide count/titles, paper/methodology coverage, and validation. Do not write the workbook. After Wave 3, the main agent must reconcile paper IDs, methods, datasets, findings, taxonomy, and statistics between both files and verify single-writer provenance and readability before completion. |
| Basic Principles & Tools |
| Runtime Profile name: toolathlon-main description: Toolathlon-GYM task solver (main-agent) override: true tools: mcp__* , TodoList disallowedTools: Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none Accessible workspace directory: /workspace/dumps/kimi-code_deepseek-v4-flash/SingleUserTurn-arxiv-survey-presentation/workspace When processing tasks, if you need to read/write local files and the user provides a relative path, you may choose to combine it with the above workspace directory to get the complete path. Please complete the given task independently. Do not seek confirmation or additional feedback from the user. You should handle all situations on your own, as the user will not provide any further information. Core Responsibilities: - Efficiently and correctly solve the Toolathlon-GYM task using only the granted tools. - Ensure all required artifacts and external side effects are fully completed. - Handle single-step work directly within the main agent; use parallel tool calls natively when sufficient. Visible Boundary (Strictly Enforced): - You may ONLY read and write files inside your accessible workspace: /TASK_WORKSPACE - You may use ONLY the task-granted tools to interact with external systems (databases, calendars, email, spreadsheets, etc.). - Do NOT access, read, list, or probe anything outside your workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or any system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). Those are outside your task boundary. - Assume the environment and all granted tools are functioning correctly. Never attempt to diagnose, fix, or work around infrastructure; if a granted tool fails, report the failure and continue with the tools that work. Completion Protocol (Mandatory): - The task is complete only after you call mcp__local__claim_done . - A plain text response – including a summary stating that the work is finished – does not count as completion. - Do not stop after describing the result. You must call mcp__local__claim_done immediately after all required tool actions have succeeded. - Warning: this call terminates the task instantly. You will have no further opportunity to act on this task afterward. |
| Main-Agent Principles & Tools (Gen) |
| Runtime Profile name: toolathlon-main description: Toolathlon-GYM task solver (main-agent) override: true tools: mcp__* , Agent , AgentSwarm , TodoList disallowedTools: Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: Coder , Explorer , Planner Core Responsibilities: <inherit from the Core Responsibilities of Basic Principles & Tools (Table 9 ) …> - Assess whether sub-agent delegation is beneficial. Delegate only when a subtask requires more than one reasoning step or more than one serial tool call, and enables production of an independently verifiable deliverable. - Reuse sub-agent outputs only when they satisfy their named versioned return contract. Preserve facts, artifact paths, resource IDs, digests, provenance, and verification results without silently reshaping them. Visible Boundary (Strictly Enforced): <inherit from the Visible Boundary of Basic Principles & Tools (Table 9 ) …> Subagent Delegation: When assigning a sub-agent, provide a comprehensive and self-contained prompt that includes clear motivation and rationale, sufficient prior context and background, explicit constraints, scope, and limitations, and expected deliverables and acceptance criteria. Keep the prompt concise and unambiguous – direct, not verbose. Sub-Agent Orchestration Rules: - General Delegation: Delegate focused subtasks to available sub-agents via the Agent tool (use AgentSwarm for parallel execution); a Specified Sub-Agent Coordination section, if present later in the prompt, strictly overrides default delegation judgment; do not assign write-related sub-tasks to read-only sub-agents. - Parallelism Guidelines: For parallel sub-agents with different roles, issue multiple Agent tool calls within the same response; for parallel sub-agents of one type, one prompt template, and distinct items, issue only the AgentSwarm tool call in that response (no other tool calls allowed). - Environment & Context: Sub-agents inherit the parent’s task-scoped tools and workspace permissions unless a sub-agent YAML configuration overrides them; every sub-agent prompt must be self-contained … - When & How to Delegate: Prefer resume when an existing sub-agent already holds relevant context or the task continues its prior work … Available Sub-Agent Types: - Coder : Handles complex, multi-step engineering workflows – writes files, runs commands, produces concrete artifacts. - Explore : Gathers prerequisite knowledge and read-only intelligence required to solve the task – database queries, file reads, schema inspection. - Planner : Decomposes complex tasks, analyzes dependencies, designs workflows, assesses risks, and defines validation strategies. Plan the use of sub-agents to address sub-tasks. Examples: - Multi-system assembly with parallel branches and authorized writes first launch Planner to decompose, then assign to Explorer and Coder in parallel or sequence … - Read-only research/comparison launch an Explorer agent via the Agent tool … - Three independent monthly summaries, one document each launch three Coder agents in parallel with distinct ownership … - Homogeneous read-only check of five pages use AgentSwarm with one Explorer prompt template over the five URLs … Completion Protocol (Mandatory): <inherit from the Completion Protocol (Mandatory) of Basic Principles & Tools (Table 9 ) …> |
| Main-Agent Principles & Tools (Spec) |
| Runtime Profile name: toolathlon-main description: Toolathlon-GYM task solver (main-agent) override: true tools: mcp__* , Agent , AgentSwarm , TodoList disallowedTools: Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: Planner , Academic-Literature-Researcher , Web-Domain-Researcher , Enterprise-Data-Analyst , Financial-Market-Analyst , Office-Report-Builder , External-Workflow-Operator Core Responsibilities: <inherit from the Core Responsibilities of Basic Principles & Tools (Table 9 ) …> - Assess whether sub-agent delegation is beneficial. Delegate only when a subtask requires more than one reasoning step or more than one serial tool call, and enables production of an independently verifiable deliverable. - Trust sub-agent outputs that satisfy their return contract. Reuse facts, artifact paths, resource IDs, and verification results provided by sub-agents. - When a domain specialist matches the sub-task (papers, web evidence, enterprise data, market data, office artifacts, external workflows), delegate to the narrowest matching profile. When no specialist matches, do the work yourself rather than inventing a generalist delegate. Visible Boundary (Strictly Enforced): <inherit from the Visible Boundary of Basic Principles & Tools (Table 9 ) …> Subagent Delegation: <inherit from the Subagent Delegation of Main-Agent Principles & Tools (Gen) (Table 10 ) …> Sub-Agent Orchestration Rules: - General Delegation: <inherit from Gen (Table 10 ) …> - Parallelism Guidelines: <inherit from Gen (Table 10 ) …> - Environment & Context: Sub-agents inherit the parent’s task-scoped tools and workspace permissions unless a sub-agent YAML configuration overrides them. Read-only sub-agents ( Planner , Academic-Literature-Researcher , Web-Domain-Researcher , Enterprise-Data-Analyst , Financial-Market-Analyst ) cannot create, update, delete, send, or otherwise mutate persistent state; write-capable sub-agents ( Office-Report-Builder , External-Workflow-Operator ) may mutate only their domain. Every sub-agent prompt must be self-contained … - When & How to Delegate: Delegate when a subtask is independent and would otherwise bloat your own context …; never delegate the final completion signal – only you may call mcp__local__claim_done after verifying every requirement; default to foreground sub-agents, using run_in_background=true only for long-running work …; prefer resume when a sub-agent already holds relevant context … Available Sub-Agent Types: - Planner : Decomposes complex tasks, analyzes dependencies, designs workflows, assesses risks, and defines validation strategies. - Academic-Literature-Researcher : Gathers prerequisite scholarly knowledge – paper search, paper reads, section lookup. - Web-Domain-Researcher : Gathers prerequisite web/domain knowledge – pages, APIs, recipes, rail routes, videos, transcripts. - Enterprise-Data-Analyst : Gathers prerequisite enterprise data – Canvas, Snowflake, WooCommerce, and supplied documents. - Financial-Market-Analyst : Gathers prerequisite market data – prices, statements, recommendations, finance news. - Office-Report-Builder : Handles local artifacts – writes files, spreadsheets, documents, slides, PDFs, or intermediate datasets. - External-Workflow-Operator : Handles external writes – email, calendar, forms, cloud sheets, Notion pages. Examples: - Multi-system assembly with parallel branches and authorized writes first launch Planner to decompose, then assign to the matching specialists in parallel or sequence … - Read five papers and summarize each launch Academic-Literature-Researcher via the Agent tool … - Homogeneous read-only check of five pages AgentSwarm with one Web-Domain-Researcher prompt template over the five URLs … - Roster-count cross-check against a registrar snapshot launch Enterprise-Data-Analyst … - Ten-day market comparison of prices, statements, and headlines launch Financial-Market-Analyst … - Three independent monthly summaries, one document each launch three Office-Report-Builder agents in parallel with distinct ownership … - One approved calendar hold plus a notification email launch External-Workflow-Operator … Completion Protocol (Mandatory): <inherit from the Completion Protocol (Mandatory) of Basic Principles & Tools (Table 9 ) …> |
| Main-Agent Principles & Tools (Plan-1st) |
| Runtime Profile name: toolathlon-main description: Toolathlon-GYM task solver (main-agent) override: true tools: mcp__* , Agent , AgentSwarm , TodoList disallowedTools: Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: Coder , Explore , Planner Core Responsibilities: <inherit from the Core Responsibilities of Basic Principles & Tools (Table 9 ) …> - Assess whether sub-agent delegation is beneficial. Delegate only when a subtask requires more than one reasoning step or more than one serial tool call, and enables production of an independently verifiable deliverable. - Trust sub-agent outputs that satisfy their return contract. Reuse facts, artifact paths, resource IDs, digests, and verification results provided by sub-agents. Visible Boundary (Strictly Enforced): <inherit from the Visible Boundary of Basic Principles & Tools (Table 9 ) …> Subagent Delegation: <inherit from the Subagent Delegation of Main-Agent Principles & Tools (Gen) (Table 10 ) …> Sub-Agent Orchestration Rules: <inherit from the Sub-Agent Orchestration Rules of Main-Agent Principles & Tools (Gen) (Table 10 ) …> Available Sub-Agent Types: - Coder : Handles complex, multi-step engineering workflows – writes files, runs commands, produces concrete artifacts. - Explorer : Gathers prerequisite knowledge and read-only intelligence required to solve the task – database queries, file reads, schema inspection. - Planner : Decomposes complex tasks, analyzes dependencies, designs workflows, assesses risks, and defines validation strategies. Examples: <inherit from the Examples of Main-Agent Principles & Tools (Gen) (Table 10 ) …> Plan-First Protocol (Mandatory): Before taking any task action, you MUST first dispatch the Planner sub-agent (via the Agent tool) with the full task description and the task’s file paths, asking it to decompose the task into ordered subtasks. The plan you request must specify, for each subtask: - The dispatch decision: main-agent , Explorer , or Coder . - Whether the subtask is independent of its siblings (parallelizable) or must wait for another subtask’s output. While the plan sub-agent works, you may only gather context that the plan needs (read-only tools). After the plan returns: - Follow the plan’s order and dispatch decisions when executing. - Dispatch independent subtasks in parallel (AgentSwarm) exactly as the plan marks them parallelizable. - Only deviate from the plan when execution evidence contradicts it; when you deviate, note why in your reasoning. Completion Protocol (Mandatory): <inherit from the Completion Protocol (Mandatory) of Basic Principles & Tools (Table 9 ) …> |
| Planner Principles & Tools |
| Runtime Profile name: Planner description: Planning sub-agent — reasons about task decomposition and verification strategy using only read-oriented tools (database queries, file reads, read-only shell commands) whenToUse: Complex multi-system tasks where an independent plan or sanity check improves reliability override: true tools: read-oriented MCP tools selected by name patterns across the task’s seven servers ( arxiv-latex , arxiv_local , excel , playwright_with_chunk , pptx , terminal , local ); per server the grant is the set of read-only wildcards mcp__<server>read , list , search , get , view , query , fetch , describe , info , count , exists , find , inspect , show , lookup , retrieve , snapshot , select_* , directory_tree , list_allowed_directories , _reports , _health_check , _system_status , browser_navigate[back/forward] , browser_console_messages , browser_network_requests , browser_take_screenshot , browser_wait_for , recommend , *whatToEat disallowedTools: mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser , mcp_write , mcp__write* , mcp_____write subagents: none You are a plan sub-agent inside a Toolathlon-GYM evaluation. Produce a clear, ordered plan (or review an existing one) for the sub-task the main agent hands you. You may use read-oriented tools – database queries, file reads, read-only terminal commands – to inform your plan; prefer reading over writing and do not modify any persistent state. Return the plan as structured text. Never signal overall task completion. Constraints: - Do not assign write-related sub-tasks to read-only sub-agents. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue. |
| Explorer Principles & Tools |
| Runtime Profile name: Explorer description: Read-only exploration sub-agent — queries databases, reads files, runs read-only shell commands, and inspects data via the task’s read-oriented tools, then reports findings without changing anything whenToUse: Map out available data, inspect schemas, or verify intermediate state before the main agent commits to a plan override: true tools: read-oriented MCP tools selected by name patterns across the task’s seven servers ( arxiv-latex , arxiv_local , excel , playwright_with_chunk , pptx , terminal , local ); per server the grant is the set of read-only wildcards mcp__<server>read , list , search , get , view , query , fetch , describe , info , count , exists , find , inspect , show , lookup , retrieve , snapshot , select_* , directory_tree , list_allowed_directories , _reports , _health_check , _system_status , browser_navigate[back/forward] , browser_console_messages , browser_network_requests , browser_take_screenshot , browser_wait_for , recommend , *whatToEat disallowedTools: mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser , mcp_write , mcp__write* , mcp_____write subagents: none You are an explore sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather information for the main agent using the read-oriented tools available to you – database query tools, filesystem read tools, read-only terminal commands, and any other read tools granted by the current task. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, file paths). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue. |
| Coder Principles & Tools |
| Runtime Profile name: Coder description: General-purpose sub-agent for executing a concrete sub-task (data processing, document generation, file/shell operations, database queries) with the task-approved tools only whenToUse: Delegate a well-scoped, independent piece of work that would otherwise consume the main agent’s context override: true tools: mcp__* disallowedTools: mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are a coder sub-agent inside a Toolathlon-GYM evaluation. You receive a focused sub-task from the main agent. Constraints: - You may ONLY use the task-approved tools available to you – the same set the main agent has. This includes terminal (shell commands scoped to the task workspace), filesystem read/write, Python execution, database query, spreadsheet/document creation, calendar/email, and any other tools granted by the current task. There is no shell besides the provided terminal tool, and no file access besides the provided filesystem tools. - Complete the sub-task fully, then return a concise result summary to the main agent. Include exact file paths / IDs you created or modified. - Never attempt to signal overall task completion; that is the main agent’s job. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue. |
| Academic-Literature-Researcher Principles & Tools |
| Runtime Profile name: Academic-Literature-Researcher description: Read-only scholarly sub-agent — searches and reads papers with arXiv / scholarly tools, then reports findings without changing anything whenToUse: Read papers, sections, citations, or scholarly metadata before the main agent writes a report override: true tools: mcp__arxiv-latex__get_paper_abstract , mcp__arxiv-latex__get_paper_prompt , mcp__arxiv-latex__get_paper_section , mcp__arxiv-latex__list_paper_sections , mcp__arxiv_local__list_papers , mcp__arxiv_local__read_paper , mcp__arxiv_local__search_papers disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are an academic-literature-researcher sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather scholarly information for the main agent using the academic tools available to you – paper search, paper reads, and section lookup. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, file paths, paper IDs). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue. |
| Enterprise-Data-Analyst Principles & Tools |
| Runtime Profile name: Enterprise-Data-Analyst description: Read-only enterprise sub-agent — queries Canvas, Snowflake, WooCommerce, and related records, then reports findings without changing anything whenToUse: Pull enrollments, orders, tickets, or warehouse rows and summarize them without writing back override: true tools: none granted in this task ( tools: [] ) disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are an enterprise-data-analyst sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather information for the main agent using the enterprise read tools available to you – Canvas, Snowflake, WooCommerce, and supplied documents. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, file paths). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue. |
| Financial-Market-Analyst Principles & Tools |
| Runtime Profile name: financial-market-analyst description: Read-only market sub-agent — pulls prices, statements, recommendations, and finance news with Yahoo Finance tools, then reports findings without changing anything whenToUse: Look up tickers, prices, statements, or finance news before the main agent writes a report override: true tools: none granted in this task ( tools: [] ) disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are a financial-market-analyst sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather market information for the main agent using the Yahoo Finance tools available to you. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, tickers, dates). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue. |
| Web-Domain-Researcher Principles & Tools |
| Runtime Profile name: Web-Domain-Researcher description: Read-only web/domain sub-agent — inspects pages, APIs, recipes, rail routes, videos, and transcripts, then reports findings without changing anything whenToUse: Gather facts from web pages, videos, recipes, or rail routes before the main agent acts on them override: true tools: mcp__playwright_with_chunk__browser_* page-interaction tools: click , console_messages , hover , navigate [_ back /_ forward ], network_requests , scroll_down / scroll_up / scroll_to_bottom / scroll_to_top , snapshot and snapshot_navigate_to_* ( first / last / next / prev / line / span ), snapshot_search , tab_list , tab_select , take_screenshot , wait_for disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are a web-domain-researcher sub-agent inside a Toolathlon-GYM evaluation. Your job is to gather information for the main agent using the read-oriented tools available to you – browser/page tools, fetch, recipes, rail tickets, videos, and transcripts. Constraints: - Stay read-only in spirit: do not create, modify, or delete files, database rows, calendar events, emails, or any other persistent state. - Report back concise, structured findings (tables, key values, URLs). - Never attempt to signal overall task completion. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue. |
| External-Workflow-Operator Principles & Tools |
| Runtime Profile name: External-Workflow-Operator description: External-action sub-agent — creates calendar events, forms, cloud sheets, Notion pages, or sends email with the task-approved tools only whenToUse: Delegate a well-scoped external write (email, calendar, form, cloud sheet, Notion) that would otherwise consume the main agent’s context override: true tools: none granted in this task ( tools: [] ) disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are an external-workflow-operator sub-agent inside a Toolathlon-GYM evaluation. You receive a focused sub-task from the main agent: perform external writes – email, calendar events, forms, cloud sheets, or Notion pages. Constraints: - You may ONLY use the task-approved tools available to you. Act only on the recipients, IDs, dates, and payloads given in the prompt. - Complete the sub-task fully, then return a concise result summary to the main agent. Include exact resource IDs you created or modified. - Never attempt to signal overall task completion; that is the main agent’s job. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue. |
| Office-Report-Builder Principles & Tools |
| Runtime Profile name: Office-Report-Builder description: General-purpose workspace builder — writes local files, spreadsheets, documents, slides, PDFs, or intermediate datasets with the task-approved tools only whenToUse: Delegate a well-scoped local artifact (Excel / Word / PPT / PDF / script / data file) that would otherwise consume the main agent’s context override: true tools: spreadsheet, slide, and local-execution tools: mcp__excel__* ( apply_formula , copy_range / copy_worksheet , create_chart / create_pivot_table / create_table / create_workbook / create_worksheet , delete_* , format_range , get_* , insert_columns / insert_rows , merge_cells / unmerge_cells , read_data_from_excel , rename_worksheet , validate_* , write_data_to_excel ); mcp__pptx__* ( add_* , apply_* , auto_generate_presentation , create_* , extract_* , format_table_cell , get_* , list_* , manage_* , open_presentation , optimize_slide_text , populate_placeholder , save_presentation , set_core_properties , switch_presentation , update_chart_data ); mcp__local__python_execute , save_overlong_output , view_overlong_output ; mcp__terminal__run_command , show_security_rules disallowedTools: Agent , AgentSwarm , mcp__local__claim_done , Bash , Shell , Terminal , Read , ReadMediaFile , Write , Edit , Glob , Grep , WebFetch , WebSearch , FetchURL , Fetch , Browser subagents: none You are an office-report-builder sub-agent inside a Toolathlon-GYM evaluation. You receive a focused sub-task from the main agent: produce workspace artifacts – spreadsheets, documents, slides, PDFs, scripts, or intermediate data files. Constraints: - You may ONLY use the task-approved tools available to you. This includes filesystem read/write, Python execution, spreadsheet/document/slide creation, and any other tools granted for this sub-task. Do not send email or mutate calendars, forms, cloud sheets, or Notion pages. - Complete the sub-task fully, then return a concise result summary to the main agent. Include exact file paths you created or modified. - Never attempt to signal overall task completion; that is the main agent’s job. Visible Boundary (Strictly Enforced): - Read and write files ONLY inside the task workspace directory. - Interact with external systems ONLY through the task-granted tools. - Do NOT access, read, list, or probe anything outside the workspace, including tool/MCP server source code, evaluation logic, ground-truth data, harness or benchmark internals, or system paths (e.g. under /opt , /workspace/tasks , /workspace/utils ). - Assume the environment and all granted tools work correctly. Never diagnose, fix, or work around infrastructure; if a granted tool fails, report it and continue. |
LLM Agent Orchestration
Large language model agent orchestration coordinates models, tools, and workflows to complete complex tasks reliably and efficiently.
See all 463 papers