MULTILOOP: Evolving Multi-Agent Harnesses for Complex Information-Seeking Tasks
Organizations: Tel Aviv University · Google Research
Abstract
The performance of LLM-based agents depends not only on the underlying model but also on the harness: the prompts, tools, and orchestration logic around it. Recent works automate harness optimization with coding agents that analyze execution traces, diagnose failures, and edit the harness code; however, those focused on single-agent systems. Multi-agent harnesses are harder to optimize as fixing them requires changing how agents interact in addition to how each behaves. We study whether automated harness optimization can diagnose and repair failures in multi-agent systems through a case study on complex information-seeking tasks from DeepSearchQA [Gupta et al., 2026]. We apply MULTILOOP, a simple optimization loop in which a coding agent iteratively analyzes, evaluates and edits the harness. With only 10 training samples, the evolved harness improves accuracy by 4.6% over the base harness and by 3.2% over the best single-agent system. However, a taxonomy of failure modes before and after optimization shows that the orchestration failures that characterize multi-agent systems largely persist. Our findings motivate harness optimization methods that encourage structural changes to how agents delegate tasks and share information.
Figures & tables
| Agent Architecture | Multi-agent | FC |
| Without harness optimization | ||
| Search Agent | no | |
| Search + Other Tools (SPOT) | no | |
| Orchestrator w/ SPOT subagents | yes | |
| With full harness optimization | ||
| SPOT | no | |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Error Category | Implemented Harness Change |
|---|---|
| Bad Orchestration | Updated the orchestrator’s system instruction to direct how to manage subagents: You are the Orchestrator. Your job is to break down complex queries into smaller, independent sub-queries and spawn subagents to solve them. When a task involves comparing data across multiple entities (e.g., countries, years, boroughs), ALWAYS dispatch separate subagent queries for each entity so they can be researched in parallel. Be highly specific in your queries to the subagents so they know exactly what data to find. |
| Ineffective Tool Usage | Modified the orchestrator’s system instruction to instruct subagents to use specific tools for specific tasks: If the task involves analyzing datasets or tabular data, explicitly instruct the subagents to use their ‘run_python‘ tool to download and process the data. Modified error messages in failed tool executions: f’ERROR: Failed to extract page {page_number} from {url} Reason: {e} The PDF may be unavailable, require authentication, or the URL may point to an HTML page instead of a PDF. Try using the ‘browse‘ tool on this URL.’ Updated the subagent’s system instruction on how to use tools: CRITICAL: When searching for specific statistics, numbers, or tabular data, Google Search snippets usually do NOT contain the exact answer. Instead of iteratively guessing numbers in your search queries, you MUST use the ‘browse‘ tool to read the full page of the most promising URLs returned by ‘google_search‘. For PDF documents, use the ‘pdf_tool_search‘ and ‘pdf_tool_extract‘ tools. For datasets, CSVs, or complex data analysis, use the ‘run_python‘ tool to download the data with ‘download_lib.download_file()‘ and analyze it using ‘pandas‘. Added a hint in the function declaration of the google_search tool: If the exact answer is not in the snippets, do NOT guess numbers in new queries; instead, use the ‘browse‘ tool on the returned URLs. Added a new tool for downloading online files, like CSVs: def download_file(url: str, save_path: str = "downloaded_file") -> str: """Downloads a file from the internet to the local sandbox.""" ... |
| Error Category | Implemented Harness Change |
|---|---|
| Tool Output Misreading | Updated the subagent’s system instruction to explain the use-cases of different tools: CRITICAL: When searching for specific statistics, numbers, or tabular data, Google Search snippets usually do NOT contain the exact answer. Instead of iteratively guessing numbers in your search queries, you MUST use the ‘browse‘ tool to read the full page of the most promising URLs returned by ‘google_search‘. For PDF documents, use the ‘pdf_tool_search‘ and ‘pdf_tool_extract‘ tools. For datasets, CSVs, or complex data analysis, use the ‘run_python‘ tool to download the data with ‘download_lib.download_file()‘ and analyze it using ‘pandas‘. Updated the function declarations of the tools to instruct the subagents how to use the tools in the sandboxed environment and parse their outputs: You MUST use the provided tools. NOTE: ALL injected tools return RAW STRINGS (str). If the returned string is in JSON format, you MUST parse it using ‘json.loads()‘ before indexing into it as a dictionary or list... str and extract_page(url: str, page_number: int) -> str - download_lib: download_file(url: str, save_path: str = "downloaded_file") -> str Use ‘download_lib‘ to download CSV, JSON, or other data files into your sandbox. DO NOT use requests/urllib, they will fail. Call the tools directly in your python code. For example: import json res = json.loads(google_search_lib.search(queries=["q1"])) download_lib.download_file( "https://example.com/data.csv", "data.csv") import pandas as pd df = pd.read_csv("data.csv") print(df.head()) |
| Final Answer Comp. | Added a strict guardrail in the orchestrator’s system instruction: CRITICAL INSTRUCTION FOR FINAL RESPONSE: When you have gathered the required information, you MUST output ONLY the exact requested answer as your final text. ABSOLUTELY NO CONVERSATIONAL FILLER. Do not use phrases like "Based on...", "According to...", or "The answer is...". If the answer is a list of items, provide a simple comma-separated list of those items and NOTHING ELSE. Any extra text, explanations, or context will cause the test to FAIL. |
| Behavioral Change | Example & Question | “Before” Rollout Quote / Action | “After” Rollout Quote / Action |
| Programmatic Data Processing (Using Python to download and parse data instead of searching) | Question: “Based on OWID sources for records of the year 2021… which country had the largest child mortality rate also in 2021.” | Quote: “I’ll use the Python tool to find the specific values or rankings by searching for ‘top age-standardized death rates 2021’ on the web or looking for the GBD 2021 publication.” Action: The agent improperly attempts to use the Python environment merely to run iterative Google search queries without downloading data, resulting in hallucinated values from snippets. | Quote: “Since I can’t browse the interactive table results easily via search snippets, I will use Python to search and maybe download if I find a CSV link.” Action: The subagent successfully uses Python to circumvent sandbox limitations, utilizing the injected download_lib tool to fetch the CSV datasets, parse them with pandas, and extract the real values for 2021. |
| Sequential Dependency Resolution (Waits for intermediate facts before proceeding) | Question: “According to Toyota, what is the part number for the oil filter cap assembly for the highest MSRP 2020 Toyota SUV that uses a battery with the same chemistry as the 2021 Toyota Aqua?” | Action: The Orchestrator attempted to solve the logically dependent steps at once by blindly guessing the target vehicle in parallel. It dispatched queries like: “oil filter cap assembly part number for 2020 Toyota Land Cruiser” and “oil filter cap assembly part number for 2020 Toyota Highlander Hybrid” before even identifying the required battery chemistry. | Quote: “What is the battery chemistry of the 2021 Toyota Aqua?” and “Which 2020 Toyota SUV models use a battery with the same chemistry as the 2021 Toyota Aqua?” Action: The Orchestrator avoids guessing and correctly assigns subagents sequentially to identify the Bipolar NiMH battery before executing a targeted search for the Highlander Hybrid’s oil filter cap. |
| Continued on next page… | |||
| Behavioral Change | Example & Question | “Before” Rollout Quote / Action | “After” Rollout Quote / Action |
|---|---|---|---|
| Page Browsing (Opening URLs to read full text instead of relying on snippets) | Question: (Implied from context) Calculate the horsepower-to-cargo-volume ratio for specific Honda vehicles based on Kelley Blue Book (KBB) data. | Quote: “I’ll present the calculation using Maximum Cargo Volume as it’s the standard for general comparison.” Action: The agent relied entirely on the google_search snippets. Without reading the full context of the spec sheet, it incorrectly chose the minivan’s maximum cargo volume (144.9 cu. ft. with seats removed) instead of the standard trunk space, skewing the math. | Quote: “I will use Google Search to find the specific kbb.com page for the 2023 Honda HR-V and then use the browse tool to extract the required specifications.” Action: Guided by a new instruction to use the browse tool, the agent establishes the correct detailed intention. This rigorous approach causes it to accurately extract the standard cargo volume behind the third row (“32.8 cu. ft.”), leading to the correct mathematical conclusion. |
| Over-Delegation & Exhaustive Verification (Degradation) (Over-thinking and getting trapped in loops) | Question: (Implied from context) Find the most recently seen species among those assessed as ‘Extinct’ (EX) on the IUCN Red List that occurred in Spain. | Action: The subagent correctly zeroes in on multiple candidate species and executes Python looping scripts to cross-reference their last-seen dates via google search, rapidly deducing that the freshwater snail Islamia ateni was the most recent. | Quote: “Are there any others? I’ll use google search within python to find the specific list of EX insects in Spain.” Action: Even though the Orchestrator established the correct answers internally, it became paranoid about proving a negative. It redundantly spawned multiple subagents to hunt for obscure species, trapping the system in an endless verification loop. |
Agent Harness Optimization
Agent harness optimization improves the prompts, tools, memory, workflows, and control code that shape how AI agents act.
See all 206 papers