OptiMAS: Automatically Optimize Multi-Agent System

Yuxin Cheng; Chang Liu; Hanxin Yu; Haochen Tan; Taiqiang Wu; Weiqiang Jin; Jie Ran; Kaibo Wang

cs.MAAug 22, 2026

OptiMAS: Automatically Optimize Multi-Agent System

Authors: Yuxin Cheng, Chang Liu, Hanxin Yu, Haochen Tan, Taiqiang Wu, Weiqiang Jin, Jie Ran, Kaibo Wang, +4 more

Organizations: The University of Hong Kong · Huawei Foundation Model Department, Huawei Technologies Ltd. · City University of Hong Kong · Xi’an Jiaotong University · The Hong Kong University of Science and Technology

Abstract

Automated evolution of Multi-Agent Systems (MAS) holds significant potential for reducing the manual effort required to design and optimize LLM-based agent architectures. However, extant search-based paradigms face a fundamental trade-off, where an expanded optimization scope exacerbates evolutionary instability, while discrete branch-and-discard search isolates insights across lineages. To address these limitations, we propose a continuous, data-driven optimization paradigm built upon a unified ReAct-based infrastructure that reconciles a broad optimization scope with operational stability. Under this paradigm, we present OptiMAS, a task-agnostic agentic optimizer that leverages textual interaction trajectories and task feedback as loss signals for end-to-end MAS evolution. Equipped with a novel dual-track memory mechanism, OptiMAS sustains performance improvement over extended optimization horizons. Evaluation on four heterogeneous agentic benchmarks with three varying scale and accessibility LLM backbones, demonstrates that OptiMAS consistently achieves competitive or superior accuracy relative to both domain-specialized hand-crafted systems and existing evolutionary methods. Our work establishes a practical milestone toward robust, automated MAS evolution.

Figures & tables

Figure 1: Comparison of MAS evolution paradigms. (Left) Search-based methods frame evolution as stochastic tree-branching ( $G_{0}\!\to\!G_{t}^{i}$ ) over full datasets, incurring low efficiency with insights isolated across branches. (Right) OptiMAS reformulates evolution as a data-driven continuous optimization over partitioned data (train/val/test), channeling action trajectories and accuracy as loss signals for OptiMAS to derive textual gradients and perform end-to-end MAS updates, yielding stable, sample-efficient, and overfit-resistant synthesis.

Figure 2: OptiMAS optimizer architecture at step $t$ . (a) The MAS $\mathcal{E}_{t}$ executes on train/val queries, producing trajectories and accuracy. (d) The optimizer diagnoses root causes via workspace inspection and evolves $\mathcal{E}_{t}$ , governed by (c) a plan-oriented short-term memory enforcing intra-step consistency and (b) a hypothesis-driven long-term memory maintaining a state-tracked ledger of falsifiable hypotheses across steps, yielding $\mathcal{E}_{t+1}$ .

Method	WorkBench (%) $\uparrow$			GAIA (%) $\uparrow$			BrowseComp* (%) $\uparrow$			SWE-Bench* (%) $\uparrow$
Method	GPT5	Qwen3.6	Gemini3	GPT5	Qwen3.6	Gemini3	GPT5	Qwen3.6	Gemini3	GPT5	Qwen3.6	Gemini3
Self-define Initial Agent
Single $\mathcal{E}_{0}$	18.6	65.6	71.7	49.5	73.1	80.7	8.3	33.9	52.2	29.4	61.1	72.8
Hand-crafted MAS
SMoA	15.8	57.4	66.3	38.7	50.0	59.7	1.7	6.7	20.0	–	–	–
MAS Debate	15.0	39.2	44.2	32.3	38.7	50.0	5.0	11.7	20.0	13.3	23.3	26.7

Table 1: Main results (task accuracy, %) across four benchmarks and three backbone LLMs: GPT5, Qwen3.6 and Gemini3 (abbreviated in column headers; full specifications in \cref sec:experiment_setting). Green and purple highlight the best and second-best results per backbone column. (–) denotes that such methodology is inapplicable or non-executable on given configurations. (*) marks evaluation subsets sampled via data curation(Appendix A ).

Figure 3: Evolution dynamics on WorkBench. Accuracy of

\mathcal{E}_{t}

\mathcal{X}_{\mathrm{train}}

(left) and

\mathcal{X}_{\mathrm{test}}

(right) over 10 epochs. Dashed lines show mean over three trials while solid lines are moving averages to highlight trend.

Figure 4: Ablation analysis of optimizer backbones. Performance is compared across combinations of two MAS backbones and two OptiMAS backbones.

Ablation Settings	WorkBench (%) $\uparrow$		SWE-Bench (%) $\uparrow$
Ablation Settings	GPT5	Qwen3.6	GPT5	Qwen3.6
Initial $\mathcal{E}_{0}$ (baseline)	18.6	65.6	29.4	61.1
OptiMAS w/ BS $n\!=\!10$	64.0 (+45.4)	84.5 (+18.9)	41.7 (+12.3)	68.9 (+7.8)
✕ w/ BS $n\!=\!4$	55.0 (+36.4)	81.1 (+15.5)	24.4 (-5.0)	62.8 (+1.7)
✕ Validation Metric	58.2 (+39.6)	82.8 (+17.2)	28.9 (-0.5)	64.4 (+3.3)
✕ Hypothesis Memory	53.9 (+35.3)	71.4 (+5.8)	22.2 (-7.2)	53.3 (-7.8)

Table 2: Ablation analysis of framework components. Bold indicates best results. “✕” signifies component removal, and “BS” represents batch size.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Source size	$\|\mathcal{X}_{\mathrm{train}}\|$	$\|\mathcal{X}_{\mathrm{val}}\|$	$\|\mathcal{X}_{\mathrm{test}}\|$	Total $\|\mathcal{X}\|$
SWE-Bench-Verified	500	36	4	60	100
BrowseComp	1,266	36	4	60	100
GAIA-text	103	37	4	62	103

Appendix

Table 3: Dataset partitioning for evolutionary experiments.

Domain	Source	$\|\mathcal{X}_{\text{train}}\|$	$\|\mathcal{X}_{\text{val}}\|$	$\|\mathcal{X}_{\text{test}}\|$
Analytics	120	10	2	108
Calendar	110	10	2	98
CRM	80	10	2	68
Email	90	10	2	78
Multi-Domain	210	10	2	198
Project Management	80	10	2	68

Appendix

Table 4: WorkBench dataset partition via stratified sampling (seed

s=42

Hyperparameter	Value	Mathematical Functional Description
$\phi$ (Retry Ratio)	0.5	Mass allocation ratio reserved for historic failed queries
$\alpha$ (Decay Factor)	0.6	Multiplicative discount multiplier for successful queries
$\beta$ (Boost Factor)	1.2	Multiplicative scaling multiplier for failed queries
$p_{\min}$ (Weight Floor)	0.2	Lower-bound boundary for sample probability clamping
$p_{\max}$ (Weight Ceiling)	2.0	Upper-bound boundary for sample probability clamping
$\tau$ (Reset Interval)	2 epochs	Temporal stride between global uniform weight resets

Appendix

Table 5: Parameter configurations for the adaptive sampling subsystem.

Operational Category	Hyperparameter Layer	Calibrated Value
Training Loop Boundaries	Max Optimization Horizon $E$	10 epochs
Training Loop Boundaries	Batch Capacity $B$	10
Adaptive Sampling Subsystem	Retry Ratio $\phi$	0.5
	Decay Factor $\alpha$	0.6
	Boost Factor $\beta$	1.2
	Weight Floor $p_{\min}$	0.2

Appendix

Table 6: Consolidated global hyperparameter specifications.

LLM Backbone	Context Limit	soft limit $C_{\text{soft}}$	hard limit $C_{\text{hard}}$	force limit $C_{\text{force}}$	Tool Output Limit
GPT-5-Nano	128K	126K	127K	128K	20K
Qwen3.6-35B-A3B	262K	260K	261K	262K	20K
Gemini-3-Flash	1M	970K	980K	990K	80K

Appendix

Table 7: Context Management hyperparameters on LLM Backbones.

Figure 5: Architectural Topography of the Evolved Multi-Agent System (MAS) on SWE-Bench-Verified. This schematic contrasts the initial configuration $\mathcal{E_{0}}$ against the evolved architecture ( $\mathcal{E^{*}}$ ). Through our OptiMAS optimization, the system undergoes severe structural transformation: transitioning from a standard flat-worker scheme into a modular, hierarchical coalition composed of specialized agents ( Worker , Reproducer , Refiner , and Verifier ). Each sub-agent customized ReAct iteration budgets, and differentiated, priority-driven tool and skill allocations.

Explore similar work

MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems
May 7, 2026Zhexuan Wang, Xuebo Liu, Li Wang +4Multi-Agent LLM Systems Multi-Agent System Optimization
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China.
Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systems
May 28, 2026Zhezheng Hao, Tianfu Wang, Huanshuo Dong +7Multi-Agent System Optimization Multi-Agent Coordination
Zhejiang University · Hong Kong University of Science and Technology · Tencent
MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?
Jun 22, 2026Juyang Bai, Laixi ShiMulti-Agent LLM Systems Multi-Agent System Optimization
Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA.

May 7, 2026J/K move · Enter open · S save

MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems

Zhexuan Wang, Xuebo Liu, Li Wang, Zifei Shan+3

Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China.

Multi-Agent LLM Systems Multi-Agent System Optimization Multi-Agent Collaboration Prompt Optimization

Open paper AskPDF arXiv

Large language model (LLM)-based Multi-agent systems (MAS) have shown promise in tackling complex collaborative tasks, where agents are typically orchestrated via role-specific prompts. While the quality of these prompts is pivotal, jointly optimizing them across interacting agents remains a non-trivial challenge, primarily due to the misalignment between local agent objectives and holistic system goals. To address this, we introduce MASPO, a novel framework designed to automatically and iteratively refine prompts across the entire system. A core innovation of MASPO is its joint evaluation mechanism, which assesses prompts not merely by their local validity, but by their capacity to facilitate downstream success for successor agents. This effectively bridges the gap between local interactions and global outcomes without relying on ground-truth labels. Furthermore, MASPO employs a data-driven evolutionary beam search to efficiently navigate the high-dimensional prompt space. Extensive empirical evaluations across 6 diverse tasks demonstrate that MASPO consistently outperforms state-of-the-art prompt optimization methods, achieving an average accuracy improvement of 2.9. We release our code at https://github.com/wangzx1219/MASPO.

Abstract

Explore similar work

MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems

Evolve as a Team: Collaborative Self-Evolution for LLM-based Multi-Agent Systems

MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems?