Multi-agent systems (MAS) built on large language models (LLMs) have shown strong performance across many tasks. Most existing approaches improve only one aspect at a time, such as the communication topology, role assignment, or LLM routing, while treating each agent as a single, indivisible unit. This misses the opportunity to use mixtures of LLMs within an agent to strengthen role-specific abilities. We propose HieraMAS, a hierarchical collaboration framework that combines intra-node LLM mixtures with an inter-node communication topology. HieraMAS introduces supernodes, where each functional role is implemented by multiple heterogeneous LLMs using a propose-synthesis structure. Optimizing HieraMAS creates unique credit-assignment challenges: final task performance depends heavily on the underlying LLMs' capabilities, which can lead reinforcement methods to incorrectly reward suboptimal configurations. To address this, we use a two-stage algorithm: (1) multi-level reward attribution, which provides fine-grained feedback at both the node level and the overall system level; (2) graph classification for topology selection, which treats choosing the communication structure as a holistic decision rather than optimizing edges one by one. Experiments on reasoning and coding benchmarks show that HieraMAS substantially outperforms existing methods while also delivering better cost-performance trade-offs.
Figures & tables
Figure 1: Illustration of two credit assignment challenges in joint optimization and our solutions. Challenge 1 : Final task rewards mask individual node errors—Node 2 produces incorrect output but receives high reward R2=0.92 . HieraMAS addresses this via multi-level rewards that provide effective per-role attribution ( R2eff=−0.23 ). Challenge 2 : Per-edge optimization suffers from entangled attribution, where edges may be falsely reinforced or suppressed. HieraMAS reformulates topology selection as a holistic graph classification task, using a graph generator to produce candidates and a graph classifier to select the optimal topology.
Figure 2: The overall framework of HieraMAS . By optimizing a policy learner πm with multi-level rewards (Stage 1) and a graph classifier fG(⋅) with contrastive rewards (Stage 2), HieraMAS learns to select optimal supernode configurations and communication topologies. During inference, the trained modules jointly determine the supernode configurations and graph topology, then execute the MAS to produce the final answer.
HumanEval++
MATH
MMLU-Redux
Method
Multi
Topo
Role
Node
GPT-5-Mini
Qwen3-80B
GPT-5-Mini
Qwen3-80B
GPT-5-Mini
Qwen3-80B
Avg.
Base
✗
✗
✗
✗
89.06
84.14
77.78
74.44
92.00
82.40
83.14
CoT
✗
✗
✗
✗
87.50
85.94
92.22
90.00
93.60
89.60
89.81
Self-Consistency
✓
✗
✗
✗
89.06
87.50
93.33
91.11
94.40
83.20
89.77
Self-Consistency+CoT
✓
✗
✗
✗
90.62
85.94
94.44
92.22
93.60
92.80
91.60
LLM-Debate
✓
✗
✗
✗
87.50
87.50
94.44
94.44
92.80
92.00
91.45
Table 1: Main results on three benchmarks. We report accuracy (%) for each dataset. ✓ indicates the method uses the corresponding component: Multi = multi-agent, Topo = optimizing topology, Role = optimizing roles and LLMs in each role, Node = optimizing intra-node configuration. Best results are in bold , second best are underlined .
MATH
MMLU-Redux
Variant
Acc
Cost
Acc
Cost
w/o Graph
93.33
1.57
92.00
1.36
w/o LLM Selection
97.78
2.56
94.40
3.92
Ours
96.67
1.52
95.20
1.29
Table 2: Ablation study on MATH and MMLU-Redux benchmarks. We evaluate the contribution of graph topology scoring and LLM selection. Acc (%) and Cost (USD) are reported.
Figure 3: Analysis of learned topologies on MMLU-Redux. (a) Visualization of the top-3 most frequently selected graph structures with their density. (b) Pairwise Jaccard similarity between top-5 graphs, showing low structural overlap.
Table 3: Generalization performance (%) on unseen MMLU categories. Best in bold , second best underlined .
Method
HumanEval++
MATH
MMLU
GDesigner
1.33
2.16
3.66
MASRouter
0.12
0.32
0.15
Ours
0.53
1.52
1.29
Table 4: Cost comparison ($). Lowest cost in bold .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Description
Symbol
Description
Search Space & Supernode Structure
S
Search space (M,R,G)
Si
Supernode i with role ri
M
Pool of Nm LLM backbones
ri
Role for supernode i , ri∈R
R
Set of Nr predefined agent roles
W
Number of proposers per supernode
G
Space of graph topologies
mi,j(w)
LLM for j -th proposer in supernode i
G
Pool of K candidate DAGs
mi(a)
LLM for synthesizer in supernode i
Appendix
Table 5: Summary of notation used in this paper.
Method
Acc (%)
Cost ($)
w/ 8 proposers
88.80
2.32
w/ 6 proposers
91.20
1.97
Ours (2 proposers)
95.20
1.29
Appendix
Table 6: Effect of the number of proposers per supernode on MMLU-Redux.
Method
Acc (%)
w/ K=50
93.60
w/ K=500
94.40
Ours ( K=200 )
95.60
Appendix
Table 7: Effect of graph pool size K on MMLU-Redux.
MMLU-Redux Case Studies
Case 1: Nutrition Question
Case 2: Communication Theory
Query: Which of the following appears to lower bad cholesterol? A. Vitamin D B. Niacin C. Thiamine D. Riboflavin
Query: According to “measurement,” what is the step that occurs between an individual gaining information and changing behavior? A. coorientation B. opinion change C. reaction formation D. semantic encoding
Appendix
Table 8: Case studies on MMLU-Redux. The typical configuration assigns GPT-5-Mini or GPT-5-Nano as proposers and GPT-5-Nano as the aggregator, favoring cost-efficient models for knowledge-intensive QA tasks.
HumanEval Case Studies
Case 1: Sum and Product
Case 2: Filter by Substring
Query: def sum_product(numbers: List[int]) -> Tuple[int, int]: """ For a given list of integers, return a tuple consisting of a sum and a product of all the integers in a list..."""
Query: def filter_by_substring(strings: List[str], substring: str) -> List[str]: """ Filter an input list of strings only for ones that contain given substring..."""
Appendix
Table 9: Case studies on HumanEval. The typical configuration assigns Qwen3-8B, Qwen3-80B, or LLaMA3.1-8B as proposers, leveraging diverse code-specialized models for programming tasks.
MATH Case Studies
Case 1: Geometry Reflection
Case 2: Cone Volume
Query: Triangle ABC with vertices A(−2,0) , B(1,4) and C(−3,2) is reflected over the y -axis to form triangle A′B′C′ . What is the length of a segment drawn from C to C′ ?
Query: The diameter of a cone is 30 decimeters. If the height is two times the radius, what is the volume of the cone in cubic decimeters?
Appendix
Table 10: Case studies on MATH. The typical configuration assigns Qwen3-8B combined with Qwen3-80B or DeepSeek-V3.2 as proposers, utilizing strong mathematical reasoning models.
Dataset
GDesigner
AFlow
MASRouter
Ours
MATH
$1.22
$18.62
$1.97
$2.03
HumanEval++
$1.95
$28.64
$0.54
$1.55
MMLU-Redux
$3.51
$13.94
$0.46
$1.85
Appendix
Table 11: Training cost comparison (in USD) across benchmarks.
Model
Profile Description
qwen3-8b
Fast and efficient 8B parameter model from Alibaba. Very fast inference, low latency, cost-effective. Good for basic text generation and simple Q&A. Very low cost.
qwen3-80b
Powerful 80B mixture-of-experts model from Alibaba. Strong reasoning, excellent at complex tasks, good at math and coding. Good for complex reasoning and mathematical problem-solving. Medium-high cost.
deepseek-r1-14b
Math-focused 14B model optimized for reasoning. Excellent mathematical reasoning and step-by-step problem solving. Good for math problems and logical reasoning. Low-medium cost.
llama-8b
General-purpose 8B model from Meta. Well-balanced capabilities with good instruction following. Good for general tasks, conversation, and basic reasoning. Very low cost.
deepseek-v3
State-of-the-art reasoning model with excellent performance. Top-tier reasoning, excellent at complex problems, strong coding ability. Good for complex reasoning and advanced math. Medium cost.
gemma-3-27b
Balanced 27B model from Google. Good balance of speed and capability with strong instruction following. Good for medium complexity tasks. Medium cost.
Appendix
Table 12: LLM profiles used for embedding-based model selection. Each profile describes model strengths, cost tier, and recommended use cases.
Role
Profile Description
Mathematical Reasoning Roles (MATH benchmark)
Mathematical Analyst
Specialized in breaking down complex math problems. Identifies mathematical structure, defines variables, and sets up equations. Best for initial problem understanding and mathematical modeling.
Math Solver
Specialized in executing mathematical computations. Solves equations, performs calculations, and verifies numerical results. Best for carrying out calculations and deriving numerical answers.
Inspector
Specialized in reviewing and validating solutions. Checks solution correctness, identifies errors, and verifies final answers. Best for final verification and error catching.
Programming Roles (HumanEval benchmark)
Project Manager
Specialized in design patterns and code structure. Coordinates solution strategy and organizes workflow. Best for complex problems requiring structured multi-step approaches.
Appendix
Table 13: Role profiles used for embedding-based role assignment. Roles are categorized by their primary application domain.
Large Language Models (LLMs) have shown remarkable performance in completing various tasks. However, solving complex problems often requires the coordination of multiple agents, raising a fundamental question: how to effectively select and interconnect these agents. In this paper, we propose Agent Q-Mix, a reinforcement learning framework that reformulates topology selection as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. Our method learns decentralized communication decisions using QMIX value factorization, where each agent selects from a set of communication actions that jointly induce a round-wise communication graph. At its core, Agent Q-Mix combines a topology-aware GNN encoder, GRU memory, and per-agent Q-heads under a Centralized Training with Decentralized Execution (CTDE) paradigm. The framework optimizes a reward function that balances task accuracy with token cost. Across seven core benchmarks in coding, reasoning, and mathematics, Agent Q-Mix achieves the highest average accuracy compared to existing methods while demonstrating superior token efficiency and robustness against agent failure. Notably, on the challenging Humanity's Last Exam (HLE) using Gemini-3.1-Flash-Lite as a backbone, Agent Q-Mix achieves 20.8% accuracy, outperforming Microsoft Agent Framework (19.2%) and LangGraph (19.2%), followed by AutoGen and Lobster by OpenClaw. These results underscore the effectiveness of learned, decentralized topology optimization in pushing the boundaries of multi-agent reasoning.
Eric Hanchen Jiang, Levina Li, Rui Sun +9
University of California Los Angeles · University of Washington · Northwestern University
LLM-based multi-agent systems (MAS) have emerged as a promising paradigm for enhancing complex reasoning through role specialization and collaboration, yet a key challenge remains: how can collaboration be adapted to a backbone's competence on the current query? This problem matters because more collaboration is not always better--richer coordination may help on difficult cases, but it also incurs additional token cost and redundant interactions. We propose a two-stage competence-aware framework for automatic MAS orchestration. It first models the LLM backbone's task-conditioned competence for the current query, and then uses this signal to adapt role activation and the communication graph. Across six benchmarks, our method achieves an average score of 93.81%. Experiments with six LLM backbones of varying competence levels further show that our method adjusts collaboration according to the inferred competence: for higher-competence backbones, it improves average accuracy from 87.22 to 89.97 while reducing token cost by 36.26%; for lower-competence backbones, it yields a further average gain of 4.58 points over existing methods. The same trend also holds on unseen backbones.
LLM-based multi-agent systems (MAS) typically optimize a single topology, restricting reasoning to a narrow trajectory and limiting comprehensive analytical capacity. Naively merging multiple topologies into a composite graph introduces redundant noise propagation across irrelevant connections, degrading solution quality. To address this dilemma, we propose Hierarchical Sparse Coordination over a Union of Complementary Topologies for MAS (HELENA), a multi-agent framework that balances diverse reasoning paths with sparse task-dependent execution. \helena{} constructs a union MAS graph from complementary candidate topologies selected via Monte Carlo Tree Search and Determinantal Point Process, broadening the reasoning trajectory for comprehensive analysis of complex problems. A Hierarchical Sparse Coordination module then activates only a sparse subgraph at each step while agents exchange compressed latent briefs to suppress redundant noise propagation. Finally, a Local Self-Refinement stage identifies decision units with discrepancy evidence and rewrites them only when contrastive evidence simultaneously confirms a reliable solution-side failure and a challenger-side improvement. Experiments across eight benchmarks show that \helena{} achieves state-of-the-art results on all benchmarks, with an average gain of \pctup{3.47} over the strongest baseline and up to \pctup{10.34} on MMLU-Pro, achieving larger improvements on harder benchmarks at a reasonable additional cost.
Zhifang Mao, Linyao Zheng, Xuhang Shi +1
1XiaoLab · 2Xi’an Jiaotong University · 3Beijing University of Posts and Telecommunications