Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning

Eric Hanchen Jiang; Levina Li; Rui Sun; Xiao Liang; Yubei Li; Yuchen Wu; Haozheng Luo; Hengli Li

cs.CLApr 1, 2026

Agent Q-Mix: Selecting the Right Action for LLM Multi-Agent Systems through Reinforcement Learning

Authors: Eric Hanchen Jiang, Levina Li, Rui Sun, Xiao Liang, Yubei Li, Yuchen Wu, Haozheng Luo, Hengli Li,

Organizations: University of California Los Angeles · University of Washington · Northwestern University

Abstract

Large Language Models (LLMs) have shown remarkable performance in completing various tasks. However, solving complex problems often requires the coordination of multiple agents, raising a fundamental question: how to effectively select and interconnect these agents. In this paper, we propose Agent Q-Mix, a reinforcement learning framework that reformulates topology selection as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. Our method learns decentralized communication decisions using QMIX value factorization, where each agent selects from a set of communication actions that jointly induce a round-wise communication graph. At its core, Agent Q-Mix combines a topology-aware GNN encoder, GRU memory, and per-agent Q-heads under a Centralized Training with Decentralized Execution (CTDE) paradigm. The framework optimizes a reward function that balances task accuracy with token cost. Across seven core benchmarks in coding, reasoning, and mathematics, Agent Q-Mix achieves the highest average accuracy compared to existing methods while demonstrating superior token efficiency and robustness against agent failure. Notably, on the challenging Humanity's Last Exam (HLE) using Gemini-3.1-Flash-Lite as a backbone, Agent Q-Mix achieves 20.8% accuracy, outperforming Microsoft Agent Framework (19.2%) and LangGraph (19.2%), followed by AutoGen and Lobster by OpenClaw. These results underscore the effectiveness of learned, decentralized topology optimization in pushing the boundaries of multi-agent reasoning.

Figures & tables

Figure 1: Overview of Agent Q-Mix. Agents select communication actions, induce a round-wise communication graph, exchange messages over that graph, and produce a final team output under centralized training with decentralized execution.

Figure 2: Illustration of the six discrete communication actions in the Agent Q-Mix action space (Section 4.1 ). Each panel depicts the induced graph structure for a representative multi-agent team. Arrows indicate the direction of information flow. From left to right: (a) solo_process —the agent works independently with no outgoing edges; (b) broadcast_all —the agent sends its message to all other agents; (c) selective_query —the agent sends a targeted query to one specific neighbor; (d) aggregate_refine —the agent collects responses from all others through incoming edges; (e) execute_verify —the agent forwards its output to the next agent for verification with minimal communication overhead; (f) debate_check —two agents engage in mutual exchange via bidirectional edges. These six primitives span a range of connectivity levels from isolated nodes to dense cliques, enabling the QMIX policy to compose task-adaptive topologies.

Figure 3: Case study of Agent Q-Mix showing a representative communication and reasoning trajectory across rounds. In practice, mathematics tasks typically converge in $T=3$ communication rounds, while coding and reasoning tasks require $T=2$ rounds (see the ablation on communication rounds in Figure 5 d and Appendix D ).

Figure 4: Humanity’s Last Exam (HLE). Accuracy (%) on the first 250 MCQ items of HLE using Gemini-3.1-Flash-Lite. Agent Q-Mix (20.8%) outperforms Microsoft Agent Framework (19.2%), LangGraph (19.2%), AutoGen, and Lobster, while using significantly lower tokens.

Base Model: GPT-OSS:120B
Model/Methods	Code		Reasoning	Mathematics
Model/Methods	LiveCode	HE	MMLU-Pro	AIME 25	AIME 26	HMMT	Beyond	Avg
Single-agent baseline
Base (direct) ( OpenAI, 2025 )	88.75	90.04	74.29	23.33	20.00	26.67	18.00	48.73
Static multi-agent
LLM-Debate ( Du et al., 2023 )	92.00	89.63	78.57	30.00	36.67	33.33	28.00	55.46

Table 1: Accuracy (%) on seven benchmarks compared to base (single-agent). Bold : best overall per model. Underline : best among prior methods.

Table 2: Robustness on MMLU-Pro (GPT-OSS:120B). One adversarial agent is injected. $\Delta$ : accuracy drop.

Table 3: Token usage comparison on GPT-OSS:120B (total tokens across all samples). Lower is better at comparable accuracy.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Split	Samples	Phase
LCB TestGen	Coding	test	15	Train
MMLU-Pro	Reasoning	val	15	Train
AIME 2024	Math	train	15	Train
LiveCodeBench	Coding	test	400	Eval
HumanEval	Coding	test	164	Eval
MMLU-Pro	Reasoning	val	70	Eval

Appendix

Table 4: Dataset statistics.

Hyperparameter	Value
GNN layers	2
GNN hidden dimension	128
GRU hidden dimension	128
QMIX mixing hidden dim	64
Hyper-network hidden dim	64
Optimizer	Adam

Appendix

Table 5: Hyperparameter settings.

Symbol	Description
$N$	Number of agents
$\mathcal{N}$	Set of agents $\{1,\dots,N\}$
$\mathcal{S}$	Global state space
$\mathcal{A}_{i}$	Action space of agent $i$ (6 communication actions)
$\mathbf{u}_{t}$	Joint action vector $(u_{t}^{1},\dots,u_{t}^{N})$ at round $t$
$P$	State transition kernel

Appendix

Table 6: Summary of notation.

Figure 5: Ablation studies on Gemini-3.1-Flash-Lite. (a) Number of agents vs. accuracy on Beyond-AIME, average token cost, and HumanEval. (b) Number of training examples vs. accuracy. (c) Accuracy reward weight $w_{\mathrm{acc}}$ vs. accuracy and average token cost (with $w_{\mathrm{tok}}=0.075$ ). (d) Number of communication rounds vs. accuracy.

Multi-Agent Communication Topology

See all 34 papers

Learning Who Talks to Whom: Role-Aware Communication Topology Scheduling for Efficient LLM Multi-Agent Collaboration
Oct 19, 2026Haibin Li, Wenya Yu, Xuefeng Jiang +4Multi-Agent Communication Topology Multi-Agent LLMs
Beihang University · Nanyang Technological University · University of Chinese Academy of Sciences
HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systems
Feb 23, 2026Tianjun Yao, Zhaoyi Li, Zhiqiang ShenCommunication Topology Optimization Multi-Agent LLM Systems
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE.
RUMAD: Reinforcement-Unifying Multi-Agent Debate
Feb 27, 2026Chao Wang, Han Lin, Huaze Tang +2Communication Topology Optimization Multi-Agent Debate
Tsinghua University, Shenzhen, China · Zhejiang University, Hangzhou, China

Oct 19, 2026J/K move · Enter open · S save

Learning Who Talks to Whom: Role-Aware Communication Topology Scheduling for Efficient LLM Multi-Agent Collaboration

Haibin Li, Wenya Yu, Xuefeng Jiang, Jia Li+3

Beihang University · Nanyang Technological University · University of Chinese Academy of Sciences

Multi-Agent Communication Topology Multi-Agent LLMs Multi-LLM Collaboration LLM Agent Efficiency Cooperative MARL

Open paper AskPDF arXiv

Multi-agent systems driven by large language models (LLMs) rely on structured communication topologies to facilitate information exchange and collaborative reasoning. However, effectively scheduling these topologies to dynamically determine who talks to whom and when to communicate remains a fundamental challenge in balancing collective reasoning performance against exponential token overhead. Existing topology scheduling approaches either enforce static communication graphs that cannot adapt to dynamic problem-solving states, or overlook agent role complementarity while failing to explicitly incorporate the synthesized graph structure into team-level decision evaluation. To tackle these limitations, we propose ACTS, a role-aware adaptive communication topology scheduling framework for efficient LLM multi-agent collaboration. Orchestrated within a Centralized Training with Decentralized Execution (CTDE) paradigm, ACTS introduces a novel dual-attention mechanism. Specifically, a Role-conditioned Relational Attention module dynamically selects variable-cardinality interaction partners based on agent roles and evolving execution states, while a Topological Attention Mixer explicitly embeds the directed communication topology into team value decomposition for precise credit assignment. Comprehensive evaluations across 9 challenging benchmarks covering code generation, multi-step reasoning, general knowledge, and competitive mathematics demonstrate that ACTS consistently outperforms state-of-the-art baselines, boosting average accuracy by up to 6.19 percentage points while reducing token consumption by over 15%. The project website is available at https://acts-mas.github.io/ACTS/.

Abstract

Multi-Agent Communication Topology

Explore similar work

Learning Who Talks to Whom: Role-Aware Communication Topology Scheduling for Efficient LLM Multi-Agent Collaboration

HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systems

RUMAD: Reinforcement-Unifying Multi-Agent Debate