Test-time Multi-agent Coordination by Decomposed Value Gradient Flow

Dongsu Lee; Haoran Xu; Amy Zhang

cs.LGOct 1, 2026

Test-time Multi-agent Coordination by Decomposed Value Gradient Flow

Authors: Dongsu Lee, Haoran Xu, Amy Zhang

Organizations: The University of Texas at Austin

Abstract

Offline multi-agent reinforcement learning (MARL) faces a persistent trade-off. Expressive generative policies can represent multi-modal coordination in the data, but cannot distinguish high-value regions, while value-optimized policies exploit the learned Q-function but collapse the multi-modal into a single dominant mode. A single agent's mode collapse can break joint coordination, and simultaneous drift across agents can push the joint policy into unseen regions of the action space. We propose scalable coordination via optimal unified transport (SCOUT), the first offline MARL framework to combine a generative foundation model with a learned value function through test-time action refinement. SCOUT trains two decoupled components: a flow-matching behavioral prior and a decomposed value function. At test-time, it transports behavioral samples toward high-value regions via Stein variational gradient descent. The number of transport steps controls adaptive test-time scaling, replacing a fixed regularization coefficient. Under the individual-global-max (IGM) principle, we prove a single-term KL bound on the joint soft-value gap that vanishes as transport converges, with an irreducible additive residual proportional to the IGM violation. Empirically, SCOUT achieves the best average performance across discrete and continuous offline MARL benchmarks and yields performance improvements in all offline-to-online configurations.

Figures & tables

Figure 1: Overview of the proposed solution. ( a ) The prerequisite for SCOUT is to train the reference policy via flow matching and decomposed value via a value decomposition network (VDN). ( b ) SCOUT transports the behavior distribution at test time via Stein variational gradient descent.

Figure 2: Toy example. ( a ) Offline dataset. ( b ) SCOUT ’s action refinement (

L_{\text{test}}=5

). ( c ) MAC-Flow ’s one-step distilled policy. ( d ) Learning curve of two algorithms. We plot

10

evaluation examples.

Scenarios		Dataset	Gaussian policies			Diffusion policies		Flow policies		w/o policy
Scenarios		Dataset	BC †	MABCQ †	MACQL †	MADiff-D	DoF	Flow BC	MAC-Flow	SCOUT
SMACv1	3m	Good	16.0 $\pm$ 1.0	16.3 $\pm$ 1.5	19.6 $\pm$ 0.3	19.3 $\pm$ 0.5	19.4 $\pm$ 0.1	20.0 $\pm$ 0.0	19.8 $\pm$ 1.0	20.0 $\pm$ 0.0
		Medium	8.2 $\pm$ 0.8	18.3 $\pm$ 1.2	18.9 $\pm$ 1.2	16.4 $\pm$ 2.6	18.2 $\pm$ 0.2	14.7 $\pm$ 1.5	16.8 $\pm$ 1.3	17.8 $\pm$ 2.2
		Poor	4.4 $\pm$ 0.1	12.4 $\pm$ 2.3	5.8 $\pm$ 0.4	9.3 $\pm$ 2.1	8.9 $\pm$ 0.5	4.5 $\pm$ 0.1	6.8 $\pm$ 1.1	17.0 $\pm$ 3.6
	8m	Good	16.7 $\pm$ 0.4	12.7 $\pm$ 6.3	11.3 $\pm$ 6.1	18.9 $\pm$ 1.1	19.1 $\pm$ 0.1	19.5 $\pm$ 0.2	19.4 $\pm$ 1.3	20.0 $\pm$ 0.0
		Medium	10.7 $\pm$ 0.5	16.0 $\pm$ 1.4	16.8 $\pm$ 3.1	16.8 $\pm$ 1.6	17.5 $\pm$ 1.2	18.2 $\pm$ 0.8	19.1 $\pm$ 1.0	19.6 $\pm$ 0.4

Table 1: A performance comparison for SMACv1 is provided, covering

4

tasks, and

12

datasets. Results are averaged over

8

seeds, with two standard deviations reported after the

\pm

sign. The superscript † denotes reported scores from [ 58 ] .

Scenarios		Dataset	Offline MARL				Generative modeling
Scenarios		Dataset	MATD3BC †	MACQL †	ICQ †	OMIGA †	MADiff-D	MAC-Flow	SCOUT
MA-MuJoCo	Ant	E	93.8 $\pm$ 1.4	95.8 $\pm$ 1.0	93.9 $\pm$ 0.9	94.2 $\pm$ 0.1	94.5 $\pm$ 0.8	94.5 $\pm$ 0.9	95.8 $\pm$ 1.0
		M	42.4 $\pm$ 1.0	11.2 $\pm$ 1.1	41.8 $\pm$ 1.2	42.1 $\pm$ 0.4	43.0 $\pm$ 0.9	43.4 $\pm$ 0.7	43.6 $\pm$ 0.7
		M-R	8.0 $\pm$ 4.2	-36.8 $\pm$ 8.7	9.8 $\pm$ 4.3	17.1 $\pm$ 7.2	31.6 $\pm$ 29.1	43.7 $\pm$ 1.4	42.5 $\pm$ 4.6
		M-E	59.2 $\pm$ 6.1	73.5 $\pm$ 0.9	55.5 $\pm$ 6.8	66.0 $\pm$ 8.9	67.7 $\pm$ 12.7	93.3 $\pm$ 0.6	94.7 $\pm$ 0.3
	Hopper	E	87.8 $\pm$ 0.1	88.9 $\pm$ 13.6	18.5 $\pm$ 19.9	21.3 $\pm$ 17.3	74.6 $\pm$ 14.3	95.2 $\pm$ 0.2	90.2 $\pm$ 0.4

Table 2: Performance evaluation for MA-MuJoCo. We present a performance comparison across

3

tasks and

12

datasets. The superscript † denotes reported scores from [ 10 ] .

Figure 3: Offline-to-online RL (8 seeds). Fine-tuning starts at normalized gradient steps $0.5$ .

Figure 4: Hyperparameter sweep. The plots show how different

L

affect offline RL performance.

Appendix

Table 8

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Table 9

Configuration	$L_{\text{train}}$	$L_{\text{test}}$	Configuration	$L_{\text{train}}$	$L_{\text{test}}$
3m - Good	$1$	$1$	2s3z - Good	$1$	$0$
3m - Medium	$1$	$3$	2s3z - Medium	$2$	$1$
3m - Poor	$2$	$3$	2s3z - Poor	$3$	$10$
8m - Good	$3$	$0$	5m_vs_6m - Good	$3$	$3$
8m - Medium	$2$	$10$	5m_vs_6m - Medium	$2$	$3$
8m - Poor	$3$	$10$	5m_vs_6m - Poor	$1$	$3$

Appendix

Table 5: Hyperparameter selection of SCOUT for offline RL configuration

Figure 5: Offline-to-online RL in MA-MuJoCo (8 seeds.)

Figure 6: Hyperparameter sweep on MA-MuJoCo.

Multi-Agent Coordination

Multi-agent coordination studies how autonomous agents align actions, share information, and divide work to achieve individual or collective goals.

See all 516 papers

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies
Oct 1, 2026Zhuoran Li, Yunzhan Li, Xun Wang +2Flow Matching Multi-Agent Coordination
Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design
CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making
May 2, 2026Guowei Zou, Haitao Wang, Beiwen Zhang +2Flow Matching Multi-Agent Coordination
Sun Yat-sen University
Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning
Apr 9, 2026Teng Pang, Zhiqiang Dong, Yan Zhang +3Offline MARL Cooperative MARL
School of Software, Shandong University, Jinan, China · Luke EI, Jinan, China

Oct 1, 2026J/K move · Enter open · S save

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

Zhuoran Li, Yunzhan Li, Xun Wang, Yihan Du+1

Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design

Flow Matching Multi-Agent Coordination Cooperative MARL Multi-Agent Reinforcement Learning

Open paper AskPDF arXiv

Multi-agent reinforcement learning (MARL) provides a powerful framework for learning coordinated behaviors through interactions with the environment. Developing MARL policies requires balancing expressive modeling of complex and multimodal action distributions with efficient training and execution. Generative policies, particularly diffusionbased policies, can faithfully capture complex and multimodal behaviors, but costly iterative sampling hinders their scalability in online multi-agent settings. We propose an Online MARL framework via one-step Flow model (OMAF) that combines expressive generative policies with efficient one-step action generation. OMAF employs a Transformer-based flow policy to capture complex coordination behaviors, while its approximate path score surrogate provides a principled route to synchronized flow policy optimization. To enable stable and sampleefficient learning, we further develop a joint optimization scheme coupling softmax Q-value estimation with a joint flow policy objective for coordinated policy learning. By eliminating iterative sampling, OMAF dramatically reduces training overhead without sacrificing policy expressiveness. Extensive experiments across 10 standard tasks from MPE and MAMuJoCo show that OMAF consistently achieves superior performance, with up to 3.4x higher returns and 10.5x sample efficiency improvement compared with baseline methods. These results validate the effectiveness of OMAF as an expressive and computationally efficient one-step flow policy paradigm for online MARL.

Abstract

Multi-Agent Coordination

Explore similar work

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning