G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policies

Guowei Zou; Haitao Wang; Guoxin Wang; Zhiquan Chen; Beiwen Zhang; Guojie Wang; Hejun Wu

cs.AISep 25, 2026

G2MAF: Test-Time Gradient Guidance for Multi-Agent Flow Policies

Authors: Guowei Zou, Haitao Wang, Guoxin Wang, Zhiquan Chen, Beiwen Zhang, Guojie Wang, Hejun Wu

Organizations: Sun Yat-sen University

Abstract

Offline multi-agent reinforcement learning (MARL) learns cooperative policies from fixed datasets without further environment interaction and a learned policy is frozen at deployment. Such a frozen policy typically proposes a single joint action and executes it directly at deployment time. However, this one-shot deployment often commits to a suboptimal proposal, even when better nearby alternatives remain consistent with the behavior data. To address this issue, we propose Gradient Guided Multi Agent Flow (G2MAF), a refinement framework for optimizing joint policies at test-time. G2MAF applies one globally normalized, projected critic gradient to guide and coordinate all agents' corrections while keeping the action both feasible and close to the frozen policy proposal. Across 24 MPE and SMAC settings, its canonical variant improves 20 frozen settings, with mean relative gains of 9.2% on MPE and 8.9% on SMAC, with model inference latency increased by about 6% only.

Figures & tables

Figure 1: Framework of G 2 MAF. Given the joint observation and target return, a frozen generative policy proposes $\bm{a}_{\theta}$ . The centralized critic evaluates the complete joint action and supplies one gradient whose blocks for individual agents depend on all teammates’ current actions. G 2 MAF globally normalizes this gradient, projects the refined action $\tilde{\bm{a}}$ onto the feasible action set, and executes it. The method uses neither policy retraining nor rollouts; the SMAC variant applies the same update to logits with illegal actions masked out.

Task	Qual.	Data	BC	ICQ	TD3+BC	CQL	OMAR	Diff	MA-SfBC	DOM2	G 2 MAF
Spread	Expert	96.2	35.0 $\pm$ 2.6	104.0 $\pm$ 3.4	108.3 $\pm$ 3.9	98.2 $\pm$ 5.2	114.9 $\pm$ 2.6	95.0 $\pm$ 5.3	87.5 $\pm$ 7.3	88.7 $\pm$ 6.3	115.2 $\pm$ 0.9
	MedRep	9.3	10.0 $\pm$ 3.8	13.6 $\pm$ 5.7	15.4 $\pm$ 5.6	31.4 $\pm$ 7.2	37.9 $\pm$ 6.1	30.3 $\pm$ 2.5	8.2 $\pm$ 4.6	63.1 $\pm$ 9.5	47.3 $\pm$ 2.2
	Medium	27.0	31.6 $\pm$ 4.8	29.3 $\pm$ 5.5	39.4 $\pm$ 3.6	34.1 $\pm$ 7.2	47.9 $\pm$ 18.9	64.9 $\pm$ 7.7	51.6 $\pm$ 14.2	78.6 $\pm$ 8.1	59.7 $\pm$ 1.7
	Random	0.1	-0.5 $\pm$ 3.2	6.3 $\pm$ 3.5	9.8 $\pm$ 4.9	24.0 $\pm$ 9.8	34.4 $\pm$ 5.3	6.9 $\pm$ 3.1	5.1 $\pm$ 3.9	37.4 $\pm$ 11.3	58.0 $\pm$ 9.0
Tag	Expert	89.5	40.0 $\pm$ 9.6	113.0 $\pm$ 14.4	115.2 $\pm$ 12.8	119.3 $\pm$ 14.0	123.9 $\pm$ 10.5	103.0 $\pm$ 12.0	77.4 $\pm$ 13.9	98.2 $\pm$ 14.4	139.7 $\pm$ 1.1
	MedRep	8.6	0.9 $\pm$ 1.4	34.5 $\pm$ 27.8	28.7 $\pm$ 20.9	41.7 $\pm$ 15.3	47.1 $\pm$ 15.3	53.9 $\pm$ 11.4	12.7 $\pm$ 7.3	68.2 $\pm$ 16.7	73.5 $\pm$ 7.5

Table 1: Main results on (a) MPE and (b) SMAC. “Data” is the offline data mean, “Diff” abbreviates MADiff, and G 2 MAF denotes the refined frozen backbone at its selected operating point. Bold and underline mark the best and second best method in each row, excluding Data. Appendix A.1 gives the reporting conventions.

Figure 2: G 2 MAF gain for each setting over G 2 MAF without refinement at test time, evaluated at Table 1 operating points. Colors denote higher, lower, or tied relative gain

\delta_{\mathrm{rel}}

. (a) MPE OMAR normalized score; (b) SMAC shaped episodic return. Labels give

\delta_{\mathrm{rel}}

; error bars show the reported uncertainty for G 2 MAF without refinement and

\pm 1

standard error for G 2 MAF.

Figure 4: Sensitivity to the step size across action spaces. Panel (a) shows the unified grid for continuous actions in OMAR normalized MPE score; panel (b) shows the SMAC grid for masked logits in shaped episodic return. Each continuous marker uses one fixed

K{=}5

protocol. Curves report gain over their matched Base evaluation, axes are specific to each task, and rings mark the best discrete step. Appendix A.3 gives the full protocol.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Map	Quality	Frozen %	G 2 MAF %	$\Delta_{\rm win}$	Map	Quality	Frozen %	G 2 MAF %	$\Delta_{\rm win}$
3m	Good	95.6	95.6	$0.0$	5m_vs_6m	Good	74.4	78.9	+4.4
	Medium	66.7	50.0	$-16.7$		Medium	68.9	77.8	+8.9
	Poor	22.2	43.3	+21.1		Poor	13.3	25.6	+12.2
2s3z	Good	97.8	95.6	$-2.2$	8m	Good	96.7	98.9	+2.2
	Medium	65.6	76.7	+11.1		Medium	76.7	88.9	+12.2
	Poor	0.0	4.4	+4.4		Poor	0.0	0.0	$0.0$

Appendix

Table 2: SMAC success rate (

\uparrow

, our policy only): Frozen backbone vs. G 2 MAF at each method’s selected denoising-step count. Success and shaped episode return measure different outcomes; success is therefore reported separately. Here

\Delta_{\rm win}=W_{\rm\mathrm{G}^{2}\mathrm{MAF}{}}-W_{\rm Frozen}

is the percentage-point change in success rate;

\Delta_{R}

is the shaped-return gain.

Figure 5: Quality and refinement diagnostics. Panels (a) and (b) show task-level Table 1 gain across dataset qualities for MPE and SMAC, respectively; MPE uses normalized-score gain and SMAC uses shaped episodic-return gain. (c) Critic pairwise accuracy against simulator-realized returns. Trajectory rows use frozen-policy trajectories; local rows use the Frozen action and

14

random proposals with matched nominal pre-projection radius from

64

restored states per MPE setting. Points and error bars are means

\pm 1

standard error across settings; right-hand labels count settings above chance. (d) Mean task-standardized MPE gain after splitting realized headroom at zero and independent local reliability at its median; gray and green denote low and high reliability. Labels report the cell mean.

Task	Quality	Base (ms)	Grad (ms)	G 2 MAF (ms)	$\times$	Task	Quality	Base (ms)	Grad (ms)	G 2 MAF (ms)	$\times$
MPE						SMAC
Spread	Expert	142.1	3.0	145.1	1.02	3m	Good	189.9	3.4	179.5	0.94
	MedReplay	137.4	3.0	142.7	1.04		Medium	175.7	3.9	188.5	1.07
	Medium	140.5	3.2	149.1	1.06		Poor	221.2	6.8	233.2	1.05
	Random	137.3	2.9	143.9	1.05	2s3z	Good	153.8	4.3	181.9	1.18
Tag	Expert	138.9	3.1	150.7	1.08		Medium	164.2	3.7	162.4	0.99

Appendix

Table 3: Model-side inference latency in milliseconds per decision for every evaluated setting. Base is the frozen backbone; Grad is the direct G 2 MAF critic-gradient step; G 2 MAF is the full refined policy.

Task	Quality	Frozen	G 2 MAF	Best- $N$	1-agent	Factor.	Task	Quality	Frozen	G 2 MAF	Best- $N$	1-agent	Factor.
MPE							SMAC
Spread	Expert	108.59	$+0.24$	$\mathbf{+1.19}$	$+0.22$	$+0.05$	3m	Good	19.17	$+0.01$	$\mathbf{+0.03}$	$+0.01$	–
	MedReplay	43.29	$\mathbf{+1.72}$	$-2.48$	$-0.03$	$+1.62$		Medium	15.12	$-1.88$	$-1.70$	$\mathbf{+0.79}$	–
	Medium	59.35	$\mathbf{+3.50}$	$-4.74$	$+1.24$	$+2.21$		Poor	9.14	$\mathbf{+1.72}$	$+1.04$	$-0.03$	–
	Random	36.69	$\mathbf{+6.84}$	$+5.93$	$+1.59$	$+6.30$	2s3z	Good	19.54	$+0.32$	$+0.26$	$\mathbf{+0.46}$	–
Tag	Expert	113.02	$+6.46$	$+7.45$	$+5.05$	$\mathbf{+10.57}$		Medium	17.08	$\mathbf{+1.62}$	$+0.75$	$-0.06$	–

Appendix

Table 4: Full-grid mechanism controls and local-step diagnostics over all

24

settings. (a) Paired mean performance gains for the full centralized update and the structure/search controls. (b) Critic-predicted value change and realized update geometry from the same rerun.

Task	Quality	Canonical $\Delta_{R}$	LOTO $\eta$	LOTO $\Delta_{R}$	Task	Quality	Canonical $\Delta_{R}$	LOTO $\eta$	LOTO $\Delta_{R}$
MPE					SMAC
Spread	Expert	$+0.30$	0.1	$-0.76$	3m	Good	$+0.00$	1	$+0.00$
	MedReplay	$+1.40$	0.1	$+2.12$		Medium	$+1.40$	2	$-1.75$
	Medium	$+2.10$	0.1	$+4.97$		Poor	$+0.10$	2	$-1.61$
	Random	$+6.10$	0.1	$+7.62$	2s3z	Good	$+1.20$	2	$+1.22$
Tag	Expert	$+2.50$	0.1	$+4.76$		Medium	$+0.40$	2	$+0.40$

Appendix

Table 5: Full-grid non-oracle step-size transfer and perturbation controls over all

24

settings. (a) Canonical gain, the step selected from the other settings in the same domain, and the independently evaluated held-out gain. (b) Canonical Base and G 2 MAF gain together with independently measured matched nominal-radius Random and Shuffled effects. MPE values use Table 1 OMAR-normalized scale; SMAC values use shaped episodic return. Results across the two panels use different evaluation batches and are compared descriptively.

Figure 6: Qualitative benchmark trajectories. Each row shows one trajectory at $0\%$ , $25\%$ , $50\%$ , $75\%$ , and $100\%$ progress. Panels (a)–(c) cover MPE and panels (d)–(g) cover SMAC.

Cooperative MARL

Cooperative multi-agent reinforcement learning (MARL) trains agents to coordinate toward shared goals under partial information and decentralized control.

See all 234 papers

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning
Apr 9, 2026Teng Pang, Zhiqiang Dong, Yan Zhang +3Offline MARL Cooperative MARL
School of Software, Shandong University, Jinan, China · Luke EI, Jinan, China
Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies
Oct 1, 2026Zhuoran Li, Yunzhan Li, Xun Wang +2Flow Matching Multi-Agent Coordination
Institute for Interdisciplinary Information Sciences, Tsinghua University · Engineering Systems and Design Pillar, Singapore University of Technology and Design
CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making
May 2, 2026Guowei Zou, Haitao Wang, Beiwen Zhang +2Flow Matching Multi-Agent Coordination
Sun Yat-sen University

Apr 9, 2026J/K move · Enter open · S save

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning

Teng Pang, Zhiqiang Dong, Yan Zhang, Rongjian Xu+2

School of Software, Shandong University, Jinan, China · Luke EI, Jinan, China

Offline MARL Cooperative MARL Flow Matching

Open paper AskPDF arXiv

Offline multi-agent reinforcement learning (MARL) aims to learn the optimal joint policy from pre-collected datasets, requiring a trade-off between maximizing global returns and mitigating distribution shift from offline data. Recent studies use diffusion or flow generative models to capture complex joint policy behaviors among agents; however, they typically rely on multi-step iterative sampling, thereby reducing training and inference efficiency. Although further research improves sampling efficiency through methods like distillation, it remains sensitive to the behavior regularization coefficient. To address the above-mentioned issues, we propose Value Guidance Multi-agent MeanFlow Policy (VGM

^2

P), a simple yet effective flow-based policy learning framework that enables efficient action generation with coefficient-insensitive conditional behavior cloning. Specifically, VGM

^2

P uses global advantage values to guide agent collaboration, treating optimal policy learning as conditional behavior cloning. Additionally, to improve policy expressiveness and inference efficiency in multi-agent scenarios, it leverages classifier-free guidance MeanFlow for both policy training and execution. Experiments on tasks with both discrete and continuous action spaces demonstrate that, even when trained solely via conditional behavior cloning, VGM

^2

P efficiently achieves performance comparable to state-of-the-art methods.

Abstract

Cooperative MARL

Explore similar work

Value-Guidance MeanFlow for Offline Multi-Agent Reinforcement Learning

Flowing Faster to Coordinate: One-Step Online Multi-Agent Flow Policies

CoFlow: Coordinated Few-Step Flow for Offline Multi-Agent Decision Making