Adaptive Robust Estimator for Multi-Agent Reinforcement Learning

Zhongyi Li; Wan Tian; Jingyu Chen; Kangyao Huang; Huiming Zhang; Hui Yang; Tao Ren; Jinyang Jiang

cs.AIMar 23, 2026

Adaptive Robust Estimator for Multi-Agent Reinforcement Learning

Authors: Zhongyi Li, Wan Tian, Jingyu Chen, Kangyao Huang, Huiming Zhang, Hui Yang, Tao Ren, Jinyang Jiang,

Organizations: Beihang University · Peking University · Chinese Academy of Sciences · Tsinghua University

Abstract

Multi-agent collaboration has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models, yet it suffers from interaction-level ambiguity that blurs generation, critique, and revision, making credit assignment across agents difficult. Moreover, policy optimization in this setting is vulnerable to heavy-tailed and noisy rewards, which can bias advantage estimation and trigger unstable or even divergent training. To address both issues, we propose a robust multi-agent reinforcement learning framework for collaborative reasoning, consisting of two components: Dual-Agent Answer-Critique-Rewrite (DACR) and an Adaptive Robust Estimator (ARE). DACR decomposes reasoning into a structured three-stage pipeline: answer, critique, and rewrite, while enabling explicit attribution of each agent's marginal contribution to its partner's performance. ARE provides robust estimation of batch experience means during multi-agent policy optimization. Across mathematical reasoning and embodied intelligence benchmarks, even under noisy rewards, our method consistently outperforms the baseline in both homogeneous and heterogeneous settings. These results indicate stronger robustness to reward noise and more stable training dynamics, effectively preventing optimization failures caused by noisy reward signals.

Figures & tables

Figure 1: The scoring results of the reward model. We randomly selected questions and answers from the MATH500 and Gaokao2023 datasets, and then used the large model to generate the answers. The first and third figures show the scoring results of the reward model for the answers generated by the large model. The second and fourth figures are the corresponding moment generating functions. It can be seen that the data distribution of the scoring by the reward model is biased.

Figure 2: The proposed DACR interaction framework. Two agents, $\pi_{1}$ and $\pi_{2}$ , act as both solvers and peer-evaluators. The reward mechanism attributes credit based on standalone accuracy and the marginal improvement induced in the partner’s revised solution.

Figure 3: The probability distribution and its negative log-likelihood corresponding to the adaptive loss function. The left panel shows the negative log-likelihood function

-\log p(\epsilon;\alpha,1)=\rho(\epsilon;\alpha,1)+\log Z(\alpha)

for different shape parameters

\alpha=0,0.5,1,2,4,\infty

; The right panel shows the corresponding probability density function

p(\epsilon;\alpha,1)=\exp\{-\rho(\epsilon;\alpha,1)\}/Z(\alpha)

Figure 4: Comparison of the effectiveness of the three-stage interaction model,The grey-filled area represents the experimental results without the three-stage interaction, while the orange part shows the results obtained with the three-stage interaction. It can be seen that the three-stage interactive solution is effective.

Figure 5: The performance of the proposed method under different noise conditions is illustrated: the left figure shows contamination by noise groups of varying proportions, while the right figure depicts contamination ratios within the same group. It can be observed that the proposed method demonstrates strong noise resistance.

Figure 6: Training reward curves of ARE and FlightGPT.

Model	Method	MATH500	AIME24	AIME25	AMC23	Gaokao 2023en	Minerva Math
Qwen2.5 (1.5B-Instruct)	Base Model	35.80	-	-	30.00	23.50	5.150
	baseline-GroupA	56.00	-	-	22.50	45.50	13.20
	baseline-GroupC	55.00	3.300	-	22.50	45.50	13.60
	ARE-GroupA	56.60	3.300	-	32.50	46.00	14.00
	ARE-GroupC	55.60	3.300	3.300	27.50	45.50	15.40
Llama3.1 (8B-Instruct)	Base Model	32.60	-	-	5.000	3.390	7.350

Table 1: Comparison of accuracy rates of various methods on the mathematical reasoning dataset

Figure 7: Visualization of the navigation task. The UAV trajectory, predicted target, and ground-truth target are annotated.

Split	Method	NE $\downarrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$
VS	Base	193.92	0.3300	1.0000	0.0000
	FlightGPT	148.19	2.6700	13.670	0.0200
	ARE	139.49	3.0000	14.330	0.0300
VUS	Base	190.84	1.3300	1.3300	0.0100
	FlightGPT	145.56	3.6700	18.000	0.0300
	ARE	124.38	6.3300	18.330	0.0600

Table 2: Aerial VLN results (higher is better for SR/OSR/SPL; lower is better for NE).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$n$	Metric	l2	Cauchy	GM	Adapt	AMB	GNC_Adapt	GNC_AMB	GNC_TLS	ARE
200	MAE	0.057287	0.074298	0.082726	0.089879	0.086309	0.089576	0.086304	0.067855	0.093291
	MSE	0.005173	0.008747	0.010770	0.012688	0.011817	0.012600	0.011710	0.007302	0.013051
	Median	0.047519	0.062271	0.068645	0.074465	0.070920	0.074508	0.072363	0.056955	0.080019
	Q90	0.117608	0.152255	0.173105	0.185062	0.181983	0.185578	0.178264	0.140570	0.187690
	Q95	0.137931	0.199400	0.216909	0.229500	0.224137	0.225030	0.220215	0.163779	0.224390
500	MAE	0.034494	0.044633	0.049594	0.053918	0.052523	0.053909	0.052433	0.041655	0.063132

Appendix

Table 3: Results on DGP1 (Gaussian): robust error summaries over

R=100

replications.

$n$	Metric	l2	Cauchy	GM	Adapt	AMB	GNC_Adapt	GNC_AMB	GNC_TLS	ARE
200	MAE	0.440732	2.383597	2.428866	2.437117	2.441007	2.457052	2.348575	2.228842	2.442037
	MSE	0.319861	5.684898	5.905568	5.977074	5.964692	6.039979	5.522237	4.972983	5.968493
	Median	0.374759	2.385970	2.433266	2.458548	2.446208	2.460386	2.348119	2.225001	2.446614
	Q90	0.835407	2.453506	2.501665	2.523363	2.513925	2.521825	2.452893	2.326485	2.530520
	Q95	1.092987	2.474307	2.514922	2.537888	2.530098	2.537393	2.473458	2.343185	2.551792
500	MAE	0.284348	2.384501	2.431850	2.456983	2.446077	2.457022	2.342375	2.230460	2.456446

Appendix

Table 4: Results on DGP2 (Centered LogNormal): robust error summaries over

R=100

replications.

$n$	Metric	l2	Cauchy	GM	Adapt	AMB	GNC_Adapt	GNC_AMB	GNC_TLS	ARE
200	MAE	0.079329	0.077027	0.085139	0.091804	0.089159	0.091806	0.088340	0.075760	0.106217
	MSE	0.010140	0.009076	0.011105	0.013011	0.012191	0.013010	0.011964	0.009004	0.017967
	Median	0.065749	0.066426	0.074241	0.079843	0.077080	0.079817	0.076304	0.065225	0.086817
	Q90	0.165889	0.156022	0.175618	0.185670	0.181116	0.185723	0.181239	0.155254	0.216114
	Q95	0.197331	0.181987	0.201629	0.219082	0.212402	0.219154	0.211227	0.181042	0.265095
500	MAE	0.053192	0.050686	0.055188	0.059119	0.057425	0.059175	0.057249	0.050691	0.070144

Appendix

Table 5: Results on DGP3 (Student-

t

): robust error summaries over

R=100

replications.

$n$	Metric	l2	Cauchy	GM	Adapt	AMB	GNC_Adapt	GNC_AMB	GNC_TLS	ARE
200	MAE	0.412362	1.438328	1.518497	1.522176	1.517095	1.500790	1.564941	1.595852	1.326587
	MSE	0.590463	2.072604	2.413169	2.459166	2.413946	2.255401	2.458244	2.645237	1.767638
	Median	0.253832	1.436787	1.502406	1.498419	1.502898	1.501094	1.564170	1.650789	1.325492
	Q90	0.848063	1.514769	1.572911	1.565906	1.574786	1.566497	1.664394	1.763932	1.438845
	Q95	1.155875	1.537037	1.602260	1.596185	1.600452	1.599216	1.694309	1.795882	1.461990
500	MAE	0.346482	1.439051	1.505764	1.513167	1.516621	1.497524	1.574814	1.638352	1.339161

Appendix

Table 6: Results on DGP4 (Symmetric Pareto): robust error summaries over

R=100

replications.

$n$	Metric	l2	Cauchy	GM	Adapt	AMB	GNC_Adapt	GNC_AMB	GNC_TLS	ARE
200	MAE	1.275762	0.076195	0.085042	0.092851	0.088960	0.092902	0.082093	0.069203	0.101623
	MSE	2.589767	0.009235	0.011385	0.013490	0.012538	0.013503	0.010758	0.007540	0.016209
	Median	1.034393	0.063265	0.071296	0.077451	0.074361	0.077476	0.064493	0.058816	0.085344
	Q90	2.562352	0.154705	0.171824	0.187716	0.182085	0.187708	0.170956	0.143705	0.219979
	Q95	3.040650	0.198719	0.218241	0.232770	0.224837	0.232767	0.207915	0.179176	0.251584
500	MAE	0.803141	0.045911	0.051108	0.055784	0.054351	0.055812	0.051871	0.043391	0.065892

Appendix

Table 7: Results on DGP5 (Random contamination): robust error summaries over

R=100

replications.

$n$	Metric	l2	Cauchy	GM	Adapt	AMB	GNC_Adapt	GNC_AMB	GNC_TLS	ARE
200	MAE	4.998228	0.076277	0.085139	0.093145	0.089440	0.093207	0.072658	0.069398	0.103908
	MSE	24.987188	0.009264	0.011434	0.013583	0.012628	0.013600	0.008495	0.007742	0.016552
	Median	4.998399	0.063285	0.072447	0.080556	0.077372	0.080661	0.059623	0.057831	0.091392
	Q90	5.084834	0.158220	0.175776	0.190045	0.185091	0.190049	0.154349	0.143999	0.212980
	Q95	5.108672	0.194569	0.210418	0.232594	0.216446	0.232717	0.181021	0.173504	0.255916
500	MAE	4.999549	0.045390	0.050404	0.054917	0.053448	0.054955	0.044563	0.042151	0.067477

Appendix

Table 8: Results on DGP5 (Adversarial contamination): robust error summaries over

R=100

replications.

Figure 8: Adaptive robust loss

\rho(\epsilon;\alpha,c)

and its gradient under different parameter settings. From left to right and top to bottom, the four panels show: ( top-left ) the loss

\rho(\epsilon;\alpha,c)

for varying shape

\alpha

with fixed

c=1

; ( top-right ) the corresponding gradient

\partial\rho/\partial\epsilon

for varying

\alpha

with fixed

c=1

; ( bottom-left ) the loss for varying scale

c

with fixed

\alpha=0

; ( bottom-right ) the corresponding gradient for varying

c

with fixed

\alpha=0

Figure 9: Three instantiations of the interpolation function

f(\beta,\alpha^{(1)})

: (a) polynomial decay

f_{1}=2-\frac{2-\alpha^{(1)}}{\beta^{p}}

with

\beta:\infty\to 1

; (b) exponential transition

f_{2}=\alpha^{(1)}e^{-1/\beta}+2(1-e^{-1/\beta})

with

\beta:0^{+}\to\infty

; (c) rational form

f_{3}=\frac{2+\alpha^{(1)}\beta^{q}}{1+\beta^{q}}

with

\beta:0^{+}\to\infty

, all smoothly interpolating between the quadratic regime (

f=2

) and the adaptive regime (

f=\alpha^{(1)}

) for different values of

\alpha^{(1)}\in\{-2,0,1\}

Hyperparameter	Value	Description
Learning rate (Agent 1)	$1\times 10^{-6}$	Adam optimizer
Learning rate (Agent 2)	$1\times 10^{-6}$	Adam optimizer
Sampling temperature	$0.5$	—
Training batch size	$32$	—
Number of episodes	$10$	—
Max prompt length	$6144$	Tokens

Appendix

Table 9: Training hyperparameters for dual-agent ILR training.

Multi-Agent Reinforcement Learning

Multi-agent reinforcement learning studies how interacting agents learn coordinated or competitive policies through shared experience and rewards.

See all 363 papers

Counterfactual Credit Policy Optimization for Multi-Agent Collaboration
Mar 23, 2026Zhongyi Li, Wan Tian, Jinju Chen +4Counterfactual Credit Assignment RL for Language Model Reasoning
Beihang University · Peking University · Beijing University of Posts and Telecommunications
Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
Aug 18, 2026Yunhao Yang, Yuexin Bian, Yunjie Tian +6RL for Language Model Reasoning Reinforcement Learning
Johns Hopkins University · UC San Diego · Independent Researcher +1
Weak-Link Optimization for Multi-Agent Reasoning and Collaboration
Apr 17, 2026Haoyu Bian, Chaoning Zhang, Jiaquan Zhang +4Multi-Agent Collaboration Inference-Time Compute Allocation
University of Electronic Science and Technology of China, Chengdu 611731, China · Beihang University, Beijing 100191, China · Xi’an University of Architecture and Technology, Xi’an 710064, China

Mar 23, 2026J/K move · Enter open · S save

Counterfactual Credit Policy Optimization for Multi-Agent Collaboration

Zhongyi Li, Wan Tian, Jinju Chen, Huiming Zhang+3

Beihang University · Peking University · Beijing University of Posts and Telecommunications

Counterfactual Credit Assignment RL for Language Model Reasoning Multi-Agent Collaboration

Open paper AskPDF arXiv

Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles, but reinforcement learning for such systems is limited by credit assignment: shared terminal rewards obscure individual contributions and can encourage free-riding. We introduce two optimizer-agnostic credit assignment methods for converting joint outcomes into agent-specific learning signals. Counterfactual Credit for Policy Optimization (CCPO) estimates an agent's marginal contribution by comparing the realized joint outcome with a counterfactual outcome where that agent is removed. Self-Evaluated Credit for Policy Optimization (SEPO) uses constrained self- and peer-evaluations as a verifier-anchored credit signal while keeping the external task outcome dominant. Both operate at the reward-construction layer rather than as policy optimizers, producing role-specific rewards or advantages for GRPO, GSPO, or REINFORCE++. We instantiate these credit signals in a sequential Think--Solve setting and evaluate them on mathematical reasoning benchmarks. Results show that explicit credit assignment often improves dual-agent reasoning, especially on MATH500 and several out-of-distribution settings, while gains vary across models and datasets. Our code is available at: https://github.com/bhai114/ccpo.

Abstract

Multi-Agent Reinforcement Learning

Explore similar work

Counterfactual Credit Policy Optimization for Multi-Agent Collaboration

Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL

Weak-Link Optimization for Multi-Agent Reasoning and Collaboration