Many emerging agentic paradigms require agents to collaborate with one another (or people) to achieve shared goals. Unfortunately, existing approaches to learning policies for such collaborative problems produce brittle solutions that fail when paired with new partners. We attribute these failures to a combination of free-riding during training and a lack of strategic robustness. To address these problems, we study the concept of strategic risk aversion and interpret it as a principled inductive bias for generalizable cooperation with unseen partners. While strategically risk-averse players are robust to deviations in their partner's behavior by design, we show that, in collaborative games, they also (1) can have better equilibrium outcomes than those at classical game-theoretic concepts like Nash, and (2) exhibit less or no free-riding. Inspired by these insights, we develop a multi-agent reinforcement learning (MARL) algorithm that integrates strategic risk aversion into standard policy optimization methods. Our empirical results across collaborative benchmarks (including an LLM collaboration task) validate our theory and demonstrate that our approach consistently achieves reliable collaboration with heterogeneous and previously unseen partners across collaborative tasks.
Figures & tables
Figure 1: (i). Expected utility of each player as a function of the degree of risk aversion τ at equilibrium, when ϵ=1 in Section 4.1 . Player’s utilities can first increase with risk aversion before decreasing due to over-conservatism, meaning that strategic risk aversion can yield better performing equilibria than Nash or QRE. (ii). Probability that a player collaborates at a RQE as a function of the level of risk aversion, for ϵ=0.2 for the game in Example 4.2 . Strategic risk aversion alleviates free riding entirely after a given threshold (i.e., δ→0 ) as our theory predicts.
Figure 2: Cross-play and ablation experiments in the overcooked environment. Each square represents the average reward across 10 episodes of length 128 for each pair of agents. Diagonal blocks represent the training performance of the agents. (i) We directly observe that IPPO ( ϵ=0.1 ) learns to free-ride while SRPO ( τ=10,ϵ=0.1) does not. Furthermore, mirroring Section 4.1 , we observe that SRPO yields higher utility strategies (i.e., risk improves performance). (ii) Results of an ablation experiment, varying τ while holding ϵ=0.1 . We empirically observe that free-riding completely disappears as risk aversion increases, mirroring the result in Section 4.2 . (iii) Difference between Training Performance (TP) and Cross-play Performance (CP) (mean and standard deviation): the performance of IPPO drastically decreases, with lower average and larger standard deviation in cross-play, while the performance of SRPO is unaffected.
Figure 3: Cross-play performances of SRPO ( τ=10,ϵ=0.01 ) and IPPO ( ϵ=0.01 ) agents in the Tag environment against a runner seen during training (i) and an unseen runner (ii). Each square represents the average reward of two agents across 100 runs of length 100. IPPO does well in training environments (yet still clearly learns free-riding like policies), but their performance degrades drastically against an unseen runner. SRPO has slightly lower training performance but clearly learns a more generalizable policy. (iii) Difference between Training Performance (TP) and Cross-play Performance (CP) (mean and standard deviation): the performance of IPPO drastically decreases, with lower average and larger standard deviation in cross-play, while the performance of SRPO is almost unaffected.
Figure 4: Cross-play performance of SRPO and IPPO agents in the Hanabi environment. We use policy sharing to validate the scalability of SRPO. During evaluation, we let agents 1 and 2 share a policy and agents 3 and 4 share a policy, enabling pairwise cross-play evaluation. In Fig. 4(a) , each square represents the average reward of the two agent groups across 100 runs, each of length 100. Fig. 4(b) shows the differences between training performance (TP) and cross-play performance (CP) (mean and standard deviation) for both IPPO and SRPO. SRPO remains more robust when paired with an unseen partner. Here, we set the entropy coefficient to be ϵ=0.001 for both IPPO and SRPO, and τ=0.01 for SRPO.
Method
Q0.5B&Q0.6B
Q0.5B&Q3B
Q0.5B&Q4B
Q0.6B&Q3B
Q0.6B&Q4B
Q3B&Q4B
SRPO
0.622
0.686
0.873
0.616
0.848
0.732
IPPO
0.605
0.651
0.846
0.573
0.711
0.709
Improvement (%)
+2.81%
+5.38%
+3.19%
+7.50%
+19.27%
+3.24%
Table 1: Cross-play performance (joint accuracy) across model scale combinations. Percentage improvement is computed relative to IPPO. Across all model combinations, SRPO consistently outperforms IPPO in terms of cross-play performance.
Method
Q0.5B
Q0.6B
Q3B
Q4B
SRPO
0.405
0.671
0.632
0.917
IPPO
0.378
0.587
0.552
0.901
Improvement (%)
+7.14%
+14.31%
+14.49%
+1.78%
Table 2: Performance (the accuracy of the trained agent itself) when paired with an untuned Llama 3.2-1B-Instruct model. Percentage improvement is computed relative to IPPO. Across all model sizes, SRPO agents consistently outperform IPPO, indicating that SRPO endows the trained the power to be robust to the teammate.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7
Figure 7: Cross-play performances of SRPO ( ϵ=0.001,τ=0.01 ) and IPPO ( ϵ=0.001 ) in both 2 -player and 4 -player hanabi games. Each square represents the average reward of two agents across 50 runs of length 100.
Figure 8: Performance change (mean and standard deviation) between Training Performance (TP) and Cross-play Performance (CP): the performance of IPPO drastically decreases, with lower average and larger standard deviation in cross-play, while the performance of SRPO drops less severely.
Figure 9: The training curve for SRPO and IPPO with Qwen2.5-0.5B-Instruct. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward.
Figure 10: The training curve for SRPO and IPPO with Qwen3-0.6B. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward.
Figure 11: The training curve for SRPO and IPPO with Qwen2.5-3B-Instruct. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward.
Figure 12: The training curve for SRPO and IPPO with Qwen3-4B-Instruct-2507. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward.
Figure 13: Ablation study on the entropy parameter ϵ .
Figure 14: Ablation study on the degree of risk aversion parameter τ ; τ=0 refers to the risk-neutral case, i.e., IPPO.
We study collaborative learning in multi-agent Bayesian bandit problems, where strategic agents collectively solve the same bandit instance. While multiple agents can accelerate learning by sharing information, strategic agents might prefer to free-ride and avoid exploration. We consider a setting with persistent agents that participate in multiple time periods. This is in contrast to most previous works on incentives in multi-agent MAB, which assume short-lived agents, namely each agent has a single decision to make and optimizes their expected reward in that single decision. As in the multi-agent MAB model with incentives, our model does not have monetary transfers, and the only incentives are through information sharing. We propose CAOS, a mechanism that sustains collaboration as a Nash equilibrium while achieving strong regret guarantees. Our results demonstrate that collaborative exploration can be sustained purely through information sharing, achieving performance close to that of fully cooperative systems despite strategic behavior.
Cooperative equilibria are fragile. When agents learn alongside each other rather than in a fixed environment, the process of learning destabilizes the cooperation they are trying to sustain: every gradient step an agent takes shifts the distribution of actions its partner will play, turning a cooperative partner into a source of stochastic noise precisely where the cooperation decision is most sensitive. We study how this co-learning noise propagates through the structure of coordination games, and find that the cooperative equilibrium, even when strongly Pareto-dominant, is exponentially unstable under standard risk-neutral learning, collapsing irreversibly once partner noise crosses the game's critical cooperation threshold. The natural response to apply distributional robustness to hedge against partner uncertainty makes things strictly worse: risk-averse return objectives penalize the high-variance cooperative action relative to defection, widening the instability region rather than shrinking it, a paradox that reveals a fundamental mismatch between the domains where robustness is applied and instability originates. We resolve this by showing that robustness should target the policy gradient update variance induced by partner uncertainty, not the return distribution. This distinction yields an algorithm whose gradient updates are modulated by an online measure of partner unpredictability, provably expanding the cooperation basin in symmetric coordination games. To unify stability, sample complexity, and welfare consequences of this approach, we introduce the Price of Paranoia as the structural dual of the Price of Anarchy. Together with a novel Cooperation Window, it precisely characterizes how much welfare learning algorithms can recover under partner noise, pinning down the optimal degree of robustness as a closed-form balance between equilibrium stability and sample efficiency.
Deep Kumar Ganguly, Chandradithya S Jonnalagadda, Pratham Chintamani +1
Technical University of Munich (TU Munich) · Brown University · Indian Institute of Technology Tirupati (IIT Tirupati)
Multi-agent collaboration has emerged as a powerful paradigm for enhancing the reasoning capabilities of large language models, yet it suffers from interaction-level ambiguity that blurs generation, critique, and revision, making credit assignment across agents difficult. Moreover, policy optimization in this setting is vulnerable to heavy-tailed and noisy rewards, which can bias advantage estimation and trigger unstable or even divergent training. To address both issues, we propose a robust multi-agent reinforcement learning framework for collaborative reasoning, consisting of two components: Dual-Agent Answer-Critique-Rewrite (DACR) and an Adaptive Robust Estimator (ARE). DACR decomposes reasoning into a structured three-stage pipeline: answer, critique, and rewrite, while enabling explicit attribution of each agent's marginal contribution to its partner's performance. ARE provides robust estimation of batch experience means during multi-agent policy optimization. Across mathematical reasoning and embodied intelligence benchmarks, even under noisy rewards, our method consistently outperforms the baseline in both homogeneous and heterogeneous settings. These results indicate stronger robustness to reward noise and more stable training dynamics, effectively preventing optimization failures caused by noisy reward signals.
Zhongyi Li, Wan Tian, Jingyu Chen +8
Beihang University · Peking University · Chinese Academy of Sciences +1