Training Generalizable Collaborative Agents via Strategic Risk Aversion

Chengrui Qu; Yizhou Zhang; Nicolas Lanzetti; Eric Mazumdar

cs.LGFeb 25, 2026

Training Generalizable Collaborative Agents via Strategic Risk Aversion

Authors: Chengrui Qu, Yizhou Zhang, Nicolas Lanzetti, Eric Mazumdar

Organizations: Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA.

Abstract

Many emerging agentic paradigms require agents to collaborate with one another (or people) to achieve shared goals. Unfortunately, existing approaches to learning policies for such collaborative problems produce brittle solutions that fail when paired with new partners. We attribute these failures to a combination of free-riding during training and a lack of strategic robustness. To address these problems, we study the concept of strategic risk aversion and interpret it as a principled inductive bias for generalizable cooperation with unseen partners. While strategically risk-averse players are robust to deviations in their partner's behavior by design, we show that, in collaborative games, they also (1) can have better equilibrium outcomes than those at classical game-theoretic concepts like Nash, and (2) exhibit less or no free-riding. Inspired by these insights, we develop a multi-agent reinforcement learning (MARL) algorithm that integrates strategic risk aversion into standard policy optimization methods. Our empirical results across collaborative benchmarks (including an LLM collaboration task) validate our theory and demonstrate that our approach consistently achieves reliable collaboration with heterogeneous and previously unseen partners across collaborative tasks.

Figures & tables

Figure 1: (i). Expected utility of each player as a function of the degree of risk aversion $\tau$ at equilibrium, when $\epsilon=1$ in Section 4.1 . Player’s utilities can first increase with risk aversion before decreasing due to over-conservatism, meaning that strategic risk aversion can yield better performing equilibria than Nash or QRE. (ii). Probability that a player collaborates at a RQE as a function of the level of risk aversion, for $\epsilon=0.2$ for the game in Example 4.2 . Strategic risk aversion alleviates free riding entirely after a given threshold (i.e., $\delta\rightarrow 0$ ) as our theory predicts.

Figure 2: Cross-play and ablation experiments in the overcooked environment. Each square represents the average reward across 10 episodes of length 128 for each pair of agents. Diagonal blocks represent the training performance of the agents. (i) We directly observe that IPPO ( $\epsilon=0.1$ ) learns to free-ride while SRPO ( $\tau=10,\epsilon=0.1)$ does not. Furthermore, mirroring Section 4.1 , we observe that SRPO yields higher utility strategies (i.e., risk improves performance). (ii) Results of an ablation experiment, varying $\tau$ while holding $\epsilon=0.1$ . We empirically observe that free-riding completely disappears as risk aversion increases, mirroring the result in Section 4.2 . (iii) Difference between Training Performance (TP) and Cross-play Performance (CP) (mean and standard deviation): the performance of IPPO drastically decreases, with lower average and larger standard deviation in cross-play, while the performance of SRPO is unaffected.

Figure 3: Cross-play performances of SRPO ( $\tau=10,\epsilon=0.01$ ) and IPPO ( $\epsilon=0.01$ ) agents in the Tag environment against a runner seen during training (i) and an unseen runner (ii). Each square represents the average reward of two agents across 100 runs of length 100. IPPO does well in training environments (yet still clearly learns free-riding like policies), but their performance degrades drastically against an unseen runner. SRPO has slightly lower training performance but clearly learns a more generalizable policy. (iii) Difference between Training Performance (TP) and Cross-play Performance (CP) (mean and standard deviation): the performance of IPPO drastically decreases, with lower average and larger standard deviation in cross-play, while the performance of SRPO is almost unaffected.

Panel of Figure 4 — Figure 4: Cross-play performance of SRPO and IPPO agents in the Hanabi environment. We use policy sharing to validate the scalability of SRPO. During evaluation, we let agents 1 and 2 share a policy and agents 3 and 4 share a policy, enabling pairwise cross-play evaluation. In Fig. 4(a) , each square represents the average reward of the two agent groups across 100 runs, each of length 100. Fig. 4(b) shows the differences between training performance (TP) and cross-play performance (CP) (mean and standard deviation) for both IPPO and SRPO. SRPO remains more robust when paired with an unseen partner. Here, we set the entropy coefficient to be $\epsilon=0.001$ for both IPPO and SRPO, and $\tau=0.01$ for SRPO.

Method	Q0.5B&Q0.6B	Q0.5B&Q3B	Q0.5B&Q4B	Q0.6B&Q3B	Q0.6B&Q4B	Q3B&Q4B
SRPO	0.622	0.686	0.873	0.616	0.848	0.732
IPPO	0.605	0.651	0.846	0.573	0.711	0.709
Improvement (%)	+2.81%	+5.38%	+3.19%	+7.50%	+19.27%	+3.24%

Table 1: Cross-play performance (joint accuracy) across model scale combinations. Percentage improvement is computed relative to IPPO. Across all model combinations, SRPO consistently outperforms IPPO in terms of cross-play performance.

Method	Q0.5B	Q0.6B	Q3B	Q4B
SRPO	0.405	0.671	0.632	0.917
IPPO	0.378	0.587	0.552	0.901
Improvement (%)	+7.14%	+14.31%	+14.49%	+1.78%

Table 2: Performance (the accuracy of the trained agent itself) when paired with an untuned Llama 3.2-1B-Instruct model. Percentage improvement is computed relative to IPPO. Across all model sizes, SRPO agents consistently outperform IPPO, indicating that SRPO endows the trained the power to be robust to the teammate.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Performance change (mean and standard deviation) between Training Performance (TP) and Cross-play Performance (CP): the performance of IPPO drastically decreases, with lower average and larger standard deviation in cross-play, while the performance of SRPO drops less severely.

Panel of Figure 9 — Figure 9: The training curve for SRPO and IPPO with Qwen2.5-0.5B-Instruct. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward.

Panel of Figure 10 — Figure 10: The training curve for SRPO and IPPO with Qwen3-0.6B. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward.

Panel of Figure 11 — Figure 11: The training curve for SRPO and IPPO with Qwen2.5-3B-Instruct. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward.

Panel of Figure 12 — Figure 12: The training curve for SRPO and IPPO with Qwen3-4B-Instruct-2507. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward.

Panel of Figure 13 — Figure 13: Ablation study on the entropy parameter $\epsilon$ .

Panel of Figure 14 — Figure 14: Ablation study on the degree of risk aversion parameter $\tau$ ; $\tau=0$ refers to the risk-neutral case, i.e., IPPO.

Zero-Shot Coordination

See all 11 papers

Collaborating in Multi-Armed Bandits with Strategic Agents
May 13, 2026Idan Barnea, Ofir Schlisselberg, Yishay MansourMulti-Armed Bandits Mechanism Design
Tel Aviv University · Google Research
The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning
Apr 17, 2026Deep Kumar Ganguly, Chandradithya S Jonnalagadda, Pratham Chintamani +1Multi-Agent Reinforcement Learning Multi-Agent Coordination
Technical University of Munich (TU Munich) · Brown University · Indian Institute of Technology Tirupati (IIT Tirupati)
Adaptive Robust Estimator for Multi-Agent Reinforcement Learning
Mar 23, 2026Zhongyi Li, Wan Tian, Jingyu Chen +8Multi-Agent Reinforcement Learning RL for Language Model Reasoning
Beihang University · Peking University · Chinese Academy of Sciences +1

May 13, 2026J/K move · Enter open · S save

Collaborating in Multi-Armed Bandits with Strategic Agents

Idan Barnea, Ofir Schlisselberg, Yishay Mansour

Tel Aviv University · Google Research

Multi-Armed Bandits Mechanism Design Multi-Agent Bandits Game Theory Multi-Agent Collaboration

Open paper AskPDF arXiv

We study collaborative learning in multi-agent Bayesian bandit problems, where strategic agents collectively solve the same bandit instance. While multiple agents can accelerate learning by sharing information, strategic agents might prefer to free-ride and avoid exploration. We consider a setting with persistent agents that participate in multiple time periods. This is in contrast to most previous works on incentives in multi-agent MAB, which assume short-lived agents, namely each agent has a single decision to make and optimizes their expected reward in that single decision. As in the multi-agent MAB model with incentives, our model does not have monetary transfers, and the only incentives are through information sharing. We propose CAOS, a mechanism that sustains collaboration as a Nash equilibrium while achieving strong regret guarantees. Our results demonstrate that collaborative exploration can be sustained purely through information sharing, achieving performance close to that of fully cooperative systems despite strategic behavior.

Abstract

Zero-Shot Coordination

Explore similar work

Collaborating in Multi-Armed Bandits with Strategic Agents

The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learning

Adaptive Robust Estimator for Multi-Agent Reinforcement Learning