A Generalist Hanabi Agent

Arjun V Sudhakar; Hadi Nekoei; Mathieu Reymond; Miao Liu; Janarthanan Rajendran; Sarath Chandar

cs.MAMar 17, 2025

A Generalist Hanabi Agent

Authors: Arjun V Sudhakar, Hadi Nekoei, Mathieu Reymond, Miao Liu, Janarthanan Rajendran, Sarath Chandar

Organizations: Chandar Research Lab · Mila – Quebec AI Institute · Polytechnique Montréal · Université de Montréal ·

Abstract

Traditional multi-agent reinforcement learning (MARL) systems can develop cooperative strategies through repeated interactions. However, these systems are unable to perform well on any other setting than the one they have been trained on, and struggle to successfully cooperate with unfamiliar collaborators. This is particularly visible in the Hanabi benchmark, a popular 2-to-5 player cooperative card-game which requires complex reasoning and precise assistance to other agents. Current MARL agents for Hanabi can only learn one specific game-setting (e.g., 2-player games), and play with the same algorithmic agents. This is in stark contrast to humans, who can quickly adjust their strategies to work with unfamiliar partners or situations. In this paper, we introduce Recurrent Replay Relevance Distributed DQN (R3D2), a generalist agent for Hanabi, designed to overcome these limitations. We reformulate the task using text, as language has been shown to improve transfer. We then propose a distributed MARL algorithm that copes with the resulting dynamic observation- and action-space. In doing so, our agent is the first that can play all game settings concurrently, and extend strategies learned from one setting to other ones. As a consequence, our agent also demonstrates the ability to collaborate with different algorithmic agents -- agents that are themselves unable to do so. The implementation code is available at: R3D2-A-Generalist-Hanabi-Agent

Figures & tables

Figure 3: Policy Transfer - Zeroshot setting. Each subplot shows the evaluation setting for a

n

-player game. Each bar combines

0<i<n

agents trained on a different setting, with

n-i

players trained on

n

-player games. R3D2 agents demonstrate strong zero-shot generalization to novel settings. Moreover, R2D2-text seems to be unable to match R3D2’s transfer performance specially when transferring from a setting with large number of actions to smaller action space.

Figure 6: The evaluation of language models performance as an agent is conducted via gameplay score, measured across various language models with 1200 game runs.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Performance comparison across different settings. The table shows both self-play (SP) and intra-cross-play (Intra-XP) scores for different methods in 3- , 4- and 5-player Hanabi settings. While IQL and OP achieve high SP scores but fail in Intra-XP, both R3D2 variants maintain consistent performance across both metrics, with R3D2-S showing particularly strong results.

Panel of Figure 8 — Figure 8: Impact of pre-trained weights and update frequency on learning efficiency. (a) Performance difference between R3D2 agents trained with pre-trained language model (LM) weights versus randomly initialized LM weights, showing significant improvements in sample efficiency with pre-trained weights. (b) The effect of varying the frequency of LM updates, highlighting that frequent updates are critical for effective learning in the Hanabi environment.

Figure 9: Policy Transfer - Zeroshot setting. Each subplot shows the evaluation setting for a

n

-player game. Each bar combines

0<i<n

agents trained on a different setting, with

n-i

players trained on

n

-player games. Comparing R3D2 and R3D2-M’s cross-play performance with R3D2-S partners trained on different settings (Figure 9 ), we observe complementary strengths: R3D2 excels in 2-player settings while R3D2-M performs better in 5-player scenarios. Despite being trained on multiple settings simultaneously, R3D2-M achieves comparable performance to its single-task counterpart, demonstrating robust generalization capabilities.

Hyper-parameters	Value
# replay buffer related
burn_in_frames	10,000
replay_buffer_size	50,000
priority_exponent	0.9
priority_weight	0.6
max_trajectory_length	80

Appendix

Table 1: Hyper-Parameters for R3D2 agents.

Figure 10: Visualizing the number of actions available in the dataset to create a diverse dataset of Hanabi gameplay in the form of text.

Panel of Figure 11 — Figure 11: Learning graph for (a) Validation accuracy plotted against(b) Game play score, for each epoch for different language model providing insights into the observed trends during the training process.

Method	System Prompt 1	System Prompt 2
GPT-4	0.0 ± 0.0	0.0 ± 0.0
o3-mini	7.66 ± 3.13	12.0 ± 0.47
GPT-4*( % Bombed )	3.0 ± 0.0 ( 100% )	2.34 ± 0.27( 100% )
o3-mini*( % Bombed )	9.0 ± 2.05( 33.3% )	12.0 ± 0.47( 0% )

Appendix

Table 2: GPT-4 and o3-mini performance with different system prompts on text-based Hanabi averaged over 3 seeds. * including the final scores before getting bombed.

Panel of Figure 12 — Figure 12: Evaluation of Low-Rank Adaptation (LoRA) in LLaMA-7B finetuning, showcasing the impact on a) Validation Accuracy and b) Game Play Score. The experiments involve varying data sizes [ $200,500,1000$ ] and LoRA ranks [ $32,64,128$ ].

Panel of Figure 13 — Figure 13: Analysis of the impact of training data amount on BERT, examining a) BERT Validation Accuracy, b) BERT Game Play Score across different percentages of training data, and c) BERT model variants with varying parameter sizes.

Figure 14: Evaluation of the discard pile’s role in the game is assessed by comparing game scores with the presence and absence of the discard pile in the observation during training.

RL for Game AI

See all 29 papers

HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning
Jun 28, 2026Runze Zhao, Dongruo Zhou, Sumit Kumar Jha +2Multi-Agent Communication Cooperative MARL
Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington Bloomington, IN 47408 · Department of Computer & Information Science & Engineering University of Florida Gainesville, FL 32611 · Department of Electrical Engineering and Computer Science United States Military Academy West Point, NY 10996 +1
Probing Dec-POMDP Reasoning in Cooperative MARL
Feb 24, 2026Kale-ab Tessera, Leonard Hinckeldey, Riccardo Zamboni +2Cooperative MARL Multi-Agent Reasoning
University of Edinburgh, Edinburgh, United Kingdom · Politecnico di Milano, Milan, Italy
Hierarchical Lead Critic based Multi-Agent Reinforcement Learning
Feb 25, 2026David Eckel, Henri MeeßMulti-Agent Reinforcement Learning Cooperative MARL
Fraunhofer Institute IVI, Ingolstadt, Germany

Jun 28, 2026J/K move · Enter open · S save

HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

Runze Zhao, Dongruo Zhou, Sumit Kumar Jha, Nathaniel D. Bastian+1

Luddy School of Informatics, Computing, and Engineering Indiana University Bloomington Bloomington, IN 47408 · Department of Computer & Information Science & Engineering University of Florida Gainesville, FL 32611 · Department of Electrical Engineering and Computer Science United States Military Academy West Point, NY 10996+1

Multi-Agent Communication Cooperative MARL

Open paper AskPDF arXiv

Cooperative multi-agent reinforcement learning (MARL) often relies on communication to mitigate partial observability, yet most existing protocols treat messages as flat dense vectors detached from the structure of the observations they summarize. This design overlooks an important source of inductive bias in many cooperative environments, where observations naturally follow a hierarchy such as groups and entities. We propose HiComm, a plug-in communication module that grounds messages in the sender's hierarchical observation. HiComm is receiver-driven: the receiver issues a query, and the hierarchy is resolved through a three-stage decoding process that first selects a group, then a sender, and then an entity within that group, returning the corresponding feature slice as the message. This converts communication from unstructured vector transmission into structured information retrieval over the sender's observation hierarchy. We instantiate this mechanism with Straight-Through Gumbel-Softmax for differentiable discrete selection and a lightweight shared projection design that attaches to standard MARL pipelines. Experiments across cooperative MARL tasks with different observation structures and coordination demands show that HiComm matches or outperforms representative learned communication baselines while reducing communication volume by up to

23\times

per receiver per episode.

Abstract

RL for Game AI

Explore similar work

HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

Probing Dec-POMDP Reasoning in Cooperative MARL

Hierarchical Lead Critic based Multi-Agent Reinforcement Learning