Improvisational Games as a Benchmark for Social Intelligence of AI Agents: The Case of Connections

Gaurav Rajesh Parikh; Angikar Ghosal

cs.AIMar 31, 2026

Improvisational Games as a Benchmark for Social Intelligence of AI Agents: The Case of Connections

Authors: Gaurav Rajesh Parikh, Angikar Ghosal

Organizations: Graduate School of Business, Stanford University, 655 Knight Way, Stanford, CA 94305

Abstract

We formally introduce a improvisational wordplay game called Connections to explore reasoning capabilities of AI agents. Playing Connections combines skills in knowledge retrieval, summarization and awareness of cognitive states of other agents. We show how the game serves as a good benchmark for social intelligence abilities of language model based agents that go beyond the agents' own memory and deductive reasoning and also involve gauging the understanding capabilities of other agents. Finally, we show how through communication with other agents in a constrained environment, AI agents must demonstrate social awareness and intelligence in games involving collaboration.

Figures & tables

Word	Reveals	Guesser Wrong	Setter Blocked	Iterations
kaleidoscope	0	1	7	8
xenophobia	1	2	4	7
labyrinthine	9	26	12	47
uppercases	8	20	18	46
entrepreneur	4	12	26	42
laboriously	5	21	23	49

Table 1: Comparisons Across Different Words, Two Guessers, One Setter (Ordered by Iterations)

Figure 1: Number of Characters Revealed over iterations for words with more than 100 iterations of game play

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,497 papers

Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments
Aug 10, 2026Keyu He, Xuhui Zhou, Maarten SapGame-Playing Agents LLM Agent Evaluation
Carnegie Mellon University
CaM-Wolf: Causal-Aware Multimodal Agents for Social Deduction Games
Jul 29, 2026Zheng Zhang, Nanjie Yao, Jiarui He +3Game-Playing Agents Social Deduction Games
The Hong Kong University of Science and Technology (Guangzhou) · Nanyang Technological University · Shanghai Jiao Tong University
MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs
May 28, 2026Kevin Wang, Anna Thöni, Benjamin Kempinski +50LLM Agent Evaluation Strategic Reasoning Risks in Language Models

Aug 10, 2026J/K move · Enter open · S save

Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

Keyu He, Xuhui Zhou, Maarten Sap

Carnegie Mellon University

Game-Playing Agents LLM Agent Evaluation LLM Agent Self-Improvement AI Agent Benchmarks Large Language Model-Based Social Simulation Social Reasoning

Open paper AskPDF arXiv

LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring and improving these social skills is hard because, unlike math or logic, social interaction offers no objective ground truth: evaluations fall back on LLM judges, which are costly, subjective, and noisy, and models get no reliable signal to learn from. To address both, we first introduce Social Gym, an environment of 21 multi-agent social games (e.g., Werewolves, Resistance, Spyfall) whose rule-decided outcomes make agent performance verifiable and objective, with an Elo tournament that produces a cross-game leaderboard. Benchmarking experiments show that while GPT-5-mini tops the leaderboard, no model excels at all games uniformly or in all game roles, pointing to limitations of social reasoning. Motivated by this, we additionally propose SPaRTan (Self-Play and Reflect-Transfer), a training-free self-improvement loop: a model plays a game, reflects on its trajectories and their outcomes to produce a transferable playbook, and applies that playbook in subsequent games. Our results show that SPaRTan playbooks help GPT-5-mini agents level their performance on weaker roles, but largely do not improve Qwen3-32B's performance. Together, Social Gym and SPaRTan offer a reproducible, verifiable foundation for measuring and improving LLM social reasoning without weight updates.

Abstract

AI Agent Benchmarks

Explore similar work

Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

CaM-Wolf: Causal-Aware Multimodal Agents for Social Deduction Games

MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs