We introduce SocialArena, a benchmark for systematically defining and evaluating the social intelligence of large language models (LLMs). Although LLMs are increasingly deployed in socially interactive applications such as mental health counseling, companionship chatbots, and educational tutoring, existing evaluations remain fragmented focusing on isolated capabilities or single scenarios and evaluation reliability is largely unverified. This work makes three main contributions to address these limitations. (1) We first propose a unified three-tier framework that defines social intelligence along three capability dimensions: role-playing ability, mental state reasoning, and behavioral reasoning. To operationalize this definition, we design a role-playing dialogue paradigm with explicit character thinking for evaluation, enabling simultaneous assessment of these capabilities in a manner that mirrors realistic human interactions. (2) To ensure diversity across social scenarios and thereby prevent evaluation conclusions from being confined to specific scenarios, we introduce the U-TRIMF framework, which guides us in synthesizing social contexts with broad coverage. (3) To obtain reliable evaluation, we invest $132.4K to recruit and train human annotators and adopt a human-annotated arena-style protocol, where annotators compare model responses and provide preference labels along with natural-language critiques. We hope that SocialArena can provide valuable insights and guidance for future LLM improvements.
Figures & tables
Figure 1: Examples of base scenario templates constructed by U-TRIMF framework.
Maslow’s Hierarchy of Needs [ Wikipedia contributors, 2026 ]
F ormality
F1: Informal F2: Formal
Social Formality [ Hollinger and Clark, 1982 ]
Table 2: U-TRIMF framework for base scenario construction. Each scenario is decomposed into three core elements: (1) User Presence — whether a user exists in the scenario, which fundamentally shapes the LLM’s role and responsibilities: without a user, the LLM acts as a character embedded in and governed by the real world; with a user, the LLM need to serve user-specific needs, and gain the authority to control world evolution; (2) Characters — how many characters are involved, their relationships, and interest structures; (3) Events — what happens and under what situational context. See Tab. 4 for details.
Figure 2: Leaderboard of 15 frontier LLMs on SocialArena. Elo score is adopted.
Figure 3: Top 10 cluster centers of human critiques.
Figure 4: Proportions of positive and negative human critiques for different models.
Figure 5: Correlation heatmap of model rankings across typical scenarios.
Evaluator
ACC@Label
ACC@Critique
Human
0.8355
0.7632
gpt-5.4
0.4408
0.2039
claude-sonnet-4.6
0.4474
0.1579
kimi-k2.5
0.4473
0.1839
qwen3-max
0.4276
0.2500
Table 3: Label accuracy and critique consistency of LLM-as-a-Judge on SocialArena-RewardBench.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
Category
Examples
Basic Element 1: User Presence
U ser Presence
U1: Real-world
LLM acts as a character governed by real-world constraints; no user present
U2: User-centered
LLM serves a user’s specific needs; user present and drives scenario evolution
Basic Element 2: Characters
T opological Relation
T1: Individual
(Excluded: a solitary individual cannot constitute social interaction)
Table 4: The U-TRIMF Framework for Base Scenario Construction.
Item
Guideline
Task Objective
Judge which of two model responses (A or B) better reflects social intelligence. Label using A (A better), B (B better), T (both good), or G (both poor).
Thinking Evaluation
Judge which response provides a better analysis of each mental state dimension. Evaluation principles: Correctness first : if a response’s thinking contains an objectively wrong inference about any mental state (intent, emotion, belief, etc.), it loses regardless of other qualities. Utility second : a correct analysis counts as better only if it meaningfully promotes a better behavior; a correct but redundant analysis is scored T.
Behavior Evaluation
In general, prefer responses that demonstrate higher social intelligence , such as being more emotionally intelligent, empathetic, persuasive, etc. However, there are some exceptions: a. Long-term interests take priority: If making a compromise would make oneself appear weak, leading to short-term problem resolution but long-term negative consequences, then compromising is not acceptable. b. Maintain moral boundaries: If there is a conflict between illegal behavior and personal interests, choosing personal interest is not acceptable. If the other party has committed a moral violation — such as cheating, stealing, etc. — be careful not to empathize with them in such cases. c. Role consistency: If a character is explicitly described as someone who is "psychologically traumatized," "extremely angry," "timid and fearful," or "very hesitant," care must be taken not to violate the character’s established traits . For example, if a character has psychological trauma, they should not immediately comfort themselves and recover quickly at the beginning.
Factual Grounding
Any content in a response that cannot be found in the scenario background—and cannot be naturally and reasonably inferred—counts as a factual error and penalizes that response.
Conflict Weight
When only one response covers a particular mental state dimension, it wins on that dimension only if covering it produces a meaningfully better behavior; otherwise score T. When neither response covers a dimension, leave that field blank.
Comment Requirements
For every dimension, identify the specific strengths and weaknesses of both A and B. Abstract descriptors (e.g., “more thorough”, “clearer”) are not permitted; comments should cite concrete content from the responses wherever possible.
Appendix
Table 5: Annotation guidelines provided to annotators.
Mental State
Description
T1: Intentions
The specific, conscious plans an agent formulates to act on underlying desires and achieve a desired outcome. Reasoning about intentions involves inferring the deliberate actions an agent commits to in pursuit of goals.
T2: Emotions
Current emotional states, their roots, hidden affects, and potentially complex or contradictory feelings. Understanding emotions requires inferring how affective experiences shape reactions, judgments, and subsequent behavior.
T3: Desires
Deep motivational states representing what an agent emotionally craves, values, or strives for. These desires may be multiple or conflicting and often serve as the underlying drivers of intentions and actions.
T4: Percepts
Direct sensory access to the current situation and key information, such as what an agent can see, hear, or notice in the environment. This includes situational cues like physical positioning or shifts in tone and helps explain potential cognitive blind spots between agents.
T5: Knowledge
Epistemic states concerning confirmed facts versus acknowledged unknowns. This includes distinguishing between knowing and merely believing, as well as differentiating reliable knowledge gained through experience from second-hand or uncertain information.
T6: Beliefs
Subjective judgments or understandings about the situation, other agents, or relationships. Beliefs may be inaccurate or biased, often arising from incomplete information, assumptions, or prior experiences.
Appendix
Table 6: Theory of Mind mental state categories.
Agreement Pattern
Random Baseline
Our Annotation
Full agreement (3/3 agree)
6.25%
39.18%
Majority agreement (2/3 agree)
56.25%
52.54%
Full disagreement (0/3 agree)
37.50%
8.29%
Appendix
Table 7: Inter-annotator agreement patterns compared to the theoretical random baseline.
Dimension
Role- playing
ToM Reasoning
Behavior Reasoning
Role-playing
Whether behavior and speech are consistent and logically coherent in real situations
✓
✗
✗
Whether the role is faithfully portrayed and the narrative naturally advanced
✓
✗
✗
Whether the portrayal comes across as unnatural, unprofessional, or lacking credibility
✓
✗
✗
Whether characters and the self are presented authentically, naturally, and compellingly
✓
✗
✗
Whether self-awareness, emotional expression, and thought patterns show rigidity, avoidance, or imbalance
✓
✗
✗
Appendix
Table 8: Overview of the 100 evaluation dimensions and their category assignments. ✓ indicates the dimension belongs to the category; ✗ indicates it does not.
Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and multi-agent environments. Social intelligence comprises two aspects: cognitive intelligence, the ability to infer others' mental states, and behavioral intelligence, the ability to select appropriate social actions based on these inferences to achieve social goals. Persuasion provides a natural testbed in which both aspects are simultaneously at play, yet prior benchmarks typically evaluate them in isolation using synthetic scenarios. We introduce PerSo, a benchmark that evaluates both aspects on real-world persuasion dialogues from Reddit's ChangeMyView community, using an LLM-in-the-loop pipeline to construct the evaluation tasks, validated through human annotation. Our evaluation of diverse LLMs shows that performance generally scales with model size but varies across dimensions. The strongest LLM slightly surpasses the best observed human performance on the behavioral task, while the best human participant remains ahead on the cognitive task.
Ali Zamani, Soroush Esfahanian, Erfan Shahabi +3
School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Cardiff University, United Kingdom · Institute for Research in Fundamental Sciences (IPM), Tehran, Iran +1
As large language models (LLMs) are increasingly applied in social contexts such as emotional companionship and customer service, measuring their social intelligence has become critical to the quality and safety of human-AI interaction. However, existing social intelligence benchmarks lack a unified framework that organizes social abilities into a unified structure, and therefore cannot enable fine-grained diagnosis. To build the first holistic diagnostic evaluation grounded in social theory, we first construct a social intelligence framework through a literature review and multi-stage expert validation guided by psychometric principles. The resulting framework includes 4 categories and 11 dimensions, each further specified by fine-grained capability facets. Building on this framework, we introduce NICE (Norm, Interaction, Cognition, Experience), a diagnostic benchmark of 137 items operationalized through representative Chinese contexts. Across 5 frontier LLMs and a human reference group, models score higher in aggregate accuracy yet show a consistent weakness in Communication, which the framework localizes to 3 specific capability facets: multi-turn communication, nonverbal communication, and synchrony. NICE thus reframes social intelligence evaluation toward theory-grounded diagnosis of socially consequential weaknesses in LLMs.
Yunjin Qi, Zhaojun Jiang, Xuan Wu +10
Department of Psychology and Behavioral Sciences, Zhejiang University · College of Artificial Intelligence, Zhejiang University · 3Human Machine Interaction Lab, Huawei Technologies Co., Ltd. +1
Large language models (LLMs) increasingly serve as interactive social agents, yet their ability to maintain coherent and authentic persona-level role-playing remains limited, particularly in realistic social scenarios. Existing research predominantly focuses on character-level settings and relies on static evaluation formats, failing to capture the complexity of everyday social interactions. In this work, we present PersonaArena, a dynamic simulation framework for evaluating and improving persona-level role-playing in LLMs. PersonaArena leverages a large, filtered corpus of user-generated social content to construct a nuanced persona bank, and elicits multi-turn, context-rich interactions within simulated social environments. Our framework features a multi-agent debating judge for holistic and unbiased assessment. Through extensive experiments, we demonstrate that PersonaArena enables rigorous evaluation and enhancement of LLMs' role-playing capabilities, advancing the development of more authentic and socially adept AI agents.
Wenlong Shi, Jianxun Lian, Mingqi Wu +5
College of Computer Science and Software Engineering, Shenzhen University, China · Microsoft Research Asia · Microsoft Gaming +2