cs.AIOct 29, 2026
SaveSocialArena: Evaluating Social Intelligence in Large Language Models
Organizations: Alibaba Token Hub, Alibaba Group · Tsinghua University · Alibaba Group
Abstract
We introduce SocialArena, a benchmark for systematically defining and evaluating the social intelligence of large language models (LLMs). Although LLMs are increasingly deployed in socially interactive applications such as mental health counseling, companionship chatbots, and educational tutoring, existing evaluations remain fragmented focusing on isolated capabilities or single scenarios and evaluation reliability is largely unverified. This work makes three main contributions to address these limitations. (1) We first propose a unified three-tier framework that defines social intelligence along three capability dimensions: role-playing ability, mental state reasoning, and behavioral reasoning. To operationalize this definition, we design a role-playing dialogue paradigm with explicit character thinking for evaluation, enabling simultaneous assessment of these capabilities in a manner that mirrors realistic human interactions. (2) To ensure diversity across social scenarios and thereby prevent evaluation conclusions from being confined to specific scenarios, we introduce the U-TRIMF framework, which guides us in synthesizing social contexts with broad coverage. (3) To obtain reliable evaluation, we invest $132.4K to recruit and train human annotators and adopt a human-annotated arena-style protocol, where annotators compare model responses and provide preference labels along with natural-language critiques. We hope that SocialArena can provide valuable insights and guidance for future LLM improvements.
Figures & tables
Figure 1: Examples of base scenario templates constructed by U-TRIMF framework.
| Dimension | Sub-categories | Source |
| U ser Presence | U1: Real-world U2: User-centered | — |
| T opological Relation | T1: Individual T2: Dyad T3: Group | Topology [ Klüver et al., 1999 ] |
| R elational Model | R1: Communal Sharing R2: Authority Ranking R3: Equality Matching R4: Market Pricing R5: Null | Relational Models Theory [ Fiske, 1992 ] |
| I nterest Structure | I1: Zero-sum I2: Negative-sum I3: Positive-sum | Game Theory [ Fudenberg and Tirole, 1991 ] |
| M otivational Type | M1: Physiological M2: Safety M3: Belonging M4: Esteem M5: Self-actualization | Maslow’s Hierarchy of Needs [ Wikipedia contributors, 2026 ] |
| F ormality | F1: Informal F2: Formal | Social Formality [ Hollinger and Clark, 1982 ] |
Table 2: U-TRIMF framework for base scenario construction. Each scenario is decomposed into three core elements: (1) User Presence — whether a user exists in the scenario, which fundamentally shapes the LLM’s role and responsibilities: without a user, the LLM acts as a character embedded in and governed by the real world; with a user, the LLM need to serve user-specific needs, and gain the authority to control world evolution; (2) Characters — how many characters are involved, their relationships, and interest structures; (3) Events — what happens and under what situational context. See Tab. 4 for details.
Figure 2: Leaderboard of 15 frontier LLMs on SocialArena. Elo score is adopted.
Figure 3: Top 10 cluster centers of human critiques.
Figure 4: Proportions of positive and negative human critiques for different models.
Figure 5: Correlation heatmap of model rankings across typical scenarios.
| Evaluator | ACC@Label | ACC@Critique |
| Human | 0.8355 | 0.7632 |
| gpt-5.4 | 0.4408 | 0.2039 |
| claude-sonnet-4.6 | 0.4474 | 0.1579 |
| kimi-k2.5 | 0.4473 | 0.1839 |
| qwen3-max | 0.4276 | 0.2500 |
Table 3: Label accuracy and critique consistency of LLM-as-a-Judge on SocialArena-RewardBench.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | Category | Examples |
| Basic Element 1: User Presence | ||
| U ser Presence | U1: Real-world | LLM acts as a character governed by real-world constraints; no user present |
| U2: User-centered | LLM serves a user’s specific needs; user present and drives scenario evolution | |
| Basic Element 2: Characters | ||
| T opological Relation | T1: Individual | (Excluded: a solitary individual cannot constitute social interaction) |
| T2: Dyad | Couple, doctor/patient, teacher/student, buyer/seller | |
Table 4: The U-TRIMF Framework for Base Scenario Construction.
| Item | Guideline |
| Task Objective | Judge which of two model responses (A or B) better reflects social intelligence. Label using A (A better), B (B better), T (both good), or G (both poor). |
| Thinking Evaluation | Judge which response provides a better analysis of each mental state dimension. Evaluation principles: Correctness first : if a response’s thinking contains an objectively wrong inference about any mental state (intent, emotion, belief, etc.), it loses regardless of other qualities. Utility second : a correct analysis counts as better only if it meaningfully promotes a better behavior; a correct but redundant analysis is scored T. |
| Behavior Evaluation | In general, prefer responses that demonstrate higher social intelligence , such as being more emotionally intelligent, empathetic, persuasive, etc. However, there are some exceptions: a. Long-term interests take priority: If making a compromise would make oneself appear weak, leading to short-term problem resolution but long-term negative consequences, then compromising is not acceptable. b. Maintain moral boundaries: If there is a conflict between illegal behavior and personal interests, choosing personal interest is not acceptable. If the other party has committed a moral violation — such as cheating, stealing, etc. — be careful not to empathize with them in such cases. c. Role consistency: If a character is explicitly described as someone who is "psychologically traumatized," "extremely angry," "timid and fearful," or "very hesitant," care must be taken not to violate the character’s established traits . For example, if a character has psychological trauma, they should not immediately comfort themselves and recover quickly at the beginning. |
| Factual Grounding | Any content in a response that cannot be found in the scenario background—and cannot be naturally and reasonably inferred—counts as a factual error and penalizes that response. |
| Conflict Weight | When only one response covers a particular mental state dimension, it wins on that dimension only if covering it produces a meaningfully better behavior; otherwise score T. When neither response covers a dimension, leave that field blank. |
| Comment Requirements | For every dimension, identify the specific strengths and weaknesses of both A and B. Abstract descriptors (e.g., “more thorough”, “clearer”) are not permitted; comments should cite concrete content from the responses wherever possible. |
Table 5: Annotation guidelines provided to annotators.
| Mental State | Description |
| T1: Intentions | The specific, conscious plans an agent formulates to act on underlying desires and achieve a desired outcome. Reasoning about intentions involves inferring the deliberate actions an agent commits to in pursuit of goals. |
| T2: Emotions | Current emotional states, their roots, hidden affects, and potentially complex or contradictory feelings. Understanding emotions requires inferring how affective experiences shape reactions, judgments, and subsequent behavior. |
| T3: Desires | Deep motivational states representing what an agent emotionally craves, values, or strives for. These desires may be multiple or conflicting and often serve as the underlying drivers of intentions and actions. |
| T4: Percepts | Direct sensory access to the current situation and key information, such as what an agent can see, hear, or notice in the environment. This includes situational cues like physical positioning or shifts in tone and helps explain potential cognitive blind spots between agents. |
| T5: Knowledge | Epistemic states concerning confirmed facts versus acknowledged unknowns. This includes distinguishing between knowing and merely believing, as well as differentiating reliable knowledge gained through experience from second-hand or uncertain information. |
| T6: Beliefs | Subjective judgments or understandings about the situation, other agents, or relationships. Beliefs may be inaccurate or biased, often arising from incomplete information, assumptions, or prior experiences. |
Table 6: Theory of Mind mental state categories.
| Agreement Pattern | Random Baseline | Our Annotation |
| Full agreement (3/3 agree) | 6.25% | 39.18% |
| Majority agreement (2/3 agree) | 56.25% | 52.54% |
| Full disagreement (0/3 agree) | 37.50% | 8.29% |
Table 7: Inter-annotator agreement patterns compared to the theoretical random baseline.
| Dimension | Role- playing | ToM Reasoning | Behavior Reasoning |
|---|---|---|---|
| Role-playing | |||
| Whether behavior and speech are consistent and logically coherent in real situations | ✓ | ✗ | ✗ |
| Whether the role is faithfully portrayed and the narrative naturally advanced | ✓ | ✗ | ✗ |
| Whether the portrayal comes across as unnatural, unprofessional, or lacking credibility | ✓ | ✗ | ✗ |
| Whether characters and the self are presented authentically, naturally, and compellingly | ✓ | ✗ | ✗ |
| Whether self-awareness, emotional expression, and thought patterns show rigidity, avoidance, or imbalance | ✓ | ✗ | ✗ |
Table 8: Overview of the 100 evaluation dimensions and their category assignments. ✓ indicates the dimension belongs to the category; ✗ indicates it does not.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,427 papers