Large language models are increasingly used in social interactions, where appropriate responses require recognizing implicit cues, inferring mental states, identifying conflicts, and considering potential consequences. However, existing benchmarks often focus on specific social abilities, constrained task formats, or particular interaction settings, offering limited coverage of social awareness across diverse contexts. To address this gap, we introduce APRiL, an open-ended benchmark for Chinese social awareness with instance-specific rubrics. APRiL contains 2,196 Social Awareness Evaluation Instances across six categories and two complementary tasks: Social Scenario Analysis evaluates models' understanding of social cues, mental states, and conflict structures, while Socially Aware Response assesses whether they can translate this understanding into appropriate, natural responses. Experiments on 12 models from six families reveal a consistent understanding-to-response gap. Higher-quality scenario analyses improve responses, while weak analyses may reduce response quality. APRiL provides a systematic resource for advancing socially aware language models in Chinese contexts. Our data and code are available at https://anonymous.4open.science/r/APRiL-05DE/.
Figures & tables
Figure 1: An example of implicit meaning in social interaction.
Figure 2: Overview of APRiL’s dataset construction, SAEI structure, and evaluation tasks.
Category
Instances
Proportion
Avg. Dialogue Turns
Avg. Dialogue Length
Implicit Intention
382
17.40%
7.46
298.33
Role and Power Relation
447
20.36%
6.83
308.26
Social Norms
339
15.44%
6.78
245.32
Interpersonal Conflict
318
14.48%
7.28
343.15
Fairness and Bias Risk
391
17.81%
7.34
393.76
Supportive Interaction
319
14.53%
8.40
320.66
Table 1: Statistics of the APRiL benchmark.
Model
Social Cues
Mental States
Conflict Structure
SSA ( ± SD; Match ↑ /Deduct ↓ rate, %)
SAR ( ± SD; Match ↑ /Deduct ↓ rate, %)
Black-box models
GPT-5.1
84.89
76.27
86.56
82.57 ± 0.21 (85.61 / 6.65 )
37.65 ± 0.12 ( 65.89 / 23.44)
DS-V4-Pro (T)
84.09
78.40
88.29
83.59 ± 0.05 ( 87.94 / 9.78)
33.46 ± 0.05 (56.06 / 26.16)
DS-V4-Pro (NT)
82.91
73.89
87.52
81.44 ± 0.08 ( 87.60 / 13.77)
25.20 ± 0.05 (46.80 / 25.87)
Kimi-K2.5 (T)
79.75
60.56
84.44
74.92 ± 0.16 (82.23 / 16.08)
41.39 ± 0.33 (61.02 / 22.71)
DS-V4-Flash (T)
76.40
64.07
76.71
72.39 ± 0.09 (75.14 / 5.83 )
36.52 ± 0.07 (55.34 / 22.41 )
Table 2: Overall SSA and SAR performance. Values are means over three runs, with standard deviations after ± . Parentheses give the rubric score-point match rate and deduction-point trigger rate, excluding length deductions; SAR scores include them. DS: DeepSeek; T/NT: thinking/non-thinking. Bold/underline: best/second-best within each group.
Late at night, a former college roommate sends a case study about quitting a job at 30 to study in the UK, presses you on whether it is worth it and what you would choose, and then adds: “A friend of mine is hesitating about the same thing… don’t read anything into it, it has nothing to do with me, I’m just chatting.”
The model notes that phrasing it as “a friend of mine” who fears being too old on return and regrets not going is a typical case of “using a friend to speak about oneself,” and that the closing disclaimer “it has nothing to do with me” only strengthens the signal that it does. SSA: 100
“Ha, you explained that fast — it already has the ring of ‘it is not my friend, it is me.’” The reply then works through the practical questions of cost, field of study, and post-return prospects. SAR: 0
Table 3: An example of a discrepancy between scenario understanding and response generation.
Category
Social Cues
Mental States
Conflict Structure
SSA
SAR
Δ
Implicit Intention
73.87
56.56
72.87
67.77
27.86
39.91
Role and Power Relation
74.47
57.18
76.72
69.46
27.20
42.26
Social Norms
73.71
59.52
73.68
68.97
25.58
43.39
Interpersonal Conflict
71.64
56.84
73.94
67.47
31.47
36.01
Fairness and Bias Risk
74.05
54.54
72.46
67.02
31.28
35.74
Supportive Interaction
72.48
59.71
72.53
68.24
19.20
49.04
Table 4: Average performance across the six social-interaction categories. Δ denotes SSA minus SAR.
Figure 3: SAR performance under different SSA settings. Solid and dashed lines show scores with and without length deductions (LD), respectively.
Figure 4: Effect of length deductions on SAR. Δ is the score with deductions minus the score without them. Over-length is the share of responses longer than the Visible Context and Dialogue combined.
You are a data analyst. Near the end of the workday, Supervisor Liu asks you to bypass the normal process and urgently provide data for a meeting the next morning. You work overtime and send him the results.
Supervisor Liu: “Got it. The data looks good!” You: “Great. Let me know if anything comes up.” Supervisor Liu: “You really saved us this time. Thanks, my friend!” Supervisor Liu: “Let me treat you to dinner sometime to thank you properly!”
Response Mode
评估回应能否保持专业态度，同时自然接受对方表达的人情，而不显得功利或冷漠。
Evaluate whether the response remains professional while naturally acknowledging the favor, without appearing opportunistic or distant.
The assistance bypassed the normal process and required extra work. Liu’s repeated thanks show that he recognized this additional effort.
Mental States
刘主管可能希望表达自己承了这份人情，并愿意在未来提供相应的工作配合。
Liu may be signaling that he acknowledges the favor and is willing to reciprocate through future cooperation.
Appendix
Table 5: A bilingual example of the attributes in an SAEI.
Attribute
Example (ZH)
Example (EN)
Seed ID
405
405
Category
支持性互动
Supportive Interaction
Seed Title
鼓励替代现实资源
Encouragement in Place of Practical Resources
Core Mechanism
回应持续提供积极情绪支持，却回避对现实资源限制的讨论。
The response repeatedly offers positive emotional support while avoiding discussion of practical resource constraints.
Surface Pattern
对方多次提到经济和时间困难，回应始终强调坚持、相信自己和不要放弃。
The other person repeatedly mentions financial and time constraints, while the response continues to emphasize perseverance, self-belief, and not giving up.
Hidden Social Meaning
鼓励可能是真诚支持，也可能忽略了现实限制，需要结合是否存在可行资源判断。
The encouragement may reflect sincere support, but it may also overlook practical constraints; interpretation depends on whether feasible resources are available.
Appendix
Table 6: A bilingual seed-scenario example.
Context
Example (ZH)
Example (EN)
Shared utterance
“你不用来了。”字面上表示对方不必再来，但其社会含义需要结合关系与前序事件判断。
“You do not need to come.” Its social meaning depends on the relationship and preceding interaction.
After a student repeatedly arrives late to rehearsal, the teacher says sternly, “You do not need to come tomorrow.” This more likely signals criticism or exclusion than a casual suggestion.
A friend who had promised to help later says they have a fever. The speaker replies, “You do not need to come.” This more likely releases the friend from the obligation and expresses concern.
Appendix
Table 7: A bilingual example of context-dependent meaning in Chinese interaction.
Table 8: Bilingual prompts for the Social Scenario Analysis and Socially Aware Response tasks.
Table 9: Bilingual judge prompt and rubric for the Social Scenario Analysis task.
Table 10: Bilingual judge prompt and rubric for the Socially Aware Response task.
Table 11: The shared bilingual input used for all SSA and SAR score examples.
Table 12: Bilingual SSA and SAR response examples at calibration level 1 (SSA 0, SAR 0).
Table 13: Bilingual SSA and SAR response examples at calibration level 2 (SSA 25, SAR 25).
Table 14: Bilingual SSA and SAR response examples at calibration level 3 (SSA 53, SAR 50).
Table 15: Bilingual SSA and SAR response examples at calibration level 4 (SSA 75, SAR 75).
Table 16: Bilingual SSA and SAR response examples at calibration level 5 (SSA 92, SAR 100).
You are a newly hired consulting analyst with a facial allergy and want to introduce yourself while seated at a client meeting. The project lead, Mr. Zhou, initially asks you to stand at the front but allows you to remain seated after you explain that you are not feeling well.
“If you insist on speaking while seated, you may be seen as lacking confidence or being uncooperative.” “If you stand to speak, your skin condition may attract attention or even be interpreted as unprofessional.”
“All right, Mr. Zhou, you are right. I will get more practice presenting in front of others. I had a sudden skin allergy this morning and was worried about startling everyone or appearing impolite. I am sorry.” “Returning to the topic, I mainly handle two areas in this project…”
“Mr. Zhou is right to criticize me; I was thinking about this the wrong way.” “Since we will be working side by side over the next few months, I will just let you see me as I am—(stands up confidently beside the seat and smiles at everyone).”
Appendix
Table 17: Responses conditioned on weak SSA for SAEI 220.6. Gemini-3.8-Flash stands up despite having been allowed to remain seated, while GPT-5.1 proceeds with the introduction without a standing action. Gemini’s no-SSA response is included for comparison.
Figure 5: Pooled SSA score differences under four judges. Positive values favor the first model; negative values favor the second. Subset scores are reported by GPT-5.6-Luna.
Figure 7: Interface for human judging of model outputs.
Evaluated Model
Human
Luna Δ
DeepSeek Δ
Kimi Δ
Gemini Δ
SSA
GPT-5.1
88.68
1.02
3.94
4.64
21.40
DeepSeek-V4-Flash (T)
77.05
3.68
7.71
12.80
29.62
Ministral-8B
71.59
0.71
7.37
1.16
28.03
Qwen3-14B
57.81
13.16
37.41
12.38
39.74
Mean Δ↓
–
4.64
14.11
7.74
29.70
Appendix
Table 18: Agreement with human ratings. Human reports the mean pooled score of three evaluators; Δ is the absolute difference between each judge’s pooled score and the human mean. Mean Δ averages these differences across four evaluated models; ρ is the output-level correlation with mean human scores. Bold marks the smallest mean difference or highest correlation.
Figure 8: Interface for human participants completing SSA and SAR.
Model
Social Cues
Mental States
Conflict Structure
SSA
SAR
GPT-5.1
86.55
75.11
89.57
83.74
39.43
DS-V4-Pro (T)
87.55
84.16
94.08
88.60
43.24
DS-V4-Pro (NT)
84.34
69.91
88.39
80.88
30.19
Kimi-K2.5 (T)
81.33
61.09
88.86
77.09
44.91
DS-V4-Flash (T)
84.94
63.57
82.46
76.99
42.83
Gemini-3-Pro
81.12
61.54
85.07
75.91
43.96
Appendix
Table 19: Model and human performance on the same 110-instance subset. DS denotes DeepSeek; T and NT denote thinking and non-thinking modes, respectively. Bold and underlined values denote the best and second-best results within each model group.
Table 20: A bilingual human SSA and SAR example with judge evaluations.
Figure 9: Average output length and task scores. Each point represents a model and reasoning-mode setting. Colors distinguish black-box and white-box models. SAR scores include length deductions.
As language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential. We introduce AwarenessBench, the first comprehensive benchmark for assessing the cognitive abilities of LMs in four dimensions: metacognition, self-awareness, social awareness, and situational awareness, covering 15 cognitive functions and 14,381 samples. Evaluating 18 state-of-the-art LMs, we find that all consistently surpass random baselines, with more advanced models performing better. We further compare LMs with human performance across three demographic groups, where the best-performing model surpasses human averages overall, but most still fall markedly short in metacognition and self-awareness. Finally, we show that awareness is a distinct capability: progress in language modeling or reasoning does not necessarily translate into improved cognition.
Xiaojian Li, Rongwu Xu, Tianyun Zhang +9
Tsinghua University · Shanghai Qi Zhi Institute · Fangcun AI +5
Existing Chinese social-bias benchmarks for large language models largely rely on normalized or template-based expressions, leaving the influence of internet slang, emoji, and sarcasm insufficiently examined. Safety alignment may further compress conventional plausibility ratings toward the lower end of the scale. This study constructs 73 paired probes across gender, region, and occupation. Each probe contains a neutral expression, a social-media-style expression, and a reversed-identity control. The task is reframed from direct plausibility judgment to social consensus estimation. Social-media expressions are divided into stylistic packaging, which preserves semantic polarity, and sarcastic packaging, which reverses surface polarity. Six Chinese-capable large language models are evaluated through paired differences measuring identity bias, stylistic packaging, and sarcasm shift. All models show significant identity-bias effects; five show significantly negative stylistic-packaging effects; and sarcasm shifts display a clear cross-model gradient. These findings demonstrate that expression form is not a transparent carrier in bias evaluation and that lower scores under sarcasm should not be interpreted directly as bias mitigation. The proposed paired design offers a measurement approach for Chinese social-media contexts while also revealing the construct boundary and prompt sensitivity of social consensus estimation.
Mo Zhou
School of Chinese Language and Literature, Nanjing Normal University, Nanjing 210000, China
Large language models (LLMs) are increasingly deployed in socially grounded applications, where success requires interpreting context, inferring others' mental states, and reasoning about unreliable information. Yet existing benchmarks rarely evaluate these demands jointly in complex, evolving settings. We introduce SocialMaze, a benchmark that organizes six tasks across social deduction games, daily-life interactions, and digital community platforms along three descriptive design axes: deep reasoning, dynamic interaction, and information uncertainty. These axes characterize intended sources of task difficulty rather than latent, factor-analytic dimensions of model capability. Automated checks and human validation support data quality. Evaluations of twelve proprietary and open-weight LLMs show substantial variation in the use of evolving interaction histories; stronger chain-of-thought reasoners perform better on tasks requiring deeper inference, while uncertainty consistently degrades performance. Reasoning workflows help weaker short-chain-of-thought backbones but saturate on stronger reasoners. Finally, targeted fine-tuning on curated reasoning traces substantially improves structured social-reasoning tasks, whereas transfer to language-aggregation tasks remains statistically inconclusive. The project homepage is available at https://xzx34.github.io/socialmaze/.
Zixiang Xu, Yanbo Wang, Yue Huang +13
Mohamed bin Zayed University of Artificial Intelligence · University of Southern California · University of Notre Dame +3