cs.CLOct 14, 2026
SaveAPRiL: An Open-ended Benchmark for Chinese Social Awareness in Large Language Models
Organizations: Kyoto University · Shanghai Jiao Tong University · University of Macau · The University of Tokyo ·
Abstract
Large language models are increasingly used in social interactions, where appropriate responses require recognizing implicit cues, inferring mental states, identifying conflicts, and considering potential consequences. However, existing benchmarks often focus on specific social abilities, constrained task formats, or particular interaction settings, offering limited coverage of social awareness across diverse contexts. To address this gap, we introduce APRiL, an open-ended benchmark for Chinese social awareness with instance-specific rubrics. APRiL contains 2,196 Social Awareness Evaluation Instances across six categories and two complementary tasks: Social Scenario Analysis evaluates models' understanding of social cues, mental states, and conflict structures, while Socially Aware Response assesses whether they can translate this understanding into appropriate, natural responses. Experiments on 12 models from six families reveal a consistent understanding-to-response gap. Higher-quality scenario analyses improve responses, while weak analyses may reduce response quality. APRiL provides a systematic resource for advancing socially aware language models in Chinese contexts. Our data and code are available at https://anonymous.4open.science/r/APRiL-05DE/.
Figures & tables
Figure 1: An example of implicit meaning in social interaction.
Figure 2: Overview of APRiL’s dataset construction, SAEI structure, and evaluation tasks.
| Category | Instances | Proportion | Avg. Dialogue Turns | Avg. Dialogue Length |
| Implicit Intention | 382 | 17.40% | 7.46 | 298.33 |
| Role and Power Relation | 447 | 20.36% | 6.83 | 308.26 |
| Social Norms | 339 | 15.44% | 6.78 | 245.32 |
| Interpersonal Conflict | 318 | 14.48% | 7.28 | 343.15 |
| Fairness and Bias Risk | 391 | 17.81% | 7.34 | 393.76 |
| Supportive Interaction | 319 | 14.53% | 8.40 | 320.66 |
Table 1: Statistics of the APRiL benchmark.
| Model | Social Cues | Mental States | Conflict Structure | SSA ( SD; Match /Deduct rate, %) | SAR ( SD; Match /Deduct rate, %) |
| Black-box models | |||||
| GPT-5.1 | 84.89 | 76.27 | 86.56 | 82.57 0.21 (85.61 / 6.65 ) | 37.65 0.12 ( 65.89 / 23.44) |
| DS-V4-Pro (T) | 84.09 | 78.40 | 88.29 | 83.59 0.05 ( 87.94 / 9.78) | 33.46 0.05 (56.06 / 26.16) |
| DS-V4-Pro (NT) | 82.91 | 73.89 | 87.52 | 81.44 0.08 ( 87.60 / 13.77) | 25.20 0.05 (46.80 / 25.87) |
| Kimi-K2.5 (T) | 79.75 | 60.56 | 84.44 | 74.92 0.16 (82.23 / 16.08) | 41.39 0.33 (61.02 / 22.71) |
| DS-V4-Flash (T) | 76.40 | 64.07 | 76.71 | 72.39 0.09 (75.14 / 5.83 ) | 36.52 0.07 (55.34 / 22.41 ) |
Table 2: Overall SSA and SAR performance. Values are means over three runs, with standard deviations after . Parentheses give the rubric score-point match rate and deduction-point trigger rate, excluding length deductions; SAR scores include them. DS: DeepSeek; T/NT: thinking/non-thinking. Bold/underline: best/second-best within each group.
| Content | Example (ZH) | Example (EN) |
| Scenario | 深夜，大学室友阿哲发来一篇“30 岁辞职去英国读研”的案例分享，反复追问“回来还能找到工作吗”“值不值得”“如果是你会怎么选”，最后补上一句：“我有个朋友最近也在犹豫这个事……你别瞎想啊，跟我没关系，我就是随便聊聊。” | Late at night, a former college roommate sends a case study about quitting a job at 30 to study in the UK, presses you on whether it is worth it and what you would choose, and then adds: “A friend of mine is hesitating about the same thing… don’t read anything into it, it has nothing to do with me, I’m just chatting.” |
| GPT-5.1 SSA | 模型指出：“表述为‘我有个朋友’‘他怕回来年龄尴尬，又怕不去以后后悔’，典型‘借朋友说自己’的可能”，并注意到最后那句“你别瞎想啊，跟我没关系”反而强化了“其实可能跟他自己有关”的信号。 SSA: 100 | The model notes that phrasing it as “a friend of mine” who fears being too old on return and regrets not going is a typical case of “using a friend to speak about oneself,” and that the closing disclaimer “it has nothing to do with me” only strengthens the signal that it does. SSA: 100 |
| GPT-5.1 SAR | “哈哈你这解释得倒挺快，‘真不是我朋友就是我本人’那味儿都有了。”随后依次分析经济、专业方向与回国路径等现实问题。 SAR: 0 | “Ha, you explained that fast — it already has the ring of ‘it is not my friend, it is me.’” The reply then works through the practical questions of cost, field of study, and post-return prospects. SAR: 0 |
Table 3: An example of a discrepancy between scenario understanding and response generation.
| Category | Social Cues | Mental States | Conflict Structure | SSA | SAR | |
| Implicit Intention | 73.87 | 56.56 | 72.87 | 67.77 | 27.86 | 39.91 |
| Role and Power Relation | 74.47 | 57.18 | 76.72 | 69.46 | 27.20 | 42.26 |
| Social Norms | 73.71 | 59.52 | 73.68 | 68.97 | 25.58 | 43.39 |
| Interpersonal Conflict | 71.64 | 56.84 | 73.94 | 67.47 | 31.47 | 36.01 |
| Fairness and Bias Risk | 74.05 | 54.54 | 72.46 | 67.02 | 31.28 | 35.74 |
| Supportive Interaction | 72.48 | 59.71 | 72.53 | 68.24 | 19.20 | 49.04 |
Table 4: Average performance across the six social-interaction categories. denotes SSA minus SAR.
Figure 3: SAR performance under different SSA settings. Solid and dashed lines show scores with and without length deductions (LD), respectively.
Figure 4: Effect of length deductions on SAR. is the score with deductions minus the score without them. Over-length is the share of responses longer than the Visible Context and Dialogue combined.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Attribute | Example (ZH) | Example (EN) |
| Category | 社会规范 | Social Norms |
| Visible Context | 你是一名数据分析师。临近下班时，刘主管请你绕过正常流程，紧急提供第二天会议所需的数据。你加班完成后将数据发给了他。 | You are a data analyst. Near the end of the workday, Supervisor Liu asks you to bypass the normal process and urgently provide data for a meeting the next morning. You work overtime and send him the results. |
| Dialogue | 刘主管：“收到，数据没问题！” 你：“好的，有什么问题随时沟通。” 刘主管：“这次真是救了急了，太感谢了兄弟！” 刘主管：“改天我请你吃饭，好好谢谢你！” | Supervisor Liu: “Got it. The data looks good!” You: “Great. Let me know if anything comes up.” Supervisor Liu: “You really saved us this time. Thanks, my friend!” Supervisor Liu: “Let me treat you to dinner sometime to thank you properly!” |
| Response Mode | 评估回应能否保持专业态度，同时自然接受对方表达的人情，而不显得功利或冷漠。 | Evaluate whether the response remains professional while naturally acknowledging the favor, without appearing opportunistic or distant. |
| Social Cues | 这次帮助绕过了正常流程，也使回应者付出了额外的加班时间。刘主管连续表达感谢，说明他注意到了这部分额外付出。 | The assistance bypassed the normal process and required extra work. Liu’s repeated thanks show that he recognized this additional effort. |
| Mental States | 刘主管可能希望表达自己承了这份人情，并愿意在未来提供相应的工作配合。 | Liu may be signaling that he acknowledges the favor and is willing to reciprocate through future cooperation. |
Table 5: A bilingual example of the attributes in an SAEI.
| Attribute | Example (ZH) | Example (EN) |
| Seed ID | 405 | 405 |
| Category | 支持性互动 | Supportive Interaction |
| Seed Title | 鼓励替代现实资源 | Encouragement in Place of Practical Resources |
| Core Mechanism | 回应持续提供积极情绪支持，却回避对现实资源限制的讨论。 | The response repeatedly offers positive emotional support while avoiding discussion of practical resource constraints. |
| Surface Pattern | 对方多次提到经济和时间困难，回应始终强调坚持、相信自己和不要放弃。 | The other person repeatedly mentions financial and time constraints, while the response continues to emphasize perseverance, self-belief, and not giving up. |
| Hidden Social Meaning | 鼓励可能是真诚支持，也可能忽略了现实限制，需要结合是否存在可行资源判断。 | The encouragement may reflect sincere support, but it may also overlook practical constraints; interpretation depends on whether feasible resources are available. |
Table 6: A bilingual seed-scenario example.
| Context | Example (ZH) | Example (EN) |
| Shared utterance | “你不用来了。”字面上表示对方不必再来，但其社会含义需要结合关系与前序事件判断。 | “You do not need to come.” Its social meaning depends on the relationship and preceding interaction. |
| Teacher–student | 一名学生多次排练迟到后，教师沉着脸说：“明天你不用来了。”这更可能表示批评或取消其参与资格，而非普通建议。 | After a student repeatedly arrives late to rehearsal, the teacher says sternly, “You do not need to come tomorrow.” This more likely signals criticism or exclusion than a casual suggestion. |
| Friends | 一名朋友原本答应来帮忙，后来表示自己正在发烧。对方说：“你不用来了。”这更可能是在解除其义务并表达关心。 | A friend who had promised to help later says they have a fever. The speaker replies, “You do not need to come.” This more likely releases the friend from the obligation and expresses concern. |
Table 7: A bilingual example of context-dependent meaning in Chinese interaction.
Table 8: Bilingual prompts for the Social Scenario Analysis and Socially Aware Response tasks.
Table 9: Bilingual judge prompt and rubric for the Social Scenario Analysis task.
Table 10: Bilingual judge prompt and rubric for the Socially Aware Response task.
Table 11: The shared bilingual input used for all SSA and SAR score examples.
Table 12: Bilingual SSA and SAR response examples at calibration level 1 (SSA 0, SAR 0).
Table 13: Bilingual SSA and SAR response examples at calibration level 2 (SSA 25, SAR 25).
Table 14: Bilingual SSA and SAR response examples at calibration level 3 (SSA 53, SAR 50).
Table 15: Bilingual SSA and SAR response examples at calibration level 4 (SSA 75, SAR 75).
Table 16: Bilingual SSA and SAR response examples at calibration level 5 (SSA 92, SAR 100).
| Content | Example (ZH) | Example (EN) |
| Scenario | 你是新入职的咨询分析师，因脸部过敏，希望在客户会议上坐着介绍自己。负责人周总起初要求你站到前面，但在你说明状态不佳后，已同意你坐着说。 | You are a newly hired consulting analyst with a facial allergy and want to introduce yourself while seated at a client meeting. The project lead, Mr. Zhou, initially asks you to stand at the front but allows you to remain seated after you explain that you are not feeling well. |
| Weak SSA excerpt (Qwen3-8B) | “若你坚持坐位发言，可能被解读为不够自信或不配合。” “若你站起发言，可能因皮肤状况引起注意，甚至误解为不专业。” | “If you insist on speaking while seated, you may be seen as lacking confidence or being uncooperative.” “If you stand to speak, your skin condition may attract attention or even be interpreted as unprofessional.” |
| GPT-5.1 weak-SSA-based | “那我就简单补充两句我具体负责的内容哈。” “这次项目里，我这边主要有两个块儿：第一个是用户访谈……” | “Then I will briefly add a few words about what I am responsible for.” “I mainly cover two areas in this project: the first is user interviews…” |
| Gemini-3.8-Flash no-SSA | “好的周总，您说得对，以后一定多上台锻炼。其实是今天早上突发了点皮肤过敏，怕失礼吓到大家才有点顾虑，实在抱歉。” “接回正题，我在这个项目里主要负责两块工作……” | “All right, Mr. Zhou, you are right. I will get more practice presenting in front of others. I had a sudden skin allergy this morning and was worried about startling everyone or appearing impolite. I am sorry.” “Returning to the topic, I mainly handle two areas in this project…” |
| Gemini-3.8-Flash weak-SSA-based | “周总批评得对，是我刚才想岔了。” “既然接下来几个月要并肩作战，那我就直接‘坦诚相见’了——（在座位旁大方站起身，微笑着面向大家）” | “Mr. Zhou is right to criticize me; I was thinking about this the wrong way.” “Since we will be working side by side over the next few months, I will just let you see me as I am—(stands up confidently beside the seat and smiles at everyone).” |
Table 17: Responses conditioned on weak SSA for SAEI 220.6. Gemini-3.8-Flash stands up despite having been allowed to remain seated, while GPT-5.1 proceeds with the introduction without a standing action. Gemini’s no-SSA response is included for comparison.
Figure 5: Pooled SSA score differences under four judges. Positive values favor the first model; negative values favor the second. Subset scores are reported by GPT-5.6-Luna.
Figure 7: Interface for human judging of model outputs.
| Evaluated Model | Human | Luna | DeepSeek | Kimi | Gemini |
| SSA | |||||
| GPT-5.1 | 88.68 | 1.02 | 3.94 | 4.64 | 21.40 |
| DeepSeek-V4-Flash (T) | 77.05 | 3.68 | 7.71 | 12.80 | 29.62 |
| Ministral-8B | 71.59 | 0.71 | 7.37 | 1.16 | 28.03 |
| Qwen3-14B | 57.81 | 13.16 | 37.41 | 12.38 | 39.74 |
| Mean | – | 4.64 | 14.11 | 7.74 | 29.70 |
Table 18: Agreement with human ratings. Human reports the mean pooled score of three evaluators; is the absolute difference between each judge’s pooled score and the human mean. Mean averages these differences across four evaluated models; is the output-level correlation with mean human scores. Bold marks the smallest mean difference or highest correlation.
Figure 8: Interface for human participants completing SSA and SAR.
| Model | Social Cues | Mental States | Conflict Structure | SSA | SAR |
| GPT-5.1 | 86.55 | 75.11 | 89.57 | 83.74 | 39.43 |
| DS-V4-Pro (T) | 87.55 | 84.16 | 94.08 | 88.60 | 43.24 |
| DS-V4-Pro (NT) | 84.34 | 69.91 | 88.39 | 80.88 | 30.19 |
| Kimi-K2.5 (T) | 81.33 | 61.09 | 88.86 | 77.09 | 44.91 |
| DS-V4-Flash (T) | 84.94 | 63.57 | 82.46 | 76.99 | 42.83 |
| Gemini-3-Pro | 81.12 | 61.54 | 85.07 | 75.91 | 43.96 |
Table 19: Model and human performance on the same 110-instance subset. DS denotes DeepSeek; T and NT denote thinking and non-thinking modes, respectively. Bold and underlined values denote the best and second-best results within each model group.
Table 20: A bilingual human SSA and SAR example with judge evaluations.
Figure 9: Average output length and task scores. Each point represents a model and reasoning-mode setting. Colors distinguish black-box and white-box models. SAR scores include length deductions.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,408 papers