CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords

Yifan Wang; Junyu Lu; Qifan Wang; Shun Zhang; Chaozhuo Li; Jiahao Liu; Zhijun Cao; Lingbin Bu

cs.CLSep 18, 2026

CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords

Authors: Yifan Wang, Junyu Lu, Qifan Wang, Shun Zhang, Chaozhuo Li, Jiahao Liu, Zhijun Cao, Lingbin Bu,

Organizations: People’s Public Security University of China · Dalian University of Technology · Meta AI · Beijing University of Posts and Telecommunications ·

Abstract

Chinese social media has generated a vast and continually evolving lexicon of internet buzzwords whose meanings are often non-literal and deeply rooted in local cultural and pragmatic contexts. Existing research has primarily focused on interpreting these buzzwords within Chinese, leaving largely unexplored whether LLMs can transfer such culturally grounded knowledge across languages and accurately convey the intended meanings in English. This cross-lingual capability is also critical for safety, as harmful expressions may obscure their offensive content through culture-specific homophony, euphemism, irony, or coded language. In this paper, we investigate the ability of advanced LLMs to understand Chinese internet buzzwords across languages. To this end, we introduce CIBuzzBench, the first benchmark for cross-lingual Chinese-to-English understanding of Chinese internet buzzwords. CIBuzzBench comprises 3,001 Chinese internet buzzwords annotated with English meaning explanations, English equivalents, category labels, and harmfulness labels. Based on these annotations, we design three evaluation tasks: Meaning Explanation, Cross-lingual Equivalent Matching, and Culturally Grounded Harmfulness Detection. We evaluate representative state-of-the-art proprietary and Chinese LLMs under both English- and Chinese-prompting settings. Our results show that LLMs continue to struggle with the cross-lingual understanding of Chinese internet buzzwords, particularly in fine-grained non-literal interpretation, robust equivalent matching under option perturbations, and calibrated harmfulness detection. These findings highlight the persistent challenges posed by culturally grounded language phenomena for multilingual LLMs and safety-oriented evaluation. The dataset and code are available at https://github.com/SuperYFan/CIBuzzBench.

Figures & tables

Figure 1: Illustration of CIBuzzBench tasks with two examples. Each Chinese internet buzzword is paired with an English meaning explanation, an equivalent-selection question, and a harmfulness label.

Figure 2: Construction process of CIBuzzBench. In the data collection stage, we collect Chinese internet buzzwords and Chinese explanations from Chinese buzzword websites and assign category labels. In the task annotation stage, GPT-5.5 assists in generating annotations for the three tasks, which are then checked by task-specific reviewers; rejected cases are re-analyzed and re-annotated, while accepted cases are confirmed by the group and finalized as task datasets.

Category
Label	Count	Percent
Stylistic device	792	26.4%
Quotation	602	20.1%
Experience	592	19.7%
Slang	481	16.0%
Homophonic pun	398	13.3%

Table 1: Label distribution in CIBuzzBench. The benchmark contains 3,001 Chinese internet buzzwords with category and harmfulness annotations.

English Prompt
Model	BLEU	ROUGE-L	BERT-F1	Human	LLM
GPT-5.5	4.01	22.33	88.35	3.23	3.69
Claude Opus 4.8	2.94	19.42	87.07	3.09	3.49
Gemini 3.1 Pro	3.22	19.95	87.72	4.10	4.12
DeepSeek-V4-Pro	3.05	19.73	87.60	3.11	3.57
Qwen3.7-Max	4.02	20.20	87.98	3.50	3.94

Table 2: Meaning Explanation results under English and Chinese prompts. LLM-judge scores use the full benchmark, while Human scores use 200 sampled predictions per model and prompt language. Average rows report unweighted means across the six models. Values in bold indicate the best results, whilst those underlined indicate the second-best.

Model	English Prompt (F1)	Chinese Prompt (F1)
GPT-5.5	87.12 $\pm$ 0.66	87.83 $\pm$ 0.18
Claude Opus 4.8	79.94 $\pm$ 1.81	83.17 $\pm$ 2.41
Gemini 3.1 Pro	88.03 $\pm$ 0.36	88.41 $\pm$ 0.49
DeepSeek-V4-Pro	79.66 $\pm$ 0.48	78.91 $\pm$ 0.54
Qwen3.7-Max	82.38 $\pm$ 0.26	84.80 $\pm$ 0.42
GLM-5.2	82.27 $\pm$ 0.48	84.40 $\pm$ 0.23

Table 3: Equivalent Selection Macro F1 under English and Chinese prompts. Average rows report unweighted means across the six models. Values in bold indicate the best results, whilst those underlined indicate the second-best.

English Prompt
Model	Macro F1	Harmful F1
GPT-5.5	76.23	60.71
Claude Opus 4.8	71.09	49.47
Gemini 3.1 Pro	80.45	66.17
DeepSeek-V4-Pro	75.36	58.94
Qwen3.7-Max	74.00	53.94

Table 4: Harmfulness Detection Macro F1 and harmful-class F1 under English and Chinese prompts. Average rows report unweighted means across the six models. Values in bold indicate the best results, whilst those underlined indicate the second-best.

Zero-shot Prompting
Model	Prompt	ME Human	ES F1	HD F1
GPT-5.5	en	3.13	87.62 $\pm$ 1.13	78.83
GPT-5.5	zh	3.03	88.19 $\pm$ 0.40	77.20
Gemini 3.1 Pro	en	4.10	88.39 $\pm$ 0.73	84.13
Gemini 3.1 Pro	zh	3.70	88.24 $\pm$ 0.62	84.98
Few-shot Prompting

Table 5: Zero-shot prompting, few-shot prompting, and LoRA fine-tuning results. ME: Meaning Explanation; ES: Equivalent Selection; HD: Harmfulness Detection. Values in bold indicate the best results within each setting, whilst those underlined indicate the second-best.

Category	Entries	LLM Score
Stylistic device	792	3.34
Quotation	602	3.53
Experience	592	4.00
Slang	481	4.19
Homophonic pun	398	3.32
Abbreviation	136	3.81

Table 6: Full-benchmark category-level Meaning Explanation scores averaged across models and prompt languages.

Figure 3: Category-level error rates for Equivalent Selection and Harmfulness Detection, averaged across evaluated models and prompt languages.

Figure 4: Average distribution of wrong answers by distractor type in Equivalent Selection. Percentages are computed over incorrect predictions and averaged across evaluated models and option-shuffle seeds.

Figure 5: Case study comparing gold annotations and representative model predictions across the three tasks.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Prompt templates used for the three CIBuzzBench tasks. [TERM] denotes the Chinese internet buzzword and [OPTIONS] denotes the four English options in Equivalent Selection.

Figure 7: LLM judge prompt used for semantic-equivalence scoring in Meaning Explanation.

Model	English Prompt	Chinese Prompt
GPT-5.5	0.7739 [0.6883, 0.8070]	0.7906 [0.7179, 0.8527]
Claude Opus 4.8	0.7212 [0.6902, 0.7465]	0.7347 [0.6817, 0.7849]
Gemini 3.1 Pro	0.7447 [0.7097, 0.7889]	0.6845 [0.6596, 0.7307]
DeepSeek-V4-Pro	0.7284 [0.6584, 0.8062]	0.6883 [0.6515, 0.7420]
Qwen3.7-Max	0.7252 [0.6859, 0.7510]	0.7690 [0.7038, 0.8277]
GLM-5.2	0.8527 [0.7930, 0.9022]	0.7814 [0.7125, 0.8356]

Appendix

Table 7: Quadratic weighted kappa between human and LLM-judge scores for Meaning Explanation. Brackets report 95% confidence intervals from 10,000 paired bootstrap resamples. The Average row reports the mean QWK across models, with intervals computed from the bootstrap distribution of the model mean.

English Prompt
Model	BLEU	ROUGE-L	BERT-F1	Human	LLM
Qwen3-4B	0.73	12.09	86.46	1.33	1.90
Qwen3-8B	1.86	17.00	87.35	1.84	2.16
Qwen3-14B	1.47	15.77	86.65	1.97	2.20
Chinese Prompt
Qwen3-4B	1.08	14.97	86.78	1.35	1.92

Appendix

Table 8: Qwen model-size results for Meaning Explanation on the full benchmark.

Model	English Prompt (F1)	Chinese Prompt (F1)
Qwen3-4B	45.18 $\pm$ 0.47	45.41 $\pm$ 0.39
Qwen3-8B	54.59 $\pm$ 0.40	54.91 $\pm$ 0.43
Qwen3-14B	56.68 $\pm$ 0.51	56.98 $\pm$ 0.63

Appendix

Table 9: Qwen model-size results for Equivalent Selection Macro F1 on the full benchmark. Values are mean

\pm

standard deviation over five option-shuffle seeds (1111, 2222, 3333, 4444, 5555).

English Prompt
Model	Macro F1	Harmful F1
Qwen3-4B	28.77	25.90
Qwen3-8B	58.62	36.51
Qwen3-14B	59.09	37.52
Chinese Prompt
Qwen3-4B	22.83	26.04

Appendix

Table 10: Qwen model-size results for Harmfulness Detection Macro F1 and harmful-class F1 on the full benchmark.

Split	Entries
Train	2,401
Test	600

Appendix

Table 11: Train–test split for few-shot prompting and LoRA fine-tuning.

Few-shot, English Prompt
Model	BLEU	ROUGE-L	BERT-F1	Human	LLM
GPT-5.5	4.91	23.66	88.75	4.02	3.76
Gemini 3.1 Pro	3.08	20.51	88.00	4.21	4.16
Few-shot, Chinese Prompt
GPT-5.5	6.19	26.20	89.29	4.01	3.67
Gemini 3.1 Pro	3.70	21.98	88.44	4.17	4.09

Appendix

Table 12: Detailed Meaning Explanation results for few-shot prompting and LoRA fine-tuning.

Few-shot Prompting
Model	English Prompt (F1)	Chinese Prompt (F1)
GPT-5.5	88.98 $\pm$ 1.39	89.45 $\pm$ 0.85
Gemini 3.1 Pro	89.89 $\pm$ 0.32	90.18 $\pm$ 0.65
LoRA Fine-tuning
Qwen3-8B	86.74 $\pm$ 0.45	86.55 $\pm$ 1.00
GLM-4-9B	86.15 $\pm$ 0.82	84.81 $\pm$ 1.06

Appendix

Table 13: Detailed Equivalent Selection Macro F1 for few-shot prompting and LoRA fine-tuning. Values are mean

\pm

standard deviation over five option-shuffle seeds (1111, 2222, 3333, 4444, 5555).

Few-shot, English Prompt
Model	Macro F1	Harmful F1
GPT-5.5	79.78	64.68
Gemini 3.1 Pro	84.82	72.19
Few-shot, Chinese Prompt
GPT-5.5	78.47	56.76
Gemini 3.1 Pro	84.78	70.52

Appendix

Table 14: Detailed Harmfulness Detection Macro F1 and harmful-class F1 for few-shot prompting and LoRA fine-tuning.

English Prompt
Model	Abbrev.	Exper.	Homoph.	Quota.	Slang	Stylistic
GPT-5.5	3.90	4.00	3.36	3.52	4.26	3.38
Claude Opus 4.8	3.54	3.90	2.73	3.36	4.28	3.18
Gemini 3.1 Pro	4.51	4.40	4.03	3.98	4.44	3.78
DeepSeek-V4-Pro	3.71	3.93	3.11	3.44	4.14	3.27
Qwen3.7-Max	4.16	4.29	3.78	3.81	4.34	3.56

Appendix

Table 15: Full-benchmark category-level LLM-judge scores for Meaning Explanation. Scores range from 0 to 5, with higher values indicating better semantic and pragmatic equivalence. Exper., Homoph., and Quota. denote Experience, Homophonic pun, and Quotation.

Figure 8: Category-level error rates for Equivalent Selection and Harmfulness Detection under all evaluated models and English and Chinese prompts. The darker the colour, the higher the error rate.

Figure 9: Distribution of wrong answers over controlled distractor types for all evaluated models under English and Chinese prompts in Equivalent Selection.

Figure 10: Equivalent Selection case studies under Chinese prompts across the six CIBuzzBench categories, showing the gold equivalent, controlled distractors, and representative model predictions.

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

Oct 6, 2025

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

As Large Language Models (LLMs) develop stronger multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly important. Prior work by Naous et al. (2024) has shown that LLMs often favor Western-associated entities in Arabic. Due to the lack of entity-centric multilingual benchmarks, it remains unclear if such biases also manifest in various non-Western languages. In this paper, we introduce Camellia, a benchmark for evaluating entity-centric cultural biases in nine Asian languages, spanning six Asian cultures. Camellia includes 19,530 manually annotated entities associated with the covered Asian or Western cultures, as well as 2,173 masked contexts for these entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLMs across three tasks: cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show that LLMs struggle with cultural adaptation across these languages, with performance differing across models developed in different regions. We further observe that different LLM families can hold distinct biases, reflected in the ways they link cultures to particular sentiments. Lastly, we find that LLMs can struggle with context understanding in some Asian languages, creating performance gaps between cultures in entity extraction.

Tarek Naous, Anagha Savit, Carlos Rafael Catalan +17

Georgia Institute of Technology · Samsung R&D Institute Philippines · Sungkyunkwan University +6

Multilingual Language Model Evaluation Cultural Bias in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models Cultural Knowledge in Language Models

Ask PDF

May 28, 2026

Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese

When Large Language Models (LLMs) are deployed in Chinese-language settings, a troubling pattern emerges: safety systems that work well in English break down. These systems struggle to cross linguistic and cultural bound-aries, leaving models exposed to adversarial prompts that exploit Chinese-specific evasion techniques, including Pinyin romanization, character decomposition, internet slang, and hedging tone. To address this gap, we introduce ChiSafe-PAS (Chinese Safety Pilot Annotation Set), a human-annotated benchmark of 1,897 adversarial Chinese prompts spanning four high-stakes domains: self-harm and violence, drug and illicit trade, fraud, and satire. Of these, 1,544 entries carry complete gold-standard annotations: a 3-class response label (REFUSE, SAFE-REDIRECT, RESPOND), a nine-category obfuscation taxonomy, a risk-level rating, and annotator rationale. We describe the dataset design, annotation process, and obfuscation taxonomy in detail. Our primary goal is practical: to give the research community a high-quality, culturally grounded resource for benchmarking LLM safety alignment. In doing so, we engage three broader tensions in the field: the blurring boundary between training and evaluation data, the need for domain coverage grounded in real-world risk, and the limits of scale as a substitute for cultural expertise.

Wajdi Zaghouani, Kholoud K. Aldous, Yicheng Gao

Northwestern University in Qatar

Multilingual Language Model Evaluation LLM Evaluation Language Model Safety Evaluation Adversarial Attacks on LLMs

Ask PDF

Apr 17, 2026

Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language

While large language models (LLMs) have achieved remarkable success in general language tasks, their performance on Chouxiang Language, a representative subcultural language in the Chinese internet context, remains largely unexplored. In this paper, we introduce Mouse, a specialized benchmark designed to evaluate the capabilities of LLMs on NLP tasks involving Chouxiang Language across six tasks. Experimental results show that, current state-of-the-art (SOTA) LLMs exhibit clear limitations on multiple tasks, while performing well on tasks that involve contextual semantic understanding. In addition, we further discuss the reasons behind the generally low performance of SOTA LLMs on Chouxiang Language, examine whether the LLM-as-a-judge approach adopted for translation tasks aligns with human judgments and values, and analyze the key factors that influence Chouxiang translation. Our study aims to promote further research in the NLP community on multicultural integration and the dynamics of evolving internet languages. Our code and data are publicly available.

Dianqing Lin, Tian Lan, Jiali Zhu +7

College of Computer Science, Inner Mongolia University, China

LLM Evaluation

Ask PDF

Abstract

Cross-Cultural Language Model Evaluation

Explore similar work

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese

Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language