CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords

Yifan Wang; Junyu Lu; Qifan Wang; Shun Zhang; Chaozhuo Li; Jiahao Liu; Zhijun Cao; Lingbin Bu

cs.CLSep 18, 2026

CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords

Authors: Yifan Wang, Junyu Lu, Qifan Wang, Shun Zhang, Chaozhuo Li, Jiahao Liu, Zhijun Cao, Lingbin Bu,

Organizations: People’s Public Security University of China · Dalian University of Technology · Meta AI · Beijing University of Posts and Telecommunications ·

Abstract

Chinese social media has generated a vast and continually evolving lexicon of internet buzzwords whose meanings are often non-literal and deeply rooted in local cultural and pragmatic contexts. Existing research has primarily focused on interpreting these buzzwords within Chinese, leaving largely unexplored whether LLMs can transfer such culturally grounded knowledge across languages and accurately convey the intended meanings in English. This cross-lingual capability is also critical for safety, as harmful expressions may obscure their offensive content through culture-specific homophony, euphemism, irony, or coded language. In this paper, we investigate the ability of advanced LLMs to understand Chinese internet buzzwords across languages. To this end, we introduce CIBuzzBench, the first benchmark for cross-lingual Chinese-to-English understanding of Chinese internet buzzwords. CIBuzzBench comprises 3,001 Chinese internet buzzwords annotated with English meaning explanations, English equivalents, category labels, and harmfulness labels. Based on these annotations, we design three evaluation tasks: Meaning Explanation, Cross-lingual Equivalent Matching, and Culturally Grounded Harmfulness Detection. We evaluate representative state-of-the-art proprietary and Chinese LLMs under both English- and Chinese-prompting settings. Our results show that LLMs continue to struggle with the cross-lingual understanding of Chinese internet buzzwords, particularly in fine-grained non-literal interpretation, robust equivalent matching under option perturbations, and calibrated harmfulness detection. These findings highlight the persistent challenges posed by culturally grounded language phenomena for multilingual LLMs and safety-oriented evaluation. The dataset and code are available at https://github.com/SuperYFan/CIBuzzBench.

Figures & tables

Figure 1: Illustration of CIBuzzBench tasks with two examples. Each Chinese internet buzzword is paired with an English meaning explanation, an equivalent-selection question, and a harmfulness label.

Figure 2: Construction process of CIBuzzBench. In the data collection stage, we collect Chinese internet buzzwords and Chinese explanations from Chinese buzzword websites and assign category labels. In the task annotation stage, GPT-5.5 assists in generating annotations for the three tasks, which are then checked by task-specific reviewers; rejected cases are re-analyzed and re-annotated, while accepted cases are confirmed by the group and finalized as task datasets.

Category
Label	Count	Percent
Stylistic device	792	26.4%
Quotation	602	20.1%
Experience	592	19.7%
Slang	481	16.0%
Homophonic pun	398	13.3%

Table 1: Label distribution in CIBuzzBench. The benchmark contains 3,001 Chinese internet buzzwords with category and harmfulness annotations.

English Prompt
Model	BLEU	ROUGE-L	BERT-F1	Human	LLM
GPT-5.5	4.01	22.33	88.35	3.23	3.69
Claude Opus 4.8	2.94	19.42	87.07	3.09	3.49
Gemini 3.1 Pro	3.22	19.95	87.72	4.10	4.12
DeepSeek-V4-Pro	3.05	19.73	87.60	3.11	3.57
Qwen3.7-Max	4.02	20.20	87.98	3.50	3.94

Table 2: Meaning Explanation results under English and Chinese prompts. LLM-judge scores use the full benchmark, while Human scores use 200 sampled predictions per model and prompt language. Average rows report unweighted means across the six models. Values in bold indicate the best results, whilst those underlined indicate the second-best.

Model	English Prompt (F1)	Chinese Prompt (F1)
GPT-5.5	87.12 $\pm$ 0.66	87.83 $\pm$ 0.18
Claude Opus 4.8	79.94 $\pm$ 1.81	83.17 $\pm$ 2.41
Gemini 3.1 Pro	88.03 $\pm$ 0.36	88.41 $\pm$ 0.49
DeepSeek-V4-Pro	79.66 $\pm$ 0.48	78.91 $\pm$ 0.54
Qwen3.7-Max	82.38 $\pm$ 0.26	84.80 $\pm$ 0.42
GLM-5.2	82.27 $\pm$ 0.48	84.40 $\pm$ 0.23

Table 3: Equivalent Selection Macro F1 under English and Chinese prompts. Average rows report unweighted means across the six models. Values in bold indicate the best results, whilst those underlined indicate the second-best.

English Prompt
Model	Macro F1	Harmful F1
GPT-5.5	76.23	60.71
Claude Opus 4.8	71.09	49.47
Gemini 3.1 Pro	80.45	66.17
DeepSeek-V4-Pro	75.36	58.94
Qwen3.7-Max	74.00	53.94

Table 4: Harmfulness Detection Macro F1 and harmful-class F1 under English and Chinese prompts. Average rows report unweighted means across the six models. Values in bold indicate the best results, whilst those underlined indicate the second-best.

Zero-shot Prompting
Model	Prompt	ME Human	ES F1	HD F1
GPT-5.5	en	3.13	87.62 $\pm$ 1.13	78.83
GPT-5.5	zh	3.03	88.19 $\pm$ 0.40	77.20
Gemini 3.1 Pro	en	4.10	88.39 $\pm$ 0.73	84.13
Gemini 3.1 Pro	zh	3.70	88.24 $\pm$ 0.62	84.98
Few-shot Prompting

Table 5: Zero-shot prompting, few-shot prompting, and LoRA fine-tuning results. ME: Meaning Explanation; ES: Equivalent Selection; HD: Harmfulness Detection. Values in bold indicate the best results within each setting, whilst those underlined indicate the second-best.

Category	Entries	LLM Score
Stylistic device	792	3.34
Quotation	602	3.53
Experience	592	4.00
Slang	481	4.19
Homophonic pun	398	3.32
Abbreviation	136	3.81

Table 6: Full-benchmark category-level Meaning Explanation scores averaged across models and prompt languages.

Figure 3: Category-level error rates for Equivalent Selection and Harmfulness Detection, averaged across evaluated models and prompt languages.

Figure 4: Average distribution of wrong answers by distractor type in Equivalent Selection. Percentages are computed over incorrect predictions and averaged across evaluated models and option-shuffle seeds.

Figure 5: Case study comparing gold annotations and representative model predictions across the three tasks.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Prompt templates used for the three CIBuzzBench tasks. [TERM] denotes the Chinese internet buzzword and [OPTIONS] denotes the four English options in Equivalent Selection.

Figure 7: LLM judge prompt used for semantic-equivalence scoring in Meaning Explanation.

Model	English Prompt	Chinese Prompt
GPT-5.5	0.7739 [0.6883, 0.8070]	0.7906 [0.7179, 0.8527]
Claude Opus 4.8	0.7212 [0.6902, 0.7465]	0.7347 [0.6817, 0.7849]
Gemini 3.1 Pro	0.7447 [0.7097, 0.7889]	0.6845 [0.6596, 0.7307]
DeepSeek-V4-Pro	0.7284 [0.6584, 0.8062]	0.6883 [0.6515, 0.7420]
Qwen3.7-Max	0.7252 [0.6859, 0.7510]	0.7690 [0.7038, 0.8277]
GLM-5.2	0.8527 [0.7930, 0.9022]	0.7814 [0.7125, 0.8356]

Appendix

Table 7: Quadratic weighted kappa between human and LLM-judge scores for Meaning Explanation. Brackets report 95% confidence intervals from 10,000 paired bootstrap resamples. The Average row reports the mean QWK across models, with intervals computed from the bootstrap distribution of the model mean.

English Prompt
Model	BLEU	ROUGE-L	BERT-F1	Human	LLM
Qwen3-4B	0.73	12.09	86.46	1.33	1.90
Qwen3-8B	1.86	17.00	87.35	1.84	2.16
Qwen3-14B	1.47	15.77	86.65	1.97	2.20
Chinese Prompt
Qwen3-4B	1.08	14.97	86.78	1.35	1.92

Appendix

Table 8: Qwen model-size results for Meaning Explanation on the full benchmark.

Model	English Prompt (F1)	Chinese Prompt (F1)
Qwen3-4B	45.18 $\pm$ 0.47	45.41 $\pm$ 0.39
Qwen3-8B	54.59 $\pm$ 0.40	54.91 $\pm$ 0.43
Qwen3-14B	56.68 $\pm$ 0.51	56.98 $\pm$ 0.63

Appendix

Table 9: Qwen model-size results for Equivalent Selection Macro F1 on the full benchmark. Values are mean

\pm

standard deviation over five option-shuffle seeds (1111, 2222, 3333, 4444, 5555).

English Prompt
Model	Macro F1	Harmful F1
Qwen3-4B	28.77	25.90
Qwen3-8B	58.62	36.51
Qwen3-14B	59.09	37.52
Chinese Prompt
Qwen3-4B	22.83	26.04

Appendix

Table 10: Qwen model-size results for Harmfulness Detection Macro F1 and harmful-class F1 on the full benchmark.

Split	Entries
Train	2,401
Test	600

Appendix

Table 11: Train–test split for few-shot prompting and LoRA fine-tuning.

Few-shot, English Prompt
Model	BLEU	ROUGE-L	BERT-F1	Human	LLM
GPT-5.5	4.91	23.66	88.75	4.02	3.76
Gemini 3.1 Pro	3.08	20.51	88.00	4.21	4.16
Few-shot, Chinese Prompt
GPT-5.5	6.19	26.20	89.29	4.01	3.67
Gemini 3.1 Pro	3.70	21.98	88.44	4.17	4.09

Appendix

Table 12: Detailed Meaning Explanation results for few-shot prompting and LoRA fine-tuning.

Few-shot Prompting
Model	English Prompt (F1)	Chinese Prompt (F1)
GPT-5.5	88.98 $\pm$ 1.39	89.45 $\pm$ 0.85
Gemini 3.1 Pro	89.89 $\pm$ 0.32	90.18 $\pm$ 0.65
LoRA Fine-tuning
Qwen3-8B	86.74 $\pm$ 0.45	86.55 $\pm$ 1.00
GLM-4-9B	86.15 $\pm$ 0.82	84.81 $\pm$ 1.06

Appendix

Table 13: Detailed Equivalent Selection Macro F1 for few-shot prompting and LoRA fine-tuning. Values are mean

\pm

standard deviation over five option-shuffle seeds (1111, 2222, 3333, 4444, 5555).

Few-shot, English Prompt
Model	Macro F1	Harmful F1
GPT-5.5	79.78	64.68
Gemini 3.1 Pro	84.82	72.19
Few-shot, Chinese Prompt
GPT-5.5	78.47	56.76
Gemini 3.1 Pro	84.78	70.52

Appendix

Table 14: Detailed Harmfulness Detection Macro F1 and harmful-class F1 for few-shot prompting and LoRA fine-tuning.

English Prompt
Model	Abbrev.	Exper.	Homoph.	Quota.	Slang	Stylistic
GPT-5.5	3.90	4.00	3.36	3.52	4.26	3.38
Claude Opus 4.8	3.54	3.90	2.73	3.36	4.28	3.18
Gemini 3.1 Pro	4.51	4.40	4.03	3.98	4.44	3.78
DeepSeek-V4-Pro	3.71	3.93	3.11	3.44	4.14	3.27
Qwen3.7-Max	4.16	4.29	3.78	3.81	4.34	3.56

Appendix

Table 15: Full-benchmark category-level LLM-judge scores for Meaning Explanation. Scores range from 0 to 5, with higher values indicating better semantic and pragmatic equivalence. Exper., Homoph., and Quota. denote Experience, Homophonic pun, and Quotation.

Figure 8: Category-level error rates for Equivalent Selection and Harmfulness Detection under all evaluated models and English and Chinese prompts. The darker the colour, the higher the error rate.

Figure 9: Distribution of wrong answers over controlled distractor types for all evaluated models under English and Chinese prompts in Equivalent Selection.

Figure 10: Equivalent Selection case studies under Chinese prompts across the six CIBuzzBench categories, showing the gold equivalent, controlled distractors, and representative model predictions.

Cross-Cultural Language Model Evaluation

Cross-cultural language model evaluation tests whether AI systems understand and respond appropriately across languages, communities, and cultural contexts.

See all 121 papers

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages
Oct 6, 2025Tarek Naous, Anagha Savit, Carlos Rafael Catalan +17Multilingual Language Model Evaluation Cultural Bias in Language Models
Georgia Institute of Technology · Samsung R&D Institute Philippines · Sungkyunkwan University +6
Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese
May 28, 2026Wajdi Zaghouani, Kholoud K. Aldous, Yicheng GaoMultilingual Language Model Evaluation LLM Evaluation
Northwestern University in Qatar
Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language
Apr 17, 2026Dianqing Lin, Tian Lan, Jiali Zhu +7LLM Evaluation
College of Computer Science, Inner Mongolia University, China

Oct 6, 2025J/K move · Enter open · S save

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

Tarek Naous, Anagha Savit, Carlos Rafael Catalan, Geyang Guo+16

Georgia Institute of Technology · Samsung R&D Institute Philippines · Sungkyunkwan University+6

Multilingual Language Model Evaluation Cultural Bias in Language Models Language Model Bias Evaluation Linguistic Bias in Language Models Cultural Knowledge in Language Models

Open paper AskPDF arXiv

As Large Language Models (LLMs) develop stronger multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly important. Prior work by Naous et al. (2024) has shown that LLMs often favor Western-associated entities in Arabic. Due to the lack of entity-centric multilingual benchmarks, it remains unclear if such biases also manifest in various non-Western languages. In this paper, we introduce Camellia, a benchmark for evaluating entity-centric cultural biases in nine Asian languages, spanning six Asian cultures. Camellia includes 19,530 manually annotated entities associated with the covered Asian or Western cultures, as well as 2,173 masked contexts for these entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLMs across three tasks: cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show that LLMs struggle with cultural adaptation across these languages, with performance differing across models developed in different regions. We further observe that different LLM families can hold distinct biases, reflected in the ways they link cultures to particular sentiments. Lastly, we find that LLMs can struggle with context understanding in some Asian languages, creating performance gaps between cultures in entity extraction.

Abstract

Cross-Cultural Language Model Evaluation

Explore similar work

Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages

Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese

Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language