cs.CLAug 28, 2026
SaveCNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms
Organizations: The University of Tokyo · Xiaohongshu Inc. · Tsinghua University
Abstract
Chinese neologisms exploit diverse and unique linguistic mechanisms, such as phonetic substitution (e.g., 886 for ``bye-bye'') and visual character decomposition that are rare in other languages. We introduce CNeo-Bench, a benchmark of 4,759 such neologisms with reference definitions, organized into five top-level categories and nine subcategories by the linguistic mechanism behind each expression. CNeo-Bench is paired with a two-tier evaluation framework that separates whether a model can describe a neologism from whether it can operate on its underlying mechanism. Evaluating 18 LLMs, we find that Chinese neologisms remain an open challenge; most models fall below 40% on definition generation, and on several subcategories a systematic recognition-manipulation gap emerges: models describe neologisms correctly but, in source-form restoration tasks, substitute a semantic equivalent (paraphrase) for the source form rather than producing the source form itself. A few-shot analysis on 1,058 hard items shows that in-context examples can solve many difficult cases, but leave a noticeable portion of errors remaining, indicating challenges beyond prompting alone can address.
Figures & tables
Figure 1: Categories and tasks in CNeo-Bench .
| Subcategory | Full | Hard | Hard Rate |
| Pinyin Abbr. | 192 | 35 | 18.23% |
| English Abbr. | 58 | 7 | 12.07% |
| Chinese Homo. | 377 | 65 | 17.24% |
| Number Homo. | 29 | 7 | 24.14% |
| Lexical Neo. | 1161 | 298 | 25.67% |
| Semantic Neo. | 850 | 185 | 21.76% |
Table 1: Statistics for CNeo-Bench . “Full” is the total number of items in each subcategory; “Hard” is the number of items on which GPT-5.1, DeepSeek-V3.2, and Kimi-K2.5 jointly fail under definition generation.
| Model | Alphabetic | Homophonic | Lexical | Stylistics | Foreign | Avg. ∗ | ||||
| Pinyin | English | Chinese | Number | Lexical | Semantic | Unconv. Char. | Formulaic | |||
| GPT-5.1 | 47.40 | 74.14 | 48.81 | 44.83 | 49.10 | 54.94 | 21.74 | 52.67 | 42.54 | 50.89 |
| DeepSeek-V3 | 64.06 | 74.14 | 55.44 | 72.41 | 51.51 | 54.59 | 65.22 | 54.35 | 45.27 | 53.81 |
| DeepSeek-V3.2 | 58.85 | 79.31 | 60.21 | 72.41 | 54.61 | 55.65 | 69.57 | 54.17 | 49.00 | 55.26 |
| Kimi-K2-Instruct | 66.15 | 70.69 | 61.27 | 72.41 | 58.14 | 64.47 | 65.22 | 62.57 | 53.48 | 61.27 |
| Kimi-K2.5 | 76.56 | 79.31 | 70.82 | 72.41 | 66.06 | 67.06 | 82.61 | 67.55 | 61.69 | 67.74 |
Table 2: Full definition generation task results. Scores are reported as accuracy (%). Bold indicates the best score in each column; underline indicates the second best. Avg. ∗ is computed as micro-average across all 4,759 instances.
| Model | Alphabetic | Homophonic | Lexical | Stylistics | Foreign | Avg. ∗ | ||||
| Pinyin | English | Chinese † | Number † | Lexical | Semantic | Unconv. Char. † | Formulaic | |||
| GPT-5.1 | 72.40 | 98.28 | 56.76 | 72.41 | 73.47 | 96.46 | 22.73 | 68.77 | 86.36 | 75.70 |
| DeepSeek-V3 | 77.60 | 98.28 | 65.25 | 82.76 | 77.43 | 96.58 | 45.45 | 70.10 | 81.06 | 77.76 |
| DeepSeek-V3.2 | 76.04 | 94.83 | 64.46 | 82.76 | 78.12 | 95.75 | 54.55 | 64.20 | 81.31 | 75.61 |
| Kimi-K2-Instruct | 78.65 | 96.55 | 68.97 | 93.10 | 83.03 | 96.58 | 59.09 | 68.17 | 84.60 | 79.20 |
| Kimi-K2.5 | 80.21 | 96.55 | 68.17 | 79.31 | 82.00 | 97.76 | 59.09 | 75.39 | 88.64 | 81.94 |
Table 3: Category-specific task results. Scores are reported as accuracy (%). Bold indicates the best score in each column; underline indicates the second best. Tasks with † use open-ended restoration; all other tasks use multiple-choice format. Avg. ∗ is computed as micro-average across all 4,759 instances.
Figure 2: Contingency analysis of Tier 1 (definition generation) vs Tier 2 (category-specific) outcomes for five representative models on three open-ended subcategories. Cells A/B/C/D denote (T1 , T2 ), (T1 , T2 ), (T1 , T2 ), (T1 , T2 ) respectively. The highlighted B/(A+B) column reports the operation failure rate (the fraction of correctly described neologisms on which restoration fails) directly quantifying the recognition-manipulation gap.
Figure 3: Few-shot recovery on 1,058 hard samples. Each bar shows the number of hard samples recovered (i.e., judged correct under the few-shot setting) under 1-shot, 2-shot, and 3-shot prompting. Bars are stacked by subcategory to show the distribution of recoveries across the nine subcategories.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Accuracy on English-based vs Pinyin-based alphabetic abbreviations in the definition generation task. English-based abbreviations are substantially easier than Pinyin-based ones for all evaluated models.
| Subcategory | Human | Kimi-K2.5 |
| Pinyin Abbr. | 91.00% | 76.56% |
| English Abbr. | 84.48% | 79.31% |
| Chinese Homo. | 94.00% | 70.82% |
| Number Homo. | 82.76% | 72.41% |
| Lexical Neo. | 87.00% | 66.06% |
| Semantic Neo. | 83.00% | 67.06% |
Table 4: Human performance on a stratified sample of CNeo-Bench Tier 1 definition generation. We leave Kimi-K2.5, the strongest model we evaluate, for comparison.
Figure 5: Tier 2 minus Tier 1 accuracy (percentage points) across 18 models and 9 subcategories. Positive gaps (red) indicate Tier 2 exceeds Tier 1; negative gaps (blue) indicate Tier 2 underperforms Tier 1. Subcategories marked with † use open-ended restoration; others use multiple-choice.
| Categories | Subcategories | Explanations | Examples |
| 1. Alphabetic Abbreviations | Pinyin Abbr. | Expressions formed by taking the initial letters of the Pinyin transliteration of Chinese phrases to create abbreviated forms. | yyds from “永远的神” (yong yuan de shen), which means forever a legend. |
| English Abbr. | Expressions formed by taking the initial letters of the English phrases to create abbreviated forms. | ACG from “Anime, Comics, Games”. | |
| 2. Homophonic Expressions | Number Homo. | Expressions that use numbers to convey meaning based on phonetic similarity. | “拜拜咯” becomes “886” as they have similar pronunciations in Chinese, which means bye bye. |
| Chinese Homo. | Expressions that replace standard Chinese characters with characters of similar pronunciation. | “与你无关” (yu ni wu guan) becomes “雨女无瓜” (yu nv wu gua), which means none of your business. | |
| 3. Lexical Innovations | Lexical Neo. | Newly coined expressions that introduce novel lexical items. | “键盘侠”, which means keyboard warrior, a person who aggressively criticizes or attacks others online without engaging in real-world action. |
| Semantic Neo. | Existing words that acquire new meanings in emerging usage. | “养鱼”, literal meaning: fish farming; new meaning: to maintain multiple romantic prospects simultaneously, often by keeping others as backups. |
Table 5: Mechanism-driven taxonomy of Chinese neologisms in CNeo-Bench , with five top-level categories and nine subcategories. ∗ Foreign Borrowings has no further subcategorization and is treated as a single class.
| Subcategory | Task | Example | Answer |
| Pinyin Abbr. | Cloze | Context : 这家楼下新开的烧烤摊味道真的太绝了，朋友吃完直接说它是夜宵界的()。 English : The new BBQ stall that just opened downstairs is absolutely amazing, my friend straight-up called it the () of the late-night food scene. Options : (A) cxk (B) xswl (C) yysy (D) yyds | (D) yyds (“forever legendary”) |
| English Abbr. | Cloze | Context : 他平时看番、追漫画还爱玩手游，朋友圈里一看就是个()重度爱好者。 English : He watches anime, follows manga, and loves playing mobile games, and you can tell he’s a hardcore () enthusiast from his social media. Options : (A) KPI (B) ACG (C) PDF (D) CPU | (B) ACG (Anime, Comics, Games) |
| Chinese Homo. † | Restoration | Neologism : 雨女无瓜 Example context : 这件事是他们部门内部的安排，雨女无瓜，别再到处打听了。 English : This is an internal arrangement within their department, none of your business, so stop asking around about it. | 与你无关 |
| Number Homo. † | Restoration | Neologism : 886 Example context : "时间不早了, 我先撤, 886, 明天见。 English : It’s getting late, I’m gonna head out. Bye-bye, see you tomorrow! | 拜拜咯 |
| Lexical Neo. | Cloze | Context : 事情刚上热搜、真相还没出来，他就在评论区里对当事人冷嘲热讽、指点江山，活脱脱一个()。 English : The story had just started trending and the truth hadn’t even come out yet, but there he was in the comments section mocking the people involved and pontificating about everything, a textbook (). Options : (A) 程序员 (B) 黑客 (C) 杠精 (D) 键盘侠 | (D) 键盘侠 (keyboard warrior) |
| Semantic Neo. | Semantic Discrimination | Target word : 养鱼 Sentences : (A) 他在院子里挖了个小池子，专门养鱼和种睡莲。(fish farming) (B) 爷爷退休后最大的乐趣就是养鱼，每天一早都去喂食。(fish farming) (C) 她总是同时和好几个人保持暧昧，朋友都说她是在养鱼。 (D) 这家农场除了养鸡养鸭，还养鱼，逢年过节卖得特别好。(fish farming) | (C) maintaining multiple romantic relationships |
Table 6: Representative examples for each Tier 2 tasks in CNeo-Bench . Subcategories marked with † require the model to complete open-ended restoration. English references are not included in our dataset.
| Model | Subcategory | Cell B Failure Mode (%) | |||
| Copy Verbatim | Explanation/Paraphrase | Wrong Restoration | Empty/Other | ||
| Kimi-K2.5 | Chinese Homo. | 26.6% | 38.0% | 32.9% | 2.5% |
| Number Homo. | 0.0% | 60.0% | 40.0% | 0.0% | |
| Unconv. Chars. | 0.0% | 71.4% | 28.6% | 0.0% | |
| DeepSeek-V3.2 | Chinese Homo. | 25.5% | 35.3% | 35.3% | 3.9% |
| Number Homo. | 0.0% | 100% | 0.0% | 0.0% | |
Table 7: Breakdown of Cell B failure modes (Tier 1 correct, Tier 2 wrong) across five models and three open-ended subcategories. Each row sums to 100%. Copy Verbatim : model outputs the neologism itself. Explanation/Paraphrase : model outputs a semantic gloss or near-synonym of the source form. Wrong Restoration : model outputs a different Chinese expression that is neither the neologism nor a paraphrase. Empty/Other : empty output or unparseable.
| Subcategory | Model | Neologism | Gold | Tier 1 Model Output (correct) | Tier 2 Output (wrong) |
| Chinese Homo. | Kimi-K2.5 | v我50 | 微我50 | 网络流行语，指通过微信转账50元人民币。源自肯德基“疯狂星期四”促销活动… Referring to transferring 50 yuan RMB via WeChat. It originated from KFC’s ‘Crazy Thursday’ promotional campaign. | 微信我50 (additional characters) |
| Chinese Homo. | Kimi-K2.5 | 石乐志 | 失了智 | 网络流行语，“失了智”的谐音，指失去理智、做出愚蠢或不理智的行为… A homophonic pun on “失了智” (lost one’s wits), meaning to lose one’s reason and act in a stupid or irrational way. | 失智 (dementia) |
| Chinese Homo. | DeepSeek-V3.2 | 河蟹 | 和谐 | 网络用语中常被用作“删除”或“屏蔽”的委婉说法，因“和谐”与“河蟹”谐音，后者常被用来讽刺内容审查。 Often used as a euphemism for ‘deleting’ or ‘blocking’ content. Because 和谐 (harmony) is a homophone of 河蟹 (river crab). | 删除 (delete) |
| Chinese Homo. | DeepSeek-V3.2 | 大清药丸 | 大清要完 | 大清药丸是网络流行语，源自“大清要完”的谐音，实际用来调侃某事物或局势即将崩溃或失败，带有讽刺或幽默意味。 A homophonic pun on ‘大清要完’ (the Qing dynasty is finished). It’s used to jokingly suggest that something or some situation is on the verge of collapse or failure, with a satirical or humorous undertone. | 大清朝要完 (additional characters) |
| Number Homo. | Kimi-K2.5 | 3q | thank you | 英文“thank you”的谐音数字缩写，“3”对应“th”音、“q”对应“you”音。 ‘3Q’ is a homophonic abbreviation of the English ‘thank you’, using numbers and letters: ‘3’ represents the ‘th’ sound and ‘q’ represents the ‘you’ sound. | 谢谢 (Chinese expression for thank you) |
| Number Homo. | DeepSeek-V3.2 | 666 | 溜溜溜 | 表示非常厉害、令人惊叹的意思，源自数字6的谐音“溜”。 ‘666’ means something is awesome or impressive. It derives from the number 6, a homophone of ‘溜’ (‘slick/skilled’) | 牛牛牛 (semantically equivalent) |
Table 8: Representative Cell B cases (Tier 1 correct, Tier 2 wrong) across the three open-ended subcategories. In each case, the model correctly describes the neologism in Tier 1 but, in Tier 2, substitutes a semantic equivalent for the source form rather than producing the source form itself (Explanation/Paraphrase), which is the dominant failure mode across all three subcategories (Table 7 ). English references are provided below the generated definition.
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,427 papers