Evaluating LLMs on Chinese Topic Constructions: A Research Proposal Inspired by Tian et al. (2024)

Xiaodong Yang

cs.CLApr 21, 2025

Evaluating LLMs on Chinese Topic Constructions: A Research Proposal Inspired by Tian et al. (2024)

Authors: Xiaodong Yang

Organizations: College of Foreign Languages Zhejiang University of Technology 288, Liuhe Rd., Xihu District, Hangzhou, Zhejiang, 310023 China

Abstract

This paper proposes a framework for evaluating large language models (LLMs) on Chinese topic constructions, focusing on their sensitivity to island constraints. Drawing inspiration from Tian et al. (2024), we outline an experimental design for testing LLMs' grammatical knowledge of Mandarin syntax. While no experiments have been conducted yet, this proposal aims to provide a foundation for future studies and invites feedback on the methodology.

Explore similar work

Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language
Apr 17, 2026Dianqing Lin, Tian Lan, Jiali Zhu +7Natural Language Processing
Evaluating Chinese Ambiguity Understanding in Large Language Models
May 15, 2026Junwen Mo, Yuanzhi Lu, Yifang Xue +2Large Language Model Performance
Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay
May 27, 2026Mariah Al Giptiah Binte Yusoff, Jakin Tan, Bocheng Chen +2Large Language Model Performance Linguistics

Apr 17, 2026 · cs.CLJ/K move · Enter open · S save

Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language

Dianqing Lin, Tian Lan, Jiali Zhu, Jiang Li+6

College of Computer Science, Inner Mongolia University, China

While large language models (LLMs) have achieved remarkable success in general language tasks, their performance on Chouxiang Language, a representative subcultural language in the Chinese internet context, remains largely unexplored. In this paper, we introduce Mouse, a specialized benchmark designed to evaluate the capabilities of LLMs on NLP tasks involving Chouxiang Language across six tasks. Experimental results show that, current state-of-the-art (SOTA) LLMs exhibit clear limitations on multiple tasks, while performing well on tasks that involve contextual semantic understanding. In addition, we further discuss the reasons behind the generally low performance of SOTA LLMs on Chouxiang Language, examine whether the LLM-as-a-judge approach adopted for translation tasks aligns with human judgments and values, and analyze the key factors that influence Chouxiang translation. Our study aims to promote further research in the NLP community on multicultural integration and the dynamics of evolving internet languages. Our code and data are publicly available.

Natural Language Processing

Open paper Ask about it PDF

Abstract

Explore similar work

Exploring the Capability Boundaries of LLMs in Mastering of Chinese Chouxiang Language

Evaluating Chinese Ambiguity Understanding in Large Language Models

Can Large Language Models Handle Discourse Particles? A Case Study of Colloquial Malay