Not X, but When: Diagnosing a Mandarin LLM Writing Tic through Contextual Appropriateness

Yihang Peng; Yi Jing; Dewen Liu; Weihan Meng; Ying Liu

cs.CLOct 15, 2026

Not X, but When: Diagnosing a Mandarin LLM Writing Tic through Contextual Appropriateness

Authors: Yihang Peng, Yi Jing, Dewen Liu, Weihan Meng, Ying Liu

Organizations: Tsinghua University · Fudan University

Abstract

Recurring expressions in model-written text are often treated as signatures of machine authorship, yet most of them serve legitimate functions. Whether such an expression is a tic depends on whether the model knows when it is called for. This judgment of appropriateness, central to communicative competence, remains largely unexamined for language models. We study it through the Mandarin corrective construction bushi X, ershi Y ("not X, but Y"). Drawing on Gricean pragmatics and corrective negation, we introduce a decision tree that classifies uses as substantive, optional, or defective by testing whether the context makes X a live alternative and whether rejecting X is informative. Five commercial models use the construction 2.5 to 9.2 times as densely as a human reference. Of 307 annotated uses, only 1.30% are defective while 52.8% are optional, that is, correct but unneeded, so models fall short in appropriateness far more often than in correctness. Several reward models favor the construction even over quality-matched variants, an LLM judge favors it even where the context does not license it, and both supervised fine-tuning and preference optimization tend to raise its frequency in open-weight models. We hope this work encourages the development of models attentive to the appropriateness of their expressions.

Figures & tables

Figure 1: One form, four discourse functions: substantive ( $S$ ), unlicensed ( $O_{L}$ ), redundant ( $O_{U}$ ), and defective ( $D$ ) uses of not X, but Y . Translated excerpts retain the target sentence; ellipses mark omitted context.

Figure 2: Diagnostic tree and experimental framework. (a) Recover the contextual reading, then test defects (T/A/P) and corrective function (R/L/U). (b) Connect generation patterns to evaluator preferences, checkpoint trajectories, and preference-data composition.

Figure 3: Frequency and corrective function. (a) Occurrences per 1,000 punctuation-delimited sentences, per 10,000 Han characters ( $I_{H}$ ), and $\log_{10}(1+I_{H})$ . Rates average equally over four shared tasks; the log is taken after averaging. Human sentence counts are unavailable (NA). (b) Optional-use percentages by task and source; NA denotes no classified cases. All LLMs pools model cases. (c) Functional shares: substantive ( $S$ ), unlicensed ( $O_{L}$ ), redundant ( $O_{U}$ ), relation failure ( $O_{R}$ ), and defective ( $D$ ). Panels (b,c) include 307 selected model occurrences across four benchmarks and 33 eligible human-authored occurrences annotated with the same tree. Their denominators include every classified case; these descriptive samples differ from the full-answer frequency panel in (a).

Figure 4: Evaluator preference and functional sensitivity. A is the original, B reorders the correction, and D retains only Y. (a) RM non-tie A-win rates for optional uses. (b) Substantive-minus-optional reward contrasts, standardized by A-arm score SD. (c) Mean ordinal preference by function, separately for Claude Sonnet 5.5 and GPT-6.1-sol. (d) Their

FS_{G}=\bar{p}_{S}-\bar{p}_{G}

, with

O

pooling

O_{L}

and

O_{U}

. Dashed lines mark neutrality (a,c) or zero sensitivity (b,d). Panels (c,d) use each judge’s complete pairs and shared scales across judges. Error bars are pointwise 95% Wilson intervals (a) or 10,000 item-bootstrap intervals (b–d).

Figure 5: Cohort gap and SFT/DPO trajectories. (a) Twelve open checkpoints from five families, the human reference, and five API models on four shared task dimensions. Points show

LF_{H}

, with

I_{H}

on the upper axis; open points are offset vertically for visibility. (b) Descriptive Base–SFT–DPO trajectories using four-benchmark macro

LF_{C}

. Points use valid answers at each checkpoint; lines connect available stages. Nonempty truncated answers are retained.

	Swimmer’s victory speech	Commentary on Messi’s title
Target	此刻我脑海里闪过的，不是终点触壁的那一瞬间，而是过去这几年的每一个清晨……‘not the moment I touched the wall, but every morning of these past years’	梅西的夺冠，不是因为他终于赢了，而是因为他用二十多年……换来了这一次最完整的救赎。‘Messi’s title is not because he finally won, but because he traded twenty-odd years … for this most complete redemption’
Tree	$S$ : winning makes the finish accessible; injury motivates rejecting it.	$O_{L}$ at L: no one reduces the title to winning; the target introduces X itself.
Scores	RMs 3/3, 3/3; judge 0.75, 0.75	RMs 2/3, 1/3; judge 0.75, 0.625
Reason	‘rejects first, then asserts; more natural, more emotional impact’	‘slightly odd logically but reads smoothly and passionately’

Table 1: Same evaluation, different function. Scores list A–B, then A–D. RMs counts how many of FsfairX, Skywork, and OffsetBias score the original higher; judge scores average two presentation orders (0.5 is a tie). Reasons are quoted from the judge’s A–B output.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task dimension	Han chars.	Count	$I_{H}$	$LF_{H}$
Four shared task dimensions (main comparison)
Professional tasks	98,450	2	0.203	0.080
General question answering	100,020	13	1.30	0.362
Chinese understanding	100,146	11	1.10	0.322
Writing	102,823	12	1.17	0.336
Macro average of four shared dimensions	—	—	0.942	—

Appendix

Table 2: Human corpus reference.

I_{H}

counts occurrences per 10,000 Han characters;

LF_{H}=\log_{10}(1+I_{H})

. The main comparison in the paper uses the macro average of the four shared dimensions (

I_{H}=0.942

). The corpus-pooled estimate over all eight dimensions is

I_{H}=0.664

. Dimensions with zero counts are shown for completeness; they are not used to compute the main comparison baseline.

	Four-benchmark generation panel					Four shared task dimensions
Model	$N$	$A$ (%)	$I$	95% CI for $I$	$LF$	$I_{H}$	Model/Human
GPT	1061	31.6	2.54	[2.15, 2.94]	0.549	4.59	$4.87\times$
DeepSeek	1058	23.5	2.97	[2.49, 3.51]	0.599	8.68	$9.21\times$
Gemini	1061	17.0	1.20	[0.987, 1.41]	0.343	3.41	$3.62\times$
GLM	1046	11.8	1.10	[0.840, 1.38]	0.321	2.38	$2.52\times$
Qwen	1059	34.2	3.13	[2.60, 3.75]	0.616	6.75	$7.17\times$

Appendix

Table 3: Five-model generation panel.

N

is the number of valid responses. Within each benchmark,

A

is the percentage of valid answers containing at least one occurrence, and

I

is occurrences per 10,000 Unicode characters; the reported

A

and

I

average equally over four benchmarks.

LF=\log_{10}(1+I)

. The two rightmost columns use four shared task dimensions:

I_{H}

is macro-average density per 10,000 Han characters, and Model/Human divides model

I_{H}

by the unrounded human baseline (0.942 when rounded; Table 2 ). These ratios correspond to Figure 3 a. Model names abbreviate the logged identifiers in the text.

Outcome / nested subtype	Count	Share (%)
1. Defective ( $D$ )	4	1.30
First positive check: $T\rightarrow A\rightarrow P$
$T$ : Truth conflict	2	0.651
$A$ : Misattribution	2	0.651
$P$ : Pragmatic defect	0	0
2. Optional ( $O$ )	162	52.8

Appendix

Table 4: Functional composition of the 307 annotated cases. S/O/D are top-level outcomes; R/L/U (optional) and T/A/P (defective) are mutually exclusive subcategories assigned in priority order (R

\rightarrow

\rightarrow

U; T

\rightarrow

\rightarrow

P). One cybersecurity-related item was excluded due to an annotation refusal. Labels were generated by an LLM and reviewed by humans; inter-rater agreement is reported in Table 5 . This table describes the current sample (

N=307

) and should not be interpreted as a precise estimate of population proportions.

Label level	$N$	Matches	Agreement	$\kappa$
Top-level ( $S/O/D$ )	100	77	77.0%	0.577
Terminal (7 labels)	100	66	66.0%	0.521

Appendix

Table 5: Inter-annotator agreement on 100 cases. Two independent annotators with linguistics backgrounds labeled all cases. Exact Matches (n) and Exact Agreement (%) report raw agreement.

\kappa

is unweighted Cohen’s

\kappa

. The terminal level includes the seven labels (S, R, L, U, T, A, P). The moderate

\kappa

values reflect the inherent difficulty of fine-grained pragmatic classification, particularly distinguishing lack of contextual licensing (

O_{L}

) from no discourse advancement (

O_{U}

Pair	RM	$N$	W/L/T	Rate
A–B	FsfairX	57	36/10/11	0.783
A–B	Skywork	67	30/11/26	0.732
A–B	OffsetBias	57	38/14/5	0.731
A–B	ArmoRM	57	24/20/13	0.545
A–B	InternLM2	67	16/28/23	0.364
A–D	FsfairX	59	39/18/2	0.684

Appendix

Table 6: General-quality score-matched subset on unchanged scored texts. W/L/T are A wins, losses, and ties; rate is W/(W+L), not W/

N

. Matching requires equality on every applicable general-quality dimension.

Contrast	Group	$n/N$	Mean score	95% CI
A–D	Overall	297/301	0.633	[0.621, 0.646]
A–D	S	138/141	0.632	[0.613, 0.650]
A–D	$O_{L}$	73/74	0.630	[0.603, 0.656]
A–D	$O_{U}$	86/86	0.638	[0.615, 0.661]
A–B	Overall	297/301	0.566	[0.554, 0.577]
A–B	S	139/141	0.567	[0.551, 0.585]

Appendix

Table 7: Claude Sonnet 5.5 rubric-guided pairwise judgments by functional group. Scores average two presentation orders after restoring variant identity; values above 0.5 favor A. Intervals use 10,000 item bootstraps.

n/N

denotes complete pairs/eligible groups. Overall covers the 301 substantive, unlicensed, and redundant cases; the two relation failures and four defective cases are excluded.

Family	Transition	Paired $N$	$\Delta I_{C}$
DeepSeekMath	Base $\to$ SFT	1,028	+0.0154
Llama/Tülu 3	Base $\to$ SFT	1,055	+0.0187
Llama/Tülu 3	SFT $\to$ DPO	1,061	+0.154
OLMo 2	Base $\to$ SFT	1,006	+0.0767
OLMo 2	SFT $\to$ DPO	1,018	+0.124
MiMo	Base $\to$ SFT	1,053	+0.131

Appendix

Table 8: Descriptive parent–child density changes for SFT and DPO.

\Delta I_{C}

is child minus parent density per 10,000 Unicode characters, averaged equally over four benchmarks. Each transition uses common successful prompts and retains nonempty truncated answers. These paired changes need not equal differences between the marginal checkpoint points in Figure 5 b.

Figure 6: Descriptive Base–SFT–DPO trajectories by family. Points show benchmark-macro log-frequency,

LF_{C}=\log_{10}(1+I_{C})

, using valid answers at each checkpoint and retaining nonempty truncated answers. Lines connect available stages; vertical scales differ across panels.

Data	Arm	Pairs	Hits	Instances	$A$ (%)	$A$ per 10k pairs	$\Delta_{\mathrm{rel}}$ vs. rejected
Tülu 3	Chosen	271,958	86	145	0.03162	3.162	+14.7%
Tülu 3	Rejected	271,958	75	91	0.02758	2.758	—
OLMo 2	Chosen	376,910	253	307	0.06712	6.712	+26.5%
OLMo 2	Rejected	376,910	200	272	0.05306	5.306	—

Appendix

Table 9: Public preference data, restricted to complete context-matched chosen/rejected pairs and final assistant responses. Hits count activated answers.

A

per 10k pairs

=10^{4}\times\mathrm{Hits}/\mathrm{Pairs}

\Delta_{\mathrm{rel}}

is the relative increase of chosen over rejected activation, computed from unrounded counts. Chosen answers consistently show higher activation than rejected answers in both datasets, with a larger gap in OLMo 2. These rates are corpus-pooled, not benchmark-macro estimates.

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 103 papers

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments
Sep 3, 2026Shiwei Hong, Junjie Ma, Emma Jiren Wang +5Pragmatic Reasoning in Language Models LLM Evaluation
George Mason University · Virginia Tech · University of Toronto +2
Artificial Epanorthosis: Why large language models overuse a classical rhetorical figure, and how to mitigate it
Jul 23, 2026Federico BoggiaLanguage Model Calibration
This Is a Good Paper, Right? How Tag Questions Affect Sycophancy Across 106 Language Models
Jul 27, 2026Tapan ParikhLLM Evaluation LLM Sycophancy
Cornell Tech

Sep 3, 2026J/K move · Enter open · S save

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments

Shiwei Hong, Junjie Ma, Emma Jiren Wang, Ethan Z. Rong+4

George Mason University · Virginia Tech · University of Toronto+2

Pragmatic Reasoning in Language Models LLM Evaluation Social Media Analysis

Open paper AskPDF arXiv

Chinese online comments often convey social meaning through indirect and playful language that is hard to interpret without context. Existing evaluations largely organize items around predefined phenomena or controlled pragmatic categories, leaving open whether models can distinguish plausible readings of what a naturally occurring comment is doing in a particular exchange. We introduce a benchmark for evaluating whether LLMs can recover such situated pragmatic meanings. From more than 200,000 public Chinese social media interaction records, we construct 4,735 human-validated diagnostic items, each pairing a target comment with reconstructed preceding context and plausible misreadings. We evaluate eight LLMs as both question writers and solvers in a cross-writer setting. The task is challenging: the strongest model achieves 81.42% leave-writer-out accuracy. Across all eight models, the mean leave-writer-out accuracy is 68.70% while human accuracy was 90.8%. Case analysis shows that models often recognize broad irony or playfulness while misidentifying the mechanism or interactional move.

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments

Artificial Epanorthosis: Why large language models overuse a classical rhetorical figure, and how to mitigate it

This Is a Good Paper, Right? How Tag Questions Affect Sycophancy Across 106 Language Models