Not X, but When: Diagnosing a Mandarin LLM Writing Tic through Contextual Appropriateness

Yihang Peng; Yi Jing; Dewen Liu; Weihan Meng; Ying Liu

cs.CLOct 15, 2026

Not X, but When: Diagnosing a Mandarin LLM Writing Tic through Contextual Appropriateness

Authors: Yihang Peng, Yi Jing, Dewen Liu, Weihan Meng, Ying Liu

Organizations: Tsinghua University · Fudan University

Abstract

Recurring expressions in model-written text are often treated as signatures of machine authorship, yet most of them serve legitimate functions. Whether such an expression is a tic depends on whether the model knows when it is called for. This judgment of appropriateness, central to communicative competence, remains largely unexamined for language models. We study it through the Mandarin corrective construction bushi X, ershi Y ("not X, but Y"). Drawing on Gricean pragmatics and corrective negation, we introduce a decision tree that classifies uses as substantive, optional, or defective by testing whether the context makes X a live alternative and whether rejecting X is informative. Five commercial models use the construction 2.5 to 9.2 times as densely as a human reference. Of 307 annotated uses, only 1.30% are defective while 52.8% are optional, that is, correct but unneeded, so models fall short in appropriateness far more often than in correctness. Several reward models favor the construction even over quality-matched variants, an LLM judge favors it even where the context does not license it, and both supervised fine-tuning and preference optimization tend to raise its frequency in open-weight models. We hope this work encourages the development of models attentive to the appropriateness of their expressions.

Figures & tables

Figure 1: One form, four discourse functions: substantive ( $S$ ), unlicensed ( $O_{L}$ ), redundant ( $O_{U}$ ), and defective ( $D$ ) uses of not X, but Y . Translated excerpts retain the target sentence; ellipses mark omitted context.

Figure 2: Diagnostic tree and experimental framework. (a) Recover the contextual reading, then test defects (T/A/P) and corrective function (R/L/U). (b) Connect generation patterns to evaluator preferences, checkpoint trajectories, and preference-data composition.

Figure 3: Frequency and corrective function. (a) Occurrences per 1,000 punctuation-delimited sentences, per 10,000 Han characters ( $I_{H}$ ), and $\log_{10}(1+I_{H})$ . Rates average equally over four shared tasks; the log is taken after averaging. Human sentence counts are unavailable (NA). (b) Optional-use percentages by task and source; NA denotes no classified cases. All LLMs pools model cases. (c) Functional shares: substantive ( $S$ ), unlicensed ( $O_{L}$ ), redundant ( $O_{U}$ ), relation failure ( $O_{R}$ ), and defective ( $D$ ). Panels (b,c) include 307 selected model occurrences across four benchmarks and 33 eligible human-authored occurrences annotated with the same tree. Their denominators include every classified case; these descriptive samples differ from the full-answer frequency panel in (a).

Figure 4: Evaluator preference and functional sensitivity. A is the original, B reorders the correction, and D retains only Y. (a) RM non-tie A-win rates for optional uses. (b) Substantive-minus-optional reward contrasts, standardized by A-arm score SD. (c) Mean ordinal preference by function, separately for Claude Sonnet 5.5 and GPT-6.1-sol. (d) Their

FS_{G}=\bar{p}_{S}-\bar{p}_{G}

, with

O

pooling

O_{L}

and

O_{U}

. Dashed lines mark neutrality (a,c) or zero sensitivity (b,d). Panels (c,d) use each judge’s complete pairs and shared scales across judges. Error bars are pointwise 95% Wilson intervals (a) or 10,000 item-bootstrap intervals (b–d).

Figure 5: Cohort gap and SFT/DPO trajectories. (a) Twelve open checkpoints from five families, the human reference, and five API models on four shared task dimensions. Points show

LF_{H}

, with

I_{H}

on the upper axis; open points are offset vertically for visibility. (b) Descriptive Base–SFT–DPO trajectories using four-benchmark macro

LF_{C}

. Points use valid answers at each checkpoint; lines connect available stages. Nonempty truncated answers are retained.

	Swimmer’s victory speech	Commentary on Messi’s title
Target	此刻我脑海里闪过的，不是终点触壁的那一瞬间，而是过去这几年的每一个清晨……‘not the moment I touched the wall, but every morning of these past years’	梅西的夺冠，不是因为他终于赢了，而是因为他用二十多年……换来了这一次最完整的救赎。‘Messi’s title is not because he finally won, but because he traded twenty-odd years … for this most complete redemption’
Tree	$S$ : winning makes the finish accessible; injury motivates rejecting it.	$O_{L}$ at L: no one reduces the title to winning; the target introduces X itself.
Scores	RMs 3/3, 3/3; judge 0.75, 0.75	RMs 2/3, 1/3; judge 0.75, 0.625
Reason	‘rejects first, then asserts; more natural, more emotional impact’	‘slightly odd logically but reads smoothly and passionately’

Table 1: Same evaluation, different function. Scores list A–B, then A–D. RMs counts how many of FsfairX, Skywork, and OffsetBias score the original higher; judge scores average two presentation orders (0.5 is a tie). Reasons are quoted from the judge’s A–B output.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task dimension	Han chars.	Count	$I_{H}$	$LF_{H}$
Four shared task dimensions (main comparison)
Professional tasks	98,450	2	0.203	0.080
General question answering	100,020	13	1.30	0.362
Chinese understanding	100,146	11	1.10	0.322
Writing	102,823	12	1.17	0.336
Macro average of four shared dimensions	—	—	0.942	—

Appendix

Table 2: Human corpus reference.

I_{H}

counts occurrences per 10,000 Han characters;

LF_{H}=\log_{10}(1+I_{H})

. The main comparison in the paper uses the macro average of the four shared dimensions (

I_{H}=0.942

). The corpus-pooled estimate over all eight dimensions is

I_{H}=0.664

. Dimensions with zero counts are shown for completeness; they are not used to compute the main comparison baseline.

	Four-benchmark generation panel					Four shared task dimensions
Model	$N$	$A$ (%)	$I$	95% CI for $I$	$LF$	$I_{H}$	Model/Human
GPT	1061	31.6	2.54	[2.15, 2.94]	0.549	4.59	$4.87\times$
DeepSeek	1058	23.5	2.97	[2.49, 3.51]	0.599	8.68	$9.21\times$
Gemini	1061	17.0	1.20	[0.987, 1.41]	0.343	3.41	$3.62\times$
GLM	1046	11.8	1.10	[0.840, 1.38]	0.321	2.38	$2.52\times$
Qwen	1059	34.2	3.13	[2.60, 3.75]	0.616	6.75	$7.17\times$

Appendix

Table 3: Five-model generation panel.

N

is the number of valid responses. Within each benchmark,

A

is the percentage of valid answers containing at least one occurrence, and

I

is occurrences per 10,000 Unicode characters; the reported

A

and

I

average equally over four benchmarks.

LF=\log_{10}(1+I)

. The two rightmost columns use four shared task dimensions:

I_{H}

is macro-average density per 10,000 Han characters, and Model/Human divides model

I_{H}

by the unrounded human baseline (0.942 when rounded; Table 2 ). These ratios correspond to Figure 3 a. Model names abbreviate the logged identifiers in the text.

Outcome / nested subtype	Count	Share (%)
1. Defective ( $D$ )	4	1.30
First positive check: $T\rightarrow A\rightarrow P$
$T$ : Truth conflict	2	0.651
$A$ : Misattribution	2	0.651
$P$ : Pragmatic defect	0	0
2. Optional ( $O$ )	162	52.8

Appendix

Table 4: Functional composition of the 307 annotated cases. S/O/D are top-level outcomes; R/L/U (optional) and T/A/P (defective) are mutually exclusive subcategories assigned in priority order (R

\rightarrow

\rightarrow

U; T

\rightarrow

\rightarrow

P). One cybersecurity-related item was excluded due to an annotation refusal. Labels were generated by an LLM and reviewed by humans; inter-rater agreement is reported in Table 5 . This table describes the current sample (

N=307

) and should not be interpreted as a precise estimate of population proportions.

Label level	$N$	Matches	Agreement	$\kappa$
Top-level ( $S/O/D$ )	100	77	77.0%	0.577
Terminal (7 labels)	100	66	66.0%	0.521

Appendix

Table 5: Inter-annotator agreement on 100 cases. Two independent annotators with linguistics backgrounds labeled all cases. Exact Matches (n) and Exact Agreement (%) report raw agreement.

\kappa

is unweighted Cohen’s

\kappa

. The terminal level includes the seven labels (S, R, L, U, T, A, P). The moderate

\kappa

values reflect the inherent difficulty of fine-grained pragmatic classification, particularly distinguishing lack of contextual licensing (

O_{L}

) from no discourse advancement (

O_{U}

Pair	RM	$N$	W/L/T	Rate
A–B	FsfairX	57	36/10/11	0.783
A–B	Skywork	67	30/11/26	0.732
A–B	OffsetBias	57	38/14/5	0.731
A–B	ArmoRM	57	24/20/13	0.545
A–B	InternLM2	67	16/28/23	0.364
A–D	FsfairX	59	39/18/2	0.684

Appendix

Table 6: General-quality score-matched subset on unchanged scored texts. W/L/T are A wins, losses, and ties; rate is W/(W+L), not W/

N

. Matching requires equality on every applicable general-quality dimension.

Contrast	Group	$n/N$	Mean score	95% CI
A–D	Overall	297/301	0.633	[0.621, 0.646]
A–D	S	138/141	0.632	[0.613, 0.650]
A–D	$O_{L}$	73/74	0.630	[0.603, 0.656]
A–D	$O_{U}$	86/86	0.638	[0.615, 0.661]
A–B	Overall	297/301	0.566	[0.554, 0.577]
A–B	S	139/141	0.567	[0.551, 0.585]

Appendix

Table 7: Claude Sonnet 5.5 rubric-guided pairwise judgments by functional group. Scores average two presentation orders after restoring variant identity; values above 0.5 favor A. Intervals use 10,000 item bootstraps.

n/N

denotes complete pairs/eligible groups. Overall covers the 301 substantive, unlicensed, and redundant cases; the two relation failures and four defective cases are excluded.

Family	Transition	Paired $N$	$\Delta I_{C}$
DeepSeekMath	Base $\to$ SFT	1,028	+0.0154
Llama/Tülu 3	Base $\to$ SFT	1,055	+0.0187
Llama/Tülu 3	SFT $\to$ DPO	1,061	+0.154
OLMo 2	Base $\to$ SFT	1,006	+0.0767
OLMo 2	SFT $\to$ DPO	1,018	+0.124
MiMo	Base $\to$ SFT	1,053	+0.131

Appendix

Table 8: Descriptive parent–child density changes for SFT and DPO.

\Delta I_{C}

is child minus parent density per 10,000 Unicode characters, averaged equally over four benchmarks. Each transition uses common successful prompts and retains nonempty truncated answers. These paired changes need not equal differences between the marginal checkpoint points in Figure 5 b.

Figure 6: Descriptive Base–SFT–DPO trajectories by family. Points show benchmark-macro log-frequency,

LF_{C}=\log_{10}(1+I_{C})

, using valid answers at each checkpoint and retaining nonempty truncated answers. Lines connect available stages; vertical scales differ across panels.

Data	Arm	Pairs	Hits	Instances	$A$ (%)	$A$ per 10k pairs	$\Delta_{\mathrm{rel}}$ vs. rejected
Tülu 3	Chosen	271,958	86	145	0.03162	3.162	+14.7%
Tülu 3	Rejected	271,958	75	91	0.02758	2.758	—
OLMo 2	Chosen	376,910	253	307	0.06712	6.712	+26.5%
OLMo 2	Rejected	376,910	200	272	0.05306	5.306	—

Appendix

Table 9: Public preference data, restricted to complete context-matched chosen/rejected pairs and final assistant responses. Hits count activated answers.

A

per 10k pairs

=10^{4}\times\mathrm{Hits}/\mathrm{Pairs}

\Delta_{\mathrm{rel}}

is the relative increase of chosen over rejected activation, computed from unrounded counts. Chosen answers consistently show higher activation than rejected answers in both datasets, with a larger gap in OLMo 2. These rates are corpus-pooled, not benchmark-macro estimates.

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 103 papers

Sep 3, 2026

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments

Chinese online comments often convey social meaning through indirect and playful language that is hard to interpret without context. Existing evaluations largely organize items around predefined phenomena or controlled pragmatic categories, leaving open whether models can distinguish plausible readings of what a naturally occurring comment is doing in a particular exchange. We introduce a benchmark for evaluating whether LLMs can recover such situated pragmatic meanings. From more than 200,000 public Chinese social media interaction records, we construct 4,735 human-validated diagnostic items, each pairing a target comment with reconstructed preceding context and plausible misreadings. We evaluate eight LLMs as both question writers and solvers in a cross-writer setting. The task is challenging: the strongest model achieves 81.42% leave-writer-out accuracy. Across all eight models, the mean leave-writer-out accuracy is 68.70% while human accuracy was 90.8%. Case analysis shows that models often recognize broad irony or playfulness while misidentifying the mechanism or interactional move.

Shiwei Hong, Junjie Ma, Emma Jiren Wang +5

George Mason University · Virginia Tech · University of Toronto +2

Pragmatic Reasoning in Language Models LLM Evaluation Social Media Analysis

Ask PDF

Jul 23, 2026

Artificial Epanorthosis: Why large language models overuse a classical rhetorical figure, and how to mitigate it

A rhetorical figure that Cicero and Quintilian catalogued two thousand years ago reappears, systematically, in the text of large language models: epanorthosis, the self-correction of the specimen «This is not a course. It is a journey of transformation». This essay argues that the overuse is a trained disposition, driven mainly by a training distribution rich in promotional prose and by preference tuning (RLHF) that rewards confident, emphatic phrasing; the left-to-right nature of generation is an amplifier rather than the root cause. Building on evidence that models diverge from human rhetorical style, and on Fontanier's classification of epanorthosis as a figure of thought, it sets out a programme that scores the figure against genre-specific human baselines through an Epanorthosis Index (density relative to the human rate). A first measurement, on three sizes of one instruction-tuned model family, finds mis-calibration by register in both directions: the models overshoot in oratory (about twofold, near threefold in Italian, concentrated in the larger tiers) and undershoot in informal question-and-answer writing, while matching humans in argument, journalism, and encyclopedic prose. Three constructive contributions follow: a survey of mitigation techniques centred on lightweight LoRA adapters; a demonstration, in Italian, that a one-line instruction cuts the figure by half to nearly three-quarters and that a supervised-fine-tuning adapter removes it almost entirely, with a scaling coefficient that dials the reduction back onto the human rate; and the argument that the target is calibration to the human rate for each genre, not elimination. It closes on the stakes: the real risk is that we begin to write like the machines.

Federico Boggia

Language Model Calibration

Ask PDF

Jul 27, 2026

This Is a Good Paper, Right? How Tag Questions Affect Sycophancy Across 106 Language Models

Whether a language model agrees with a user depends on how the user asks. We measure this on 106 models with a grid of three wordings and three cues, on 20 personal decisions, counterbalanced over both options and scored by exact match with no LLM judge. Since a no cannot be told apart from a refusal, we measure only agreement. In every wording, a tentative tag ("Luna is the better choice, maybe?") draws more agreement than a confident one ("..., right?") in nearly every model. The confident tag's own effect depends on model generation and wording. Against the question "Is Luna the better choice?" older models agree more under the confident tag and newer models agree less, in five of six families, and release order within families predicts the change. The resistance is keyed to the construction: a synonym tag reproduces it, and the same preference stated without a tag raises agreement by 24 points, in 105 of 106 models, including those that resist the tag most.

Tapan Parikh

Cornell Tech

LLM Evaluation LLM Sycophancy Language Model Bias Evaluation

Ask PDF

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments

Artificial Epanorthosis: Why large language models overuse a classical rhetorical figure, and how to mitigate it

This Is a Good Paper, Right? How Tag Questions Affect Sycophancy Across 106 Language Models