Honeyed Words, Hidden Daggers: Benchmarking and Training LLMs to See Through Mock Politeness

Caixin Kang; Sitong Gong; Tianyu Yan; Kaipeng Zhang; Yoichi Sato; Yifei Huang

cs.CLOct 19, 2026

Honeyed Words, Hidden Daggers: Benchmarking and Training LLMs to See Through Mock Politeness

Authors: Caixin Kang, Sitong Gong, Tianyu Yan, Kaipeng Zhang, Yoichi Sato, Yifei Huang

Organizations: The University of Tokyo · Alaya Lab

Abstract

The Chinese idiom "kou mi fu jian" ("honeyed words, hidden daggers") names speech that is polite in form but hostile in function. We present ZhenhuanBench, a 1,592-item benchmark built from expert commentary on Empresses in the Palace, covering speaker intent, emotion-language mismatch, and per-listener uptake. Text-only evaluation covers 34 LLMs. GPT-6 Astra leads the leaderboard at 73.48% overall, while its listener-reaction score is 58.37%. The annotations also supply supervised traces and string-verifiable training rewards. With 338 gold traces, an 8B model outperforms teacher distillation on identical items. Subsequent RL saturates, and chance correction reduces the estimated recoverable performance gap by a factor of 7.5. Our CPR-GRPO loop (Compress-Probe-Replenish) uses this estimate to weight gold supervision, reaching 67.0 on public multiple-choice tasks and 74.5 under the training-matched hidden protocol. Without target-task training, CPR outperforms Base, SFT, and GRPO across six Chinese and English pragmatic and sarcasm evaluations. On three external datasets, training with CPR yields higher mean Macro-F1 than SFT and GRPO.

Figures & tables

Figure 1: Overview of ZhenhuanBench and CPR-GRPO. (1) 47 hours of expert commentary become 1,506 structured annotations and four tasks, with 81.4 percent of candidate items rejected by quality gates. (2) A sample scene with multi-listener context translated into derived tasks T1–T4. (3) Training that reuses annotations as rewards and traces through Compress, Probe, and Replenish iterations (§ 5.4 ).

Open-weight models
Model	All	T1	T2	T3	T4	Model	All	T1	T2	T3	T4
Kimi K3	66.8	72.4	67.9	53.4	77.4	Qwen3-8B (base)	34.2	41.4	34.2	26.9	32.1
DeepSeek V4 Pro	62.5	76.2	58.8	49.4	66.0	GLM-4.6V-Flash	33.2	33.9	33.8	28.8	38.7
Kimi-K2.6	62.0	66.1	65.4	48.2	71.7	Qwen3.5-4B	32.9	38.5	35.8	15.8	47.2
Gemma-4-31B	59.6	64.4	62.9	42.0	75.5	InternVL3-8B	31.8	29.7	32.5	26.0	46.2
DeepSeek-V4-Flash	56.8	67.8	52.1	45.3	65.1	GLM-4.1V-9B	31.1	30.5	35.8	25.8	32.1

Table 1: Public-set leaderboard (percent): 33 models and a trained 8B reference. Qwen3.8-Max judges the models’ free-text T3 responses. All is the mean score over all 791 items. Within each group, models rank by All down the left block, then the right. Best scores across all models are bold.

Figure 2: ZhenhuanBench composition (1,592 items). Inner ring, the three splits (training and hidden episode-disjoint); outer ring, tasks T1–T4 shaded dark to light. Full counts in Appendix A .

Model	Consist.	Per-align. mean	None right
Kimi-K2.6	42.9	65.5	13.5
Gemma-4-31B	39.7	63.7	14.1
Sonnet-4.6	36.5	60.4	16.7
GPT-5	27.6	53.2	21.2
Haiku-4.5	19.9	42.2	34.6
Qwen3-8B (base)	12.2	35.4	41.0

Table 2: Schema-enabled diagnostics over 156 alignments with at least two MCQ items (T1/T2/T4; percent). “Consist.” counts alignments with all sibling MCQs correct, “None right” with all wrong. The shaded row is our trained 8B model (§ 5 ).

Figure 3: Leaderboard-protocol MCQ accuracy (public set). Training raises 8B accuracy from 36.8 to 67.0; the shaded band spans GPT-5 to Kimi-K2.6.

Cold-start variant	Score	Paired $\Delta$
Gold scaffold (full recipe)	62.0	$+25.5$
Answer + intent, no structure	59.3	$+22.7$
Gold, matched 253 items	55.6	$+18.1$
Teacher CoT, same 253 items	49.5	$+10.6$
Shuffled content	40.8	$+2.8$

Table 3: SFT-stage attribution on the hidden set (percent; paired

\Delta

against the base of the same evaluation session). Content drives the gain; gold beats teacher CoT on identical items.

Figure 4: Pure GRPO plateaus near 70 on the hidden set (shaded band); adding Replenish (orange, two seeds) reaches 74.5 at round 4.

Figure 5: After Replenish, correct-answer rates are highest among recoverable errors (orange) and lower among low-recovery errors (gray); most previously correct answers are retained (blue).

Dataset	Base	SFT	GRPO	CPR
MultiPragEval (Zh)	65.78	64.00	65.78	66.33
MultiPragEval (En)	65.56	65.89	65.78	67.11
ToSarcasm	42.79	46.68	45.93	47.73
without topic	45.90	51.34	50.84	51.89
iSarcasmEval A	28.07	28.96	30.88	31.40
iSarcasmEval C	82.00	83.50	85.50	86.00

Table 4: Transfer without target-data training. Scores are accuracy, except ToSarcasm (Macro-F1) and iSarcasmEval A (sarcastic-class F1). Details in Appendix G.2 .

Method	FGV	CPED-B	CPED-N	SemEval
Base	17.77	23.97	23.97	19.05
SFT	32.09	28.75	31.61	38.80
GRPO	38.19	29.07	32.48	43.16
Wrong-only	32.59	28.19	37.36	41.17
CPR	40.52	32.10	39.98	47.26

Table 5: External retraining: mean Macro-F1 over three seeds for trained methods; Base uses one fixed checkpoint. B/N: balanced/natural CPED sampling. Details in Appendix G.3 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Size	Tasks	ToM	Listener	Train.
SwordsmanImp	200	1	–	–	–
Mask of Civility	100	1	–	–	–
PUB (En)	28k	14	partial	–	–
EmotionQueen (En)	10k	4	–	–	–
ZhenhuanBench	1,592	4	✓	✓	✓

Appendix

Table A.1: Pragmatics benchmarks. “Listener” marks per-listener uptake structure; “Train.” marks annotations designed to double as training signal.

Split	T1	T2	T3	T4	Total
Public	239	240	206	106	791
Hidden test	92	82	103	42	319
Training	140	158	120	64	482
All	471	480	429	212	1,592

Appendix

Table A.2: ZhenhuanBench items by task and split. Aligning 76 episodes (18,297 utterances, 1,214 scenes) with the 110-part expert commentary (47 hours) yields 1,506 alignments. The 68 podcast transcripts and 1.26M-character novel provide cross-source corroboration. Training and hidden episodes are disjoint.

Stage	Count
Aligned annotations	1,506
flagged by audit pass (repaired)	292 (269)
Candidate items (two assembly passes)	8,544
rejected: option/gold length balance	4,471
rejected: options-only probe †	1,816
rejected: anti-leakage and consistency	665

Appendix

Table A.3: Construction funnel. † still solved from options alone after two rewrites. Anti-leakage rejections include circular evidence and future-tense narration.

Prompt condition	Base 8B	CPR-GRPO	Gemma-31B
Dialogue only	44.0	52.8	62.0
+ Full scene (main)	40.7	59.3	65.7
$-$ cast block	41.2	58.3	66.2
+ Knowledge (defn.)	29.2	43.5	61.1
+ Oracle commentary	60.6	75.5	85.2

Appendix

Table B.1: Text ablations on the hidden set (percent). The injected definition lowers all three scores; expert commentary yields the highest accuracy for each model.

Figure C.1: The CPR-GRPO loop. Compress runs GRPO, Probe estimates chance-corrected recovery, and Replenish weights gold traces using those estimates.

Model	Frames	T	TV	VC	TV $-$ VC
Qwen3-VL-8B	comm.	46.7	51.0	51.0	$+0.0$
Qwen3-VL-8B	epis.	40.5	44.6	43.5	$+1.2$
Qwen3-VL-8B	crop	39.9	44.6	43.5	$+1.1$
Sonnet-4.6	epis.	58.3	60.7	65.5	$-4.8^{*}$
Sonnet-4.6	crop	58.3	62.5	62.5	$+0.0$
Gemini-2.5-Pro	epis.	67.9	70.8	74.4	$-3.6$

Appendix

Table E.1: Vision probe (percent; paired). ∗ 95% CI

[-8.9,-0.6]

; all other TV

-

VC differences include zero.

Model	Q-T3	S-T3	Q-All	S-All	Model	Q-T3	S-T3	Q-All	S-All
GPT-6 Astra	58.37	61.08	73.48	74.19	Qwen3-32B	34.43	36.65	44.11	44.69
Gemini 3.7 Flash	53.28	54.37	72.28	72.57	Qwen3.5-9B	31.07	31.03	41.59	41.58
Kimi K3	53.44	55.83	66.76	67.38	Qwen3-14B	30.91	32.40	39.15	39.54
DeepSeek V4 Pro	49.39	52.99	62.55	63.49	GPT-5-mini	35.68	35.88	38.12	38.17
Qwen3.8-Max	49.84	49.47	62.54	62.44	Qwen3-VL-8B	31.96	36.29	37.40	38.53
Kimi-K2.6	48.18	46.20	61.98	61.46	Qwen3-8B	26.94	30.99	34.20	35.25

Appendix

Table F.1: Cross-judge scores on identical responses (percent). Q: Qwen3.8-Max; S: Sonnet-4.6. MCQ scores are fixed. Both judges use the same rubric with thinking disabled.

Cold start	SFT	+GRPO	Observation
Full recipe	62.0/61.1	66.7/66.7	baseline
Interface only	56.9/55.6	60.2/58.8	GRPO gain
Content, no iface	1.9/3.7	5.6/25.0	Parse failures
None (base)	—	40.3	No gain

Appendix

Table F.2: Interface-alignment two-by-two on the hidden set, two seeds. Interface-only SFT enables GRPO gains in both seeds; unparseable gold prose yields low scores.

Figure F.1: AGVR SFT data efficiency on the hidden set. Accuracy crosses the base band (mean and SD of 62 evaluations) between 96 and 168 traces and saturates near 62.5.

Figure F.2: Naive pass@

k

(solid) vs. chance-corrected coverage (dashed). Chance-corrected coverage approaches zero at

k{=}16

Control	Score	Factor tested
GA-GRPO (gold in groups)	65.3	re-exposure
Round-5 pure iteration	69.4	more compute
RFT (rejection sampling)	63.0	self-distill
DPO ( $\beta{=}0.05$ )	53.7	preference opt.
More SFT (5 epochs)	61.6	more gradient
Ans-only vs. 3-head	65.1/66.2	reward richness

Appendix

Table F.3: Controls on the hidden set. Vanilla GRPO from the same start scores 66.7; CPR-GRPO reaches 71.8/74.5 in two seeds.

Model	Off	Low	High/Med.	Max/Xhigh
DeepSeek V4 Pro	58.36	61.13	65.53	68.44
Qwen3.8-Max	63.43	63.42	61.60	62.31
Qwen3.8-27B	51.57	54.64	54.09	52.13

Appendix

Table G.1: Overall score across reasoning settings, with Qwen3.8-Max judging T3. DeepSeek uses off/low/high/max; Qwen uses off/low/medium/xhigh, with a 16K thinking budget for Qwen-Max medium.

Dataset	Warm	Pool	Dev	Test
FGVIrony	1,024	2,048	958	1,973
CPED (each regime)	456	912	1,687	27,438
SemEval Task 3B	192	384	767	784

Appendix

Table G.2: External training and evaluation sizes. Warm and Pool are disjoint supervised-start and subsequent training sets.

Model	Orig.	A	B	Mean	$\Delta$
Base	36.22	32.09	32.87	32.48	$-3.74$
CPR	63.39	64.17	63.98	64.07	$+0.69$
DeepSeek V4 Pro	61.81	56.69	59.84	58.27	$-3.54$
Qwen3.8-Max	67.91	58.07	58.27	58.17	$-9.74$
Qwen3.8-27B	56.50	52.36	52.95	52.66	$-3.84$

Appendix

Table G.3: Accuracy under two character-renaming maps on the same 508 items. Changes are percentage points.

Pragmatic Reasoning in Language Models

Pragmatic reasoning in language models studies how models infer and communicate meanings shaped by context, intent, shared knowledge, and social norms.

See all 98 papers

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments
Sep 3, 2026Shiwei Hong, Junjie Ma, Emma Jiren Wang +5Pragmatic Reasoning in Language Models LLM Evaluation
George Mason University · Virginia Tech · University of Toronto +2
Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms
Sep 24, 2026Rong Wang, Kun Sun, Yadong GuoLLM Evaluation Pragmatic Reasoning in Language Models
University of Tübingen · Tongji University
No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus
Apr 17, 2026Hitesh Mehta, Arjit Saxena, Garima Chhikara +1Multilingual Language Model Evaluation LLM Evaluation
Department of Computer Science Engineering, Delhi Technological University, New Delhi, India · Department of Electronics and Communication Engineering, Delhi Technological University, New Delhi, India

Sep 3, 2026J/K move · Enter open · S save

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments

Shiwei Hong, Junjie Ma, Emma Jiren Wang, Ethan Z. Rong+4

George Mason University · Virginia Tech · University of Toronto+2

Pragmatic Reasoning in Language Models LLM Evaluation Social Media Analysis

Open paper AskPDF arXiv

Chinese online comments often convey social meaning through indirect and playful language that is hard to interpret without context. Existing evaluations largely organize items around predefined phenomena or controlled pragmatic categories, leaving open whether models can distinguish plausible readings of what a naturally occurring comment is doing in a particular exchange. We introduce a benchmark for evaluating whether LLMs can recover such situated pragmatic meanings. From more than 200,000 public Chinese social media interaction records, we construct 4,735 human-validated diagnostic items, each pairing a target comment with reconstructed preceding context and plausible misreadings. We evaluate eight LLMs as both question writers and solvers in a cross-writer setting. The task is challenging: the strongest model achieves 81.42% leave-writer-out accuracy. Across all eight models, the mean leave-writer-out accuracy is 68.70% while human accuracy was 90.8%. Case analysis shows that models often recognize broad irony or playfulness while misidentifying the mechanism or interactional move.

Abstract

Pragmatic Reasoning in Language Models

Explore similar work

You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Comments

Polite but Misaligned: Evaluating LLM Politeness Judgments Against Human Pragmatic Norms

No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus