Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry
Authors: Jiang Li, Tian Lan, Shanshan Wang, Dongxing Zhang, Dianqing Lin, Guanglai Gao, Derek F. Wong, Xiangdong Su
Organizations: College of Computer Science, Inner Mongolia University, China · National & Local Joint Engineering Research Center of Intelligent Information Processing Technology for Mongolian, China · NLP2CT Lab, Department of Computer and Information Science, University of Macau, China
The rapid development of large language models (LLMs) has extended text generation tasks into the literary domain. However, AI-generated literary creations has raised increasingly prominent issues of creative authenticity and ethics in literary world, making the detection of LLM-generated literary texts essential and urgent. While previous works have made significant progress in detecting AI-generated text, it has yet to address classical Chinese poetry. Due to the unique linguistic features of classical Chinese poetry, such as strict metrical regularity, a shared system of poetic imagery, and flexible syntax, distinguishing whether a poem is authored by AI presents a substantial challenge. To address these issues, we introduce ChangAn, a benchmark for detecting LLM-generated classical Chinese poetry that containing total 30,664 poems, 10,276 are human-written poems and 20,388 poems are generated by four popular LLMs. Based on ChangAn, we conducted a systematic evaluation of 12 AI detectors, investigating their performance variations across different text granularities and generation strategies. Our findings highlight the limitations of current Chinese text detectors, which fail to serve as reliable tools for detecting LLM-generated classical Chinese poetry. These results validate the effectiveness and necessity of our proposed ChangAn benchmark. Our dataset and code are available at https://github.com/VelikayaScarlet/ChangAn.
Figures & tables
Figure 1: The overall structure of our proposed benchmark
Author
Num. Poems
Num. Sents
Avg. S/P
Avg. C/S
Total Characters
Human
10,276
105,828
10.30
5.70
603,034
Doubao Seed-1.6
5,136
48,232
9.39
6.13
295,742
DeepSeek-V3.2
5,138
51,858
10.09
5.89
305,285
Kimi-K2
5,017
51,129
10.19
5.92
302,812
GPT-4.1
5,097
50,713
9.95
5.83
295,503
Overall
30,664
307,760
10.04
5.86
1,802,646
Table 1: Statistical distribution of the ChangAn dataset. Abbreviations include Num. Poems (Number of Poems), Num. Sents (Number of Sentences), Avg. S/P (Average Sentences Per Poem), and Avg. C/S (Average Characters Per Sentence).
Detectors
Acc.
Recall (AI)
Recall (Human)
Macro-Recall
Macro-F1
Deepseek-V3.2
39.56
31.79
60.96
46.37
39.42
Kimi-K2
31.98
16.06
75.83
45.94
31.30
Doubao Seed-1.6
33.80
22.08
66.07
44.07
33.96
GPT-4.1
26.02
5.10
83.63
44.37
23.31
LLM-Detector-Small-zh
29.38
3.86
98.74
51.30
25.16
Average
32.15
15.78
77.05
46.41
30.63
Table 2: Overall performance comparison of LLM detector. The metrics include Accuracy (Acc.), Recall for AI-generated and Human-written poetry, and Macro-F1.
Detectors
AUROC
Macro-F1
Log-Likelihood
83.94
73.77
Log-Rank
85.86
75.56
LRR
71.09
64.81
Roberta
95.03
86.18
Zh-v3
72.37
65.44
Zh-v3-short
72.25
65.06
Table 3: Performance comparison of detection methods. Note that "Zh-v3" and "Zh-v3-short" refer to the AIGC-Detector-v3 (Zh) and its short-text variant, respectively.
Deepseek-V3.2
Kimi-K2
Doubao Seed-1.6
GPT-4.1
Overall
Detectors
AUROC
Macro-F1
AUROC
Macro-F1
AUROC
Macro-F1
AUROC
Macro-F1
AUROC
Macro-F1
Log-Likelihood
82.39
75.30
84.62
77.42
84.62
77.22
84.19
76.85
83.94
73.77
Log-Rank
83.55
76.86
86.78
80.06
86.60
79.16
86.60
79.53
85.86
75.56
LRR
72.27
66.56
70.98
65.98
70.89
65.38
70.15
64.89
71.09
64.81
Roberta
93.61
81.69
93.90
81.89
97.95
84.12
94.64
82.52
95.03
86.18
Zh-v3
77.16
69.91
66.11
61.49
69.42
63.74
79.38
71.48
72.37
65.44
Table 4: AUROC and Macro-F1 comparison across LLM generators. Note that "Zh-v3" and "Zh-v3-short" refer to the AIGC-Detector-v3 (Zh) and its short-text variant, respectively.
Figure 2: Cross-model recall performance for poetry detection. (A) denotes Author models; (D) denotes Detector models. Numerical values of Recall (AI) represent the percentage of correctly identified AI samples.
Detectors
Recall (Gene.)
Recall (Crit.)
Doubao Seed-1.6
21.88
22.29 (+0.41)
Deepseek-V3.2
36.72
26.87 ( − 9.85)
Kimi-K2
18.33
13.79 ( − 4.54)
GPT-4.1
4.64
5.56 (+0.92)
LLM-Detector
52.13
50.48 ( − 1.65)
Average
26.74
23.80 ( − 2.94)
Table 5: Performance comparison of decision-based detectors across different generation strategies, we report the recall of AI-genereted poetry. LLM-Detector denotes the LLM-Detector-Small-zh. Values in parentheses denote the recall shift of the latter relative to the former.
Generation
Critique
Detectors
AUROC
Macro-F1
AUROC
Macro-F1
Log-Likelihood
93.33
86.54
74.70 ( − 18.63)
68.61 ( − 17.93)
Log-Rank
94.73
88.53
77.12 ( − 17.61)
70.65 ( − 17.88)
LRR
77.76
71.13
64.53 ( − 13.23)
61.21 ( − 9.92)
Roberta
97.69
87.44
92.38 ( − 5.31)
83.58 ( − 3.86)
Zh-v3
75.62
69.18
70.49 ( − 5.13)
64.88 ( − 4.30)
Table 6: Performance comparison of probability-based detectors across generation strategies. We report the AUROC and Macro-F1 score. Parentheses indicate the decrease in performance from direct generation to critique-driven refinement.
SPD
MPD-6
MPD-12
Detectors
Acc
R-AI
R-Hu
M-F1
Acc
R-AI
R-Hu
M-F1
Acc
R-AI
R-Hu
M-F1
Deepseek-V3.2
39.56
31.79
60.96
39.42
61.75
74.51
26.58
50.54
62.83
74.18
31.53
52.83
Kimi-K2
31.98
16.06
75.83
31.30
50.84
42.81
72.97
50.12
54.92
55.23
54.05
51.61
Doubao Seed-1.6
33.80
22.08
66.07
33.96
55.52
49.35
72.52
54.21
54.20
48.69
69.37
52.79
GPT-4.1
26.02
5.10
83.63
23.31
69.54
92.48
6.31
45.80
71.94
94.12
10.81
50.07
LLM-Detector-S-zh
29.38
3.86
98.74
25.16
42.86
31.57
84.24
42.60
28.73
15.67
85.51
28.65
Table 7: Performance comparison across MPD, SPD-6, and SPD-12 strategies. R-AI and R-Hu denote Recall for AI and Human samples, respectively. M-F1 stands for Macro-F1 score.
SPD
MPD-6
MPD-12
Detectors
AUROC
M-F1
AUROC
M-F1
AUROC
M-F1
Log-Likelihood
83.94
73.77
98.90
95.71
99.96
99.08
Log-Rank
85.86
75.56
99.29
96.97
100.0
99.70
LRR
71.09
64.81
99.63
97.24
100.0
99.70
Roberta
95.03
86.18
99.96
84.89
99.99
84.76
Zh-v3
72.37
65.44
87.98
80.51
89.90
83.43
Table 8: Comparison of detection performance (AUROC and Macro-F1) across different window strategies. M-F1 stands for Macro-F1 score. Zh-v3 refers to AIGC-Detector-v3 (Zh-v3).
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Examples of classical Chinese poetry from human and various LLMs.
Figure 4: Prompts for generation, refinement, and detection
Figure 5: Prompts for direct generation and critique-driven refinement
Figure 6: Semantic Clustering of AI vs. Human Poetry
Figure 7: Visual comparison of poetic imagery distributions. The left column (a, c) shows human compositions, while the right column (b, d) represents AI-generated poetry.
Top 20 High-Frequency Short Imagery Words
Top 20 High-Frequency Long Imagery Words
Rank
Human
Count
Rank
AI
Count
Rank
Human
Count
Rank
AI
Count
1
人间 (World)
466
1
凭栏 (Leaning)
1035
1
三千里 (3k Miles)
22
1
明月照 (Moonlight)
122
2
相思 (Longing)
371
2
万里 (10k Miles)
1030
2
二十年 (20 Years)
19
2
寄相思 (Send Love)
108
3
天涯 (Horizon)
369
3
何处 (Where)
1019
3
今何在 (Where now)
17
3
夜未央 (Deep night)
100
4
当年 (Past days)
311
4
西风 (W. Wind)
1010
4
读书人 (Scholar)
17
4
烛影摇红 (Candle)
76
5
何处 (Where)
310
5
斜阳 (Setting sun)
999
5
明月照 (Moonlight)
14
5
玉箫吹 (Flute)
71
Appendix
Table 9: Top 20 high-frequency poetic imagery (with translations) in human-written vs. AI-generated poetry.
Figure 8: The example of the process of critique-driven refinement.
Human Readers
Acc.
Recall (AI)
Recall (Human)
Macro-Recall
Macro-F1
HR1
47.00
66.67
13.51
40.09
38.59
HR2
51.67
76.19
21.05
48.62
39.57
HR3
52.50
69.44
8.69
39.07
36.04
HR4
53.83
74.28
16.67
45.48
42.91
Appendix
Table 10: The Human Performance on the ChangAn Dataset.
Graduate School of Informatics, Kyoto University · NLP2CT Lab, Department of Computer and Information Science, University of Macau · College of Computer Science, Inner Mongolia University