Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

Keunhyeung Park; Seunguk Yu; YoungBin Kim

cs.CLAug 29, 2026

Detecting and Guiding LLM-Generated Korean Poetry with Interpretable Form-level Features

Authors: Keunhyeung Park, Seunguk Yu, YoungBin Kim

Organizations: Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University

Abstract

LLMs often struggle with modern Korean poetry, producing outputs that resemble "line-broken prose." We address two coupled tasks: detecting whether a Korean poem is human- or LLM-authored, and guiding LLMs to generate poetry closer in form to human writing. We quantify the human-LLM gap along four form-level linguistic dimensions: output length (Volume), the diversity and connective use of line-final forms (Structure Variation), the irregularity of line lengths (Rhythmic Irregularity), and adherence to standard orthography (Normative Adherence). We operationalize these dimensions as five interpretable features. For detection, a logistic regression classifier over these five features attains an average AUC-ROC of 83.60 in zero-shot out-of-distribution detection across seven unseen LLMs, versus 75.84 for the strongest baseline in our comparison, KatFishNet, an absolute gain of 7.76 AUC points and a 10.23% relative improvement; one generator-specific punctuation pattern outside our taxonomy remains a boundary case. For generation, expert evaluation on GPT-5.2 prefers feature-guided poems over the unconstrained baseline, and analyses across GPT-5.2 and Gemini-3 show that targeted length, rhythm, and ending statistics move toward the human distribution. These results suggest that interpretable, language-specific features can bridge the diagnosis and guidance of LLM-generated poetry.

Figures & tables

Figure 1: Overview of the introduced framework. Documented LLM biases motivate four form-level dimensions: Volume , Structure Variation , Rhythmic Irregularity , and Normative Adherence . We operationalize these as five interpretable features that support both zero-shot OOD detection and feature-guided generation.

Detection Methods	Qwen2-72B	Solar	Llama-3.1-70B	Gemini-3	GPT-5.2	EXAONE-3.5-7.8B	EEVE-10.8B	Average
KatFishNet ( Punctuation )	93.45	62.65	63.22	59.48	58.76	81.38	75.10	70.58
KatFishNet ( Punctuation + Spacing )	95.41	73.78	58.64	64.88	71.31	87.12	79.77	75.84
Ours	79.78	86.39	73.22	82.73	96.42	89.85	76.84	83.60

Table 1: Zero-shot OOD AUC-ROC across target LLMs. Our five features average 83.60, compared with 75.84 for the strongest KatFishNet configuration. Bold marks the best result for each model.

Source	Size	Origin
Human	189	KatFishNet
GPT-4o	189	KatFishNet
Solar	189	KatFishNet
Qwen2-72B	189	KatFishNet
Llama-3.1-70B	189	KatFishNet
Gemini-3	189	Ours

Table 2: Dataset composition. “KatFishNet” denotes the original benchmark data, while “Ours” denotes newly collected and screened poems.

Target LLM	KatFishNet	Ours	$\Delta$ (%)	$p_{\text{holm}}$	Result
Qwen2-72B	95.41	79.78	-16.38	$<$ 0.001	Base
Solar	73.78	86.39	+17.09	$<$ 0.001	Ours
Llama-3.1-70B	58.64	73.22	+24.87	$<$ 0.001	Ours
Gemini-3	64.88	82.73	+27.51	$<$ 0.001	Ours
GPT-5.2	71.31	96.42	+35.21	$<$ 0.001	Ours
EXAONE-3.5-7.8B	87.12	89.85	+3.13	0.0799	Tie

Table 3: Paired DeLong tests comparing our method with KatFishNet using punctuation and spacing. Holm correction is applied, and our gain is significant on four of seven targets.

Detection Methods	Qwen2-72B	Solar	Llama-3.1-70B	Gemini-3	GPT-5.2	EXAONE-3.5-7.8B	EEVE-10.8B	Average
Ours	79.78	86.39	73.22	82.73	96.42	89.85	76.84	83.60
− Volume	80.48	85.87	68.55	80.62	95.58	89.32	81.10	83.07
− Rhythmic Irregularity	78.73	77.87	71.02	81.91	97.15	84.21	69.56	80.06
− Normative Adherence	75.19	87.01	65.18	76.04	74.87	82.70	77.22	76.89
− Structure Variation	67.22	74.12	67.64	65.55	78.40	80.54	69.56	71.86

Table 4: AUC-ROC after removing each linguistic dimension. Every removal lowers the mean score. Bold marks the best result for each model.

Table 6

Train Source	Ours	KatFishNet
Qwen2-72B	81.96	68.90
Solar	79.53	72.66
Llama-3.1-70B	75.77	72.50
Gemini-3	83.67	70.73
GPT-5.2	73.73	69.35
EXAONE-3.5-7.8B	83.13	66.16

Table 7: Average cross-source AUC-ROC by training source. Each row trains on one LLM and evaluates on all remaining targets.

Category	E1	E2	E3	E4	E5	Avg
Baseline	31	32	16	26	26	26.2
Refined	46	42	49	45	43	45.0
Human	43	46	55	49	51	48.8

Table 8: Aggregated Borda scores from five evaluators over 40 triplets. Baseline and Refined are GPT-5.2 outputs. First-, second-, and third-ranked poems receive 2, 1, and 0 points, with a maximum of 80 per evaluator. Refined ranks above Baseline and closer to Human . E denotes evaluator.

Figure 2: Distribution of $f_{\text{burst}}$ for human-authored poems and poems generated under Baseline and Refined aggregated across Gemini-3 and GPT-5.2.

Source	Mean	Std
Baseline	0.357	0.154
Refined	0.248	0.173
Human	0.206	0.186

Table 9: Mean Syntactic Connection

f_{\text{conn}}

across generation conditions. The generated conditions pool Gemini-3 and GPT-5.2 outputs.

Detection Method	Gemini-3			GPT-5.2
	Baseline	Refined	$\Delta$ (%)	Baseline	Refined	$\Delta$ (%)
Log-Likelihood ( Solaiman et al., 2019 )	26.82	15.57	-41.95	59.28	36.71	-38.07
Rank ( Gehrmann et al., 2019 )	13.85	18.47	+33.36	47.32	32.06	-32.25
Entropy ( Gehrmann et al., 2019 )	18.16	12.25	-32.54	43.28	31.29	-27.70
LRR ( Su et al., 2023 )	19.44	16.07	-17.34	34.86	26.96	-22.66
NPR ( Su et al., 2023 )	47.31	37.79	-20.12	81.70	71.16	-12.90

Table 10: Detector AUC-ROC under Baseline and Refined prompts. Most detectors decrease under Refined . Raw AUC values follow each detector’s score orientation, so values below 50 may reflect reversal rather than weak separability. LRR and NPR denote Log-Likelihood Log-Rank Ratio and Normalized Perturbed log-Rank.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Generated	Excluded	Final
Gemini-3	189	0	189
GPT-5.2	189	0	189
EXAONE-3.5-7.8B	189	8	181
EEVE-10.8B	189	12	177

Appendix

Table 11: Per-model screening results for newly generated poems. Exclusions are due to instruction violations, quality degradation, or meta-commentary.

Detection Methods	Qwen2-72B	Solar	Llama-3.1-70B	Gemini-3	GPT-5.2	EXAONE-3.5-7.8B	EEVE-10.8B	Average
KatFishNet ( Punctuation + Spacing )	95.41	73.78	58.64	64.88	71.31	87.12	79.77	75.84
Ours	79.78	86.39	73.22	82.73	96.42	89.85	76.84	83.60
Fusion ( KatFishNet + Ours )	91.99	86.00	71.93	74.64	79.97	91.59	81.22	82.48

Appendix

Table 12: Feature-level fusion of KatFishNet and our features. The KatFishNet and Ours rows reproduce Table 1 .

Dimension Group	Qwen2-72B	Solar	Llama-3.1-70B	Gemini-3	GPT-5.2	EXAONE-3.5-7.8B	EEVE-10.8B	Average
Volume	75.21	72.55	58.13	74.49	73.22	79.33	63.59	70.93
Rhythmic Irregularity	59.33	72.72	62.89	58.64	43.55	70.51	62.50	61.46
Structure Variation	75.58	77.57	59.21	75.16	86.41	73.30	73.03	74.32
Normative Adherence	65.88	48.93	61.64	61.41	96.05	74.12	59.46	66.79
Ours	79.78	86.39	73.22	82.73	96.42	89.85	76.84	83.60

Appendix

Table 13: Zero-shot AUC-ROC under the out-of-distribution protocol when using each dimension group alone, compared to the full Ours dimension set.

Train Source	Qwen2-72B	Solar	Llama-3.1-70B	Gemini-3	GPT-5.2	EXAONE-3.5-7.8B	EEVE-10.8B	GPT-4o	Average
Qwen2-72B	—	80.41	68.55	82.33	98.28	87.70	79.35	77.13	81.96
Solar	77.01	—	65.83	79.67	88.27	86.43	78.54	80.97	79.53
Llama-3.1-70B	69.93	75.08	—	75.14	92.91	80.97	59.74	76.61	75.77
Gemini-3	82.66	86.14	69.10	—	97.92	89.19	79.20	81.45	83.67
GPT-5.2	77.10	78.26	56.39	77.98	—	79.72	75.15	71.54	73.73
EXAONE-3.5-7.8B	82.18	84.63	71.14	83.47	97.82	—	80.72	81.97	83.13

Appendix

Table 14: Cross-source AUC-ROC matrix for our method. Each row uses a different LLM as the training source (paired with human poems), and columns indicate the target LLM at test time. Dashes indicate same-source entries excluded from evaluation.

Train Source	Qwen2-72B	Solar	Llama-3.1-70B	Gemini-3	GPT-5.2	EXAONE-3.5-7.8B	EEVE-10.8B	GPT-4o	Average
Qwen2-72B	—	70.15	54.80	59.72	64.35	81.20	74.60	77.45	68.90
Solar	90.25	—	55.30	60.45	65.80	82.55	75.40	78.90	72.66
Llama-3.1-70B	88.50	68.90	—	58.15	62.78	79.85	73.25	76.10	72.50
Gemini-3	87.15	67.45	53.20	—	61.50	78.40	72.10	75.30	70.73
GPT-5.2	86.30	66.20	52.45	57.80	—	77.15	71.05	74.50	69.35
EXAONE-3.5-7.8B	85.10	65.30	51.70	56.95	60.25	—	70.20	73.65	66.16

Appendix

Table 15: Cross-source AUC-ROC matrix for KatFishNet (Punctuation + Spacing) baseline under the same cross-source protocol. Dashes indicate same-source entries excluded from evaluation.

Temperature	AUC-ROC
0.5	95.63
0.8	98.12
1.0 (default)	96.42
1.2	96.39

Appendix

Table 16: Detection AUC-ROC of our method on GPT-5.2 poems generated at varying decoding temperatures. The classifier is trained on the default GPT-4o source without retraining.

Figure 3: Raw Volume distributions across sources.

Figure 4: Raw Volume density for human-authored poems and LLM generations.

Figure 5: Kernel density of Ending Type Diversity,

f_{\text{div}}

, for human-authored poems and LLM generations.

Figure 6: Burstiness ECDF for human-authored poems and LLM generations.

Figure 7: Spacing Ratio distributions across sources.

Figure 8: Spacing Ratio density for human-authored poems and LLM generations.

Generation prompt	Detector AUC-ROC
Baseline , no guidance	96.42
Refined , original	84.28
Refined , paraphrase 1	79.46
Refined , paraphrase 2	85.00

Appendix

Table 17: Detector AUC-ROC for Baseline , the original Refined prompt, and two instruction-preserving paraphrases on GPT-5.2.

Instruction removed	Target	Full distance	Removed distance
Volume Synchronization	$f_{\text{vol}}$	0.49	18.93
Rhythmic Variation	$f_{\text{burst}}$	0.018	0.017
Connective Reduction	$f_{\text{conn}}$	0.017	0.031
Ending Diversity	$f_{\text{div}}$	0.038	0.047
Flexible Spacing	$f_{\text{space}}$	0.074	0.093

Appendix

Table 18: Target-feature distance from the human mean under full and leave-one-instruction-out guidance on GPT-5.2. Four removals increase the distance, while

f_{\text{burst}}

remains unchanged.

Figure 9: Rank-position distributions over 40 expert-evaluation triplets. Each panel represents one evaluator. Cells show the share of each condition assigned to each rank, with higher ranks indicating greater perceived naturalness.

Figure 10: Raw Volume distributions across the Human , Baseline , and Refined conditions.

Figure 11: Raw Volume density across the Human , Baseline , and Refined conditions.

Figure 12: Kernel density of Ending Type Diversity

f_{\text{div}}

across the Human , Baseline , and Refined .

Figure 13: Spacing Ratio distributions across the Human , Baseline , and Refined conditions.

Figure 14: Spacing Ratio density across the Human , Baseline , and Refined conditions.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 167 papers

Apr 4, 2026

POEMetric: The Last Stanza of Humanity

Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs. Data and codes are released at https://github.com/Bingru-Li/POEMetric.

Bingru Li, Han Wang, Hazel Wilkinson

Department of Linguistics and Communication University of Birmingham · Department of Information Engineering and Computer Science University of Trento · Department of English Literature Institute of Data and AI (IDAI) University of Birmingham

Language Model Generation Evaluation Generative AI Evaluation

Ask PDF

May 21, 2026

Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs

Previous detection studies have shown that LLMs cannot be effectively used as detectors, but these studies have not addressed modern Chinese poetry. Moreover, no relevant research has explored the performance of LLMs in detecting modern Chinese poetry. This paper evaluates and enhances the performance of LLMs as detectors for modern Chinese poetry, and proposes an image-semantic guided poetry detection method. Compared with traditional detection approaches, our method innovatively incorporates images that reflect the content of the poetry. Through example-driven approaches, our method effectively integrates information such as meaning, imagery, and feeling from the image, then forms a complementary judgment with the poem text. Experimental results demonstrate that the LLM detectors based on our method outperform baseline detectors based on plain text, and even surpass the best-performing traditional detector, RoBERTa. The Gemini detector using our method achieves a Macro-F1 score of 85.65%, reaching the state-of-the-art level. The performance improvements of different LLM detectors on multiple LLMs-generated data prove the effectiveness of our method.

Shanshan Wang, Fengying Ye, Hanjia Lyu +6

Department of Computer and Information Science, University of Macau · University of Rochester · Sichuan University +1

Vision-Language Models AI-Generated Text Detection

Ask PDF

Jul 28, 2026

Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study

With the advancement of AI technologies, Generative AI (GenAI) and human written text have become nearly indistinguishable. Additionally, the global standardization of AI chatbots made academic malpractice more frequent. Furthermore, existing research indicates GenAI poems are the most difficult to distinguish even without any modification thus, GenAI poems are naturally deemed human-like by modern detectors. However, the objectivity of such dissertations needs to be verified against modern detection tools but the subjectivity of poetry and the black-box nature of the modern LLMs (Large Language Models) architectures made verification of such work quite complicated. Hence, the main objective of the research is to deduce the attributes of English poetry that contribute classification and misclassification of both human and AI poems and provide corroborating or contradicting evidence to the poetry distinguishability claim. For such characterizations, we propose a Zero-shot detection pipeline with a dataset consisting of both human and AI poems to verify the distinguishability of human and AI creation and extract the aforementioned crucial attributes for accurate classification. Extraction of such attributes provides benefits in two ways: firstly, it reduces the margin of training needed as only the poems based on misclassifying attributes need to be trained and fine tuned and finally provides a critical insight to the GenAI detection dilemma to strengthen the modern detection pipelines.

A. N. Biswas, T. Tabassum, A. A. Shohid +4

Department of Computer Science and Engineering BRAC University

AI-Generated Text Detection Generative AI Evaluation

Ask PDF

Abstract

AI-Generated Text Detection

Explore similar work

POEMetric: The Last Stanza of Humanity

Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs

Characterizing Human-Likeness in AI Generated Poetry: A Zero-shot Classification Study