Beyond Surface Cues: Robust LLM-Generated Korean Text Detection
Authors: SungHo Kim, SangKeun Lee
Organizations: Department of Artificial Intelligence, Korea University, Seoul, South Korea · Department of Computer Science and Engineering, Korea University, Seoul, South Korea
As large language models (LLMs) generate increasingly fluent text, the need for reliable detection of LLM-generated content has grown substantially. However, existing research remains heavily English-centric, leaving Korean largely underexplored. Moreover, current evaluations of Korean text detection often rely on superficial cues, leading to overly optimistic estimates of detection performance. To address these limitations, we introduce KoLing, a bias-controlled adversarial benchmark comprising 50.9k instances that suppresses superficial shortcuts while preserving semantic content and separately evaluates LLM-generated and LLM-modified text detection. We further propose KoLingProbe, an interpretable detector grounded in theoretical linguistics that models Korean-specific properties across multiple linguistic levels. Experiments show that KoLingProbe substantially outperforms strong baselines on all detection tasks while remaining robust to adversarial attacks. Our analyses further reveal that integrating complementary linguistic features across multiple levels yields both improved performance and interpretability.
Figures & tables
Human-Written Text
성공율 이 낮다고 해도 따지고 보면 함께 해결할
문제거든요 . 근데 서로 마음을 알 길이 없 어요 .
LLM-Modified Text (GPT-4o)
성공률 이 낮더라도 함께 해결할 문제라고 생각
합니다 . 그러나 마음을 알 수 없 습니다 .
English Translated: ‘Even if the success rate is low, it is a problem to solve together if you think about it. But there’s no way to know how they really feel.’
Table 1: A human–LLM sentence pair with similar syntactic patterns. Colors indicate distinct linguistic features captured by KoLingProbe : Phonological (phonological alternation), Morphological (connective ending), Semantic (idiom), and Pragmatic (discourse marker; polite ending) features.
Figure 1: Comparison of dataset biases between KatFish and KoLing . (a) Length distributions; (b) Human vs. LLM class balance; (c) Paired human–LLM sample ratio per genre; (d) Shortcut classifier accuracy.
Figure 2: Surface variation (bars) and semantic preservation (lines) across KoLing modification levels.
Figure 3: Overview of KoLingProbe : five independent linguistic sub-detectors each produce a detection probability, which a meta-learner fuses into the final decision.
HLT-vs-HWT
LGT-vs-HWT
Ternary
Domain
Train
Test
Train
Test
Train
Test
Total
Essay
10220
3900
5566
1397
13056
4304
38443
Paper Abs.
1089
262
338
79
1068
194
3030
Editorial
3301
852
1006
202
3407
661
9429
Total
14610
5014
6910
1678
17531
5159
50902
Table 2: Statistics of the KoLing benchmark: evaluation instances per domain and task after shortcut control. The last column sums each row.
Method
HLT vs. HWT
LGT vs. HWT
Avg.
Essay
Paper Abs.
Editorial
Avg.
Essay
Paper Abs.
Editorial
Avg.
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
Confidence-based
Log-Likelihood
54.0
52.4
1.1
56.0
58.2
5.7
52.1
53.5
2.0
54.0
54.7
2.9
47.9
51.2
1.0
55.1
60.4
6.4
52.1
59.1
4.6
51.7
56.9
4.0
52.9
55.8
3.5
Rank
54.9
52.5
0.7
56.6
56.7
6.5
52.1
53.6
2.4
54.5
54.3
3.2
48.5
51.1
1.7
54.8
58.5
11.1
49.2
56.6
2.4
50.8
55.4
5.1
52.7
54.8
4.1
Log-Rank
54.1
52.8
1.0
55.6
58.3
5.7
52.2
53.6
1.8
54.0
54.9
2.8
48.0
50.6
1.5
55.9
64.0
6.4
50.5
58.1
2.9
51.5
57.6
3.6
52.7
56.2
3.2
Table 3: Binary detection: AUROC (AUC), macro-F1 (F1), and TPR@FPR=1% (T@1) in %, averaged over 5 generators × 5 seeds. Avg. columns give per-task and overall averages; per-column best is bolded .
Method
Essay
Paper Abs.
Editorial
Avg.
Confidence-based
Log-Likelihood
32.6/51.7
38.5/53.1
35.1/52.7
35.4/52.5
Rank
32.6/47.9
38.4/52.1
35.0/48.1
35.3/49.4
Log-Rank
32.6/45.7
36.3/51.3
34.1/49.7
34.3/48.9
Entropy
33.8/43.9
33.5/42.0
34.0/38.2
33.8/41.4
LRR
34.2/44.1
33.4/43.1
34.4/39.8
34.0/42.3
Table 4: Ternary HWT/HLT/LGT classification: macro-F1 / Accuracy (%). The last column averages the three domains; per-column best is bolded .
Figure 4: Ternary confusion matrices (row-normalized %; rows = true) for the two native 3-class classifiers ( KatFishNet , KoLingProbe ) and the strongest single-score baseline (RepreGuard, threshold-mapped).
Figure 5: Essay LGT-vs-HWT detection as the typo injection rate increases (mean over 5 generators × 5 seeds; shaded: ± 1 std; dotted: chance).
Figure 6: Greedy forward selection of KoLingProbe sub-detectors per domain.
Figure 7: Rescue analysis per domain: the fraction of KatFishNet’s misclassified samples that each KoLingProbe sub-detector alone classifies correctly.
Figure 8: Cross-domain portability of KoLingProbe ’s features: effect sizes ( ∣ Cohen’s d∣ , human vs. LLM) of the top-15 features across the three domains.
Models
Essay
Poetry
Abstract
Avg.
Confidence-based
Log-Likelihood
51.58
43.32
64.11
53.00
Rank
53.00
40.78
67.61
53.79
Log-Rank
55.94
44.64
66.31
55.63
Entropy
51.09
47.76
61.57
53.47
LRR
64.45
46.65
66.18
59.10
Perturbation
NPR
82.30
65.49
60.21
69.33
Table 5: Detection performance (AUROC %) on the KatFish benchmark. Best results per column are bolded .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Source
Avg. Len
Std. Len
Essay
AI-Hub Essay Evaluation
629
251
Paper Abs.
KatFish (KIISE 2016–2018)
407
120
Editorial
AI-Hub Doc. Summarization
1068
251
Appendix
Table 6: Human-written source texts per KoLing domain. Text length is measured in characters. All domains use an 8:2 train–test split.
Writing Type
Total
Train
Test
Avg. Len
Std. Len
Creative
300
240
60
671
303
Problem-Solving
300
240
60
730
236
Expository
300
240
60
504
213
Argumentative
300
240
60
625
209
Pro-Con
300
240
60
613
223
Total
1500
1200
300
629
251
Appendix
Table 7: Statistics of the source human essay dataset used in KoLing . Text length is measured in characters.
Input Regime
Prompt
System Message
[KO] 당신은 한국어 글쓰기 전문가입니다. 주어진 지시에 따라 글을 작성하세요. 작성한 글만 출력하고, 부가 설명이나 메타 정보, 제목 표지는 포함하지 마세요. [EN] You are a Korean writing expert. Write a text according to the given instructions. Output only the written text, without any additional explanation, meta-information, or title page.
Grounded (Paper Abs.)
[KO] 다음은 한 학술 논문의 본문입니다(초록 제외). 이 논문을 대표하는 초록(abstract)을 작성해 주세요. 연구의 배경, 목적, 방법, 결과가 드러나도록 학술 논문 초록의 문어체로 작성하되, 반드시 ‘-다’로 끝나는 평서체(예: ‘제안한다’, ‘분석하였다’)로 쓰고 존댓말(‘-습니다’, ‘-요’)은 절대 사용하지 마세요. 분량은 한국어 기준 약 {length}자 내외로 맞춰 주세요. 본문: {body} [EN] The following is the body of an academic paper (abstract excluded). Write an abstract that represents this paper. Use the formal written style of academic abstracts, covering the background, objective, method, and results. Always use the plain ‘-다’ declarative ending and never the polite register. Keep the length to about { length } Korean characters. Body: { body }
Topic (Editorial)
[KO] 다음 제목으로 신문 사설을 작성해 주세요. 하나의 논지를 명확히 세우고 근거를 들어 설득력 있게 전개하는 신문 사설의 문어체로 작성하되, 반드시 ‘-다’로 끝나는 평서체로 쓰고 존댓말(‘-습니다’, ‘-요’)은 절대 사용하지 마세요. 분량은 한국어 기준 약 {length}자 내외로 맞춰 주세요. 제목: {title} [EN] Write a newspaper editorial under the following title. Establish one clear thesis and develop it persuasively with supporting arguments, in the formal written style of newspaper editorials. Always use the plain ‘-다’ declarative ending and never the polite register. Keep the length to about { length } Korean characters. Title: { title }
Persona (Essay)
[KO] 당신은 {grade} 학생입니다. 다음 논제와 제시문을 바탕으로 ‘{type}’을 작성해 주세요. 제시문의 내용을 참고하되 자신의 생각을 담아 쓰고, 분량은 한국어 기준 약 {length}자 내외로 맞춰 주세요. 논제: {topic} 제시문: {passage} [EN] You are a { grade } student. Based on the following topic and reading passage, write a { type } essay. Refer to the passage but express your own thoughts, and keep the length to about { length } Korean characters. Topic: { topic } Passage: { passage }
Appendix
Table 8: System message and domain-specific user prompts for LGT (from-scratch) generation. {length} is set to the original human text’s character length; {type} is the essay writing type (e.g., argumentative); the grounded/topic/persona input regimes mirror what the human author of each domain received.
Figure 9: Per-sample inference latency (single sample, batch=1; log scale). Colors denote CPU, GPU zero-shot, and GPU task-fine-tuned methods. KoLingProbe (38 ms, CPU) needs no GPU and is far faster than the strongest zero-shot competitors (e.g., NPR 2,476 ms).
Level
Prompt
System Message
[KO] 당신은 한국어 글쓰기 전문가입니다. 주어진 지시에 따라 텍스트를 변환하세요. 변환된 글만 출력하고, 부가 설명이나 메타 정보는 포함하지 마세요. [EN] You are a Korean writing expert. Transform the text according to the given instructions. Output only the transformed text without any additional explanation or meta-information.
Proofread (Level 1)
[KO] 다음 글의 맞춤법, 문법 오류를 교정하고, 반복적이거나 부적절한 단어를 교체해 주세요. 원문의 문체와 어조를 유지하고, 전체 내용과 구조는 변경하지 마세요. 분량은 원문과 비슷하게 유지하세요. 원문: {text} [EN] Correct any spelling and grammar errors in the following text and replace any repetitive or inappropriate words. Maintain the original style and tone, and do not change the overall content or structure. Keep the length similar to the original. Original: { text }
Polish (Level 2)
[KO] 다음 글의 문장들을 유창하고 명확하게 다듬어 주세요. 전체 내용과 구조는 유지하되, 표현을 개선해 주세요. 원문의 문체와 어조를 유지하고, 분량은 원문과 비슷하게 유지하세요. 원문: {text} [EN] Refine the sentences in the following text to be fluent and clear. Maintain the overall content and structure, but improve the expressions. Preserve the original style and tone, and keep the length similar to the original. Original: { text }
Paraphrase (Level 3)
[KO] 다음 글을 같은 의미를 전달하되, 문장 구조와 표현을 완전히 다르게 재작성해 주세요. 원문의 단어와 문장 구조를 그대로 사용하지 말고, 다른 어휘와 문형으로 바꿔 주세요. 전체 내용과 논지는 유지하되, 분량은 원문과 비슷하게 유지하세요. 원문: {text} [EN] Rewrite the following text to convey the same meaning, but with completely different sentence structures and expressions. Do not reuse the original words or sentence patterns; use different vocabulary and syntactic forms. Maintain the overall content and argument, and keep the length similar to the original. Original: { text }
Appendix
Table 9: System message and user prompts for each modification level used in the KoLing benchmark. {text} is replaced with the original human-written essay.
Category
Sub-category
Feature
Description
Phonology (12d)
G2P Rules (6d)
ratio_spell_pron
Ratio of content words whose spelling matches their G2P-converted pronunciation.
n_palatalization
Count of words exhibiting palatalization (ㄷ/ㅌ + ㅣ → ㅈ/ㅊ).
n_tensification
Count of words exhibiting tensification (lenis → fortis after obstruent codas).
n_liquid_assimilation
Count of words with liquid assimilation (ㄴ+ㄹ or ㄹ+ㄴ → ㄹㄹ).
n_nasal_assimilation
Count of words with nasal place assimilation.
n_initial_law
Count of initial-law (두음법칙) violations in spelling.
Appendix
Table 10: Detailed description of all 47 linguistic features. Each feature is computed at the text level.
Figure 10: Top-10 features by Cohen’s d on (a) KatFish and (b) KoLing . KatFish is dominated by syntax, while KoLing draws on all five sub-detectors. All features shown are significant at p<.001 (Welch’s t -test).
Design
HLT vs. HWT
LGT vs. HWT
Avg.
Essay
Abs.
Edit.
Essay
Abs.
Edit.
Concat-47
87.6
64.9
69.2
93.1
80.6
97.8
82.2
Fusion (ours)
85.1
66.9
70.1
91.2
74.0
99.6
81.1
Under typo attack (Essay); columns = injection rate
HLT: 1% / 3% / 5% / 10%
LGT: 1% / 3% / 5% / 10%
Concat-47
83.8 / 75.1 / 66.6 / 46.7
89.3 / 80.0 / 69.1 / 46.5
Appendix
Table 11: Concatenated 47-feature classifier vs. hierarchical fusion (AUROC %, same protocol as the main tables). Top: clean KoLing . Bottom: Essay under increasing typo injection.
Method
Essay
Poetry
Abstract
Avg.
Log-Likelihood
51.6 → 48.3
43.3 → 46.9
64.1 → 56.7
53.0 → 50.7
Binoculars
96.4 → 93.8
63.1 → 54.7
68.6 → 68.0
76.0 → 72.2
KatFishNet
95.6 → 81.3
72.1 → 56.8
74.0 → 64.9
80.6 → 67.7
KoLingProbe
98.1 → 93.7
79.3 → 74.6
74.8 → 66.4
84.1 → 78.2
Appendix
Table 12: AUROC (%) on KatFish, original → shortcut-controlled (length- and comma-matched, class-balanced subsets). Best shortcut-controlled result per column is bolded .
Sub-Detectors
Essay
Paper Abs.
Editorial
Avg.
HLT
LGT
HLT
LGT
HLT
LGT
1-Sub-Detector
Phon
68.0
68.8
58.0
72.9
55.9
39.6
60.5
Morph
79.9
79.2
65.8
76.5
69.8
95.6
77.8
Syn
65.5
76.5
50.8
43.4
50.9
79.6
61.1
Sem
64.0
72.2
49.5
59.1
57.2
91.9
65.6
Appendix
Table 13: AUROC (%) for all 31 sub-detector combinations on KoLing . The per-column best is bolded . Sub-Detectors: Phon = Phonology, Morph = Morphology, Syn = Syntax, Sem = Semantics, Prag = Pragmatics.
Method
HLT vs. HWT
LGT vs. HWT
Avg.
Claude
GPT-4o
Qwen
Mistral
Solar
Claude
GPT-4o
Qwen
Mistral
Solar
Confidence-based
Log-Likelihood
54.8/50.8/0.9
51.0/51.2/0.4
50.8/51.7/0.7
50.1/50.7/0.5
63.2/57.6/2.9
44.6/48.2/0.0
44.3/49.1/0.0
47.4/50.8/0.7
38.2/44.0/0.0
64.8/64.1/4.1
50.9/51.8/1.0
Rank
52.8/50.5/0.3
51.8/51.0/0.4
52.6/51.7/0.7
52.5/50.7/0.7
65.0/58.7/1.4
37.8/44.0/0.0
47.2/50.0/0.9
50.0/50.9/2.9
43.6/46.3/0.0
64.1/64.5/4.5
51.7/51.8/1.2
Log-Rank
55.2/51.7/0.7
51.2/51.7/0.8
51.1/52.7/0.5
50.6/51.0/0.5
62.6/57.0/2.2
40.2/45.2/0.0
44.4/48.5/0.9
49.9/50.9/0.4
41.1/44.7/0.0
64.2/63.7/6.4
51.1/51.7/1.2
Entropy
53.5/51.6/1.7
50.9/51.4/1.0
51.2/50.2/1.8
50.7/49.2/0.5
56.2/54.1/1.8
33.6/38.8/0.8
46.3/50.1/0.9
59.4/58.7/4.7
51.5/55.4/1.7
56.4/58.7/5.0
51.0/51.8/2.0
Appendix
Table 14: Per-generator binary results on the Essay domain: AUROC/F1/TPR@FPR=1% (%; 5-seed average for seeded methods). “Claude” is the in-distribution generator (training-dependent methods are trained on Claude-Sonnet-4.6 outputs); the other four are OOD. The last column averages the ten generator–task cells. Per column and metric, best is bolded .
Method
HLT vs. HWT
LGT vs. HWT
Avg.
Claude
GPT-4o
Qwen
Mistral
Solar
Claude
GPT-4o
Qwen
Mistral
Solar
Confidence-based
Log-Likelihood
58.3/58.1/4.3
53.3/58.5/6.5
56.5/60.2/4.2
53.1/54.0/7.8
58.8/60.5/5.6
59.4/57.6/18.2
62.8/70.0/8.3
41.1/49.9/5.6
53.3/61.1/0.0
58.7/63.2/0.0
55.5/59.3/6.0
Rank
57.4/57.2/6.5
58.1/58.7/6.5
57.2/56.7/6.2
52.4/54.0/7.8
57.7/56.9/5.6
55.8/57.6/18.2
58.9/59.0/8.3
44.8/52.3/11.1
55.9/58.1/7.7
58.7/65.3/10.0
55.7/57.6/8.8
Log-Rank
56.4/57.4/4.3
54.3/59.9/6.5
56.1/57.8/4.2
53.1/57.2/7.8
58.3/58.8/5.6
55.2/60.6/18.2
62.2/72.7/8.3
45.9/52.0/5.6
55.4/65.7/0.0
60.7/68.8/0.0
55.8/61.1/6.0
Entropy
51.0/54.8/2.2
47.7/53.0/0.0
46.4/49.5/4.2
54.2/56.1/2.0
47.9/53.3/0.0
58.8/57.6/9.1
56.1/58.3/0.0
44.4/53.0/0.0
35.4/44.7/0.0
53.3/59.9/10.0
49.5/54.0/2.8
Appendix
Table 15: Per-generator binary results on the Paper Abs. domain; format and protocol as in Table 14 .
Method
HLT vs. HWT
LGT vs. HWT
Avg.
Claude
GPT-4o
Qwen
Mistral
Solar
Claude
GPT-4o
Qwen
Mistral
Solar
Confidence-based
Log-Likelihood
43.6/49.9/1.8
58.4/56.5/1.3
56.5/56.2/3.2
52.4/52.6/2.6
49.8/52.2/1.2
57.6/59.8/13.5
46.9/56.9/0.0
67.4/68.5/4.5
46.2/56.0/0.0
42.4/54.2/4.8
52.1/56.3/3.3
Rank
43.0/50.2/2.4
58.6/55.4/1.9
57.3/56.2/1.3
51.8/53.6/2.6
49.9/52.5/3.8
52.7/56.6/5.4
44.2/52.4/0.0
63.3/63.3/6.8
47.5/56.7/0.0
38.3/54.3/0.0
50.7/55.1/2.4
Log-Rank
44.0/50.2/3.0
58.6/56.5/0.6
56.4/56.2/1.9
52.0/52.8/2.6
50.0/52.5/0.6
55.2/59.8/5.4
45.9/56.9/0.0
66.5/67.7/4.5
45.6/56.0/0.0
39.4/50.0/4.8
51.4/55.9/2.3
Entropy
42.8/46.7/2.4
58.9/57.6/1.9
56.1/56.5/1.9
50.8/51.6/2.6
50.2/50.5/0.6
45.2/53.8/0.0
38.9/48.3/0.0
62.3/67.6/4.5
44.0/53.1/0.0
34.4/46.5/0.0
48.4/53.2/1.4
Appendix
Table 16: Per-generator binary results on the Editorial domain; format and protocol as in Table 14 .
Method
HLT vs. HWT
LGT vs. HWT
Avg.
Essay
Paper Abs.
Editorial
Essay
Paper Abs.
Editorial
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
Confidence-based
Log-Likelihood
54.0 ± 4.9
52.4 ± 2.6
1.1 ± 0.9
56.0 ± 2.4
58.2 ± 2.3
5.7 ± 1.4
52.1 ± 5.2
53.5 ± 2.5
2.0 ± 0.8
47.9 ± 9.0
51.2 ± 6.8
1.0 ± 1.6
55.1 ± 7.6
60.4 ± 6.6
6.4 ± 6.7
52.1 ± 9.2
59.1 ± 5.0
4.6 ± 4.9
52.8
55.8
3.5
Rank
54.9 ± 5.0
52.5 ± 3.1
0.7 ± 0.4
56.6 ± 2.1
56.7 ± 1.5
6.5 ± 0.7
52.1 ± 5.6
53.6 ± 2.1
2.4 ± 0.8
48.5 ± 8.8
51.1 ± 7.1
1.7 ± 1.8
54.8 ± 5.2
58.5 ± 4.1
11.1 ± 3.8
49.2 ± 8.5
56.6 ± 3.7
2.4 ± 3.0
52.7
54.8
4.1
Log-Rank
54.1 ± 4.6
52.8 ± 2.1
1.0 ± 0.6
55.6 ± 1.8
58.3 ± 1.0
5.7 ± 1.4
52.2 ± 5.1
53.6 ± 2.4
1.8 ± 1.0
48.0 ± 8.8
50.6 ± 6.9
1.5 ± 2.4
55.9 ± 5.7
64.0 ± 7.2
6.4 ± 6.7
50.5 ± 9.5
58.1 ± 5.8
2.9 ± 2.4
52.7
56.2
3.2
Appendix
Table 17: Main binary table with standard deviations: AUROC (AUC), F1, and TPR@FPR=1% (T@1), each reported as mean ± std over the 5 generator test splits (and 5 seeds for seeded methods). Same methods and training protocol as Table 3 ; the rightmost block averages the six domain–task means. Per column and metric, best is bolded .
Method
HLT vs. HWT
LGT vs. HWT
Creative
Prob.-Solv.
Expository
Argument.
Pro-Con
Creative
Prob.-Solv.
Expository
Argument.
Pro-Con
Confidence-based
Log-Likelihood
54.4/54.2/0.3
56.8/57.9/3.1
52.5/52.4/1.8
57.8/56.7/7.8
51.6/54.4/0.9
38.8/47.8/0.0
58.5/62.1/2.4
38.8/46.6/0.3
55.7/60.1/4.6
46.9/53.9/0.3
Rank
54.4/54.6/0.6
58.1/57.7/5.9
53.9/51.7/3.0
58.2/58.0/8.2
52.2/54.3/6.2
38.4/46.1/0.4
60.3/62.8/6.1
39.3/45.0/1.6
56.2/58.5/9.9
48.0/53.1/1.6
Log-Rank
52.8/53.3/1.4
56.1/58.9/1.2
52.3/52.0/5.5
59.4/57.0/14.7
53.7/55.2/1.2
36.0/44.3/0.9
59.2/61.5/3.0
38.3/45.7/3.2
57.9/61.7/13.7
47.0/52.8/1.2
Entropy
47.2/53.0/0.5
53.6/56.7/1.7
48.0/49.2/1.4
59.2/54.6/12.7
56.9/56.0/1.3
35.1/46.2/0.0
62.0/65.4/5.4
42.2/50.6/2.4
60.1/61.4/17.1
47.1/51.8/2.9
Appendix
Table 18: Binary results on the Essay domain by writing type: AUROC/F1/TPR@FPR=1% (%; averaged over 5 generators, and 5 seeds for seeded methods). Types: Creative(글짓기), Problem-Solving(대안제시), Expository(설명글), Argumentative(주장), Pro-Con(찬성반대). Per column and metric, best is bolded .
Method
Essay
Paper Abs.
Editorial
Proofread
Polish
Paraphrase
Proofread
Polish
Paraphrase
Proofread
Polish
Paraphrase
Confidence-based
Log-Likelihood
51.4/52.1/0.4
52.2/52.9/0.8
58.5/58.3/2.1
53.3/58.0/9.3
55.2/59.2/5.7
58.9/61.9/2.2
49.4/51.3/0.4
50.9/53.1/1.5
56.2/58.3/4.3
Rank
53.1/52.9/0.5
52.7/52.5/0.7
59.1/57.8/0.9
55.7/57.5/9.3
55.9/57.8/3.6
57.8/61.5/7.5
50.1/52.0/2.1
51.0/53.3/1.7
55.6/58.0/3.4
Log-Rank
52.6/53.0/0.6
51.8/52.2/1.1
58.0/57.1/1.3
53.8/57.7/9.3
54.6/58.9/5.7
58.0/61.7/2.2
49.8/52.1/0.4
50.9/53.1/1.1
56.2/58.5/3.9
Entropy
53.6/54.5/1.9
50.1/52.2/1.1
53.7/54.5/1.1
49.6/53.6/3.9
46.5/53.9/1.2
53.3/58.3/0.0
50.3/52.6/1.9
50.0/53.5/1.8
55.4/58.3/1.9
Appendix
Table 19: HLT-vs-HWT binary results by modification level (Proofread → Polish → Paraphrase; increasing LLM intervention): AUROC/F1/TPR@FPR=1% (%; averaged over 5 generators, and 5 seeds for seeded methods). For each level, the negative class is the same HWT set and the positive class is restricted to HLT texts of that level. Per column and metric, best is bolded .
Method
HLT vs. HWT
LGT vs. HWT
Avg.
Essay
Paper Abs.
Editorial
Avg.
Essay
Paper Abs.
Editorial
Avg.
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
Confidence-based
Log-Likelihood
54.0
52.4
1.1
56.0
58.2
5.7
52.1
53.5
2.0
54.0
54.7
2.9
47.9
51.2
1.0
55.1
60.4
6.4
52.1
59.1
4.6
51.7
56.9
4.0
52.9
55.8
3.5
Rank
54.9
52.5
0.7
56.6
56.7
6.5
52.1
53.6
2.4
54.5
54.3
3.2
48.5
51.1
1.7
54.8
58.5
11.1
49.2
56.6
2.4
50.8
55.4
5.1
52.7
54.8
4.1
Log-Rank
54.1
52.8
1.0
55.6
58.3
5.7
52.2
53.6
1.8
54.0
54.9
2.8
48.0
50.6
1.5
55.9
64.0
6.4
50.5
58.1
2.9
51.5
57.6
3.6
52.7
56.2
3.2
Appendix
Table 20: Binary detection with the training generator switched to GPT-4o (instead of Claude-Sonnet-4.6 used in the main table): AUROC (AUC), F1, and TPR@FPR=1% (T@1), in %, averaged over the same five generator test splits as the main table. Methods marked † depend on task training and are re-trained on the new source; all other methods are zero-shot, so their scores are identical to Table 3 and are repeated for completeness. Per-column best is bolded .
Method
HLT vs. HWT
LGT vs. HWT
Avg.
Essay
Paper Abs.
Editorial
Avg.
Essay
Paper Abs.
Editorial
Avg.
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
AUC
F1
T@1
Confidence-based
Log-Likelihood
54.0
52.4
1.1
56.0
58.2
5.7
52.1
53.5
2.0
54.0
54.7
2.9
47.9
51.2
1.0
55.1
60.4
6.4
52.1
59.1
4.6
51.7
56.9
4.0
52.9
55.8
3.5
Rank
54.9
52.5
0.7
56.6
56.7
6.5
52.1
53.6
2.4
54.5
54.3
3.2
48.5
51.1
1.7
54.8
58.5
11.1
49.2
56.6
2.4
50.8
55.4
5.1
52.7
54.8
4.1
Log-Rank
54.1
52.8
1.0
55.6
58.3
5.7
52.2
53.6
1.8
54.0
54.9
2.8
48.0
50.6
1.5
55.9
64.0
6.4
50.5
58.1
2.9
51.5
57.6
3.6
52.7
56.2
3.2
Appendix
Table 21: As Table 20 , with the training generator switched to Qwen2.5-32B .
Method
Train
HLT vs. HWT (test)
LGT vs. HWT (test)
Essay
Paper Abs.
Editorial
Essay
Paper Abs.
Editorial
KatFishNet
Essay
60.9
55.7
55.4
73.7
57.3
89.6
Paper Abs.
50.0
50.0
50.0
55.9
53.1
56.4
Editorial
48.5
54.0
50.0
73.7
59.3
91.9
RepreGuard
Essay
81.6
62.7
55.8
89.3
65.4
89.4
Paper Abs.
64.9
66.7
51.0
55.4
69.4
57.7
Appendix
Table 22: Cross-domain transfer of all training-dependent detectors (AUROC, %): every detector is trained entirely on the row domain with its full original protocol and evaluated on the column domain’s test splits (5 generators × 5 seeds). Shaded cells are the in-domain reference. For academic-HLT, KatFishNet’s stability selection retains no comma feature; following the main-table convention, all features are used as a fallback, yielding chance-level scores.
The effective detection and governance of Large Language Model (LLM) generated content has become increasingly critical due to the growing risk of misuse. Despite the impressive performance of existing detectors, their reliability and potential in multilingual, real-world scenarios remain largely underexplored. In this study, we introduce DetectRL-X, a comprehensive multilingual benchmark designed to evaluate advanced detectors across 8 dimensions. The benchmark encompasses 8 languages commonly used in commercial contexts and collects human-written texts from 6 domains highly susceptible to LLM misuse. To better aligned with real-world applications, We create LLM-generated texts using 4 popular commercial LLMs, and include typical AI-assisted writing operations such as polishing, expanding, and condensing to capture authentic usage patterns. Furthermore, we develop a multilingual framework for paraphrasing and perturbation attacks to simulate diverse human modifications and writing noise, enabling stress testing of detectors across languages. Experimental results on DetectRL-X reveal the strengths and limitations of current state-of-the-art detectors when applied to diverse linguistic resources. We further analyze how domains, generators, attack strategies, text length, and refinement operations influence performance in different languages, underscoring DetectRL-X as an effective benchmark for strengthening multilingual and language-specific detectors.
Junchao Wu, Yefeng Liu, Chenyu Zhu +8
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Alibaba Group · Xiamen University
Large Language Models (LLMs) are widely applied across various domains due to their powerful text generation capabilities. While LLM-generated texts often resemble human-written ones, their misuse can lead to significant societal risks. Detecting such texts is an essential technique for mitigating LLM misuse, and many detection methods have shown promising results across different datasets. However, real-world scenarios often involve out-of-domain inputs or adversarial samples, which can affect the performance of detection methods to varying degrees. Furthermore, most existing research has focused on English texts, with limited work addressing Chinese text detection. In this study, we propose EnsemJudge, a robust framework for detecting Chinese LLM-generated text by incorporating tailored strategies and ensemble voting mechanisms. We trained and evaluated our system on a carefully constructed Chinese dataset provided by NLPCC2025 Shared Task 1. Our approach outperformed all baseline methods and achieved first place in the task, demonstrating its effectiveness and reliability in Chinese LLM-generated text detection. Our code is available at https://github.com/johnsonwangzs/MGT-Mini.
Zhuoshang Wang, Yubing Ren, Guoyu Zhao +3
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Large language models (LLMs) have demonstrated remarkable capabilities across various tasks. However, their ability to generate human-like text has raised concerns about potential misuse. This underscores the need for reliable and effective methods to detect LLM-generated text. In this paper, we propose IRM, a novel zero-shot approach that leverages Implicit Reward Models for LLM-generated text detection. Such implicit reward models can be derived from publicly available instruction-tuned and base models. Previous reward-based method relies on preference construction and task-specific fine-tuning. In comparison, IRM requires neither preference collection nor additional training. We evaluate IRM on the DetectRL benchmark and demonstrate that IRM can achieve superior detection performance, outperforms existing zero-shot and supervised methods in LLM-generated text detection.
Runheng Liu, Heyan Huang, Xingchen Xiao +1
School of Computer Science and Technology, Beijing Institute of Technology