Beyond Surface Cues: Robust LLM-Generated Korean Text Detection

SungHo Kim; SangKeun Lee

cs.CLOct 16, 2026

Beyond Surface Cues: Robust LLM-Generated Korean Text Detection

Authors: SungHo Kim, SangKeun Lee

Organizations: Department of Artificial Intelligence, Korea University, Seoul, South Korea · Department of Computer Science and Engineering, Korea University, Seoul, South Korea

Abstract

As large language models (LLMs) generate increasingly fluent text, the need for reliable detection of LLM-generated content has grown substantially. However, existing research remains heavily English-centric, leaving Korean largely underexplored. Moreover, current evaluations of Korean text detection often rely on superficial cues, leading to overly optimistic estimates of detection performance. To address these limitations, we introduce KoLing, a bias-controlled adversarial benchmark comprising 50.9k instances that suppresses superficial shortcuts while preserving semantic content and separately evaluates LLM-generated and LLM-modified text detection. We further propose KoLingProbe, an interpretable detector grounded in theoretical linguistics that models Korean-specific properties across multiple linguistic levels. Experiments show that KoLingProbe substantially outperforms strong baselines on all detection tasks while remaining robust to adversarial attacks. Our analyses further reveal that integrating complementary linguistic features across multiple levels yields both improved performance and interpretability.

Figures & tables

Human-Written Text

성공율 이 낮다고 해도 따지고 보면 함께 해결할

문제거든요 . 근데 서로 마음을 알 길이 없 어요 .

LLM-Modified Text (GPT-4o)

성공률 이 낮더라도 함께 해결할 문제라고 생각

합니다 . 그러나 마음을 알 수 없 습니다 .

English Translated: ‘Even if the success rate is low, it is a problem to solve together if you think about it. But there’s no way to know how they really feel.’

Table 1: A human–LLM sentence pair with similar syntactic patterns. Colors indicate distinct linguistic features captured by KoLingProbe : Phonological (phonological alternation), Morphological (connective ending), Semantic (idiom), and Pragmatic (discourse marker; polite ending) features.

Figure 1: Comparison of dataset biases between KatFish and KoLing . (a) Length distributions; (b) Human vs. LLM class balance; (c) Paired human–LLM sample ratio per genre; (d) Shortcut classifier accuracy.

Figure 2: Surface variation (bars) and semantic preservation (lines) across KoLing modification levels.

Figure 3: Overview of KoLingProbe : five independent linguistic sub-detectors each produce a detection probability, which a meta-learner fuses into the final decision.

	HLT-vs-HWT		LGT-vs-HWT		Ternary
Domain	Train	Test	Train	Test	Train	Test	Total
Essay	10220	3900	5566	1397	13056	4304	38443
Paper Abs.	1089	262	338	79	1068	194	3030
Editorial	3301	852	1006	202	3407	661	9429
Total	14610	5014	6910	1678	17531	5159	50902

Table 2: Statistics of the KoLing benchmark: evaluation instances per domain and task after shortcut control. The last column sums each row.

Method	HLT vs. HWT												LGT vs. HWT												Avg.
	Essay			Paper Abs.			Editorial			Avg.			Essay			Paper Abs.			Editorial			Avg.			Avg.
	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1
Confidence-based
Log-Likelihood	54.0	52.4	1.1	56.0	58.2	5.7	52.1	53.5	2.0	54.0	54.7	2.9	47.9	51.2	1.0	55.1	60.4	6.4	52.1	59.1	4.6	51.7	56.9	4.0	52.9	55.8	3.5
Rank	54.9	52.5	0.7	56.6	56.7	6.5	52.1	53.6	2.4	54.5	54.3	3.2	48.5	51.1	1.7	54.8	58.5	11.1	49.2	56.6	2.4	50.8	55.4	5.1	52.7	54.8	4.1
Log-Rank	54.1	52.8	1.0	55.6	58.3	5.7	52.2	53.6	1.8	54.0	54.9	2.8	48.0	50.6	1.5	55.9	64.0	6.4	50.5	58.1	2.9	51.5	57.6	3.6	52.7	56.2	3.2

Table 3: Binary detection: AUROC (AUC), macro-F1 (F1), and TPR@FPR=1% (T@1) in %, averaged over 5 generators

\times

5 seeds. Avg. columns give per-task and overall averages; per-column best is bolded .

Method	Essay	Paper Abs.	Editorial	Avg.
Confidence-based
Log-Likelihood	32.6/51.7	38.5/53.1	35.1/52.7	35.4/52.5
Rank	32.6/47.9	38.4/52.1	35.0/48.1	35.3/49.4
Log-Rank	32.6/45.7	36.3/51.3	34.1/49.7	34.3/48.9
Entropy	33.8/43.9	33.5/42.0	34.0/38.2	33.8/41.4
LRR	34.2/44.1	33.4/43.1	34.4/39.8	34.0/42.3

Table 4: Ternary HWT/HLT/LGT classification: macro-F1 / Accuracy (%). The last column averages the three domains; per-column best is bolded .

Figure 4: Ternary confusion matrices (row-normalized %; rows = true) for the two native 3-class classifiers ( KatFishNet , KoLingProbe ) and the strongest single-score baseline (RepreGuard, threshold-mapped).

Figure 5: Essay LGT-vs-HWT detection as the typo injection rate increases (mean over 5 generators

\times

5 seeds; shaded:

\pm

1 std; dotted: chance).

Figure 6: Greedy forward selection of KoLingProbe sub-detectors per domain.

Figure 7: Rescue analysis per domain: the fraction of KatFishNet’s misclassified samples that each KoLingProbe sub-detector alone classifies correctly.

Figure 8: Cross-domain portability of KoLingProbe ’s features: effect sizes ( $|$ Cohen’s $d|$ , human vs. LLM) of the top-15 features across the three domains.

Models		Essay	Poetry	Abstract	Avg.
Confidence-based	Log-Likelihood	51.58	43.32	64.11	53.00
	Rank	53.00	40.78	67.61	53.79
	Log-Rank	55.94	44.64	66.31	55.63
	Entropy	51.09	47.76	61.57	53.47
	LRR	64.45	46.65	66.18	59.10
Perturbation	NPR	82.30	65.49	60.21	69.33

Table 5: Detection performance (AUROC %) on the KatFish benchmark. Best results per column are bolded .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Source	Avg. Len	Std. Len
Essay	AI-Hub Essay Evaluation	629	251
Paper Abs.	KatFish (KIISE 2016–2018)	407	120
Editorial	AI-Hub Doc. Summarization	1068	251

Appendix

Table 6: Human-written source texts per KoLing domain. Text length is measured in characters. All domains use an 8:2 train–test split.

Writing Type	Total	Train	Test	Avg. Len	Std. Len
Creative	300	240	60	671	303
Problem-Solving	300	240	60	730	236
Expository	300	240	60	504	213
Argumentative	300	240	60	625	209
Pro-Con	300	240	60	613	223
Total	1500	1200	300	629	251

Appendix

Table 7: Statistics of the source human essay dataset used in KoLing . Text length is measured in characters.

Input Regime	Prompt
System Message	[KO] 당신은 한국어 글쓰기 전문가입니다. 주어진 지시에 따라 글을 작성하세요. 작성한 글만 출력하고, 부가 설명이나 메타 정보, 제목 표지는 포함하지 마세요. [EN] You are a Korean writing expert. Write a text according to the given instructions. Output only the written text, without any additional explanation, meta-information, or title page.
Grounded (Paper Abs.)	[KO] 다음은 한 학술 논문의 본문입니다(초록 제외). 이 논문을 대표하는 초록(abstract)을 작성해 주세요. 연구의 배경, 목적, 방법, 결과가 드러나도록 학술 논문 초록의 문어체로 작성하되, 반드시 ‘-다’로 끝나는 평서체(예: ‘제안한다’, ‘분석하였다’)로 쓰고 존댓말(‘-습니다’, ‘-요’)은 절대 사용하지 마세요. 분량은 한국어 기준 약 {length}자 내외로 맞춰 주세요. 본문: {body} [EN] The following is the body of an academic paper (abstract excluded). Write an abstract that represents this paper. Use the formal written style of academic abstracts, covering the background, objective, method, and results. Always use the plain ‘-다’ declarative ending and never the polite register. Keep the length to about { length } Korean characters. Body: { body }
Topic (Editorial)	[KO] 다음 제목으로 신문 사설을 작성해 주세요. 하나의 논지를 명확히 세우고 근거를 들어 설득력 있게 전개하는 신문 사설의 문어체로 작성하되, 반드시 ‘-다’로 끝나는 평서체로 쓰고 존댓말(‘-습니다’, ‘-요’)은 절대 사용하지 마세요. 분량은 한국어 기준 약 {length}자 내외로 맞춰 주세요. 제목: {title} [EN] Write a newspaper editorial under the following title. Establish one clear thesis and develop it persuasively with supporting arguments, in the formal written style of newspaper editorials. Always use the plain ‘-다’ declarative ending and never the polite register. Keep the length to about { length } Korean characters. Title: { title }
Persona (Essay)	[KO] 당신은 {grade} 학생입니다. 다음 논제와 제시문을 바탕으로 ‘{type}’을 작성해 주세요. 제시문의 내용을 참고하되 자신의 생각을 담아 쓰고, 분량은 한국어 기준 약 {length}자 내외로 맞춰 주세요. 논제: {topic} 제시문: {passage} [EN] You are a { grade } student. Based on the following topic and reading passage, write a { type } essay. Refer to the passage but express your own thoughts, and keep the length to about { length } Korean characters. Topic: { topic } Passage: { passage }

Appendix

Table 8: System message and domain-specific user prompts for LGT (from-scratch) generation. {length} is set to the original human text’s character length; {type} is the essay writing type (e.g., argumentative); the grounded/topic/persona input regimes mirror what the human author of each domain received.

Figure 9: Per-sample inference latency (single sample, batch=1; log scale). Colors denote CPU, GPU zero-shot, and GPU task-fine-tuned methods. KoLingProbe (38 ms, CPU) needs no GPU and is far faster than the strongest zero-shot competitors (e.g., NPR 2,476 ms).

Level	Prompt
System Message	[KO] 당신은 한국어 글쓰기 전문가입니다. 주어진 지시에 따라 텍스트를 변환하세요. 변환된 글만 출력하고, 부가 설명이나 메타 정보는 포함하지 마세요. [EN] You are a Korean writing expert. Transform the text according to the given instructions. Output only the transformed text without any additional explanation or meta-information.
Proofread (Level 1)	[KO] 다음 글의 맞춤법, 문법 오류를 교정하고, 반복적이거나 부적절한 단어를 교체해 주세요. 원문의 문체와 어조를 유지하고, 전체 내용과 구조는 변경하지 마세요. 분량은 원문과 비슷하게 유지하세요. 원문: {text} [EN] Correct any spelling and grammar errors in the following text and replace any repetitive or inappropriate words. Maintain the original style and tone, and do not change the overall content or structure. Keep the length similar to the original. Original: { text }
Polish (Level 2)	[KO] 다음 글의 문장들을 유창하고 명확하게 다듬어 주세요. 전체 내용과 구조는 유지하되, 표현을 개선해 주세요. 원문의 문체와 어조를 유지하고, 분량은 원문과 비슷하게 유지하세요. 원문: {text} [EN] Refine the sentences in the following text to be fluent and clear. Maintain the overall content and structure, but improve the expressions. Preserve the original style and tone, and keep the length similar to the original. Original: { text }
Paraphrase (Level 3)	[KO] 다음 글을 같은 의미를 전달하되, 문장 구조와 표현을 완전히 다르게 재작성해 주세요. 원문의 단어와 문장 구조를 그대로 사용하지 말고, 다른 어휘와 문형으로 바꿔 주세요. 전체 내용과 논지는 유지하되, 분량은 원문과 비슷하게 유지하세요. 원문: {text} [EN] Rewrite the following text to convey the same meaning, but with completely different sentence structures and expressions. Do not reuse the original words or sentence patterns; use different vocabulary and syntactic forms. Maintain the overall content and argument, and keep the length similar to the original. Original: { text }

Appendix

Table 9: System message and user prompts for each modification level used in the KoLing benchmark. {text} is replaced with the original human-written essay.

Category	Sub-category	Feature	Description
Phonology (12d)	G2P Rules (6d)	ratio_spell_pron	Ratio of content words whose spelling matches their G2P-converted pronunciation.
		n_palatalization	Count of words exhibiting palatalization (ㄷ/ㅌ + ㅣ $\rightarrow$ ㅈ/ㅊ).
		n_tensification	Count of words exhibiting tensification (lenis $\rightarrow$ fortis after obstruent codas).
		n_liquid_assimilation	Count of words with liquid assimilation (ㄴ+ㄹ or ㄹ+ㄴ $\rightarrow$ ㄹㄹ).
		n_nasal_assimilation	Count of words with nasal place assimilation.
		n_initial_law	Count of initial-law (두음법칙) violations in spelling.

Appendix

Table 10: Detailed description of all 47 linguistic features. Each feature is computed at the text level.

Panel of Figure 10 — Figure 10: Top-10 features by Cohen’s $d$ on (a) KatFish and (b) KoLing . KatFish is dominated by syntax, while KoLing draws on all five sub-detectors. All features shown are significant at $p<.001$ (Welch’s $t$ -test).

Design	HLT vs. HWT			LGT vs. HWT			Avg.
Design	Essay	Abs.	Edit.	Essay	Abs.	Edit.	Avg.
Concat-47	87.6	64.9	69.2	93.1	80.6	97.8	82.2
Fusion (ours)	85.1	66.9	70.1	91.2	74.0	99.6	81.1
Under typo attack (Essay); columns = injection rate
	HLT: 1% / 3% / 5% / 10%			LGT: 1% / 3% / 5% / 10%
Concat-47	83.8 / 75.1 / 66.6 / 46.7			89.3 / 80.0 / 69.1 / 46.5

Appendix

Table 11: Concatenated 47-feature classifier vs. hierarchical fusion (AUROC %, same protocol as the main tables). Top: clean KoLing . Bottom: Essay under increasing typo injection.

Method	Essay	Poetry	Abstract	Avg.
Log-Likelihood	51.6 $\to$ 48.3	43.3 $\to$ 46.9	64.1 $\to$ 56.7	53.0 $\to$ 50.7
Binoculars	96.4 $\to$ 93.8	63.1 $\to$ 54.7	68.6 $\to$ 68.0	76.0 $\to$ 72.2
KatFishNet	95.6 $\to$ 81.3	72.1 $\to$ 56.8	74.0 $\to$ 64.9	80.6 $\to$ 67.7
KoLingProbe	98.1 $\to$ 93.7	79.3 $\to$ 74.6	74.8 $\to$ 66.4	84.1 $\to$ 78.2

Appendix

Table 12: AUROC (%) on KatFish, original

\to

shortcut-controlled (length- and comma-matched, class-balanced subsets). Best shortcut-controlled result per column is bolded .

Sub-Detectors	Essay		Paper Abs.		Editorial		Avg.
Sub-Detectors	HLT	LGT	HLT	LGT	HLT	LGT	Avg.
1-Sub-Detector
Phon	68.0	68.8	58.0	72.9	55.9	39.6	60.5
Morph	79.9	79.2	65.8	76.5	69.8	95.6	77.8
Syn	65.5	76.5	50.8	43.4	50.9	79.6	61.1
Sem	64.0	72.2	49.5	59.1	57.2	91.9	65.6

Appendix

Table 13: AUROC (%) for all 31 sub-detector combinations on KoLing . The per-column best is bolded . Sub-Detectors: Phon = Phonology, Morph = Morphology, Syn = Syntax, Sem = Semantics, Prag = Pragmatics.

Method	HLT vs. HWT					LGT vs. HWT					Avg.
Method	Claude	GPT-4o	Qwen	Mistral	Solar	Claude	GPT-4o	Qwen	Mistral	Solar	Avg.
Confidence-based
Log-Likelihood	54.8/50.8/0.9	51.0/51.2/0.4	50.8/51.7/0.7	50.1/50.7/0.5	63.2/57.6/2.9	44.6/48.2/0.0	44.3/49.1/0.0	47.4/50.8/0.7	38.2/44.0/0.0	64.8/64.1/4.1	50.9/51.8/1.0
Rank	52.8/50.5/0.3	51.8/51.0/0.4	52.6/51.7/0.7	52.5/50.7/0.7	65.0/58.7/1.4	37.8/44.0/0.0	47.2/50.0/0.9	50.0/50.9/2.9	43.6/46.3/0.0	64.1/64.5/4.5	51.7/51.8/1.2
Log-Rank	55.2/51.7/0.7	51.2/51.7/0.8	51.1/52.7/0.5	50.6/51.0/0.5	62.6/57.0/2.2	40.2/45.2/0.0	44.4/48.5/0.9	49.9/50.9/0.4	41.1/44.7/0.0	64.2/63.7/6.4	51.1/51.7/1.2
Entropy	53.5/51.6/1.7	50.9/51.4/1.0	51.2/50.2/1.8	50.7/49.2/0.5	56.2/54.1/1.8	33.6/38.8/0.8	46.3/50.1/0.9	59.4/58.7/4.7	51.5/55.4/1.7	56.4/58.7/5.0	51.0/51.8/2.0

Appendix

Table 14: Per-generator binary results on the Essay domain: AUROC/F1/TPR@FPR=1% (%; 5-seed average for seeded methods). “Claude” is the in-distribution generator (training-dependent methods are trained on Claude-Sonnet-4.6 outputs); the other four are OOD. The last column averages the ten generator–task cells. Per column and metric, best is bolded .

Method	HLT vs. HWT					LGT vs. HWT					Avg.
Method	Claude	GPT-4o	Qwen	Mistral	Solar	Claude	GPT-4o	Qwen	Mistral	Solar	Avg.
Confidence-based
Log-Likelihood	58.3/58.1/4.3	53.3/58.5/6.5	56.5/60.2/4.2	53.1/54.0/7.8	58.8/60.5/5.6	59.4/57.6/18.2	62.8/70.0/8.3	41.1/49.9/5.6	53.3/61.1/0.0	58.7/63.2/0.0	55.5/59.3/6.0
Rank	57.4/57.2/6.5	58.1/58.7/6.5	57.2/56.7/6.2	52.4/54.0/7.8	57.7/56.9/5.6	55.8/57.6/18.2	58.9/59.0/8.3	44.8/52.3/11.1	55.9/58.1/7.7	58.7/65.3/10.0	55.7/57.6/8.8
Log-Rank	56.4/57.4/4.3	54.3/59.9/6.5	56.1/57.8/4.2	53.1/57.2/7.8	58.3/58.8/5.6	55.2/60.6/18.2	62.2/72.7/8.3	45.9/52.0/5.6	55.4/65.7/0.0	60.7/68.8/0.0	55.8/61.1/6.0
Entropy	51.0/54.8/2.2	47.7/53.0/0.0	46.4/49.5/4.2	54.2/56.1/2.0	47.9/53.3/0.0	58.8/57.6/9.1	56.1/58.3/0.0	44.4/53.0/0.0	35.4/44.7/0.0	53.3/59.9/10.0	49.5/54.0/2.8

Appendix

Table 15: Per-generator binary results on the Paper Abs. domain; format and protocol as in Table 14 .

Method	HLT vs. HWT					LGT vs. HWT					Avg.
Method	Claude	GPT-4o	Qwen	Mistral	Solar	Claude	GPT-4o	Qwen	Mistral	Solar	Avg.
Confidence-based
Log-Likelihood	43.6/49.9/1.8	58.4/56.5/1.3	56.5/56.2/3.2	52.4/52.6/2.6	49.8/52.2/1.2	57.6/59.8/13.5	46.9/56.9/0.0	67.4/68.5/4.5	46.2/56.0/0.0	42.4/54.2/4.8	52.1/56.3/3.3
Rank	43.0/50.2/2.4	58.6/55.4/1.9	57.3/56.2/1.3	51.8/53.6/2.6	49.9/52.5/3.8	52.7/56.6/5.4	44.2/52.4/0.0	63.3/63.3/6.8	47.5/56.7/0.0	38.3/54.3/0.0	50.7/55.1/2.4
Log-Rank	44.0/50.2/3.0	58.6/56.5/0.6	56.4/56.2/1.9	52.0/52.8/2.6	50.0/52.5/0.6	55.2/59.8/5.4	45.9/56.9/0.0	66.5/67.7/4.5	45.6/56.0/0.0	39.4/50.0/4.8	51.4/55.9/2.3
Entropy	42.8/46.7/2.4	58.9/57.6/1.9	56.1/56.5/1.9	50.8/51.6/2.6	50.2/50.5/0.6	45.2/53.8/0.0	38.9/48.3/0.0	62.3/67.6/4.5	44.0/53.1/0.0	34.4/46.5/0.0	48.4/53.2/1.4

Appendix

Table 16: Per-generator binary results on the Editorial domain; format and protocol as in Table 14 .

Method

HLT vs. HWT

LGT vs. HWT

Avg.

Essay

Paper Abs.

Editorial

Essay

Paper Abs.

Editorial

AUC

T@1

AUC

T@1

AUC

T@1

AUC

T@1

AUC

T@1

AUC

T@1

AUC

T@1

Confidence-based

Log-Likelihood

54.0

\pm

4.9

52.4

\pm

2.6

1.1

\pm

0.9

56.0

\pm

2.4

58.2

\pm

2.3

5.7

\pm

1.4

52.1

\pm

5.2

53.5

\pm

2.5

2.0

\pm

0.8

47.9

\pm

9.0

51.2

\pm

6.8

1.0

\pm

1.6

55.1

\pm

7.6

60.4

\pm

6.6

6.4

\pm

6.7

52.1

\pm

9.2

59.1

\pm

5.0

4.6

\pm

4.9

52.8

55.8

3.5

Rank

54.9

\pm

5.0

52.5

\pm

3.1

0.7

\pm

0.4

56.6

\pm

2.1

56.7

\pm

1.5

6.5

\pm

0.7

52.1

\pm

5.6

53.6

\pm

2.1

2.4

\pm

0.8

48.5

\pm

8.8

51.1

\pm

7.1

1.7

\pm

1.8

54.8

\pm

5.2

58.5

\pm

4.1

11.1

\pm

3.8

49.2

\pm

8.5

56.6

\pm

3.7

2.4

\pm

3.0

52.7

54.8

4.1

Log-Rank

54.1

\pm

4.6

52.8

\pm

2.1

1.0

\pm

0.6

55.6

\pm

1.8

58.3

\pm

1.0

5.7

\pm

1.4

52.2

\pm

5.1

53.6

\pm

2.4

1.8

\pm

1.0

48.0

\pm

8.8

50.6

\pm

6.9

1.5

\pm

2.4

55.9

\pm

5.7

64.0

\pm

7.2

6.4

\pm

6.7

50.5

\pm

9.5

58.1

\pm

5.8

2.9

\pm

2.4

52.7

56.2

3.2

Appendix

Table 17: Main binary table with standard deviations: AUROC (AUC), F1, and TPR@FPR=1% (T@1), each reported as mean

\pm

std over the 5 generator test splits (and 5 seeds for seeded methods). Same methods and training protocol as Table 3 ; the rightmost block averages the six domain–task means. Per column and metric, best is bolded .

Method	HLT vs. HWT					LGT vs. HWT
Method	Creative	Prob.-Solv.	Expository	Argument.	Pro-Con	Creative	Prob.-Solv.	Expository	Argument.	Pro-Con
Confidence-based
Log-Likelihood	54.4/54.2/0.3	56.8/57.9/3.1	52.5/52.4/1.8	57.8/56.7/7.8	51.6/54.4/0.9	38.8/47.8/0.0	58.5/62.1/2.4	38.8/46.6/0.3	55.7/60.1/4.6	46.9/53.9/0.3
Rank	54.4/54.6/0.6	58.1/57.7/5.9	53.9/51.7/3.0	58.2/58.0/8.2	52.2/54.3/6.2	38.4/46.1/0.4	60.3/62.8/6.1	39.3/45.0/1.6	56.2/58.5/9.9	48.0/53.1/1.6
Log-Rank	52.8/53.3/1.4	56.1/58.9/1.2	52.3/52.0/5.5	59.4/57.0/14.7	53.7/55.2/1.2	36.0/44.3/0.9	59.2/61.5/3.0	38.3/45.7/3.2	57.9/61.7/13.7	47.0/52.8/1.2
Entropy	47.2/53.0/0.5	53.6/56.7/1.7	48.0/49.2/1.4	59.2/54.6/12.7	56.9/56.0/1.3	35.1/46.2/0.0	62.0/65.4/5.4	42.2/50.6/2.4	60.1/61.4/17.1	47.1/51.8/2.9

Appendix

Table 18: Binary results on the Essay domain by writing type: AUROC/F1/TPR@FPR=1% (%; averaged over 5 generators, and 5 seeds for seeded methods). Types: Creative(글짓기), Problem-Solving(대안제시), Expository(설명글), Argumentative(주장), Pro-Con(찬성반대). Per column and metric, best is bolded .

Method	Essay			Paper Abs.			Editorial
Method	Proofread	Polish	Paraphrase	Proofread	Polish	Paraphrase	Proofread	Polish	Paraphrase
Confidence-based
Log-Likelihood	51.4/52.1/0.4	52.2/52.9/0.8	58.5/58.3/2.1	53.3/58.0/9.3	55.2/59.2/5.7	58.9/61.9/2.2	49.4/51.3/0.4	50.9/53.1/1.5	56.2/58.3/4.3
Rank	53.1/52.9/0.5	52.7/52.5/0.7	59.1/57.8/0.9	55.7/57.5/9.3	55.9/57.8/3.6	57.8/61.5/7.5	50.1/52.0/2.1	51.0/53.3/1.7	55.6/58.0/3.4
Log-Rank	52.6/53.0/0.6	51.8/52.2/1.1	58.0/57.1/1.3	53.8/57.7/9.3	54.6/58.9/5.7	58.0/61.7/2.2	49.8/52.1/0.4	50.9/53.1/1.1	56.2/58.5/3.9
Entropy	53.6/54.5/1.9	50.1/52.2/1.1	53.7/54.5/1.1	49.6/53.6/3.9	46.5/53.9/1.2	53.3/58.3/0.0	50.3/52.6/1.9	50.0/53.5/1.8	55.4/58.3/1.9

Appendix

Table 19: HLT-vs-HWT binary results by modification level (Proofread

\to

Polish

\to

Paraphrase; increasing LLM intervention): AUROC/F1/TPR@FPR=1% (%; averaged over 5 generators, and 5 seeds for seeded methods). For each level, the negative class is the same HWT set and the positive class is restricted to HLT texts of that level. Per column and metric, best is bolded .

Method	HLT vs. HWT												LGT vs. HWT												Avg.
	Essay			Paper Abs.			Editorial			Avg.			Essay			Paper Abs.			Editorial			Avg.			Avg.
	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1
Confidence-based
Log-Likelihood	54.0	52.4	1.1	56.0	58.2	5.7	52.1	53.5	2.0	54.0	54.7	2.9	47.9	51.2	1.0	55.1	60.4	6.4	52.1	59.1	4.6	51.7	56.9	4.0	52.9	55.8	3.5
Rank	54.9	52.5	0.7	56.6	56.7	6.5	52.1	53.6	2.4	54.5	54.3	3.2	48.5	51.1	1.7	54.8	58.5	11.1	49.2	56.6	2.4	50.8	55.4	5.1	52.7	54.8	4.1
Log-Rank	54.1	52.8	1.0	55.6	58.3	5.7	52.2	53.6	1.8	54.0	54.9	2.8	48.0	50.6	1.5	55.9	64.0	6.4	50.5	58.1	2.9	51.5	57.6	3.6	52.7	56.2	3.2

Appendix

Table 20: Binary detection with the training generator switched to GPT-4o (instead of Claude-Sonnet-4.6 used in the main table): AUROC (AUC), F1, and TPR@FPR=1% (T@1), in %, averaged over the same five generator test splits as the main table. Methods marked † depend on task training and are re-trained on the new source; all other methods are zero-shot, so their scores are identical to Table 3 and are repeated for completeness. Per-column best is bolded .

Method	HLT vs. HWT												LGT vs. HWT												Avg.
	Essay			Paper Abs.			Editorial			Avg.			Essay			Paper Abs.			Editorial			Avg.			Avg.
	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1	AUC	F1	T@1
Confidence-based
Log-Likelihood	54.0	52.4	1.1	56.0	58.2	5.7	52.1	53.5	2.0	54.0	54.7	2.9	47.9	51.2	1.0	55.1	60.4	6.4	52.1	59.1	4.6	51.7	56.9	4.0	52.9	55.8	3.5
Rank	54.9	52.5	0.7	56.6	56.7	6.5	52.1	53.6	2.4	54.5	54.3	3.2	48.5	51.1	1.7	54.8	58.5	11.1	49.2	56.6	2.4	50.8	55.4	5.1	52.7	54.8	4.1
Log-Rank	54.1	52.8	1.0	55.6	58.3	5.7	52.2	53.6	1.8	54.0	54.9	2.8	48.0	50.6	1.5	55.9	64.0	6.4	50.5	58.1	2.9	51.5	57.6	3.6	52.7	56.2	3.2

Appendix

Table 21: As Table 20 , with the training generator switched to Qwen2.5-32B .

Method	Train	HLT vs. HWT (test)			LGT vs. HWT (test)
Method	Train	Essay	Paper Abs.	Editorial	Essay	Paper Abs.	Editorial
KatFishNet	Essay	60.9	55.7	55.4	73.7	57.3	89.6
	Paper Abs.	50.0	50.0	50.0	55.9	53.1	56.4
	Editorial	48.5	54.0	50.0	73.7	59.3	91.9
RepreGuard	Essay	81.6	62.7	55.8	89.3	65.4	89.4
	Paper Abs.	64.9	66.7	51.0	55.4	69.4	57.7

Appendix

Table 22: Cross-domain transfer of all training-dependent detectors (AUROC, %): every detector is trained entirely on the row domain with its full original protocol and evaluated on the column domain’s test splits (5 generators

\times

5 seeds). Shaded cells are the in-domain reference. For academic-HLT, KatFishNet’s stability selection retains no comma feature; following the main-table convention, all features are used as a fallback, yielding chance-level scores.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 170 papers

DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection
May 15, 2026Junchao Wu, Yefeng Liu, Chenyu Zhu +8Multilingual Language Model Evaluation AI-Generated Text Detection
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Alibaba Group · Xiamen University
EnsemJudge: Enhancing Reliability in Chinese LLM-Generated Text Detection through Diverse Model Ensembles
Mar 30, 2026Zhuoshang Wang, Yubing Ren, Guoyu Zhao +3AI-Generated Text Detection Robustness of Machine-Generated Text Detection
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model
Apr 23, 2026Runheng Liu, Heyan Huang, Xingchen Xiao +1Zero-Shot Learning AI-Generated Text Detection
School of Computer Science and Technology, Beijing Institute of Technology

May 15, 2026J/K move · Enter open · S save

DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection

Junchao Wu, Yefeng Liu, Chenyu Zhu, Hao Zhang+7

NLP2CT Lab, Department of Computer and Information Science, University of Macau · Alibaba Group · Xiamen University

Multilingual Language Model Evaluation AI-Generated Text Detection Adversarial Robustness Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

The effective detection and governance of Large Language Model (LLM) generated content has become increasingly critical due to the growing risk of misuse. Despite the impressive performance of existing detectors, their reliability and potential in multilingual, real-world scenarios remain largely underexplored. In this study, we introduce DetectRL-X, a comprehensive multilingual benchmark designed to evaluate advanced detectors across 8 dimensions. The benchmark encompasses 8 languages commonly used in commercial contexts and collects human-written texts from 6 domains highly susceptible to LLM misuse. To better aligned with real-world applications, We create LLM-generated texts using 4 popular commercial LLMs, and include typical AI-assisted writing operations such as polishing, expanding, and condensing to capture authentic usage patterns. Furthermore, we develop a multilingual framework for paraphrasing and perturbation attacks to simulate diverse human modifications and writing noise, enabling stress testing of detectors across languages. Experimental results on DetectRL-X reveal the strengths and limitations of current state-of-the-art detectors when applied to diverse linguistic resources. We further analyze how domains, generators, attack strategies, text length, and refinement operations influence performance in different languages, underscoring DetectRL-X as an effective benchmark for strengthening multilingual and language-specific detectors.

Abstract

AI-Generated Text Detection

Explore similar work

DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection

EnsemJudge: Enhancing Reliability in Chinese LLM-Generated Text Detection through Diverse Model Ensembles

Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model