Organizations: Guilin University of Electronic Technology, China · Jinan Unaiversity, China · Nanjing University of Science and Technology, China · Jilin Unaiversity, China
Large Language Models (LLMs) increasingly participate in writing by modifying or extending human drafts, causing machine involvement to vary in both form and extent. Yet most Machine-Generated Text (MGT) detectors are evaluated only on fully human-written versus fully AI-generated text. Because human--AI collaboration can weaken or redistribute cues associated with machine generation, strong performance under this binary setting may overstate detector reliability. This mismatch remains underexplored in Chinese: detection cues are shaped by tokenization and language-specific text distributions, yet controlled resources spanning production settings, domains, and generators remain limited. To fill this gap, we present a Chinese Human-AI Collaborative Text Detection Benchmark (C-HAT-Bench), a unified benchmark that links 5,000 human-written source texts from five domains to more than 240,000 variants produced using six generative models under Prefix-Conditioned Continuation as a reference setting and three collaborative production modes. We evaluate 21 detectors through four protocols spanning zero-shot and pretrained supervised document-level detection, boundary localization, and cross-condition generalization. Relative to Prefix-Conditioned Continuation, mean AUROC across document-level detectors is 12.0% lower on the collaborative production modes, with the largest detector-specific relative decrease reaching 44.4%. Transfer across collaborative production modes is also asymmetric, indicating that performance in a given production setting is not a reliable predictor of performance in other production settings.
Figures & tables
Figure 1: Overview of the C-HAT-Bench framework.
Figure 2: Composition of C-HAT-240K across production settings.
Detector
MGT
Content Filling
Machine Rewriting
Prompt-based Generation
LC-C
LC-D
HC-C
HC-D
Light
Deep
L1
L2
L3
L3+Ref.
Zero-shot detectors
Log-Likelihood
0.743
0.651
0.631
0.715
0.695
0.634
0.629
0.758
0.730
0.694
0.697
LogRank
0.746
0.644
0.624
0.709
0.689
0.623
0.616
0.762
0.727
0.688
0.691
Entropy
0.317
0.374
0.386
0.333
0.334
0.393
0.396
0.303
0.324
0.341
0.338
DetectLLM-LRR
0.712
0.578
0.560
0.635
0.610
0.529
0.516
0.723
0.661
0.606
0.608
Table 1: Detector performance under human–AI collaboration, measured by AUROC. The best two results are highlighted by 1st and 2nd . Here, MGT denotes the prefix-conditioned continuation condition; LC/HC indicate low/high coverage (15–35%/50–75%), and C/D indicate single-suffix/separated-span filling; Light/Deep denote expression polishing/structural rewriting; and L1–L3 indicate increasing prompt detail, with L3+Ref additionally using a same-domain reference.
Figure 3: Detector performance relative to MGT across human–AI collaboration conditions. Top and bottom rows show zero-shot and supervised detectors, respectively.
Figure 4: AUROC distributions across five text domains for zero-shot (top) and supervised (bottom) detectors on MGT and human–AI collaborative text. Box colors indicate median AUROC.
Figure 5: Boundary-localization performance of WCP and DeBERTa–BiGRU–CRF across Content Filling conditions.
Figure 6: Cross-domain accuracies of AIGC-detector-zhv2 across five text domains.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Criterion
Rule
Human-text selection
Length-based selection
Starting from 1250 characters, progressively expand the selection range toward shorter and longer texts until 1,000 samples are collected per domain.
Human-text selection
Chinese-character ratio
For news, law, Q&A, and encyclopedia, start at 90% and relax the threshold when insufficient samples are available, with a lower bound of 60%. The denominator excludes whitespace.
Format cleaning
Whitespace and markup
Normalize line endings, remove redundant whitespace and invisible characters, and strip HTML/Markdown markup, code fences, and recognizable task-description prefixes. Text lengths are computed after cleaning.
Appendix
Table 2: HWT cleaning and validation rules for C-HAT-240K.
Figure 7: Character composition of the selected human corpus. Percentages are averaged over 1,000 texts per domain, excluding whitespace.
Figure 8: stride-sensitivity analysis for representative zero-shot and supervised detectors. The left panel shows zero-shot detectors, and the right panel shows supervised detectors. The highlighted x-axis value indicates the selected stride for the corresponding experiment.
Detector
MGT
Content Filling
Machine Rewriting
Prompt-based Generation
LC-C
LC-D
HC-C
HC-D
Light
Subst.
L1
L2
L3
L3+Ref.
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
1% FPR Zero-shot detectors
Log-Likelihood
58.5
22.5
52.6
10.4
52.3
9.6
53.8
14.1
54.8
15.7
52.5
10.0
52.6
10.3
56.7
18.9
53.4
12.2
53.6
12.7
53.9
13.5
LogRank
58.4
21.5
52.4
9.6
52.3
9.1
53.4
12.8
54.5
14.8
52.2
9.0
52.4
9.4
56.9
18.5
53.2
11.4
53.4
11.8
53.7
12.4
Entropy
50.1
2.3
49.9
1.7
49.9
1.6
49.9
1.4
50.0
1.7
49.9
1.4
50.0
1.7
50.0
1.7
49.8
1.1
50.0
1.7
50.0
1.6
Appendix
Table 6: Detector performance under human–AI collaboration, measured by ACC and F1 at 1%, 5%, and 10% FPR. The best two results are highlighted by 1st and 2nd . Values are reported as percentages and macro-averaged over 30 domain–generation-model combinations.
Detector
Law
Literature
News
Encyclopedia
Q&A
Zero-shot detectors
Log-Likelihood
0.891
0.575
0.618
0.676
0.698
LogRank
0.873
0.583
0.614
0.669
0.698
Entropy
0.182
0.479
0.399
0.346
0.337
DetectLLM-LRR
0.663
0.602
0.572
0.602
0.661
Fast-DetectGPT
0.783
0.602
0.552
0.604
0.655
Appendix
Table 7: Domain-wise AUROC of zero-shot and off-the-shelf supervised detectors. Each value is macro-averaged over six generation models and the fine-grained conditions within each of the four text-production settings, followed by an equal-weight average over the four settings. Within each detector family and domain, the best and second-best results are shown in boldface and underlined, respectively.
Condition
Detector
n
Sentence κ↑
WindowDiff ↓
Boundary F1@1 ↑
LC-C
WCP
1962
0.194 ± 0.343
0.428 ± 0.206
0.168 ± 0.226
LC-C
DeBERTa–BiGRU–CRF
1962
-0.054 ± 0.101
0.386 ± 0.184
0.004 ± 0.051
LC-D
WCP
1886
0.019 ± 0.163
0.381 ± 0.101
0.380 ± 0.232
LC-D
DeBERTa–BiGRU–CRF
1886
0.258 ± 0.250
0.309 ± 0.102
0.373 ± 0.271
HC-C
WCP
1877
0.166 ± 0.373
0.451 ± 0.185
0.171 ± 0.231
HC-C
DeBERTa–BiGRU–CRF
1877
0.005 ± 0.152
0.336 ± 0.089
0.341 ± 0.327
Appendix
Table 8: Detailed boundary-localization performance of the two fine-tuned detectors on the common content-filling test set. Here, n denotes the number of evaluated documents, and values are document-level means ± population standard deviations. Boundary F1@1 uses one-to-one matching between predicted and gold boundaries within a tolerance of one sentence; arrows indicate the preferred direction.
Condition
Detector
n
No predicted boundary (%)
Mean predicted
Mean gold
Mean count error
LC-C
WCP
1962
4.18
4.497
1.000
-3.497
LC-C
DeBERTa–BiGRU–CRF
1962
70.90
0.629
1.000
0.371
LC-D
WCP
1886
5.51
4.073
4.914
0.841
LC-D
DeBERTa–BiGRU–CRF
1886
24.23
1.694
4.914
3.221
HC-C
WCP
1877
3.94
4.384
1.000
-3.384
HC-C
DeBERTa–BiGRU–CRF
1877
31.43
1.477
1.000
-0.477
Appendix
Table 9: Boundary-prediction diagnostics for the two fine-tuned detectors on the common content-filling test set. Here, n denotes the number of evaluated documents. No predicted boundary is the percentage of documents with zero predicted boundaries; Mean predicted and Mean gold are the document-level average numbers of predicted and gold boundaries, respectively. Mean count error is defined as gold minus predicted boundaries, so negative and positive values indicate over-segmentation and missed boundaries, respectively.
Figure 9: Cross-domain accuracy (%) of AIGC-detector-zhv2 and IMBD with cellwise Youden-optimal thresholds. Rows and columns denote training and test domains, respectively.
Figure 10: Accuracy (%) across collaboration types for AIGC-detector-zhv2 and IMBD with cellwise Youden-optimal thresholds. Rows and columns denote training and test collaboration types, respectively.
Figure 11: Cross-generator accuracy (%) of AIGC-detector-zhv2 and IMBD with cellwise Youden-optimal thresholds. Rows and columns denote training and test generators, respectively.
The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.
Madhav S. Baidya, S. S. Baidya, Chirag Chawla
Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India
Recently, large language models (LLMs) are capable of generating highly fluent textual content. While they offer significant convenience to humans, they also introduce various risks, like phishing and academic dishonesty. Numerous research efforts have been dedicated to developing algorithms for detecting AI-generated text and constructing relevant datasets. However, in the domain of Chinese corpora, challenges remain, including limited model diversity and data homogeneity. To address these issues, we propose C-ReD: a comprehensive Chinese Real-prompt AI-generated Detection benchmark. Experiments demonstrate that C-ReD not only enables reliable in-domain detection but also supports strong generalization to unseen LLMs and external Chinese datasets-addressing critical gaps in model diversity, domain coverage, and prompt realism that have limited prior Chinese detection benchmarks. We release our resources at https://github.com/HeraldofLight/C-ReD.
Chenxi Qing, Junxi Wu, Zheng Liu +5
Tsinghua University · Nankai University · Harbin Institute of Technology, Shenzhen +2
Although it is generally agreed that AI-generated text poses a broad societal risk, there is no common understanding in the AI-generated text detection literature on what constitutes harmful use. Rather, existing datasets and approaches often define their own criteria and make their own assumptions, sometimes implicitly, and often only loosely related to real-world needs and applications. To address this gap, we here systematically define various notions of AI-generated text and their characteristics. To study these, we collect AITDNA - a new benchmark of human-machine co-constructed texts that is annotated with detailed genesis information, such as the entire edit and AI-interaction history. We benchmark various machine-generated text detectors and find that they often only perform well for specific notions but not as broad detectors. We release code and data publicly.
Nils Dycke, Marina Sakharova, Nico Daheim +1
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany · Zuse School ELIZA