Beyond the Final Actor: Modeling the Dual Roles of Creator and Editor for Fine-Grained LLM-Generated Text Detection
Organizations: Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Abstract
The misuse of large language models (LLMs) requires precise detection of synthetic text. Existing works mainly follow binary or ternary classification settings, which can only distinguish pure human/LLM text or collaborative text at best. This remains insufficient for the nuanced regulation, as the LLM-polished human text and humanized LLM text often trigger different policy consequences. In this paper, we explore fine-grained LLM-generated text detection under a rigorous four-class setting. To handle such complexities, we propose RACE (Rhetorical Analysis for Creator-Editor Modeling), a fine-grained detection method that characterizes the distinct signatures of creator and editor. Specifically, RACE utilizes Rhetorical Structure Theory (RST) to construct a logic graph for the creator's foundation while extracting Elementary Discourse Unit (EDU)-level features for the editor's style. Experiments show that RACE outperforms 12 baselines in identifying fine-grained types with low false alarms, offering a policy-aligned solution for LLM regulation.
Figures & tables
| Reference | Target | Mean | Std |
| Human-Written | LLM-Polished | 0.92 | 0.08 |
| Human-Written | LLM-Generated | 0.84 | 0.12 |
| Human-Written | Humanized | 0.86 | 0.12 |
| LLM-Generated | Human-Written | 0.84 | 0.12 |
| LLM-Generated | LLM-Polished | 0.89 | 0.09 |
| LLM-Generated | Humanized | 0.95 | 0.07 |
| Method | AUROC | TPR@1%FPR | ||||
| Human-Written | LLM-Polished | LLM-Generated | Humanized | Avg | ||
| RoBERTa ( Solaiman et al., 2019 ) | 92.22 | 99.36 | 68.06 | 63.14 | 70.92 | 75.37 |
| RoBERTa ( Abassy et al., 2024 ) | 96.17 | 96.88 | 75.03 | 48.89 | 71.78 | 73.14 |
| CoCo ( Liu et al., 2023 ) | 97.67 | 99.68 | 75.77 | 63.93 | 79.43 | 79.70 |
| SeqXGPT ( Wang et al., 2023 ) | 89.87 | 98.38 | 15.23 | 14.32 | 31.68 | 39.90 |
| DeTeCtive ( Guo et al., 2024 ) | 95.74 | 98.62 | 0.00 | 0.00 | 77.23 | 43.96 |
| Method | AUROC | TPR@1%FPR | ||||
| Human-Written | LLM-Polished | LLM-Generated | Humanized | Avg | ||
| RACE | 97.99 ±0.13 | 99.04 ±0.40 | 83.60 ±1.61 | 74.18 ±0.95 | 75.41 ±1.03 | 83.06 ±0.57 |
| w/o CL | 97.73 ±0.44 | 98.21 ±0.19 | 78.07 ±1.06 | 69.10 ±2.66 | 73.43 ±1.59 | 79.70 ±1.12 |
| w/o Relation | 96.78 ±0.65 | 97.42 ±0.75 | 78.24 ±2.08 | 65.35 ±9.19 | 74.92 ±1.74 | 78.98 ±2.95 |
| w/o RGCN | 97.91 ±0.20 | 98.92 ±0.19 | 82.27 ±4.56 | 65.35 ±6.71 | 74.42 ±1.14 | 80.24 ±2.07 |
| w/o Bottleneck | 98.07 ±0.22 | 98.54 ±0.26 | 80.82 ±2.45 | 74.68 ±0.86 | 75.74 ±0.99 | 82.45 ±0.60 |
| Variant | AUROC | Avg. TPR@1%FPR |
| RACE (DMRST) | 97.64 | 82.93 |
| RACE (IsaNLP) | 97.99 | 83.06 |
| Metric | CoCo | RACE |
| Davies-Bouldin Index ( ) | 0.9286 | 0.8042 |
| Calinski-Harabasz Index ( ) | 2289.40 | 4333.32 |
| Domain | Method | AUROC | Avg. TPR@1%FPR |
| Arxiv | CoCo | 93.67 | 55.91 |
| LF-Motifs | 94.07 | 58.64 | |
| RACE | 96.61 | 76.28 | |
| Essay | CoCo | 89.84 | 28.72 |
| LF-Motifs | 91.12 | 47.85 | |
| RACE | 95.88 | 59.73 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Category | Train | Val | Test | Total |
| Arxiv | Human-Written | 700 | 100 | 200 | 1,000 |
| LLM-Polished | 700 | 100 | 200 | 1,000 | |
| LLM-Generated | 1,229 | 174 | 352 | 1,755 | |
| Humanized | 172 | 25 | 48 | 245 | |
| Essay | Human-Written | 700 | 100 | 200 | 1,000 |
| LLM-Polished | 700 | 100 | 200 | 1,000 |
| Method | Avg. TPR@5%FPR |
| RoBERTa | 92.53 |
| CoCo | 94.13 |
| SeqXGPT | 54.41 |
| DeTeCtive | 92.82 |
| LF-Motifs | 91.90 |
| Binoculars | 28.00 |
| Method | AUROC | TPR@1%FPR | ||||
| Human-Written | LLM-Polished | LLM-Generated | Humanized | Avg | ||
| RoBERTa | 98.39 | 99.72 | 72.63 | 70.18 | 71.69 | 78.55 |
| CoCo | 97.51 | 99.33 | 77.48 | 44.03 | 65.00 | 71.46 |
| LF-Motifs | 97.13 | 94.95 | 39.19 | 55.74 | 61.45 | 62.83 |
| RACE (Ours) | 96.59 | 99.13 | 86.25 | 75.20 | 67.00 | 81.90 |
| Method | Training | Inference | Params |
| RoBERTa | 2.119 | 220.8 | 125.2 |
| CoCo | 2.138 | 32.4 | 125.6 |
| LF-Motifs | 0.587 | 50.6 | 148.8 |
| RACE (Ours) | 1.071 | 90.0 | 128.6 |
AI-Generated Text Detection
AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.
See all 170 papers