Authors: George Alexandru Adam, Alexander Cui, Edwin Thomas, Emily Napier, Nazar Shmatko, Yue Gao, Zikun Chen, Jacob Schnell, Jacob Junqi Tian, Alekhya Dronavalli, Edward Tian, Dongwon Lee
Organizations: GPTZero · University of Waterloo · Vector Institute · Mila · Pennsylvania State University
While historical considerations surrounding text authenticity revolved primarily around plagiarism, the advent of large language models (LLMs) has introduced a new challenge: distinguishing human-authored from AI-generated text. This shift raises significant concerns, including the undermining of skill evaluations, the mass-production of low-quality content, and the proliferation of misinformation. Addressing these issues, we introduce GPTZero 5.0, a state-of-the-art industrial AI detection solution, offering reliable discernment between human and LLM-generated text. Our key contributions include: introducing a hierarchical, multi-task architecture enabling a flexible taxonomy of human and AI texts, setting a new state-of-the-art for AI detection accuracy on 11 independent third-party benchmarks and 18 writing domains, and achieving superior robustness to adversarial attacks and paraphrasing via multi-tiered automated red teaming. GPTZero offers accurate and explainable detection, and educates users on its responsible use, ensuring fair and transparent assessment of text.
Figures & tables
Figure 1: The process of developing and improving our deep learning model for detecting LLM-generated texts.
Figure 2: Distribution of GPTZero 5.0b’s human-written training texts by writing domain. Essay types (expository, argumentative, descriptive, persuasive and admissions essays) are grouped as Essays. The 16 domains that each account for less than 3% of texts are grouped as Smaller domains and broken down below, all percentages are shares of all human-written training texts.
Figure 3: Hierarchical Classifier Heads
Figure 4: GPTZero’s multi-tiered red teaming approach covers a variety of adversarial threats, providing unprecedented robustness.
Figure 5: Drops in AI probability after removing top-k% of high AI impact sentences predicted by our Advanced Scan Feature (Blue) and Detector Sentence Head (Orange)
Figure 6: Levensthein ratio for polished sample misclassifications.
Figure 7: Accuracy of mixed class ablation models on classifying documents evaluated on internal dataset of 40k examples (details in Appendix).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Human texts
Share (%)
Essays
1.48M
26.8
Expository essay
1.08M
19.5
Argumentative essay
176K
3.2
Descriptive essay
125K
2.3
Persuasive essay
69.9K
1.3
Admissions essay
31.2K
0.6
Appendix
Table 10: Human-written training texts of GPTZero 5.0b by writing domain, the data behind Figure 2 .
Example
Label
Dataset
many lepidopteran insects are agricultural pests that affect stored grains , food and fiber crops .
Human
Scientific Papers
The sine-Gordon field theory and its associated massive Thirring model are quantum field theories that have been extensively studied by researchers.
AI
Scientific Papers
Suddenly , Google could find itself in a position where it has to explain that they can not perform miracles to a fully committed audience invested in the idea that they can . All of these and many , many more ways to fall off exist . There ’s no rest for the weary , as they say . :-)
Human
HC3 Plus
These include the financial performance and prospects of the company, the strength of its management and leadership, the stability and growth potential of its industry, and the overall state of the economy.Additionally, the value of a share may be influenced by the demand for the stock among investors, as well as the supply of the stock that is available for purchase.
AI
HC3 Plus
Appendix
Table 11: Example of formatting biases. The human document in Scientific Papers is all lowercase with extraneous spacing around punctuation, while the AI document has correct capitalization and spacing. The human document in HC3 Plus has an emoji and extraneous spacing, while the AI document is more formal, and is missing space before "Additionally".
Figure 8: Main user interface for document scans. An overall Classification is presented along with a user-friendly confidence category. Per-class likelihoods are shown in the Probability Breakdown section.
Figure 9: Our user feedback collection interface for improving predictions.
Figure 10: Example of the GPTZero Advanced Scan feature showing the most important sentences contributing to a prediction.
Class
Precision (%)
Recall (%)
Accuracy (%)
AI
90.0
91.0
91.0
Human
91.0
90.0
90.0
Appendix
Table 12: Classification metrics (in %) for AI and Human classes using mined features.
Figure 11: Confusion Matrix representing AI vs Pure AI classes.
Figure 12: Remapping outputs to reduce overconfident predictions limits the negative effects of false positives.
Prompt
Polished Text
Original Human Document
The author gave a loving mood in the memoir. I know this becacus he did not talk badily about anything and he talked about how his parents helped him and other people as a kit. in paragraph two he was talking about how his house was nice and comsortabur. I know this because he sais. the love of cooking and the passioniate Cuban music.
Polish this text:
The author creates a warm, loving tone in the memoir. I know this because he doesn’t speak negatively about anything; instead, he focuses on how his parents helped him and others when he was a child. In paragraph two, he describes his home as nice and comfortable. I can tell because he mentions “the love of cooking” and “the passionate Cuban music,” which help paint a cozy, caring atmosphere.
Make this text sound more fluid:
The author creates a warm, loving tone throughout the memoir. I can tell because he doesn’t speak negatively about anything—instead, he focuses on how his parents supported him and cared for others when he was a child. In paragraph two, he describes his home as nice and comfortable, shown through details like the love of cooking and the passionate Cuban music.
Improve the quality of this text while maintaining a human voice:
The author creates a warm, loving tone throughout the memoir. I can tell because he doesn’t speak negatively about anything. Instead, he focuses on how his parents supported him and how other people helped him when he was a kid. In paragraph two, he describes his home as nice and comfortable. You can see this in the details he shares, like the love of cooking and the passionate Cuban music.
Improve the spelling and grammar in this text:
The author created a loving mood in the memoir. I know this because he did not speak badly about anything. He talked about how his parents helped him and other people when he was a kid. In paragraph two, he talked about how his house was nice and comfortable. I know this because he says he loved cooking and passionate Cuban music.
Appendix
Table 13: A comparison of some polishing prompts used for generated texts trained under the Polished class.
Service
Bypassing ability on naive detectors
GPTinf
Low
Grubby AI
Medium
HIX
Low
HumanizeAIText
Medium
Phrasely
High
QuillBot
Low
Appendix
Table 14: Commercial humanizer and paraphrasing services used to create the bypassed AI texts in the AI bypasser evaluation (Section 5.5), with their bypassing ability on naive AI detection methods.