Robust and Fine-Grained Detection of AI Generated Texts

Ram Mohan Rao Kadiyala; Siddartha Pullakhandam; Kanwal Mehreen; Drishti Sharma; Siddhant Gupta; Jebish Purbey; Ashay Srivastava; Subhasya TippaReddy

cs.CLApr 16, 2025

Robust and Fine-Grained Detection of AI Generated Texts

Authors: Ram Mohan Rao Kadiyala, Siddartha Pullakhandam, Kanwal Mehreen, Drishti Sharma, Siddhant Gupta, Jebish Purbey, Ashay Srivastava, Subhasya TippaReddy,

Organizations: Cohere for AI Community · Vantager · IIT Roorkee · University of Maryland ·

Abstract

An ideal detection system for machine generated content is supposed to work well on any generator as many more advanced LLMs come into existence day by day. Existing systems often struggle with accurately identifying AI-generated content over shorter texts. Further, not all texts might be entirely authored by a human or LLM, hence we focused more over partial cases i.e human-LLM co-authored texts. Our paper introduces a set of models built for the task of token classification which are trained on an extensive collection of human-machine co-authored texts, which performed well over texts of unseen domains, unseen generators, texts by non-native speakers and those with adversarial inputs. We also introduce a new dataset of over 2.4M such texts mostly co-authored by several popular proprietary LLMs over 23 languages. We also present findings of our models' performance over each texts of each domain and generator. Additional findings include comparison of performance against each adversarial method, length of input texts and characteristics of generated texts compared to the original human authored texts.

Figures & tables


Source	Dataset/Benchmark	Samples	Languages	Generators

( Lee et al., 2022 )	CoAuthor	1,445	1	1
( Zhang et al., 2024 )	MixSet	3,600	1	12
( Dugan et al., 2022 )	RoFT	21,646	1	5
( Macko et al., 2024b )	MultiTude	4,070	11	8

Table 1: Comparison with other Human-LLM co-authored datasets & benchmarks

Figure 1: Dataset distribution per each generator and language in our dataset


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	97.16	90.69	97.55	96.44
Chinese*	93.13	76.28	91.40	86.58
Czech	96.23	79.63	93.84	94.98
Dutch	96.83	77.60	94.13	95.31

Table 2: Word-Level Accuracy (.2f) of the models on the test dataset for each case


Metrics $\rightarrow$	Accuracy	Precision	Recall	F1

Initial Model	86.51	91.61	87.46	89.49
Final Model	86.00	87.16	92.25	89.63

Table 3: Word level Metrics over Mgtd-bench (.2f) through our models (zero-shot, unseen domains, unseen generators)


Metrics $\rightarrow$	Accuracy	Precision	Recall	F1

Arabic	95.9	96.1	94.5	95.2
English	99.1	98.7	99.3	99.0
Arabic-Best	96.1	96.1	95.0	95.5
English-Best	99.3	99.0	99.2	99.1

Table 4: Overall Metrics over ETS essays (.1f) through our detectors (zero-shot, unseen generators, unseen domain) VS best submissions (fine-tuned on same generators and domain)


System	Source	M	E	R	Avg

GLTR	( Gehrmann et al., 2019a )	0.706	0.943	0.773	0.807
Binoculars	( Hans et al., 2024 )	0.822	0.995	0.613	0.778
RADAR	( Gu et al., 2025 )	0.768	0.939	0.716	0.794

Table 5: Comparisons with other systems on the same benchmarks: MGTD-Bench ( M ), Academic-Essays ( E ), and Raid-Bench ( R ) through F1 Scores, and the average scores from the 3 benchmarks ( AVG ). The results cover a randomly sampled subset of each benchmark spanning over 13k samples, including those with adversarial methods ( R ) and non-native speakers ( E ). Randomly sampled subsets were used with equal distribution of all generators, domains, and adversarial methods distribution in the subset for each benchmark.

Figure 2: F1 scores VS text sampling method used : Sampling strategy did not effect detection capability

Figure 3: F1 scores VS the generator’s texts : instruct/chat variants were harder to detect than the corresponding base models.

Figure 4: F1 scores VS each domain’s texts : news i.e the only training data domain was easier to detect.

Figure 5: F1 scores VS adversarial method used in the input texts : homo-glyphs are the only real issue, and paraphrases to a small extent, while the rest can be handled through pre-processing.


Language	Length of	Length of
	Original part	generated part

Arabic	17	13
Czech	11	8
Dutch	12	10

Table 6: Median length (words) of original & newly generated parts of the sentences - Language wise : Models tend to finish off current sentence after authorship switch quickly before continuing with the rest of the text.


Generator	Length of	Length of
	original part	generated part

Amazon-Nova-Pro	14	10
Amazon-Nova-Lite	12	10
Aya-23-35B	11	10

Table 7: Median length (words) of original & newly generated parts of the sentences : Generator wise

Figure 6: Accuracy VS length of input texts (sentence count)


Generator	Accuracy

Amazon-Nova-Pro	94.90
Amazon-Nova-Lite	95.26
Aya-23-35B	91.75
Claude-3.5-Haiku	96.07

Table 8: Word level accuracy (.2f) of our models over our dataset (English)

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

continue this text in Language directly :

complete this text in Language , respond directly :

Appendix

Table 9: Prompts used in dataset creation : Partial cases

Rewrite this in Language a different way :

Generate an alternative version of this in Language :

Generate a later update to this in Language :

Generate a previous version of this in Language ;

Appendix

Table 10: Prompts used in dataset creation : Rewritten cases


Hyperparameter	Value

Seed (Training)	1024
Seed (Shuffling)	1024
Number of Epochs	5
Per Device Batch Size (Train)	12

Appendix

Table 11: Training Hyper-parameters used


Language $\downarrow$	Mean length of	Mean length of	Median length of	Median length of
	old text portion	new text portion	Old text portion	New text portion

Arabic	18.73	16.25	17	13
Czech	12.02	9.52	11	8
Dutch	13.73	13.39	12	10

Appendix

Table 12: Comparison of replaced and generated text portion lengths (word count) : Language wise


Generator $\downarrow$	Mean length of	Mean length of	Median length of	Median length of
	old text portion	new text portion	Old text portion	New text portion

Amazon-Nova-Pro	16.02	13.27	12	10
Amazon-Nova-Lite	18.12	12.87	14	10
Aya-23-35B	13.70	12.87	11	10

Appendix

Table 13: Comparison of replaced and generated text portion lengths (word count) : Generator wise


Language $\downarrow$	Accuracy	Precision	Recall	F1-score

Arabic	96.44	92.50	97.17	94.78
Chinese*	86.58	87.03	86.46	86.75
Czech	94.98	94.57	97.96	96.23
Dutch	95.31	93.34	97.97	95.60

Appendix

Table 14: Word-level Metrics of our models over each language : our test set


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	97.56	88.10	98.09	97.17
Chinese*	93.70	75.35	91.60	87.00
Czech	96.63	80.10	94.36	95.20
Dutch	95.21	78.00	92.10	95.23

Appendix

Table 15: Case wise accuracies over all languages for each generator : amazon-nova-pro


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	96.56	90.93	95.62	95.88
Chinese*	93.20	76.99	93.57	87.77
Czech	97.81	79.40	94.58	96.43
Dutch	97.07	78.10	92.85	95.19

Appendix

Table 16: Case wise accuracies over all languages for each generator : amazon-nova-lite


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	95.14	92.22	97.76	96.05
Chinese*	85.65	75.66	89.10	82.40
Czech	89.75	79.02	87.94	89.24
Dutch	92.45	79.97	92.99	93.12

Appendix

Table 17: Case wise accuracies over all languages for each generator : Aya-23


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	98.82	91.63	95.75	97.18
Chinese*	86.51	75.93	86.18	87.75
Czech	99.80	80.78	91.56	97.55
Dutch	99.49	77.57	86.16	96.41

Appendix

Table 18: Case wise accuracies over all languages for each generator : Claude-3.5-Haiku


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	98.63	91.82	100.00	96.58
Chinese*	92.64	78.36	95.14	88.43
Czech	99.30	77.22	99.88	97.62
Dutch	99.30	77.53	99.52	97.30

Appendix

Table 19: Case wise accuracies over all languages for each generator : Claude-3.5-Sonnet


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	87.12	85.34	86	82
Chinese*	88.90	86.45	89	84
English	92.45	90.12	91	88
French	89.78	87.21	90	85

Appendix

Table 20: Case wise accuracies over all languages for each generator : Command-R-Plus


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	95.74	91.26	96.31	95.01
Chinese*	92.63	77.87	92.51	86.36
Czech	93.30	80.96	91.67	91.87
Dutch	94.14	74.85	90.84	92.58

Appendix

Table 21: Case wise accuracies over all languages for each generator : GPT-4o


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	99.10	90.46	97.08	97.92
Chinese*	95.08	76.01	86.18	87.30
Czech	98.84	80.76	86.30	97.05
Dutch	98.80	77.47	89.03	97.12

Appendix

Table 22: Case wise accuracies over all languages for each generator : GPT-o1


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	93.90	88.40	98.59	94.86
Chinese*	89.29	75.98	94.92	85.09
Czech	91.43	78.82	97.15	92.04
Dutch	95.08	78.10	91.91	94.86

Appendix

Table 23: Case wise accuracies over all languages for each generator : Gemini-1.5-Pro


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	98.55	89.55	98.00	95.88
Chinese*	89.94	75.81	94.03	87.75
Czech	98.45	80.42	99.78	96.67
Dutch	98.22	78.00	99.39	95.86

Appendix

Table 24: Case wise accuracies over all languages for each generator : Gemini-1.5-Flash


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

Arabic	96.53	92.00	99.40	95.68
Chinese*	95.05	76.30	97.63	88.07
Czech	96.99	78.78	94.73	94.98
Dutch	94.46	78.10	91.01	94.46

Appendix

Table 25: Case wise accuracies over all languages for each generator : Mistral-Large-2411


Language $\downarrow$	Partial cases	Unchanged cases	Rewritten cases	Overall

English	97.10	91.08	99.71	96.64
French	95.53	72.58	99.49	93.94
German	94.98	77.21	99.50	94.29
Portuguese	92.66	89.06	98.17	94.03

Appendix

Table 26: Case wise accuracies over all languages for each generator : Perplexity-Sonar-Large

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 170 papers

Detecting LLM-Generated Tokens in Human--LLM Coauthored Text
Jul 23, 2026Yangjun Lu, Hongyi Zhou, Fabian Spill +3AI-Generated Text Detection Human-AI Co-Creation
School of Mathematics, University of Birmingham, Birmingham, UK · School of Statistics and Data Science, Shanghai University of Finance and Economics, Shanghai, China · Department of Statistics, The London School of Economics and Political Science, London, UK
Breaking the Generator Barrier: Disentangled Representation for Generalizable AI-Text Detection
Apr 15, 2026Xiao Pu, Zepeng Cheng, Lin Yuan +2AI-Generated Text Detection Robustness of Machine-Generated Text Detection
Chongqing University of Posts and Telecommunications, China
Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions
Mar 18, 2026Madhav S. Baidya, S. S. Baidya, Chirag ChawlaAI-Generated Text Detection Robustness of Machine-Generated Text Detection
Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India

Jul 23, 2026J/K move · Enter open · S save

Detecting LLM-Generated Tokens in Human--LLM Coauthored Text

Yangjun Lu, Hongyi Zhou, Fabian Spill, Kai Ye+2

School of Mathematics, University of Birmingham, Birmingham, UK · School of Statistics and Data Science, Shanghai University of Finance and Economics, Shanghai, China · Department of Statistics, The London School of Economics and Political Science, London, UK

AI-Generated Text Detection Human-AI Co-Creation

Open paper AskPDF arXiv

The rise of human-AI collaborative writing has created a growing need for fine-grained detection methods that support localizing likely LLM-generated content in mixed-authorship documents. Existing methods for detecting LLM-generated text mainly focus on document-level classification and cannot identify which parts of the text are generated by LLMs. This paper introduces a new method to address this urgent need. Our method operates at the token level, the natural unit of modern language models, and builds on existing token-level detection scores. The key idea is to smooth adjacent token scores to reduce their variability, while using an adaptive Lepski-type rule to select the bandwidth according to the local authorship structure. Our method is simple to implement and does not require token-level labeled data for training. Theoretically, we characterize this trade-off and show that the proposed method achieves favorable mean square error performance in estimating the underlying signal. Empirically, we demonstrate strong performance of our method against a wide range of baselines in both synthetic datasets and a realistic dataset. We deploy a publicly accessible website that implements the methods as well.

Abstract

AI-Generated Text Detection

Explore similar work

Detecting LLM-Generated Tokens in Human--LLM Coauthored Text

Breaking the Generator Barrier: Disentangled Representation for Generalizable AI-Text Detection

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions