Unchecked or Falsely Accused? Evaluating AI-Generated Text Detectors on Kazakh

Mansur Ozaman

cs.CLOct 11, 2026

Unchecked or Falsely Accused? Evaluating AI-Generated Text Detectors on Kazakh

Authors: Mansur Ozaman

Organizations: School of Computing and Artificial Intelligence, Nazarbayev University, Astana, Kazakhstan

Abstract

AI-text detectors are already applied to Kazakh writing, yet no study reports how often publicly available detectors flag Kazakh text whose human authorship is dated before ChatGPT. We present AINA-KK: 3,811 human passages from encyclopedic, news, web and educational pages that provably existed before ChatGPT, and 5,597 machine and gray-zone passages from 9 generators, including the Kazakh-built KazLLM and Sherkala, with script and length stress tests. Five English-trained detectors from the Hugging Face Hub flag up to 81.4% of human passages and 79.6% of published student essays; recalibrated, they catch at most 1.3% of machine text. Calibrated zero-shot detectors catch at most 50.3% of machine text from in-distribution generators. Fine-tuned encoders rank text almost perfectly, but mDeBERTa at its default threshold flags 24.4% of 50-word human excerpts, partly because machine outputs are shorter than human text; calibrated, it catches only 47.1% of KazLLM and 62.7% of Sherkala passages. Without script handling, machine-transliterated Latin-2021 text defeats every detector. Fixes that need no new data (calibration, script canonicalization, script-augmented and length-matched training, and an OR with a zero-shot scorer) form AINA-KK-Det. On test it flags 0.9% of human passages (95% CI 0.4-2.7; 1.5% on development data) and catches 85.2% of machine text from known generators, 92.9% from two unseen Gemini-family Gemma-4 models and 61.3% from the Kazakh-built ones; its configuration was chosen with test results in view. We release the data, code and detector at https://github.com/ozxmn/aina-kk.

Figures & tables

Generator	Access	M	MIX	HP	MT	MH
Gemini 3.5 Flash-Lite	Gemini	831	388	0	0	147
Gemini 3.1 Flash-Lite	Gemini	758	351	127	86	0
Gemma 4 31B †	Gemini	404	185	0	0	0
gpt-oss-120b	Groq	316	149	60	0	0
Qwen3.8-27B	Groq	313	146	0	58	0
Gemma 4 26B-A4B †	Gemini	310	137	0	0	0

Table 1: Generators and passage counts after QC. M: machine text (C1, C2); MIX: human opening + machine continuation; HP: AI-polished human text; MT: machine translation from Russian (C4 LLM, C5 NLLB); MH: humanizer rewrite of M. Access: Gemini or Groq API, or open weights. † Held out: never used for training except in the last three rows of Table 4 . Model cards: Google DeepMind (2026a) ; Google DeepMind (2026b) ; Google DeepMind (2026c) ; OpenAI (2025) ; Groq (2026) . ‡ Gray-zone text only.

	AUROC			FPR on human test passages (%)						TPR (%)
Detector	in-dist. [95% CI]	held-out	thr.	all	enc.	news	web	edu.	50 words	in-dist.	held-out
Zero-shot, threshold at 1% FPR on training-split human text
Gemma-4-E2B / Binoculars	0.931 [0.921, 0.940]	0.981	cal	0.8	1.1	0.2	0.9	1.9	3.0	50.3	79.9
Gemma-4-E2B / Fast-DetectGPT	0.761 [0.743, 0.777]	0.817	cal	0.7	0.3	0.6	1.3	0.0	0.3	10.8	19.5
Llama-3.2-3B / Binoculars	0.664 [0.646, 0.681]	0.745	cal	1.4	2.6	0.4	0.6	3.2	4.3	3.6	9.3
Llama-3.2-3B / Fast-DetectGPT	0.814 [0.799, 0.828]	0.902	cal	1.0	2.1	0.6	0.6	0.5	0.8	16.5	29.9

Table 2: Detectors as an institution would deploy them. Human: 1,530 test-split passages (50 words: 394 of them cut to their first 50 words). In-distribution machine text: 1,274 passages from the four training generators; held-out: 442 passages from the four unseen generators on test-split sources. AUROC CIs: bootstrap over source documents.

Figure 1: Share of human test passages flagged at each detector’s threshold, at full length and cut to their first 100 or 50 words (symmetric-log scale; dashed line: the 1% target). Both the default 0.5 and a threshold calibrated on full-length text over-flag short text. AINA-KK-Det (§ 6 ) uses length-matched training and length-specific thresholds.

	Gray zones: flagged (%)				Kazakh-built: TPR		Latin-2021		Homoglyphs: TPR
Detector	HP	MT	MIX	MH	KazLLM	Sherkala	AUROC	TPR	before	after
Gemma-4-E2B / Binoculars	7.7	8.2	35.3	20.3	50.6	74.7	0.346	0.0	49.1	12.5
mGPT-1.3B-kazakh / Fast-DetectGPT	5.1	0.3	10.0	3.1	27.1	40.0	0.480	0.0	9.8	0.0
XLM-R-base (FT)	6.4	17.0	68.8	98.4	69.4	64.0	0.875	0.0	99.3	27.1
mDeBERTa-v3-base (FT)	7.7	7.9	82.5	100.0	47.1	62.7	0.384	0.0	99.0	63.3

Table 3: Stress tests at the calibrated threshold (1% FPR on unseen human text). HP: AI-polished human text; MT: machine-translated from Russian; MIX: human opening (

\sim

50 words) continued by a model; MH: machine text rewritten by a second model to sound human. Latin-2021: both classes transliterated (503 human + 398 machine passages). Homoglyphs: 409 machine passages before and after 10% of eligible Cyrillic letters are replaced by Latin look-alikes.

	FPR on human text (%)			TPR (%)			Kazakh-built: TPR		Script: TPR
Configuration	all [95% CI]	web	50 words	in-dist.	held-out ∗	50 words	KazLLM	Sherkala	Latin	homoglyph
mDeBERTa, threshold 0.5	7.6	10.3	24.4	100.0	94.8	99.5	85.9	85.3	–	–
+ threshold calibrated on unseen humans	0.8 [0.1, 3.7]	1.7	1.0	98.8	83.5	67.5	47.1	62.7	0.0	63.3
+ script canonicalization	0.8 [0.1, 3.7]	1.7	1.0	98.8	83.5	67.5	47.1	62.7	99.0	99.0
+ script-augmented training	0.8 [0.0, 2.9]	1.7	5.6	99.7	84.6	95.7	48.2	68.0	99.7	99.8
+ length-matched training	1.9 [0.4, 3.2]	3.2	3.6	99.8	84.8	94.2	48.2	69.3	99.7	99.8

Table 4: Fixes, applied cumulatively to mDeBERTa-v3-base (3 seeds, probabilities averaged). Calibrated thresholds give 1% FPR on development humans (391 passages from training topics held out of training); length-specific thresholds are fitted on the same passages cut to 100 or 50 words and applied to every input by its length (§ 4 ). Script canonicalization maps Latin-2021 text back to Cyrillic and folds Latin homoglyphs before scoring. OR: a text is flagged if either detector exceeds its own threshold, the two set jointly to 1% FPR on development humans. ∗ Held-out: test-source outputs of the four held-out generators; in the last three rows KazLLM and Sherkala outputs for training-split sources are also used for training, so only the two Gemma-4 models remain unseen there. FPR CI: bootstrap over source documents that also resamples the development humans and re-fits the thresholds, so it includes their sampling error. Appendix A.3 repeats this table on development data alone.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Cue	Human	Machine	Highest generator
non-breaking hyphen (U+2011)	0.0	3.8	gpt-oss-120b (41.7)
narrow no-break space	0.0	0.7	gpt-oss-120b (7.8)
soft hyphen	1.6	0.1	gpt-oss-120b (0.3)
ellipsis character	2.2	0.3	gpt-oss-120b (1.3)
standalone title line	0.9	2.4	Gemma 4 26B-A4B (6.7)
markdown header	0.0	0.1	gpt-oss-120b (0.9)

Appendix

Table 5: Share of passages (%) containing each surface cue before normalization: 3,299 encyclopedic, news and web passages before the second cleaning pass vs. 3,483 raw C1/C2 outputs.

	AUROC		TPR (%)
Generator	out	in	out	in
Gemini 3.1 Flash-Lite	0.998	0.996	99.8	100.0
Gemini 3.5 Flash-Lite	1.000	0.997	99.6	99.8
Qwen3.8-27B	0.998	0.997	100.0	100.0
gpt-oss-120b	0.997	0.996	94.4	99.2

Appendix

Table 6: Leave-one-generator-out among the four training generators. mDeBERTa with script augmentation and length matching, trained on three of them (out) or all four (in), scored on that generator’s outputs for test-split sources against test humans; TPR at the calibrated 1%-FPR threshold. One seed for out, three for in; single seeds are unstable at a 1% threshold (§ 6 ).

	FPR on development humans (%)			TPR on development machine text (%)
Configuration	all	web	50 words	in-dist.	Gemma-4	KazLLM	Sherkala
mDeBERTa, threshold 0.5	9.5	11.7	21.7	99.3	100.0	77.8	78.6
+ threshold calibrated on unseen humans	1.3	0.0	2.3	95.6	100.0	33.3	57.1
+ script-augmented training	1.3	0.0	6.4	98.5	100.0	33.3	57.1
+ length-matched training	1.5	0.9	3.3	98.5	100.0	33.3	57.1
+ length-specific thresholds	1.5	0.9	1.3	97.1	98.3	29.6	57.1

Appendix

Table 7: Table 4 repeated on development data alone: 391 human passages from the 7 development topic clusters and the 249 machine passages prompted from them, none of which any configuration was trained on (the last three rows included). Each passage is judged with thresholds, and for the OR-combination the empirical distributions, fitted on the other six clusters; length-specific thresholds apply to every input by its length. Development data have no script variants, so the canonicalization row is omitted. Machine passages per column: 136, 58, 27 and 28.


Statistic	Gemma-4-E2B	Llama-3.2-3B	mGPT-1.3B	Qwen3-4B
Binoculars	50.3 (0.931)	3.6 (0.664)	0.9 (0.731)	1.3 (0.509)
log-likelihood	1.9 (0.770)	2.1 (0.637)	5.2 (0.782)	1.8 (0.681)
log-rank	1.9 (0.756)	2.7 (0.638)	5.4 (0.784)	2.0 (0.687)
entropy	0.9 (0.713)	0.1 (0.536)	1.6 (0.706)	2.7 (0.698)
likelihood/log-rank	1.6 (0.616)	2.0 (0.599)	3.4 (0.701)	2.7 (0.659)

Appendix

Table 8: All 44 zero-shot statistics: TPR (%) on in-distribution test machine text at a threshold giving 1% FPR on training-split humans, and AUROC in parentheses. Each statistic’s direction is fixed on training-split data. Test FPRs range from 0.6% to 3.9%. For mGPT the performer is mGPT-1.3B-kazakh .

	KazLLM			Sherkala			Kazakh-built, pooled
Detector	caught	exact CI	re-fit CI	caught	exact CI	re-fit CI	caught	exact CI	re-fit CI
mDeBERTa, calibrated	40/85	[36.1, 58.2]	[4.7, 80.0]	47/75	[50.7, 73.6]	[9.3, 84.0]	87/160	[46.3, 62.3]	[8.1, 79.8]
Gemma-4-E2B Binoculars	43/85	[39.5, 61.6]	–	56/75	[63.3, 84.0]	–	99/160	[53.9, 69.4]	–
+ script aug., length matching and thresholds	39/85	[35.0, 57.0]	[24.1, 58.8]	46/75	[49.4, 72.4]	[43.3, 73.3]	85/160	[45.1, 61.0]	[34.8, 63.4]
AINA-KK-Det	49/85	[46.4, 68.3]	[44.1, 74.1]	49/75	[53.5, 76.0]	[58.7, 87.4]	98/160	[53.2, 68.8]	[54.5, 76.7]

Appendix

Table 9: Detection of the Kazakh-built test outputs with 95% intervals: exact (Clopper–Pearson, passages only) and a bootstrap over source documents that also resamples the development humans and re-fits the thresholds (as for FPRs). Rows 3–4 use length-specific thresholds on every input. The re-fit intervals of the single classifiers are wide because their thresholds sit in a band of saturated probabilities. Binoculars has no re-fit interval because its threshold is fitted on training-split humans, which this bootstrap does not resample.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 170 papers

Jun 3, 2026

'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions

Although it is generally agreed that AI-generated text poses a broad societal risk, there is no common understanding in the AI-generated text detection literature on what constitutes harmful use. Rather, existing datasets and approaches often define their own criteria and make their own assumptions, sometimes implicitly, and often only loosely related to real-world needs and applications. To address this gap, we here systematically define various notions of AI-generated text and their characteristics. To study these, we collect AITDNA - a new benchmark of human-machine co-constructed texts that is annotated with detailed genesis information, such as the entire edit and AI-interaction history. We benchmark various machine-generated text detectors and find that they often only perform well for specific notions but not as broad detectors. We release code and data publicly.

Nils Dycke, Marina Sakharova, Nico Daheim +1

Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany · Zuse School ELIZA

AI-Generated Text Detection Generative AI Evaluation

Ask PDF

May 24, 2026

READER: Reasoning-Enhanced AI-Generated Text Detection

Recent advances in large language models (LLMs) have made it increasingly difficult to distinguish human-written text from AI-generated content. Many existing detectors train supervised neural classifiers that achieve strong in-distribution performance but are often opaque and can degrade substantially under distribution shift. We present READER, a reasoning-enhanced AI text detector that outputs both a human/AI label and a structured rationale describing the evidence for its decision. A key component of our approach is READ, a curated supervision set of rationales and verdicts. We fine-tune an LLM on READ to build READER, which reasons before detecting at inference time. Despite having only 1.5B parameters, READER consistently outperforms existing detectors as well as prompted, high-capacity LLM baselines (GPT-5.2, Gemini-3-Pro, and DeepSeek-V3.2), which are 100 to 1000 times larger in scale.

Pingfan Su, Kai Ye, Shijin Gong +4

Department of Statistics, London School of Economics and Political Science · School of Management, University of Science and Technology of China · School of Mathematics, University of Birmingham

AI-Generated Text Detection Large Language Model-Based Text Classification

Ask PDF

Mar 18, 2026

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

The rapid proliferation of large language models (LLMs) has created an urgent need for robust and generalizable detectors of machine-generated text. Existing benchmarks typically evaluate a single detector on a single dataset under ideal conditions, leaving open questions about cross-domain transfer, cross-LLM generalization, and adversarial robustness. We present a comprehensive benchmark evaluating diverse detection approaches across two corpora: HC3 (23,363 human-ChatGPT pairs) and ELI5 (15,000 human-Mistral-7B pairs). Methods include classical classifiers, fine-tuned transformer encoders (BERT, RoBERTa, ELECTRA, DistilBERT, DeBERTa-v3), a CNN, an XGBoost stylometric model, perplexity-based detectors, and LLM-as-detector prompting. Results show that transformer models achieve near-perfect in-distribution performance but degrade under domain shift. The XGBoost stylometric model matches performance while remaining interpretable. LLM-based detectors underperform and are affected by generator-detector identity bias. Perplexity-based methods exhibit polarity inversion, with modern LLM outputs showing lower perplexity than human text, but remain effective when corrected. No method generalizes robustly across domains and LLM sources.

Madhav S. Baidya, S. S. Baidya, Chirag Chawla

Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India

AI-Generated Text Detection Robustness of Machine-Generated Text Detection

Ask PDF

Abstract

AI-Generated Text Detection

Explore similar work

'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions

READER: Reasoning-Enhanced AI-Generated Text Detection

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions