cs.CLOct 11, 2026
SaveUnchecked or Falsely Accused? Evaluating AI-Generated Text Detectors on Kazakh
Organizations: School of Computing and Artificial Intelligence, Nazarbayev University, Astana, Kazakhstan
Abstract
AI-text detectors are already applied to Kazakh writing, yet no study reports how often publicly available detectors flag Kazakh text whose human authorship is dated before ChatGPT. We present AINA-KK: 3,811 human passages from encyclopedic, news, web and educational pages that provably existed before ChatGPT, and 5,597 machine and gray-zone passages from 9 generators, including the Kazakh-built KazLLM and Sherkala, with script and length stress tests. Five English-trained detectors from the Hugging Face Hub flag up to 81.4% of human passages and 79.6% of published student essays; recalibrated, they catch at most 1.3% of machine text. Calibrated zero-shot detectors catch at most 50.3% of machine text from in-distribution generators. Fine-tuned encoders rank text almost perfectly, but mDeBERTa at its default threshold flags 24.4% of 50-word human excerpts, partly because machine outputs are shorter than human text; calibrated, it catches only 47.1% of KazLLM and 62.7% of Sherkala passages. Without script handling, machine-transliterated Latin-2021 text defeats every detector. Fixes that need no new data (calibration, script canonicalization, script-augmented and length-matched training, and an OR with a zero-shot scorer) form AINA-KK-Det. On test it flags 0.9% of human passages (95% CI 0.4-2.7; 1.5% on development data) and catches 85.2% of machine text from known generators, 92.9% from two unseen Gemini-family Gemma-4 models and 61.3% from the Kazakh-built ones; its configuration was chosen with test results in view. We release the data, code and detector at https://github.com/ozxmn/aina-kk.
Figures & tables
| Generator | Access | M | MIX | HP | MT | MH |
|---|---|---|---|---|---|---|
| Gemini 3.5 Flash-Lite | Gemini | 831 | 388 | 0 | 0 | 147 |
| Gemini 3.1 Flash-Lite | Gemini | 758 | 351 | 127 | 86 | 0 |
| Gemma 4 31B † | Gemini | 404 | 185 | 0 | 0 | 0 |
| gpt-oss-120b | Groq | 316 | 149 | 60 | 0 | 0 |
| Qwen3.8-27B | Groq | 313 | 146 | 0 | 58 | 0 |
| Gemma 4 26B-A4B † | Gemini | 310 | 137 | 0 | 0 | 0 |
Table 1: Generators and passage counts after QC. M: machine text (C1, C2); MIX: human opening + machine continuation; HP: AI-polished human text; MT: machine translation from Russian (C4 LLM, C5 NLLB); MH: humanizer rewrite of M. Access: Gemini or Groq API, or open weights. † Held out: never used for training except in the last three rows of Table 4 . Model cards: Google DeepMind (2026a) ; Google DeepMind (2026b) ; Google DeepMind (2026c) ; OpenAI (2025) ; Groq (2026) . ‡ Gray-zone text only.
| AUROC | FPR on human test passages (%) | TPR (%) | |||||||||
| Detector | in-dist. [95% CI] | held-out | thr. | all | enc. | news | web | edu. | 50 words | in-dist. | held-out |
| Zero-shot, threshold at 1% FPR on training-split human text | |||||||||||
| Gemma-4-E2B / Binoculars | 0.931 [0.921, 0.940] | 0.981 | cal | 0.8 | 1.1 | 0.2 | 0.9 | 1.9 | 3.0 | 50.3 | 79.9 |
| Gemma-4-E2B / Fast-DetectGPT | 0.761 [0.743, 0.777] | 0.817 | cal | 0.7 | 0.3 | 0.6 | 1.3 | 0.0 | 0.3 | 10.8 | 19.5 |
| Llama-3.2-3B / Binoculars | 0.664 [0.646, 0.681] | 0.745 | cal | 1.4 | 2.6 | 0.4 | 0.6 | 3.2 | 4.3 | 3.6 | 9.3 |
| Llama-3.2-3B / Fast-DetectGPT | 0.814 [0.799, 0.828] | 0.902 | cal | 1.0 | 2.1 | 0.6 | 0.6 | 0.5 | 0.8 | 16.5 | 29.9 |
Table 2: Detectors as an institution would deploy them. Human: 1,530 test-split passages (50 words: 394 of them cut to their first 50 words). In-distribution machine text: 1,274 passages from the four training generators; held-out: 442 passages from the four unseen generators on test-split sources. AUROC CIs: bootstrap over source documents.
Figure 1: Share of human test passages flagged at each detector’s threshold, at full length and cut to their first 100 or 50 words (symmetric-log scale; dashed line: the 1% target). Both the default 0.5 and a threshold calibrated on full-length text over-flag short text. AINA-KK-Det (§ 6 ) uses length-matched training and length-specific thresholds.
| Gray zones: flagged (%) | Kazakh-built: TPR | Latin-2021 | Homoglyphs: TPR | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Detector | HP | MT | MIX | MH | KazLLM | Sherkala | AUROC | TPR | before | after |
| Gemma-4-E2B / Binoculars | 7.7 | 8.2 | 35.3 | 20.3 | 50.6 | 74.7 | 0.346 | 0.0 | 49.1 | 12.5 |
| mGPT-1.3B-kazakh / Fast-DetectGPT | 5.1 | 0.3 | 10.0 | 3.1 | 27.1 | 40.0 | 0.480 | 0.0 | 9.8 | 0.0 |
| XLM-R-base (FT) | 6.4 | 17.0 | 68.8 | 98.4 | 69.4 | 64.0 | 0.875 | 0.0 | 99.3 | 27.1 |
| mDeBERTa-v3-base (FT) | 7.7 | 7.9 | 82.5 | 100.0 | 47.1 | 62.7 | 0.384 | 0.0 | 99.0 | 63.3 |
Table 3: Stress tests at the calibrated threshold (1% FPR on unseen human text). HP: AI-polished human text; MT: machine-translated from Russian; MIX: human opening ( 50 words) continued by a model; MH: machine text rewritten by a second model to sound human. Latin-2021: both classes transliterated (503 human + 398 machine passages). Homoglyphs: 409 machine passages before and after 10% of eligible Cyrillic letters are replaced by Latin look-alikes.
| FPR on human text (%) | TPR (%) | Kazakh-built: TPR | Script: TPR | |||||||
| Configuration | all [95% CI] | web | 50 words | in-dist. | held-out ∗ | 50 words | KazLLM | Sherkala | Latin | homoglyph |
| mDeBERTa, threshold 0.5 | 7.6 | 10.3 | 24.4 | 100.0 | 94.8 | 99.5 | 85.9 | 85.3 | – | – |
| + threshold calibrated on unseen humans | 0.8 [0.1, 3.7] | 1.7 | 1.0 | 98.8 | 83.5 | 67.5 | 47.1 | 62.7 | 0.0 | 63.3 |
| + script canonicalization | 0.8 [0.1, 3.7] | 1.7 | 1.0 | 98.8 | 83.5 | 67.5 | 47.1 | 62.7 | 99.0 | 99.0 |
| + script-augmented training | 0.8 [0.0, 2.9] | 1.7 | 5.6 | 99.7 | 84.6 | 95.7 | 48.2 | 68.0 | 99.7 | 99.8 |
| + length-matched training | 1.9 [0.4, 3.2] | 3.2 | 3.6 | 99.8 | 84.8 | 94.2 | 48.2 | 69.3 | 99.7 | 99.8 |
Table 4: Fixes, applied cumulatively to mDeBERTa-v3-base (3 seeds, probabilities averaged). Calibrated thresholds give 1% FPR on development humans (391 passages from training topics held out of training); length-specific thresholds are fitted on the same passages cut to 100 or 50 words and applied to every input by its length (§ 4 ). Script canonicalization maps Latin-2021 text back to Cyrillic and folds Latin homoglyphs before scoring. OR: a text is flagged if either detector exceeds its own threshold, the two set jointly to 1% FPR on development humans. ∗ Held-out: test-source outputs of the four held-out generators; in the last three rows KazLLM and Sherkala outputs for training-split sources are also used for training, so only the two Gemma-4 models remain unseen there. FPR CI: bootstrap over source documents that also resamples the development humans and re-fits the thresholds, so it includes their sampling error. Appendix A.3 repeats this table on development data alone.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Cue | Human | Machine | Highest generator |
|---|---|---|---|
| non-breaking hyphen (U+2011) | 0.0 | 3.8 | gpt-oss-120b (41.7) |
| narrow no-break space | 0.0 | 0.7 | gpt-oss-120b (7.8) |
| soft hyphen | 1.6 | 0.1 | gpt-oss-120b (0.3) |
| ellipsis character | 2.2 | 0.3 | gpt-oss-120b (1.3) |
| standalone title line | 0.9 | 2.4 | Gemma 4 26B-A4B (6.7) |
| markdown header | 0.0 | 0.1 | gpt-oss-120b (0.9) |
Table 5: Share of passages (%) containing each surface cue before normalization: 3,299 encyclopedic, news and web passages before the second cleaning pass vs. 3,483 raw C1/C2 outputs.
| AUROC | TPR (%) | |||
|---|---|---|---|---|
| Generator | out | in | out | in |
| Gemini 3.1 Flash-Lite | 0.998 | 0.996 | 99.8 | 100.0 |
| Gemini 3.5 Flash-Lite | 1.000 | 0.997 | 99.6 | 99.8 |
| Qwen3.8-27B | 0.998 | 0.997 | 100.0 | 100.0 |
| gpt-oss-120b | 0.997 | 0.996 | 94.4 | 99.2 |
Table 6: Leave-one-generator-out among the four training generators. mDeBERTa with script augmentation and length matching, trained on three of them (out) or all four (in), scored on that generator’s outputs for test-split sources against test humans; TPR at the calibrated 1%-FPR threshold. One seed for out, three for in; single seeds are unstable at a 1% threshold (§ 6 ).
| FPR on development humans (%) | TPR on development machine text (%) | ||||||
| Configuration | all | web | 50 words | in-dist. | Gemma-4 | KazLLM | Sherkala |
| mDeBERTa, threshold 0.5 | 9.5 | 11.7 | 21.7 | 99.3 | 100.0 | 77.8 | 78.6 |
| + threshold calibrated on unseen humans | 1.3 | 0.0 | 2.3 | 95.6 | 100.0 | 33.3 | 57.1 |
| + script-augmented training | 1.3 | 0.0 | 6.4 | 98.5 | 100.0 | 33.3 | 57.1 |
| + length-matched training | 1.5 | 0.9 | 3.3 | 98.5 | 100.0 | 33.3 | 57.1 |
| + length-specific thresholds | 1.5 | 0.9 | 1.3 | 97.1 | 98.3 | 29.6 | 57.1 |
Table 7: Table 4 repeated on development data alone: 391 human passages from the 7 development topic clusters and the 249 machine passages prompted from them, none of which any configuration was trained on (the last three rows included). Each passage is judged with thresholds, and for the OR-combination the empirical distributions, fitted on the other six clusters; length-specific thresholds apply to every input by its length. Development data have no script variants, so the canonicalization row is omitted. Machine passages per column: 136, 58, 27 and 28.
| Statistic | Gemma-4-E2B | Llama-3.2-3B | mGPT-1.3B | Qwen3-4B | |
|---|---|---|---|---|---|
| Binoculars | 50.3 (0.931) | 3.6 (0.664) | 0.9 (0.731) | 1.3 (0.509) | |
| observer | |||||
| log-likelihood | 1.9 (0.770) | 2.1 (0.637) | 5.2 (0.782) | 1.8 (0.681) | |
| log-rank | 1.9 (0.756) | 2.7 (0.638) | 5.4 (0.784) | 2.0 (0.687) | |
| entropy | 0.9 (0.713) | 0.1 (0.536) | 1.6 (0.706) | 2.7 (0.698) | |
| likelihood/log-rank | 1.6 (0.616) | 2.0 (0.599) | 3.4 (0.701) | 2.7 (0.659) | |
Table 8: All 44 zero-shot statistics: TPR (%) on in-distribution test machine text at a threshold giving 1% FPR on training-split humans, and AUROC in parentheses. Each statistic’s direction is fixed on training-split data. Test FPRs range from 0.6% to 3.9%. For mGPT the performer is mGPT-1.3B-kazakh .
| KazLLM | Sherkala | Kazakh-built, pooled | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Detector | caught | exact CI | re-fit CI | caught | exact CI | re-fit CI | caught | exact CI | re-fit CI |
| mDeBERTa, calibrated | 40/85 | [36.1, 58.2] | [4.7, 80.0] | 47/75 | [50.7, 73.6] | [9.3, 84.0] | 87/160 | [46.3, 62.3] | [8.1, 79.8] |
| Gemma-4-E2B Binoculars | 43/85 | [39.5, 61.6] | – | 56/75 | [63.3, 84.0] | – | 99/160 | [53.9, 69.4] | – |
| + script aug., length matching and thresholds | 39/85 | [35.0, 57.0] | [24.1, 58.8] | 46/75 | [49.4, 72.4] | [43.3, 73.3] | 85/160 | [45.1, 61.0] | [34.8, 63.4] |
| AINA-KK-Det | 49/85 | [46.4, 68.3] | [44.1, 74.1] | 49/75 | [53.5, 76.0] | [58.7, 87.4] | 98/160 | [53.2, 68.8] | [54.5, 76.7] |
Table 9: Detection of the Kazakh-built test outputs with 95% intervals: exact (Clopper–Pearson, passages only) and a bootstrap over source documents that also resamples the development humans and re-fits the thresholds (as for FPRs). Rows 3–4 use length-specific thresholds on every input. The re-fit intervals of the single classifiers are wide because their thresholds sit in a band of saturated probabilities. Binoculars has no re-fit interval because its threshold is fitted on training-split humans, which this bootstrap does not resample.
More in this topic
AI-Generated Text Detection
AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.
See all 170 papers