SITO: Sifting a Commercial Text-Safety API for Missed Harm and False Flags in English, Ukrainian and Surzhyk

Andrii Tuzhykov

cs.CLOct 19, 2026

SITO: Sifting a Commercial Text-Safety API for Missed Harm and False Flags in English, Ukrainian and Surzhyk

Authors: Andrii Tuzhykov

Organizations: AI/ML Consultant / PhD Researcher The State Institution South Ukrainian National Pedagogical University named after K. D. Ushynsky

Abstract

Platforms that host young users screen messages with commercial text-safety APIs, but independent evidence on how often such systems miss harm or flag harmless text is scarce, especially outside English, and the documentation of the one studied here leaves its false-positive rate unquantified. We evaluated one such API, Tuteliq Detect as deployed under the release identifier tuteliq-detect-v5.20261011 (not the separately published v6 model), on six harm categories: authorised push-payment fraud, money-mule recruitment, social engineering, romance scams, gambling harm and bullying. We built SITO (Synthetic Independent Text-safety benchmark with Overflagging tests; sito is Ukrainian for 'sieve'), 2,332 synthetic cases in English, Ukrainian and mixed Ukrainian, with each harmful case matched by hard harmless ones: same-topic messages, minimal-edit contrasts, warnings and retrospective accounts. On English text the system flagged 84.9-94.1% of harmful cases and 0-4.3% of harmless ones per category. It missed third-party reports of ongoing harm far more often than perpetrators' messages (63.6% against 95.1% recall, all languages), when sent without the settings the API offers for reports of harm. Ukrainian and mixed-language input was detected at least as often as English, recall was lower on text from a held-out generator family, so headline recall may be optimistic, and 3.2% of cases received a different decision in repeated runs. A platform should measure false positives on its own traffic and check how bystander reports are handled. The datasets, generation prompts and code are released.

Figures & tables

Detector	Total	Harmful	en	uk	uk-par.	surzhyk	uk–ru	uk–en	translit
app-fraud	390	197	234	58	40	16	14	14	14
mule-recruitment	387	197	234	55	40	16	14	14	14
social-engineering	392	199	237	56	40	16	14	15	14
romance-scam	385	193	235	54	39	16	14	13	14
gambling-harm	391	194	236	58	39	16	14	14	14
bullying	387	199	234	56	40	15	14	15	13

Table 1: Composition of the test sets after review. uk-par.: Ukrainian translations of English cases from the same set, used for paired comparison; translit: Ukrainian in Latin script.

Role or type	Label	Cases
Perpetrator	harmful	814
Victim disclosure, ongoing	harmful	258
Third-party report, ongoing	harmful	107
Same-topic legitimate message (hard negative)	harmless	624
Minimal-edit contrast of a harmful case	harmless	171
Educational or protective text	harmless	133

Table 2: Label policy and counts after review. The planned mix of harmless types was 40/25/15/10/10%.

Detector	$n$	Recall [95% CI]	FPR [95% CI]	Precision bound @1 / 5 / 10%	ROC-AUC
app-fraud	234	84.9 [77.4, 90.2]	0.0 [0.0, 3.2]	19.5 / 55.7 / 72.7	0.959
mule-recruitment	234	94.1 [88.3, 97.1]	1.7 [0.5, 6.1]	12.8 / 43.4 / 61.8	0.997
social-engineering	237	92.5 [86.4, 96.0]	4.3 [1.8, 9.6]	8.3 / 32.1 / 49.9	0.952
romance-scam	235	85.5 [78.0, 90.7]	0.0 [0.0, 3.2]	20.0 / 56.5 / 73.3	0.985
gambling-harm	236	85.6 [78.1, 90.8]	0.0 [0.0, 3.2]	20.0 / 56.6 / 73.4	1.000
bullying	234	89.1 [82.2, 93.5]	1.7 [0.5, 6.1]	11.9 / 41.4 / 59.9	0.960

Table 3: Primary result: English cases, run 1. Intervals are 95% Wilson intervals; the range over the three runs is given in Appendix A . Precision at prevalence is the conservative bound computed from the lower recall limit and the upper FPR limit; the point estimates are 100% wherever no false positive was observed. All values in per cent except ROC-AUC.

Detector	Precision	Accuracy	F1	ROC-AUC [95% CI]
app-fraud	100.0	92.3	0.918	0.959 [0.932, 0.981]
mule-recruitment	98.2	96.2	0.961	0.997 [0.992, 1.000]
social-engineering	95.7	94.1	0.941	0.952 [0.918, 0.981]
romance-scam	100.0	92.8	0.922	0.985 [0.966, 0.998]
gambling-harm	100.0	92.8	0.922	1.000 [1.000, 1.000]
bullying	98.1	93.6	0.934	0.960 [0.933, 0.982]

Table 4: Metrics on the balanced sets: English cases, run 1. Precision and accuracy in per cent. On a balanced set precision overstates deployment precision (Table 3 ). The ROC-AUC interval is a 1,000-resample bootstrap.

Group	$n$	Recall [95% CI]	FPR [95% CI]	ROC-AUC [95% CI]
English	1,410	88.6 [86.1, 90.7]	1.3 [0.7, 2.4]	0.975 [0.966, 0.983]
Ukrainian	337	92.6 [87.8, 95.6]	4.3 [2.1, 8.7]	0.975 [0.956, 0.990]
Ukrainian (translation of English)	238	94.1 [88.3, 97.1]	0.8 [0.1, 4.6]	0.988 [0.970, 1.000]
Surzhyk	95	97.9 [89.1, 99.6]	0.0 [0.0, 7.6]	0.998 [0.993, 1.000]
Ukrainian–Russian	84	95.2 [84.2, 98.7]	0.0 [0.0, 8.4]	0.986 [0.956, 1.000]
Ukrainian–English	85	100.0 [91.6, 100.0]	2.3 [0.4, 12.1]	0.998 [0.990, 1.000]

Table 5: Results by language group, pooled over the six categories, run 1, in per cent with 95% Wilson intervals. Small groups have wide intervals.

Panel of Figure 1 — Figure 1: Recall by speaker role (left) and false-positive rate by type of harmless case (right), all languages, run 1, no setting declared, with 95% Wilson intervals; numbers of cases in parentheses. Third party: third-party report of ongoing harm; victim, ongoing: victim disclosure of ongoing harm; same topic: same-topic legitimate message; contrast: minimal-edit contrast; educational: educational or protective text; past incident: victim account of a past incident; unrelated: unrelated everyday message (Table 2 ).

Run 3 $\rightarrow$ run with setting ( v5.20261011 $\rightarrow$ v5.20261012 )
Cases	$n$	Harmful	Harmless	Recall	FPR
		lost / gained	lost / gained
All	2,332	16 / 61	3 / 13	90.5 $\rightarrow$ 94.3	1.6 $\rightarrow$ 2.4
No setting declared	443	2 / 18	1 / 3	90.0 $\rightarrow$ 97.3	2.2 $\rightarrow$ 3.1
Setting declared	1,889	14 / 43	2 / 10	90.6 $\rightarrow$ 93.6	1.4 $\rightarrow$ 2.3
of which helpline	144	7 / 1	0 / 0	92.9 $\rightarrow$ 87.6	3.2 $\rightarrow$ 3.2

Table 6: The run with a declared setting compared with run 3 on the same cases, all languages. “Lost” counts cases flagged in the earlier run and not in the later one; “gained” the reverse. Recall and FPR in per cent. Setting by channel: direct_message (one-to-one messengers, marketplace chat; 919 cases), group_chat (group, family, parents’ and class chats, Discord; 453), public_post (posts, comments, forums, job boards; 282), helpline (144), dating (42), gaming_chat (20), workplace (17), education (school newsletter; 12); none for SMS, e-mail, phone transcripts and pop-up text (443).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Detector	Recall, runs 1 / 2 / 3	FPR, runs 1 / 2 / 3
app-fraud	84.9 / 84.0 / 84.9	0.0 / 0.0 / 0.0
mule-recruitment	94.1 / 94.1 / 95.8	1.7 / 1.7 / 1.7
social-engineering	92.5 / 91.7 / 92.5	4.3 / 4.3 / 4.3
romance-scam	85.5 / 82.9 / 82.1	0.0 / 0.8 / 0.0
gambling-harm	85.6 / 87.3 / 87.3	0.0 / 0.0 / 0.0
bullying	89.1 / 85.7 / 86.6	1.7 / 1.7 / 1.7

Appendix

Table 7: English cases, per run (per cent). Over all 2,332 cases, 74 (3.2%) changed decision in at least one run; Fleiss’

\kappa=0.957

. No call failed in any run.

Content Moderation

Content moderation studies how to detect, interpret, and respond to harmful or policy-violating material across digital platforms.

See all 87 papers

Jun 3, 2026

'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions

Although it is generally agreed that AI-generated text poses a broad societal risk, there is no common understanding in the AI-generated text detection literature on what constitutes harmful use. Rather, existing datasets and approaches often define their own criteria and make their own assumptions, sometimes implicitly, and often only loosely related to real-world needs and applications. To address this gap, we here systematically define various notions of AI-generated text and their characteristics. To study these, we collect AITDNA - a new benchmark of human-machine co-constructed texts that is annotated with detailed genesis information, such as the entire edit and AI-interaction history. We benchmark various machine-generated text detectors and find that they often only perform well for specific notions but not as broad detectors. We release code and data publicly.

Nils Dycke, Marina Sakharova, Nico Daheim +1

Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany · Zuse School ELIZA

AI-Generated Text Detection Generative AI Evaluation

Ask PDF

Apr 17, 2026

Spotlights and Blindspots: Evaluating Machine-Generated Text Detection

With the rise of generative language models, machine-generated text detection has become a critical challenge. A wide variety of models is available, but inconsistent datasets, evaluation metrics, and assessment strategies obscure comparisons of model effectiveness. To address this, we evaluate 15 different detection models from six distinct systems, as well as seven trained models, across seven English-language textual test sets and three creative human-written datasets. We provide an empirical analysis of model performance, the influence of training and evaluation data, and the impact of key metrics. We find that no single system excels in all areas and nearly all are effective for certain tasks, and the representation of model performance is critically linked to dataset and metric choices. We find high variance in model ranks based on datasets and metrics, and overall poor performance on novel human-written texts in high-risk domains. Across datasets and metrics, we find that methodological choices that are often assumed or overlooked are essential for clearly and accurately reflecting model performance.

Kevin Stowe, Kailash Patil

Pindrop

AI-Generated Text Detection Language Model Generation Evaluation OOD Generalization Robustness of Machine-Generated Text Detection Generative AI Evaluation

Ask PDF

Aug 18, 2026

Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study

Online video platforms can expose young users to harmful content, but independent audits remain difficult because video annotation is costly and moderation judgments vary across languages. We audit TikTok in France, Italy, and Sweden with sockpuppet accounts representing four age personas (13, 16, 19, 40), collecting 36,971 videos from passive For-You-page scrolling and active sessions that scroll, search for harm keywords, and scroll again. To scale annotation, we validate four multimodal LLMs against native-speaker labels on a 300-video reference set. Gemini 2.5 Flash with eight sampled frames plus text performs best (aggregate kappa = 0.42), at half the per-call cost of native-video upload, and we apply it to a 10% sample for approximately $50 in total API spend across both modalities. Keyword search returns 35-56% harmful content, a 1.5-7.5x increase over the scrolling baseline in ten of twelve country-age combinations; the spike is temporary and flattens the age differences observed in France and Sweden. Under passive scrolling, Italy has the highest harm rate at every age, with Italian age-19 reaching 48.6%. Overall, MLLM-based auditing offers a scalable approach for cross-national youth-safety audits, while provider safety filters (1.1% refusal rate) under-count the most explicit harms.

Hamidreza Saffari, Francesco Pierri

Politecnico di Milano

Algorithmic Auditing Video Content Moderation Content Moderation Multimodal Content Moderation Multimodal Large Language Models

Ask PDF

Abstract

Content Moderation

Explore similar work

'Your AI Text is not Mine': Redefining and Evaluating AI-generated Text Detection under Realistic Assumptions

Spotlights and Blindspots: Evaluating Machine-Generated Text Detection

Auditing Exposure to Harmful Content on TikTok using Multimodal Language Models: A Cross-National, Age-Stratified Study