TRIDENT and SHAKTI: A Framework and Benchmark for Hindi AI-Text Detection
Organizations: Department of CSE, Indian Institute of Technology Guwahati, Assam, India · Institute of Advanced Intelligence and Computing (IAIC), A*STAR, Singapore
Abstract
As Large Language Models (LLMs) become increasingly multilingual, distinguishing human-written from AI-generated text is essential for maintaining information integrity. However, research remains English-centric, with limited benchmarks for low-resource languages such as Hindi. To address this gap, we introduce SHAKTI, a large-scale Hindi benchmark covering four genres, eight LLM families, multiple prompting strategies, and adversarial transformations. We evaluate its quality through human annotation and analyze linguistic, grammatical, and sentiment-based differences between human- and AI-generated texts. Building on these findings, we propose TRIDENT, a round-trip translation-based detection framework with three variants: TRIDENT-S (static threshold), TRIDENT-D (dynamic threshold), and TRIDENT-C (logistic regression using multi-distance RTT features). TRIDENT leverages semantic consistency between original and round-trip translated texts to identify AI-generated content. Extensive experiments show that TRIDENT outperforms conventional and supervised baselines, with TRIDENT-S achieving the best overall performance. Further evaluations across seven languages, adversarial settings, and unseen LLMs demonstrate its multilingual applicability and robustness, while highlighting sensitivity to the translation model.
Figures & tables
| Name | Size | Langs | Multi-Domain | Multi-Generator | Adversarial | Language-Specific LLMs |
|---|---|---|---|---|---|---|
| TuringBench [ 30 ] | 200k | EN | ✗ | ✓ | ✗ | ✗ |
| RuATD [ 31 ] | 215k | RU | ✓ | ✓ | ✗ | ✗ |
| HC3 [ 32 ] | 26.9k | EN/ZH | ✓ | ✗ | ✗ | ✗ |
| MGTBench [ 33 ] | 2.8k | EN | ✓ | ✓ | ✓ | ✗ |
| MULTITuDE [ 34 ] | 74.1k | Multi | ✓ | ✓ | ✗ | ✗ |
| M4 [ 35 ] | 122k | Multi | ✓ | ✓ | ✗ | ✗ |
| Dimension | [ 9 ] | IndicDetect [ 10 ] | SHAKTI (Ours) |
|---|---|---|---|
| Language coverage | Hindi (BBC/ NDTV) | Hindi/Telugu/Tamil | Hindi |
| Domain coverage | News | Google News, IJHR Journal, Poetry, Bollywood Hungama | Essay, news, Wikipedia, QA |
| LLM diversity | 26 LLM, GPT versions, Bloom, BARD, mGPT, Mistral, Gemma, mT5 versions | GPT-4.1, Qwen-Plus, DeepSeek-v3.2 | GPT3.5, Llama 3.2, Mistral, Gemma 3, GPT-5, Gemini, Aya , Sarvam |
| Attack coverage | — | Perturbation, Whitespace, Insert-Paragraph, Alternative Spelling, Misspelling, Synonym Swap | Rewrite, Homoglyph, Synonym, Misspelling, Whitespace |
| Prompt for generation | Simple text generation method | research-style, open-ended narrative, opinion-oriented prompts | Instruction-based, Task-oriented prompt |
| Human-written samples | 7,043 | 4,000 | 83,900 |
| Agreement Metric | Fluency | Origin | Coherence | Factuality |
|---|---|---|---|---|
| Krippendorff’s Alpha ( ) | 0.769 | 0.671 | 0.535 | 0.776 |
| Fleiss’ Kappa ( ) | 0.719 | 0.543 | 0.456 | 0.543 |
| Feature | Human | Gemma QA | GPT | LLaMA | Mistral | ChatGPT | Gemini | Aya | Sarvam |
|---|---|---|---|---|---|---|---|---|---|
| # Characters | 170.73 | 825.08 | 740.44 | 4236.07 | 4572.91 | 895.42 | 1028.65 | 768.31 | 692.84 |
| # Words | 29.19 | 135.91 | 127.18 | 709.69 | 772.70 | 148.26 | 169.83 | 126.74 | 114.52 |
| # Sentences | 1.15 | 12.70 | 8.27 | 53.29 | 57.71 | 11.84 | 13.62 | 10.17 | 9.46 |
| # Paragraphs | 1.00 | 22.56 | 6.74 | 42.05 | 47.24 | 7.31 | 8.26 | 6.12 | 5.48 |
| Special Char | 138.77 | 646.58 | 588.56 | 2857.06 | 3087.26 | 704.18 | 810.36 | 601.72 | 542.63 |
| Punctuation | 3.51 | 14.49 | 14.85 | 93.57 | 99.61 | 15.82 | 17.64 | 13.27 | 12.18 |
| Feature | Human | Gemma QA | GPT | LLaMA | Mistral | ChatGPT | Gemini | Aya | Sarvam |
|---|---|---|---|---|---|---|---|---|---|
| # Unique Words | 23.20 | 58.66 | 64.47 | 131.52 | 133.27 | 102.34 | 96.18 | 75.42 | 47.68 |
| TTR | 0.84 | 0.40 | 0.60 | 0.49 | 0.48 | 0.52 | 0.55 | 0.58 | 0.55 |
| Avg. Word Len. | 4.26 | 3.38 | 4.11 | 3.88 | 3.85 | 4.02 | 3.95 | 3.72 | 1.52 |
| Vowel Ratio | 0.00051 | 0.00089 | 0.00117 | 0.01105 | 0.01100 | 0.01012 | 0.00985 | 0.00872 | 0.06 |
| Punctuation Var. | 11.80 | 15.75 | 15.85 | 17.70 | 17.61 | 16.45 | 16.12 | 14.80 | 12.45 |
| Repetition | 0.11 | 0.14 | 0.16 | 0.15 | 0.15 | 0.13 | 0.14 | 0.12 | 0.23 |
| Feature | Human | Gemma QA | GPT | LLaMA | Mistral | ChatGPT | Gemini | Aya | Sarvam |
|---|---|---|---|---|---|---|---|---|---|
| Avg. Word per Sent. | 26.64 | 18.94 | 23.31 | 19.59 | 40.08 | 27.85 | 25.12 | 21.34 | 26.45 |
| Avg. Syll. per Word | 1.50 | 1.42 | 1.43 | 1.46 | 1.02 | 1.38 | 1.36 | 1.41 | 1.12 |
| Flesch Reading | 53.15 | 67.27 | 62.02 | 63.78 | 79.63 | 58.40 | 61.12 | 65.35 | 80.34 |
| Fog Index | 12.06 | 8.75 | 10.46 | 9.09 | 16.72 | 11.25 | 10.85 | 9.72 | 11.14 |
| SMOG Index | 7.38 | 5.90 | 6.64 | 6.27 | 6.22 | 6.91 | 6.58 | 6.12 | 4.55 |
| Sentence Len. Var. | 4.18 | 8.54 | 5.81 | 9.23 | 7.91 | 6.42 | 5.97 | 7.10 | 6.35 |
| Feature | Human | Gemma QA | GPT | LLaMA | Mistral | ChatGPT | Gemini | Aya | Sarvam |
|---|---|---|---|---|---|---|---|---|---|
| Stopword Count | 6.60 | 31.29 | 23.44 | 128.68 | 136.94 | 88.52 | 75.63 | 52.74 | 18.35 |
| Stopword Density | 0.20 | 0.15 | 0.16 | 0.15 | 0.15 | 0.14 | 0.14 | 0.17 | 0.09 |
| Matra Count | 44.35 | 185.87 | 179.34 | 835.04 | 912.75 | 522.18 | 480.64 | 265.41 | 198.22 |
| BERT Dist. | 0.87 | 0.86 | 0.86 | 0.86 | 0.85 | 0.86 | 0.85 | 0.87 | 0.88 |
| Divergence Metric | Gemma QA | GPT | LLaMA | Mistral | ChatGPT | Gemini | Aya | Sarvam |
|---|---|---|---|---|---|---|---|---|
| 3.0207 | 2.1466 | 1.0611 | 3.2102 | 2.3846 | 2.6173 | 2.7935 | 2.4581 | |
| 2.3997 | 3.0345 | 2.4147 | 3.0036 | 2.7684 | 2.5831 | 2.9417 | 2.6259 | |
| JS Divergence | 0.3640 | 0.3729 | 0.3151 | 0.5567 | 0.3918 | 0.4086 | 0.4362 | 0.3785 |
| Metric | SHAKTI | AGhi | Urdu | Arabic | Korean | Chinese | Russian | Bulgarian | English |
|---|---|---|---|---|---|---|---|---|---|
| Train | 352k | 22k | 600 | 4.8k | 1.2k | 3.6k | 3.6k | 3.6k | 16k |
| Dev | 117k | 7k | 200 | 1.6k | 400 | 1.2k | 1.2k | 1.2k | 4k |
| Test | 117k | 7k | 200 | 1.6k | 400 | 1.2k | 1.2k | 1.2k | 4k |
| Threshold | 0.875 | 0.861 | 0.758 | 0.856 | 0.802 | 0.831 | 0.689 | 0.806 | 0.816 |
| Model | SHAKTI | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Genre | QA | News | Wiki | Essay | News | ||||||||||
| Acc | F1-H | F1-AI | Acc | F1-H | F1-AI | Acc | F1-H | F1-AI | Acc | F1-H | F1-AI | Acc | F1-H | F1-AI | |
| Log-Likelihood [ 17 ] | 0.29 | 0.28 | 0.31 | 0.29 | 0.26 | 0.32 | 0.32 | 0.30 | 0.34 | 0.27 | 0.25 | 0.29 | 0.30 | 0.28 | 0.31 |
| Rank [ 17 ] | 0.43 | 0.41 | 0.44 | 0.45 | 0.44 | 0.46 | 0.44 | 0.43 | 0.45 | 0.42 | 0.41 | 0.43 | 0.45 | 0.44 | 0.47 |
| Log-Rank [ 17 ] | 0.45 | 0.44 | 0.47 | 0.47 | 0.46 | 0.49 | 0.47 | 0.46 | 0.48 | 0.45 | 0.44 | 0.47 | 0.47 | 0.45 | 0.50 |
| Entropy [ 17 ] | 0.49 | 0.48 | 0.50 | 0.50 | 0.49 | 0.52 | 0.51 | 0.50 | 0.51 | 0.48 | 0.47 | 0.49 | 0.51 | 0.50 | 0.53 |
| Model Type | Model | Urdu | Arabic | Korean | Russian | Bulgarian | Chinese | RAID (English) | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | ||
| Static Threshold | GPT-3.5 | 0.90 | 0.88 | 0.91 | 0.85 | 0.89 | 0.81 | 0.77 | 0.76 | 0.90 | 0.89 | 0.87 | 0.86 | 0.82 | 0.81 |
| TRIDENT-S | LLaMA 3.3 | 0.86 | 0.83 | 0.88 | 0.81 | 0.89 | 0.84 | 0.75 | 0.71 | 0.72 | 0.74 | 0.84 | 0.86 | 0.86 | 0.87 |
| Static Threshold | Gemma 3 | 0.85 | 0.82 | 0.77 | 0.73 | 0.81 | 0.72 | 0.70 | 0.69 | 0.87 | 0.88 | 0.83 | 0.82 | 0.71 | 0.72 |
| Phi-4 | 0.84 | 0.81 | 0.79 | 0.79 | 0.82 | 0.78 | 0.71 | 0.68 | 0.81 | 0.81 | 0.81 | 0.79 | 0.76 | 0.75 | |
| Dynamic Threshold | GPT-3.5 | 0.86 | 0.81 | 0.87 | 0.83 | 0.86 | 0.81 | 0.76 | 0.75 | 0.88 | 0.88 | 0.84 | 0.83 | 0.82 | 0.79 |
| Attack | TRIDENT-S | TRIDENT-D | TRIDENT-C | |||
|---|---|---|---|---|---|---|
| Acc | Macro-F1 | Acc | Macro-F1 | Acc | Macro-F1 | |
| Rewrite | 0.88 | 0.815 | 0.71 | 0.755 | 0.75 | 0.795 |
| Homoglyph | 0.80 | 0.745 | 0.73 | 0.675 | 0.77 | 0.715 |
| Synonym Replacement | 0.72 | 0.665 | 0.66 | 0.605 | 0.70 | 0.645 |
| Misspelling | 0.52 | 0.475 | 0.45 | 0.405 | 0.49 | 0.445 |
| Whitespace | 0.76 | 0.705 | 0.70 | 0.645 | 0.74 | 0.685 |
| RTT Model | Training Generators | Test Generator | Accuracy | F1-H | F1-AI |
|---|---|---|---|---|---|
| GPT-3.5 | GPT-3.5, Mistral, Gemma-3 | LLaMA-3.2 | 0.87 | 0.83 | 0.89 |
| GPT-3.5 | GPT-3.5, LLaMA-3.2, Gemma-3 | Mistral | 0.84 | 0.80 | 0.89 |
| GPT-3.5 | GPT-3.5, LLaMA-3.2, Mistral | Gemma-3 | 0.86 | 0.82 | 0.88 |
| GPT-3.5 | LLaMA-3.2, Mistral, Gemma-3 | GPT-3.5 | 0.82 | 0.78 | 0.87 |
| Metric | Score Labels and Descriptions |
|---|---|
| Fluency | 3 – Excellent: No grammatical errors; text is clear and easy to understand. 2 – Moderate: Some grammatical errors or awkward phrasing, but the meaning remains understandable. 1 – Poor: Frequent grammatical errors; text is difficult to understand. |
| Origin | 1 – Human-like: Text appears natural and resembles human-written content. 0 – LLM-like: Text appears formulaic or resembles LLM-generated content. |
| Coherence | 1 – Coherent: Ideas are logically connected, consistent, and flow smoothly. 0 – Incoherent: Ideas are disconnected, inconsistent, or difficult to follow. |
| Factuality | 5 – Completely accurate: All relevant factual claims are correct and truthful. 4 – Mostly accurate: Main claims are correct, with minor omissions or inaccuracies. 3 – Partially accurate: Some claims are correct, but others are vague or imprecise. 2 – Substantially inaccurate: Noticeable factual errors or misleading statements are present. 1 – Mostly incorrect: Most claims are incorrect, irrelevant, or substantially misleading. |
| Variant | Component | Configuration |
| Shared Configuration | ||
| All variants | RTT transformation | |
| All variants | Sentence encoder | XLM-RoBERTa (pretrained) |
| All variants | Similarity measure | Cosine similarity |
| Variant 1: TRIDENT-S (Static Threshold Classification) | ||
| Static | Threshold selection | ROC-based validation |
AI-Generated Text Detection
AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.
See all 185 papers