Humanized but Still True? Auditing Facts and Detection in AI-Text Humanizers
Organizations: Independent Researcher · Georgia Institute of Technology · University of Massachusetts Amherst
Abstract
AI "humanizers" rewrite machine-generated text so that detectors classify it as human-written. Public leaderboards score evasion and embedding similarity, but not whether checkable details survive. We check whether numbers, dates, person and organization names, and quoted spans survive each rewrite, using a rule-based checker validated by two AI annotators (precision 0.84-1.00, recall 0.93-0.95; no human annotation). In the October 2026 cycle of a public humanizer leaderboard, all 11 commercial tools changed or dropped at least one of these units in 35-65% of the documents that contained one. Most changes were to quoted spans; counting numbers and names alone, 10 of the 11 tools still altered at least one in 20% or more of the documents. The September 2026 cycle was similar (19-50%). An open 12B humanizer kept at least 97% of numbers and names in fact-dense U.S. federal agency text, but reworded 67-78% of the quoted spans, against its own prompt. Unit retention and detector evasion were not clearly related across tools (Spearman rho = -0.46, 95% CI -0.65 to 0.08), and the open model's four released files (two of them quantization-aware low-bit files) did not show the pre-registered monotone trade-off. On federal texts with matched human, AI-styled and humanized versions, a general-purpose decision model, Jev, asked a single zero-shot question, separated human originals from humanized rewrites with AUROC 0.864 (0.827-0.902), 0.18-0.22 higher than Binoculars, Fast-DetectGPT and the adversarially trained RADAR, while calling 24% of the human originals AI at a 0.5 threshold. On the AI-styled rewrites by 2026 models, both perplexity-based detectors scored below chance (AUROC 0.37 and 0.39), also with line breaks removed. Hypotheses, data rules and analyses were frozen before the main runs; two earlier data contacts are disclosed. We release the checker and the per-item results.
Figures & tables
| Tool | Altered docs | Rate [95% CI] | Numbers | Names | Quoted | Evasion |
|---|---|---|---|---|---|---|
| aihumanize | 13/20 | 0.65 [0.43, 0.82] | 11/56 | 2/23 | 15/15 | 0.79 |
| writehuman | 13/20 | 0.65 [0.43, 0.82] | 13/56 | 8/23 | 15/15 | 0.96 |
| stealthgpt | 11/20 | 0.55 [0.34, 0.74] | 13/56 | 6/23 | 14/15 | 0.60 |
| undetectable-ai | 11/20 | 0.55 [0.34, 0.74] | 5/56 | 1/23 | 12/15 | 0.13 |
| cleverhumanizer | 10/20 | 0.50 [0.30, 0.70] | 7/56 | 5/23 | 14/15 | 0.81 |
| gptinf | 9/20 | 0.45 [0.26, 0.66] | 8/56 | 1/23 | 15/15 | 0.79 |
| Detector | AUROC, draft [95% CI] | AUROC, rewrite [95% CI] | TPR@5%FPR, draft | TPR@5%FPR, rewrite |
|---|---|---|---|---|
| Jev (decision model) | 0.943 [0.915, 0.968] | 0.864 [0.827, 0.902] | 0.728 | 0.565 |
| Fast-DetectGPT | 0.567 [0.494, 0.640] | 0.688 [0.623, 0.752] | 0.076 | 0.201 |
| Binoculars | 0.543 [0.468, 0.617] | 0.654 [0.587, 0.720] | 0.054 | 0.082 |
| RoBERTa detector | 0.435 [0.373, 0.496] | 0.661 [0.610, 0.713] | 0.043 | 0.196 |
| RADAR † | 0.717 [0.666, 0.779] | 0.650 [0.599, 0.708] | 0.043 | 0.052 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Tool | Eligible docs | With quoted spans | Altered, all units | Altered, numbers and names only |
| aihumanize | 20 | 6 | 13/20 (0.65) | 9/19 (0.47) |
| writehuman | 20 | 6 | 13/20 (0.65) | 11/19 (0.58) |
| stealthgpt | 20 | 6 | 11/20 (0.55) | 10/19 (0.53) |
| undetectable-ai | 20 | 6 | 11/20 (0.55) | 6/19 (0.32) |
| cleverhumanizer | 20 | 6 | 10/20 (0.50) | 7/19 (0.37) |
| gptinf | 20 | 6 | 9/20 (0.45) | 5/19 (0.26) |
| Quantity | Estimate | Passages (frozen) | Articles |
|---|---|---|---|
| Jev AUROC, human vs. rewrite (H3) | 0.864 | [0.826, 0.902] | [0.827, 0.902] |
| Jev Binoculars, rewrite | 0.211 | [0.138, 0.284] | [0.132, 0.294] |
| Jev Fast-DetectGPT, rewrite | 0.176 | [0.107, 0.248] | [0.099, 0.256] |
| Jev RADAR † , rewrite | 0.215 | [0.166, 0.263] | [0.157, 0.264] |
| RADAR † , flattened original, rewrite | 0.067 | [0.037, 0.098] | [0.035, 0.098] |
| Retention, bf16 IQ2_XS-QAT (H2b) | 0.008 | [-0.004, 0.021] | [-0.004, 0.020] |
Language Model Auditing
Language model auditing tests whether models behave reliably, fairly, and safely using behavioral probes and internal or deployment evidence.
See all 174 papers