How Short Is Too Short? Length-Conditional Calibration and Abstention for Detecting AI-Generated Text
Organizations: University of Cincinnati, USA
Abstract
Detectors of AI-generated text are usually tuned and tested on passages of several hundred tokens, yet many real cases are short. We study score-based detection as a function of passage length. We give a closed form for how the false-positive rate of a threshold set on long text changes as length falls, and a formula for the number of tokens a detector needs to reach a target true-positive rate. We calibrate thresholds within length bins with conformal p-values, and we add a bin-level rule that abstains when the detector cannot be informative at that length. On synthetic scores the formulas agree with simulation, and binned calibration holds the false-positive rate at its nominal value at every length. On three zero-shot detectors (log-likelihood, log-rank, Binoculars), three corpora and two open-weight generators, binned calibration gives a mean false-positive rate of 5.0% in every detector-length cell from 10 to 200 tokens, averaged over 300 random document-level resplits. A single threshold set at 200 tokens gives 29% for Binoculars at 10 tokens and about 1% for log-likelihood, as the closed form predicts. No detector reaches an 80% true-positive rate, and the best, Binoculars at 200 tokens, reaches 59%, so the rule abstains in every cell at that floor and releases only the 200-token bins at floors of 0.3 and 0.5.
Figures & tables
| Single threshold | Per-length calibration | ||||
|---|---|---|---|---|---|
| FPR, measured (Eq. 2 ) | TPR | FPR | TPR | Decision | |
| 10 | 29.5% (30.1%) | 52.5% | 1.02% [0.58, 1.78] | 4.3% | abstain |
| 20 | 22.6% (23.1%) | 53.5% | 1.00% [0.40, 1.69] | 6.8% | abstain |
| 30 | 18.2% (18.4%) | 54.4% | 1.00% [0.44, 1.58] | 9.5% | abstain |
| 50 | 12.3% (12.2%) | 55.6% | 0.97% [0.46, 1.63] | 14.8% | abstain |
| 100 | 5.0% (5.0%) | 57.9% | 0.99% [0.51, 1.69] | 31.2% | abstain |
| Detector | FPR | TPR | UCB | Decision | |
|---|---|---|---|---|---|
| Log-lik. | 10 | 4.84% [3.82, 6.02] | 3.71% [3.25, 4.21] | 0.038 | A/A/A |
| 20 | 3.80% [2.90, 4.88] | 5.11% [4.57, 5.69] | 0.065 | A/A/A | |
| 30 | 5.46% [4.36, 6.76] | 9.73% [8.99, 10.52] | 0.101 | A/A/A | |
| 50 | 4.88% [3.85, 6.10] | 9.73% [8.99, 10.51] | 0.097 | A/A/A | |
| 100 | 4.42% [3.42, 5.61] | 20.80% [19.77, 21.87] | 0.231 | A/A/A | |
| 200 | 5.79% [4.46, 7.37] | 36.31% [34.86, 37.79] | 0.361 | R/A/A |
| Detector | Per-length calibration | Single threshold | Coarse bin | |
|---|---|---|---|---|
| Log-lik. | 10 | 4.97% [3.57, 6.69] | 1.0% | 2.1% |
| 20 | 5.01% [3.62, 6.75] | 1.3% | 2.9% | |
| 30 | 4.96% [3.58, 6.44] | 2.0% | 4.1% | |
| 50 | 4.98% [3.60, 6.77] | 3.5% | 6.6% | |
| 100 | 5.00% [3.59, 6.55] | 3.4% | 8.3% | |
| 200 | 4.95% [3.37, 6.76] | 5.8% | – |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Realized FPR | |
|---|---|
| 50 | no valid threshold |
| 100 | 0.93% [0.02, 3.71] |
| 200 | 0.94% [0.09, 2.40] |
| 500 | 1.05% [0.28, 2.15] |
| 1000 | 1.03% [0.52, 1.78] |
| 2000 | 1.00% [0.64, 1.37] |
| Log-likelihood | Binoculars | |
|---|---|---|
| 50 | 4.09% [0.27, 10.10] | 3.70% [0.33, 10.44] |
| 100 | 5.11% [1.87, 9.77] | 4.73% [1.87, 10.17] |
| 200 | 5.19% [3.14, 8.49] | 4.72% [2.68, 7.42] |
| 500 | 5.14% [3.68, 7.56] | 4.79% [3.01, 6.42] |
| 1000 | 5.09% [4.15, 6.42] | 4.83% [3.68, 5.89] |
| 1500 | 5.08% [4.28, 5.95] | 4.82% [4.01, 5.69] |
| Detector | FPR by corpus | TPR by corpus | TPR by decoding | TPR by generator | |
|---|---|---|---|---|---|
| Log-lik. | 50 | 0.8 / 0.6 / 13.2 | 11.9 / 4.8 / 12.3 | 7.2 / 12.3 | 15.8 / 3.6 |
| 200 | 34.6 / 0.0 / 10.1 | 53.8 / 25.7 / 46.0 | 27.5 / 45.2 | 61.1 / 11.5 | |
| Log-rank | 50 | 0.8 / 2.1 / 13.0 | 11.4 / 5.2 / 11.4 | 7.2 / 11.6 | 15.6 / 3.2 |
| 200 | 11.5 / 0.2 / 10.3 | 52.9 / 27.5 / 45.7 | 27.7 / 46.3 | 63.3 / 10.7 | |
| Binoculars | 50 | 6.7 / 5.6 / 2.4 | 13.5 / 41.4 / 10.5 | 21.8 / 21.3 | 18.6 / 24.6 |
| 200 | 3.8 / 12.5 / 1.4 | 42.3 / 85.4 / 33.8 | 58.3 / 59.9 | 58.5 / 59.7 |
AI-Generated Text Detection
AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.
See all 185 papers