Detectors of AI-generated text are usually tuned and tested on passages of several hundred tokens, yet many real cases are short. We study score-based detection as a function of passage length. We give a closed form for how the false-positive rate of a threshold set on long text changes as length falls, and a formula for the number of tokens a detector needs to reach a target true-positive rate. We calibrate thresholds within length bins with conformal p-values, and we add a bin-level rule that abstains when the detector cannot be informative at that length. On synthetic scores the formulas agree with simulation, and binned calibration holds the false-positive rate at its nominal value at every length. On three zero-shot detectors (log-likelihood, log-rank, Binoculars), three corpora and two open-weight generators, binned calibration gives a mean false-positive rate of 5.0% in every detector-length cell from 10 to 200 tokens, averaged over 300 random document-level resplits. A single threshold set at 200 tokens gives 29% for Binoculars at 10 tokens and about 1% for log-likelihood, as the closed form predicts. No detector reaches an 80% true-positive rate, and the best, Binoculars at 200 tokens, reaches 59%, so the rule abstains in every cell at that floor and releases only the 200-token bins at floors of 0.3 and 0.5.
Figures & tables
Single threshold
Per-length calibration
n
FPR, measured (Eq. 2 )
TPR
FPR
TPR
Decision
10
29.5% (30.1%)
52.5%
1.02% [0.58, 1.78]
4.3%
abstain
20
22.6% (23.1%)
53.5%
1.00% [0.40, 1.69]
6.8%
abstain
30
18.2% (18.4%)
54.4%
1.00% [0.44, 1.58]
9.5%
abstain
50
12.3% (12.2%)
55.6%
0.97% [0.46, 1.63]
14.8%
abstain
100
5.0% (5.0%)
57.9%
0.99% [0.51, 1.69]
31.2%
abstain
Table 1. Synthetic check at nominal 1% FPR. The calibrated column gives the mean realized FPR over 200 calibration draws, with the central 95% range in brackets.
Detector
n
FPR
TPR
UCB
Decision
Log-lik.
10
4.84% [3.82, 6.02]
3.71% [3.25, 4.21]
0.038
A/A/A
20
3.80% [2.90, 4.88]
5.11% [4.57, 5.69]
0.065
A/A/A
30
5.46% [4.36, 6.76]
9.73% [8.99, 10.52]
0.101
A/A/A
50
4.88% [3.85, 6.10]
9.73% [8.99, 10.51]
0.097
A/A/A
100
4.42% [3.42, 5.61]
20.80% [19.77, 21.87]
0.231
A/A/A
200
5.79% [4.46, 7.37]
36.31% [34.86, 37.79]
0.361
R/A/A
Table 2. Real-detector results at α=0.05 on the held-out test split, with 95% Clopper–Pearson intervals. UCB is the upper confidence bound of the validation-set TPR. The last column gives the decision at γ=0.3/0.5/0.8 (R: report, A: abstain).
Detector
n
Per-length calibration
Single threshold
Coarse bin
Log-lik.
10
4.97% [3.57, 6.69]
1.0%
2.1%
20
5.01% [3.62, 6.75]
1.3%
2.9%
30
4.96% [3.58, 6.44]
2.0%
4.1%
50
4.98% [3.60, 6.77]
3.5%
6.6%
100
5.00% [3.59, 6.55]
3.4%
8.3%
200
4.95% [3.37, 6.76]
5.8%
–
Table 3. Test-split FPR at nominal 5%. Per-length calibration: mean over 300 random document-level resplits, with the central 95% range of the realized FPR in brackets. Single threshold: set on the n=200 calibration set. Coarse bin: one threshold from the pooled calibration sets of n=10 to 100.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
m
Realized FPR
50
no valid threshold
100
0.93% [0.02, 3.71]
200
0.94% [0.09, 2.40]
500
1.05% [0.28, 2.15]
1000
1.03% [0.52, 1.78]
2000
1.00% [0.64, 1.37]
Appendix
Table 4. Synthetic calibration-set size ablation at n=50 , nominal 1% FPR. Mean realized FPR over 200 draws, with the central 95% range in brackets.
m
Log-likelihood
Binoculars
50
4.09% [0.27, 10.10]
3.70% [0.33, 10.44]
100
5.11% [1.87, 9.77]
4.73% [1.87, 10.17]
200
5.19% [3.14, 8.49]
4.72% [2.68, 7.42]
500
5.14% [3.68, 7.56]
4.79% [3.01, 6.42]
1000
5.09% [4.15, 6.42]
4.83% [3.68, 5.89]
1500
5.08% [4.28, 5.95]
4.82% [4.01, 5.69]
Appendix
Table 5. Real-data calibration-set size ablation at n=50 , nominal 5% FPR. Mean realized FPR on the fixed test split over 1,000 subsamples of m passages from the 4,500 calibration and train human passages, with the central 95% range in brackets.
Detector
n
FPR by corpus
TPR by corpus
TPR by decoding
TPR by generator
Log-lik.
50
0.8 / 0.6 / 13.2
11.9 / 4.8 / 12.3
7.2 / 12.3
15.8 / 3.6
200
34.6 / 0.0 / 10.1
53.8 / 25.7 / 46.0
27.5 / 45.2
61.1 / 11.5
Log-rank
50
0.8 / 2.1 / 13.0
11.4 / 5.2 / 11.4
7.2 / 11.6
15.6 / 3.2
200
11.5 / 0.2 / 10.3
52.9 / 27.5 / 45.7
27.7 / 46.3
63.3 / 10.7
Binoculars
50
6.7 / 5.6 / 2.4
13.5 / 41.4 / 10.5
21.8 / 21.3
18.6 / 24.6
200
3.8 / 12.5 / 1.4
42.3 / 85.4 / 33.8
58.3 / 59.9
58.5 / 59.7
Appendix
Table 6. Pooled-threshold rates (%) by subgroup. FPR is by corpus; TPR is by corpus, by decoding setting, and by generator.
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt · National Research Center for Applied Cybersecurity ATHENE, Germany · Zuse School ELIZA
Beijing Institute of Computer Technology and Application, Beijing, China · Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, China · Southeast University, Nanjing, China +1