Unveiling Spectral Mechanisms in Training-Free LLM Text Detection
Organizations: Institute of Computing Technology, Chinese Academy of Sciences
Abstract
The rapid advancement of Large Language Models (LLMs) makes it increasingly difficult to distinguish human writing from machine-generated text. Training-free detection offers a scalable solution, yet common confidence-based metrics mainly measure average token probabilities and often miss the signal fluctuations that characterize human writing, which we call "generative vitality". Spectral analysis offers a way to capture this vitality, but its mechanism and practical boundaries remain underexplored. In this paper, we analyze spectral detection from both theoretical and empirical perspectives. We connect spectral energy to variance in proxy log-probability trajectories and explain how broader human token choices create the fluctuations used by frequency-domain indicators. We further show that the strength of this signal depends on text length and sampling range: spectral evidence is clearest for long, continuous, constrained generation, while short, fragmented, mixed, and edited settings require complementary confidence and fluctuation views. These findings clarify when frequency-domain detection works and provide guidance for future multi-dimensional detector design.
Figures & tables
| Metric | Statistical Interpretation | Conf. | Fluct. |
| LogLikelihood (LL) | Measures average token log-probability. | ||
| LogRank (LR) | Average of log-rank scores for token positions. | ||
| LRR | Ratio of log-likelihood to log-rank (relative confidence). | ||
| Entropy | Average predictive uncertainty of the distribution. | ||
| SpecDetect | Frequency-domain modeling of fluctuations. | ||
| Lastde | Ratio of log-likelihood to multi-scale entropy. |
| Metric | Pure Generation | Collaborative Mixed | ||
|---|---|---|---|---|
| SemEval | CoAuthor (H vs. L) | CoAuthor (H vs. C) | CoAuthor (L vs. C) | |
| Confidence-based | ||||
| LogLikelihood | 0.9085 | 0.8044 | 0.6459 | 0.6951 |
| LogRank | 0.9038 | 0.8138 | 0.6514 | 0.7010 |
| LRR | 0.2544 | 0.3698 | 0.4425 | 0.4081 |
| Entropy | 0.8655 | 0.4322 | 0.4701 | 0.4759 |
| Metric | AI-Polishing (GPT-4) | AI-Polishing (Llama-2) | Humanizing | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Polish | Polish | Complete | Rewrite | Polish | Polish | Complete | Rewrite | Adapt | Adapt | Llama-2 | GPT-4 | |
| Token | Sentence | (1/3+2/3) | (Re-gen) | Token | Sentence | (1/3+2/3) | (Re-gen) | Token | Sentence | Humanize | Humanize | |
| Confidence-based Metrics | ||||||||||||
| LogLikelihood | 0.6500 | 0.6167 | 0.6500 | 0.4067 | 0.7433 | 0.8233 | 0.8100 | 0.9533 | 0.7267 | 0.8300 | 0.7333 | 0.9667 |
| LogRank | 0.6033 | 0.5800 | 0.6667 | 0.4233 | 0.6967 | 0.8300 | 0.7967 | 0.9600 | 0.7133 | 0.8300 | 0.7333 | 0.9600 |
| LRR | 0.3067 | 0.3133 | 0.5300 | 0.4933 | 0.3067 | 0.2700 | 0.1733 | 0.2767 | 0.2100 | 0.2167 | 0.2900 | 0.0800 |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Sub-task | LLM Engine | Samples | Avg. Len. | Level | Mixed | Collab |
|---|---|---|---|---|---|---|---|
| SemEval | Prefix-Comp. | GPT-3 | 4,154 | 19.76 | Sent. | ✓ | |
| CoAuthor | Interleaved | GPT-3.5 | 1,445 | 14.71 | Sent. | ✓ | ✓ |
| MixText | AI-Polishing | GPT-4, Llama-2 | 3,000 | 130.18 | Doc. | ✓ | |
| Humanizing | GPT-4, Llama-2 | 1,500 | 128.66 | Doc. | ✓ |
| Metric | Pure Generation | Collaborative Mixed | ||
|---|---|---|---|---|
| SemEval | CoAuthor (H vs. L) | CoAuthor (H vs. C) | CoAuthor (L vs. C) | |
| Confidence-based | ||||
| LogLikelihood | 0.8698 | 0.6415 | 0.7083 | 0.5839 |
| LogRank | 0.8449 | 0.6483 | 0.7221 | 0.5982 |
| LRR | 0.2190 | 0.5152 | 0.5152 | 0.5276 |
| Entropy | 0.2981 | 0.5306 | 0.5306 | 0.4751 |
| Metric | AI-Polishing (GPT-4) | AI-Polishing (Llama-2) | Humanizing | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Polish | Polish | Complete | Rewrite | Polish | Polish | Complete | Rewrite | Adapt | Adapt | Llama-2 | GPT-4 | |
| Token | Sentence | (1/3+2/3) | (Re-gen) | Token | Sentence | (1/3+2/3) | (Re-gen) | Token | Sentence | Humanize | Humanize | |
| Confidence-based Metrics | ||||||||||||
| LogLikelihood | 0.3733 | 0.6267 | 0.5667 | 0.5300 | 0.6933 | 0.8133 | 0.9633 | 0.7733 | 0.7600 | 0.8833 | 0.7267 | 0.9900 |
| LogRank | 0.5033 | 0.5167 | 0.6467 | 0.3767 | 0.6833 | 0.7867 | 0.9767 | 0.7567 | 0.7200 | 0.8600 | 0.7100 | 0.9733 |
| LRR | 0.3200 | 0.3367 | 0.5100 | 0.5333 | 0.3133 | 0.2100 | 0.1067 | 0.2600 | 0.1467 | 0.1267 | 0.2967 | 0.0433 |
| Setting | Direction | Operation | Mean density | Best metric | Pair acc. |
|---|---|---|---|---|---|
| GPT-4 | H AI | Polish-token | 0.336 | Entropy | 0.7867 |
| GPT-4 | H AI | Polish-sentence | 0.800 | Entropy | 0.7067 |
| GPT-4 | H AI | Complete | 1.089 | SpecDetect | 0.8100 |
| GPT-4 | H AI | Rewrite | 1.051 | LRR | 0.4933 |
| Llama-2 | H AI | Polish-token | 0.635 | Entropy | 0.8233 |
| Llama-2 | H AI | Polish-sentence | 0.810 | LogRank / Lastde | 0.8300 |
AI-Generated Text Detection
AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.
See all 171 papers