Local Entropy Fluctuations for Black-Box LLM-Generated Text Detection

Xinpeng Wang; Bruce Rushing; Zhizhen Chen; Zhe Zeng

cs.AIOct 24, 2026

Local Entropy Fluctuations for Black-Box LLM-Generated Text Detection

Authors: Xinpeng Wang, Bruce Rushing, Zhizhen Chen, Zhe Zeng

Organizations: University of Virginia

Abstract

Zero-shot detection of machine-generated text is vital for mitigating the risks of large language models (LLMs) misuse. While the state-of-the-art dual-model detectors and resampling-based detectors outperform single-model statistical detectors on unprompted continuations, they suffer a catastrophic breakdown on instruction-conditioned text in the black-box setting, where the source model is unknown. We analyze this failure and trace it to poor prompt conditioning for resampling-based methods and diminished cross-model discrepancy for dual-model detectors. Motivated by recent studies that human writing maintains a steady, uniform information flow, whereas machine-generated text exhibits sharp, localized entropy fluctuations, we propose a new metric, the Relative Peak Prominence Rate (RelPPR) to capture this structural contrast along the entropy trajectory and further leverage it as a signal for LLM-generated text detection. Extensive evaluations across diverse model families and datasets demonstrate that RelPPR achieves state-of-the-art black-box detection performance and remains the most reliable detector under various practical situations. Our code is available at https://github.com/brushing-git/entropy-llm-detector.

Figures & tables

Panel of Figure 1 — Figure 1: (a) RelPPR is robust under instruction-conditioning where existing detectors degrade; (b) Machine-generated text exhibits stronger entropy fluctuations than human text.

Figure 2: Analysis of the DNA-DetectLLM score $\mathrm{Score}_{\mathrm{D}}(s)$ on XSum, with OLMo-3.1-32B-Instruct as the source model. $\mu$ and $\sigma$ denote the mean and standard deviation. Under instruction conditioning, machine text scores shift lower, converging toward the human score distribution.

Panel of Figure 3 — Figure 3: Distributions of RelPPR across four datasets for human text alongside machine-generated text from OLMo-3.1-32B-Instruct. RelPPR consistently separates human writing and machine text generated under instruction conditioning. Results on additional datasets are provided in Figure 12 .

Methods	XSum	WritingP	SQuAD	WikiText	CNN/DM	ELI5	PubMedQA	Avg
Entropy	0.822	0.812	0.813	0.788	0.817	0.929	0.703	0.812
Likelihood	0.787	0.801	0.787	0.752	0.775	0.902	0.600	0.772
LogRank	0.786	0.764	0.782	0.767	0.782	0.894	0.595	0.767
LRR	0.739	0.613	0.731	0.781	0.756	0.831	0.550	0.714
GECScore	0.712	0.673	0.684	0.648	0.688	0.839	0.558	0.686
LastDE	0.755	0.767	0.783	0.785	0.767	0.890	0.590	0.762

Table 1: Detection performance (AUROC) by dataset (averaged across five source models) under black-box setting. The best and second-best results are shown in bold and underlined , respectively.

	OLMo3	OLMo3.1	Gemma4	Mistral	Qwen3.5	Avg
	7B	32B	31B	119B	35B	Avg
Entropy	0.780	0.837	0.811	0.841	0.790	0.812
Likelihood	0.693	0.794	0.803	0.848	0.723	0.772
LogRank	0.699	0.789	0.786	0.838	0.723	0.767
LRR	0.687	0.740	0.692	0.763	0.689	0.714
GECScore	0.646	0.679	0.743	0.631	0.731	0.686

Table 2: AUROC across source models, averaged over seven datasets under the black-box setting. The best and second-best results are shown in bold and underlined , respectively.

Panel of Figure 4 — Figure 4: Detection Performance across decision thresholds, sequence length and decoding configurations on OLMo-3.1-32B-Instruct. Across all evaluations, RelPPR consistently outperforms baselines and exhibits notable robustness.

Panel of Figure 5 — Figure 5: Post-attack AUROC (Left) and AUROC drop (Right) under paraphrasing attack, on OLMo-3.1-32B-Instruct.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	Variant	XSum	SQuAD	WritingP	CNN/DM	ELI5	WikiText	PubMedQA	Average	$\Delta$
w/o Instruction	$\mathrm{Score}_{\mathrm{F}}(s)$	0.766	0.753	0.769	0.708	0.683	0.609	0.719	0.715	$+$ 0.007
w/o Instruction	$N(s)$	0.794	0.749	0.774	0.707	0.687	0.615	0.728	0.722	$+$ 0.007
w/ Instruction	$\mathrm{Score}_{\mathrm{F}}(s)$	0.569	0.514	0.494	0.497	0.556	0.509	0.465	0.515	$0.000$
w/ Instruction	$N(s)$	0.569	0.502	0.501	0.500	0.552	0.510	0.469	0.515	$0.000$

Appendix

Table 3: Per-dataset ROC AUC comparison between

\mathrm{Score}_{\mathrm{F}}(s)

and

N(s)

for OLMo-3-32B-generated text.

\Delta=\mathrm{Avg}_{N}-\mathrm{Avg}_{\mathrm{Score}_{\mathrm{F}}}

. Removing the variance denominator produces virtually no change in AUROC (

|\Delta|\leq 0.007

Panel of Figure 6 — Figure 6: Distributions of the Fast-DetectGPT numerator $N(s)$ and Binoculars score $\mathrm{Score}_{\mathrm{B}}(s)$ on XSum (OLMo-3.1-32B-Instruct).

Panel of Figure 7 — Figure 7: Distributions of the five information-theoretic components from Section 3.2 on ELI5, for machine text generated by OLMo-3.1-32B-Instruct without and with the instruction prompt. The y-axis shows the percentage of samples per bin. $\mu$ and $\sigma$ denote the mean and standard deviation.

Panel of Figure 8 — Figure 8: Distributions of the five information-theoretic components from Section 3.2 on SQuAD, for machine text generated by OLMo-3.1-32B-Instruct without and with the instruction prompt. The y-axis shows the percentage of samples per bin. $\mu$ and $\sigma$ denote the mean and standard deviation.

Panel of Figure 9 — Figure 9: Distributions of the five information-theoretic components from Section 3.2 on CNN/DailyMail, for machine text generated by OLMo-3.1-32B-Instruct without (orange) and with (blue) the instruction prompt. The y-axis shows the percentage of samples per bin. $\mu$ and $\sigma$ denote the mean and standard deviation.

Panel of Figure 10 — Figure 10: Distributions of the five information-theoretic components from Section 3.2 on WikiText, for machine text generated by OLMo-3.1-32B-Instruct without (orange) and with (blue) the instruction prompt. The y-axis shows the percentage of samples per bin. $\mu$ and $\sigma$ denote the mean and standard deviation.

Panel of Figure 11 — Figure 11: Distributions of the five information-theoretic components from Section 3.2 on WritingPrompts, for machine text generated by OLMo-3.1-32B-Instruct without (orange) and with (blue) the instruction prompt. The y-axis shows the percentage of samples per bin. $\mu$ and $\sigma$ denote the mean and standard deviation.

Panel of Figure 12 — Figure 12: Distributions of RelPPR across SQuAD, WikiText, and PubMedQA for human text alongside machine-generated text from OLMo-3.1-32B-Instruct.

	OLMo-7B	OLMo-32B	Gemma-4-31B	Qwen-35B-MoE	Mistral-119B
ELI5	$178.9\pm 12.8$	$174.4\pm 13.6$	$171.2\pm 9.7$	$170.2\pm 11.8$	$182.8\pm 14.6$
PubMedQA	$33.3\pm 6.1$	$34.3\pm 8.9$	$32.0\pm 4.7$	$32.9\pm 6.1$	$32.8\pm 6.2$

Appendix

Table 4: Token length distributions (Mean

\pm

Std) on the ELI5 and PubMedQA across all evaluated instruction models (

n=500

pairs per model), measured with each source model’s own tokenizer.

Dataset	Instruction Prompt Template
XSum	“ Continue the following text for about {word_target} words. Output ONLY the continuation – do not repeat or restate any of the given text: {prefix} ”
CNN/DM	“ Continue the following text for about {word_target} words. Output ONLY the continuation – do not repeat or restate any of the given text: {prefix} ”
WikiText	“ Continue the following text for about {word_target} words. Output ONLY the continuation – do not repeat or restate any of the given text: {prefix} ”
SQuAD	“ Continue the following text for about {word_target} words. Output ONLY the continuation – do not repeat or restate any of the given text: {prefix} ”
WritingPrompts	“ Please write a story of about {word_target} words based on the following writing prompt: {prefix} . Output ONLY the story – do not repeat or restate any of the given text: {prefix} ”
ELI5	“ Please explain the following question in about {word_target} words: {prefix} . Output ONLY the explanation – do not repeat or restate any of the given text: {prefix} ”

Appendix

Table 5: Instruction prompt templates across all seven benchmark datasets. {prefix} represents the conditioning context, and {word_target} denotes the expected word count (150 words, except PubMedQA at 30 words).

Methods	Qwen3.5	Qwen3.5	Qwen3.5	Gemma4	Gemma4	Gemma4	Falcon3	Ministral3	Falcon	Avg
Methods	2B	9B	35B-MoE	12B	31B	26B-MoE	7B	8B	7B	Avg
Entropy	0.837	0.847	0.788	0.804	0.780	0.765	0.761	0.788	0.414	0.754
Likelihood	0.794	0.814	0.746	0.722	0.728	0.720	0.671	0.772	0.526	0.721
LogRank	0.789	0.814	0.749	0.732	0.731	0.723	0.687	0.770	0.535	0.726
LastDE	0.781	0.812	0.755	0.761	0.769	0.760	0.684	0.796	0.657	0.753
Fast-DetectGPT	0.515	0.513	0.456	0.507	0.230	0.301	0.653	0.625	0.825	0.514

Appendix

Table 6: Zero-shot detection AUROC on OLMo-3.1-32B-Instruct (black-box, avg. over all 7 datasets) as the evaluator model varies in size and family (Qwen3.5 ( Qwen Team, 2026 ) , Gemma 4 ( Team et al., 2026 ) , Ministral 3 ( Liu et al., 2026 ) , and Falcon ( Almazrouei et al., 2023 ; Falcon-LLM Team, 2024 ) ). Best result per column is shown in bold , second-best is underlined .

Baseline	XSum	WritingP	SQuAD	WikiText	CNN/DM	ELI5	PubMedQA	Avg
Entropy	0.803	0.856	0.589	0.683	0.724	0.855	0.667	0.740
Likelihood	0.931	0.911	0.708	0.913	0.920	0.907	0.574	0.838
LogRank	0.926	0.901	0.697	0.909	0.915	0.896	0.601	0.835
LRR	0.798	0.797	0.614	0.767	0.777	0.811	0.620	0.741
LastDE	0.887	0.882	0.794	0.941	0.908	0.890	0.534	0.834
LastDE++	0.416	0.555	0.245	0.414	0.485	0.381	0.426	0.418

Appendix

Table 7: Detection performance (AUROC) by dataset (averaged across four source models) under white-box setting. The best and second-best results are shown in bold and underlined , respectively.

Baseline	OLMo3	OLMo3.1	Gemma4	Qwen3.5	Avg
Baseline	7B	32B	31B	35B	Avg
Entropy	0.791	0.805	0.552	0.810	0.740
Likelihood	0.901	0.870	0.776	0.803	0.838
LogRank	0.906	0.866	0.758	0.809	0.835
LRR	0.791	0.791	0.590	0.790	0.741
LastDE	0.893	0.884	0.745	0.813	0.834

Appendix

Table 8: Detection performance (AUROC) by source model (averaged across seven datasets) under white-box setting. The best and second-best results are shown in bold and underlined , respectively.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 170 papers

Once a Response, Always a Response: Detecting LLM-generated Text via Latent Prompt Restoration
Aug 6, 2026Hongrui Bao, Yubing Ren, Yanan Cao +3AI-Generated Text Detection Robustness of Machine-Generated Text Detection
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · College of Computer Science and Technology, Zhejiang University +1
Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions
Mar 18, 2026Madhav S. Baidya, S. S. Baidya, Chirag ChawlaAI-Generated Text Detection Robustness of Machine-Generated Text Detection
Indian Institute of Technology (BHU), Varanasi, India · Indian Institute of Technology Guwahati, India
Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model
Apr 23, 2026Runheng Liu, Heyan Huang, Xingchen Xiao +1Zero-Shot Learning AI-Generated Text Detection
School of Computer Science and Technology, Beijing Institute of Technology

Aug 6, 2026J/K move · Enter open · S save

Once a Response, Always a Response: Detecting LLM-generated Text via Latent Prompt Restoration

Hongrui Bao, Yubing Ren, Yanan Cao, Jinhan You+2

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · College of Computer Science and Technology, Zhejiang University+1

AI-Generated Text Detection Robustness of Machine-Generated Text Detection

Open paper AskPDF arXiv

Large language models (LLMs) can generate fluent and convincing text at scale, creating growing risks for misinformation dissemination, educational misuse, and platform governance. These concerns make robust detection of machine-generated text increasingly necessary. Recent zero-shot detectors mainly exploit probability-based statistical discrepancies, but they do not explicitly account for the training process of LLMs, which leaves a distinct generation mechanism insufficiently modeled and limits detection robustness. To address this issue, we propose EchoPrompt, a training-free detector based on latent prompt restoration. Our key intuition is that machine-generated text is typically produced conditioned on an upstream prompt, and this hidden dependency can be partially reactivated by prepending a unified generic prefix. Specifically, EchoPrompt restores a generic assistant-response context, measures the induced likelihood gain with an instruction-tuned model, calibrates it against the corresponding base model, and aggregates the resulting differences into a score that quantifies latent prompt dependency. Extensive experiments show that EchoPrompt achieves state-of-the-art performance among zero-shot detectors while maintaining strong robustness across challenging evaluation settings.

Abstract

AI-Generated Text Detection

Explore similar work

Once a Response, Always a Response: Detecting LLM-generated Text via Latent Prompt Restoration

Detecting the Machine: A Comprehensive Benchmark of AI-Generated Text Detectors Across Architectures, Domains, and Adversarial Conditions

Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model