Who Wrote the Book? Detecting and Attributing LLM Ghostwriters

Anudeex Shetty; Qiongkai Xu; Olga Ohrimenko; Jey Han Lau

cs.CLMar 30, 2026

Who Wrote the Book? Detecting and Attributing LLM Ghostwriters

Authors: Anudeex Shetty, Qiongkai Xu, Olga Ohrimenko, Jey Han Lau

Organizations: School of Computing and Information Systems, The University of Melbourne, Australia · School of Computing, FSE, Macquarie University, Australia

Abstract

In this paper, we introduce GhostWriteBench, a dataset for LLM authorship attribution. It comprises long-form texts (50K+ words per book) generated by frontier LLMs, and is designed to test generalisation across multiple out-of-distribution (OOD) dimensions, including domain and unseen LLM author. We also propose TRACE -- a novel fingerprinting method that is interpretable and lightweight -- that works for both open- and closed-source models. TRACE creates the fingerprint by capturing token-level transition patterns (e.g., word rank) estimated by another lightweight language model. Experiments on GhostWriteBench demonstrate that TRACE achieves state-of-the-art performance, remains robust in OOD settings, and works well in limited training data scenarios.

Figures & tables

Dataset	Task	LLM Generators		OOD		Long	Avg.	Num.
Dataset	Type	Num.	Frontier †	Domain	Author	Form	Length	Docs
TuringBench ${}_{(\cite[citeyear]{\@@bibref{Year}{uchendu2021turingbench}{}{}})}$	Multi	20	✗	✗	✗	✗	<200	160K
MGTBench ${}_{(\cite[citeyear]{\@@bibref{Year}{he2024mgtbench}{}{}})}$	Binary	6	✗	✗	✗	✗	<500	21K
RAID ${}_{(\cite[citeyear]{\@@bibref{Year}{dugan-etal-2024-raid}{}{}})}$	Binary	11	✗	✓	✗	✗	<250	500K
M4GT-Bench ${}_{(\cite[citeyear]{\@@bibref{Year}{wang2024m4gt}{}{}})}$	Multi	6	✗	✓	✗	✗	<500	87K
OpenTuring ${}_{(\cite[citeyear]{\@@bibref{Year}{la2025openturingbench}{}{}})}$	Multi	7	✗	✓	✗	✗	<500	497K

Table 1: Overview of existing datasets for LLM-generated text attribution. Most prior work focuses on binary classification rather than multi-class authorship attribution (AA) tasks. LLMs included in these benchmarks are largely outdated (i.e., they are small-scale,

7

B–

10

B parameters, models and/or not the latest frontier models), as indicated by the ‘Frontier’ column. While some studies include out-of-distribution (OOD) evaluation, this is rarely considered for AA tasks. ‘Author’ OOD denotes open-set evaluation where authors unseen during training are evaluated. ‘Long Form’ refers to documents longer than

10

K words. We report only AI-generated texts for ‘Avg. Length’ (in words) and ‘Num. Docs’.

Model	Total	Avg. #	Text Quality and Diversity
Model	Books	Words	PPL ( $\downarrow$ )	S-B ( $\downarrow$ )	S-R ( $\downarrow$ )	NGD ( $\uparrow$ )	CR ( $\downarrow$ )
deepseek-chat	$42$	$33.4$ K	$21.62$	$0.001$	$8.22$	$2.29$	$2.96$
glm-4.6	$32$	$28.4$ K	$33.06$	$0.002$	$7.69$	$2.46$	$2.90$
kimi-k2	$37$	$37.1$ K	$27.87$	$0.002$	$8.76$	$2.24$	$2.81$
qwen3-235b	$38$	$67.1$ K	$21.30$	$0.004$	$10.04$	$2.18$	$2.72$
qwen3-max	$34$	$41.2$ K	$24.96$	$0.003$	$9.01$	$2.29$	$2.73$

Table 2: GhostWriteBench statistics: total books (genre splits in Appendix Table 10 ), average number of words in book, and text quality and diversity metrics for each LLM (and 100 random human books). Metrics include perplexity (PPL), self-BLEU (S-B), repetition (S-R),

n

-gram diversity (NGD), and compression ratio (CR) (more details in Appendix A.2 ). Arrows (

\uparrow

;

\downarrow

) indicate whether higher or lower values are better.

Figure 1: Overview of TRACE fingerprint construction. We model transitions of consecutive token scores (using another lightweight evaluator language model; GPT-2 used in illustration). The process is shown for both Rank-based (§ 4.1 ) and Entropy-based (§ 4.2 ) fingerprint variant. We construct a pool of reference fingerprints from an LLM’s training texts. At inference time, attribution is a straightforward comparison between the test fingerprint and the LLM reference fingerprint.

Figure 2: t-SNE visualisation comparing semantic embeddings ( openai/text-embedding-3-large ; left) and our Entropy-based TRACE fingerprints (right) on single-genre books in GhostWriteBench . Marker shape denotes genre and colour denotes the generating LLM. The left plot shows content-based clustering by genre, while the right plot presents clusters by LLM across genres, indicating that TRACE captures model-specific stylistic patterns rather than genre-specific content features.

Method	Low Resource			High Resource
Method	ID	OOD-Domain	OOD-Author	ID	OOD-Domain	OOD-Author
Rank	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.40\pm 0.19$	$0.27\pm 0.08$	$0.25\pm 0.06$	$0.38\pm 0.21$
Entropy	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.20\pm 0.20$	$0.33\pm 0.06$	$0.35\pm 0.04$	$0.10\pm 0.08$
GLTR	$0.00\pm 0.00$	$0.00\pm 0.00$	$0.44\pm 0.17$	$0.30\pm 0.02$	$0.31\pm 0.02$	$0.28\pm 0.03$
n-gram	$0.40\pm 0.16$	$0.20\pm 0.16$	$0.36\pm 0.31$	$0.96\pm 0.01$	$\bm{0.92\pm 0.04}$	$0.28\pm 0.35$
Bert-AA	$0.44\pm 0.17$	$0.27\pm 0.19$	$0.04\pm 0.06$	$0.96\pm 0.04$	$0.90\pm 0.05$	$0.29\pm 0.05$

Table 3: Results for attribution techniques on GhostWriteBench under low-resource (left) and high-resource (right) training settings for different OOD dimensions (OOD-Domain and OOD-Author). Reported scores are

\text{macro-F}_{1}

(mean

\pm

std for three random splits; see Appendix Table 18 ); bold denotes the best result and underline the second best. Overall, TRACE has strong performance with several existing methods showing limited generalisation.

Method	OOD-Author
Method	LLM	Human
n-gram	$0.32\pm 0.33$	$0.48\pm 0.29$
DeTeCtive	$0.45\pm 0.15$	$0.16\pm 0.07$
TRACE (Ours)
Ranks-JS	$0.19\pm 0.03$	$0.11\pm 0.01$
Entropy-JS	$\bm{0.54\pm 0.15}$	$\bm{1.00\pm 0.00}$

Table 4: Performance on the OOD-Author variant consisting of unseen human authors. TRACE achieves near-perfect performance, demonstrating its ability to distinguish human-written vs. LLM-generated texts.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Code	Genre
LF	Literature & Fiction
HB	History & Biographies
BLP	Business, Law & Politics
ST	Science & Technology
ACM	Art, Culture & Media
SSP	Social Sciences & Philosophy

Appendix

Table 5: Different genres used in this work, sourced from Project Gutenberg ( Momen et al., 2025 ) .

LLM	Book Generation
LLM	Steps	Outline	Segment	Total
deepseek-chat	$39.90$	$0.6$ K	$0.8$ K	$33.4$ K
glm-4.6	$14.12$	$1.8$ K	$2.0$ K	$28.4$ K
kimi-k2	$15.65$	$2.4$ K	$2.4$ K	$37.1$ K
qwen3-235b	$38.47$	$2.0$ K	$1.7$ K	$67.1$ K
qwen3-max	$27.53$	$1.4$ K	$1.5$ K	$41.2$ K

Appendix

Table 6: Book generation statistics across different LLMs. All reported numbers (other than steps) are in words.

LLM	Genre	Passage-level					Overall
LLM	Genre	Rel.	Eng.	Coh.	Flu.	Div.	Coh.	Emp.	Sur.	Eng.	Comp.
deepseek-chat	SSP	$3.2$	$2.6$	$3.4$	$3.4$	$4.0$	$3.0$	$3.5$	$4.0$	$3.0$	$3.0$
glm-4.6	HB,SSP,JR	$4.9$	$3.7$	$4.3$	$4.4$	$4.0$	$4.0$	$4.0$	$3.0$	$4.0$	$3.0$
kimi-k2	LF	$4.8$	$4.4$	$4.2$	$4.2$	$4.4$	$4.0$	$5.0$	$3.0$	$5.0$	$4.0$
qwen3-235b	SSP,LF	$4.6$	$3.7$	$4.7$	$4.8$	$4.7$	$4.0$	$3.0$	$3.5$	$3.5$	$4.0$
qwen3-max	LHH,ST	$3.7$	$3.0$	$4.4$	$4.0$	$3.7$	$3.0$	$3.5$	$4.0$	$2.5$	$2.5$

Appendix

Table 7: Human evaluation scores across passage-level and overall narrative quality dimensions for different LLMs. Rel. - Relevance, Eng. - Engagement, Coh. - Coherence, Flu. - Fluency, Div. - Diversity, Emp. - Empathy, Sur. - Surprise, and Comp. - Complexity. 1 is the lowest and 5 is the highest score.

LLM	API Cost ( $/$ 1$ M)		Total
LLM	Input	Output	(in $)
deepseek-chat	$0.2$	$0.5$	$10.6$
glm-4.6	$0.5$	$1.8$	$10.9$
kimi-k2	$0.5$	$2.0$	$22.1$
qwen3-235b	$0.1$	$0.1$	$9.0$
qwen3-max	$1.2$	$6.0$	$60.2$

Appendix

Table 8: GhostWriteBench generation cost breakdown. ‘Input’ and ‘Output’ denote API cost per

1

M tokens. ‘Total’ is the cost of generating all books for a given LLM (as per the statistics in Table 2 ). OpenRouter API costs are as of September 2025.

Figure 3: An example where TRACE captures similar gpt-4.1 fingerprints across domains, demonstrating generalisation. Left: a training reference fingerprint; Middle: a test sample from seen domain ( HB ); Right: a test sample from an unseen domain ( SSP,LHH ).

Model Name	OpenRouter Checkpoint	Context	Max. Out.	Type	Reference
deepseek-chat	deepseek/deepseek-chat-v3-0324	163.8K	163.8K	Open	DeepSeek-AI et al. (2025)
glm-4.6	z-ai/glm-4.6	204.8K	204.8K	Open	GLM Team et al. (2025)
kimi-k2	moonshotai/kimi-k2-thinking	131.1K	131.1K	Open	Kimi Team et al. (2026)
qwen3-235b	qwen/qwen3-235b-a22b-2507	262.1K	262.1K	Open	Yang et al. (2025)
qwen3-max	qwen/qwen3-max	262.1K	32.8K	Closed	Qwen Team (2026)
claude-sonnet	anthropic/claude-sonnet-4.5	1M	64K	Closed	Anthropic (2025)

Appendix

Table 9: Details for all the LLMs used in GhostWriteBench . ‘Context’ is the total input context supported and ‘Max. Out.’ is the maximum number of output tokens supported by the LLM. ‘Type’ indicates whether model weights are public ( Open) or if it is proprietary LLM ( Closed).

Model	Split	Genres	# Books
deepseek-chat	ID	LF	$33$
	OOD-Domain	LHH, HB	$2$
		BLP, HB	$2$
		SSP	$5$
glm-4.6	ID	HB, SSP, JR	$3$
		HB, JR	$2$

Appendix

Table 10: Detailed genre splits (ID vs. OOD-Domain) for different LLMs in GhostWriteBench , complementing Table 2 . These genre splits replicate human book distribution from Project Gutenberg.

Method	OOD-Author
Method	AUROC	AUPR
Rank	$0.49$	$0.62$
Entropy	$0.58$	$0.66$
GLTR	$0.51$	$0.63$
n-gram	$0.90$	$0.89$
Bert-AA	$0.83$	$0.86$

Appendix

Table 11: OOD-Author performance on balanced test set containing both unknown and known author samples. Reported numbers are average threshold-independent metrics (AUROC and AUPR) over three random splits. Entropy-based TRACE is remains performant as seen in the main results.

Method	n-gram	Bert-AA	DeTeCt.	TopForm.	TRACE (Ours)			Avg.
Method	n-gram	Bert-AA	DeTeCt.	TopForm.	Ranks-JS	Entropy-JS	Entropy-Norm	Avg.
deepseek-chat	$0.31$	$0.22$	$0.26$	$0.00$	$0.95$	$0.99$	$0.99$	$0.53$
glm-4.6	$0.28$	$0.14$	$0.46$	$0.25$	$0.32$	$0.31$	$0.28$	$0.29$
kimi-k2	$0.33$	$0.28$	$0.80$	$0.84$	$0.40$	$0.32$	$0.31$	$0.47$
qwen3-235b	$0.33$	$0.00$	$0.22$	$0.22$	$0.00$	$0.22$	$0.44$	$0.20$
qwen3-max	$0.54$	$0.36$	$0.32$	$0.27$	$0.00$	$0.17$	$0.05$	$0.24$

Appendix

Table 12: OOD-Author per-LLM

\text{macro-F}_{1}

scores (averaged over the three splits). ‘Avg.’ column is row-wise average, highlighting some LLMs are particularly more difficult to differentiate.

Figure 4: The impact of using different fingerprint sizes ( $K$ clusters for Rank-based and $G$ grid-size for Entropy-based) for TRACE . The red dashed line denotes the selected configuration.

Figure 5: The impact of $\alpha$ in power law approximation for Rank-based TRACE . The red dashed line denotes the chosen $\alpha=1.5$ .

Figure 6: The impact of the number of tokens considered per text in TRACE .

Figure 7: Evaluation of different metrics for comparing TRACE fingerprints.

Panel of Figure 8 — Figure 8: The impact of different orders used in TRACE fingerprints.

Panel of Figure 9 — Figure 9: Detailed normalised OOD-Domain confusion matrices for different techniques (in sub-captions) without selecting thresholds.

Method	ID	OOD-Domain	OOD-Author
Rank	$0.13\pm 0.04$	$0.13\pm 0.03$	$0.39\pm 0.19$
Entropy	$0.16\pm 0.03$	$0.18\pm 0.02$	$0.15\pm 0.12$
GLTR	$0.15\pm 0.01$	$0.15\pm 0.01$	$0.36\pm 0.07$
n-gram	$0.68\pm 0.08$	$0.56\pm 0.10$	$0.32\pm 0.33$
Bert-AA	$0.70\pm 0.10$	$0.59\pm 0.12$	$0.17\pm 0.05$
TopFormer	$0.73\pm 0.12$	$0.56\pm 0.10$	$0.38\pm 0.05$

Appendix

Table 13: Results on GhostWriteBench for all authors (including both low-resource and high-resource), complementing Table 3 .

Method	ID				OOD-Domain
Method	Top- $1$	Top- $2$	Top- $3$	Top- $5$	Top- $1$	Top- $2$	Top- $3$	Top- $5$
Rank	$0.33\pm 0.04$	$0.46\pm 0.04$	$0.59\pm 0.05$	$0.81\pm 0.05$	$0.33\pm 0.03$	$0.51\pm 0.05$	$0.62\pm 0.04$	$0.86\pm 0.02$
Entropy	$0.42\pm 0.03$	$0.52\pm 0.00$	$0.69\pm 0.08$	$0.86\pm 0.01$	$0.44\pm 0.03$	$0.47\pm 0.01$	$0.75\pm 0.07$	$0.86\pm 0.01$
GLTR	$0.41\pm 0.07$	$0.55\pm 0.03$	$0.67\pm 0.09$	$0.82\pm 0.06$	$0.43\pm 0.08$	$0.52\pm 0.02$	$0.70\pm 0.03$	$0.85\pm 0.04$
n-gram	$0.91\pm 0.02$	$0.97\pm 0.00$	$0.97\pm 0.00$	$0.97\pm 0.00$	$0.87\pm 0.04$	$0.94\pm 0.02$	$0.95\pm 0.01$	$0.97\pm 0.00$
Bert-AA	$0.92\pm 0.05$	$0.96\pm 0.01$	$0.97\pm 0.02$	$0.98\pm 0.03$	$0.85\pm 0.05$	$0.92\pm 0.02$	$0.97\pm 0.04$	$0.98\pm 0.03$

Appendix

Table 14: Top-

k

accuracy results on GhostWriteBench for all authors, complementing Table 3 and Table 13 .

Method	Low Resource		High Resource
Method	ID	OOD-Domain	ID	OOD-Domain
n-gram	$0.47\pm 0.09$	$0.20\pm 0.16$	$0.95\pm 0.01$	$0.92\pm 0.04$
Bert-AA	$0.51\pm 0.23$	$0.24\pm 0.17$	$0.95\pm 0.04$	$0.91\pm 0.06$
TopFormer	$0.69\pm 0.06$	$0.31\pm 0.08$	$0.96\pm 0.04$	$0.91\pm 0.05$
DeTeCtive	$0.58\pm 0.14$	$0.31\pm 0.08$	$0.96\pm 0.04$	$\bm{0.93\pm 0.04}$
TRACE (Ours)

Appendix

Table 15: Without thresholding results for attribution techniques on GhostWriteBench , showing ID and OOD-Domain for Low and High resource settings, complementing Table 3 . Bold denotes the best result and underline the second best.

Method	Low Resource		High Resource
Method	Reject	Family	Reject	Family
n-gram	$0.36\pm 0.31$	$0.19\pm 0.26$	$0.28\pm 0.35$	$0.03\pm 0.04$
Bert-AA	$0.04\pm 0.06$	$0.55\pm 0.10$	$0.29\pm 0.05$	$0.09\pm 0.08$
TopFormer	$0.29\pm 0.17$	$0.55\pm 0.10$	$0.46\pm 0.27$	$0.15\pm 0.13$
DeTeCtive	$0.40\pm 0.20$	$0.49\pm 0.18$	$0.49\pm 0.22$	$0.15\pm 0.11$
TRACE (Ours)

Appendix

Table 16: Variant of OOD-Author evaluation. ‘Family’ column: if the predicted class for the unseen LLM belongs to the same family, we treat as a correct prediction (instead of rejecting: ‘Reject’ column).

Table 17: TRACE performance for different evaluator language models. Each sub-table is for a variant of the TRACE fingerprint. Overall, we observe comparable performance using different evaluator language models.

Scenario	Split 1	Split 2	Split 3
low-resource	gemini-flash	gemini-flash	gemini-pro
	glm-4.6	claude-sonnet	gemini-flash
	deepseek-chat	kimi-k2	qwen3-235b
	gpt-5.1	gpt-5.1	qwen3-max
	qwen3-235b	qwen3-235b	deepseek-chat
high-resource	gemini-pro	gemini-pro	glm-4.6

Appendix

Table 18: Random LLMs splits for low-resource and high-resource scenarios. These splits are released with the GhostWriteBench dataset.

Method	Split 1	Split 2	Split 3
Rank	0.30	0.30	0.35
Entropy	0.30	0.25	0.25
GLTR	0.30	0.30	0.25
n-gram	0.40	0.75	0.40
Bert-AA	0.50	0.50	0.50
TopFormer	0.60	0.60	0.70

Appendix

Table 19: Threshold (tuned as per development set) for different splits (from Table 18 ) and attribution methods used in this work. ‘Norm’ represents the norm-mean metric (as angle) as defined Appendix B.1 and for this lower is better (hence negative values reported). All norm-mean values are multiplied by 100 for presentation.

Figure 10: TRACE Entropy-based reference fingerprints learned for different LLMs (in subcaption) in GhostWriteBench . Each axis represents the entropy of consecutive token transitions estimated by an independent language model; denser regions indicate more frequent transition types. We note distinct structural patterns for each LLM, suggesting they capture their generation styles.

Figure 11: TRACE Entropy-based reference fingerprints (which are very different from LLM counterparts in Figure 10 ) learned for different human authors (in subcaption; sourced from Project Gutenberg) used for unseen authors ablation study Section 6.1 .

Figure 12: TRACE Rank-based reference fingerprints (in log-scale) learned for different LLMs (in subcaption). Although not as pronounced as Entropy-based (as seen in Figure 10 ), some structural patterns are there for each LLM. Note: Rank-based fingerprints have a grid structure due to the discrete nature of rank compared to continuous (scatter plot) entropy values.

Panel of Figure 13 — Figure 13: Generate Initial Summary Prompt, adapted from ( Chang et al., 2024 ) .

Panel of Figure 14 — Figure 14: Generate Intermediate Summary Prompt, adapted from ( Chang et al., 2024 ) .

Panel of Figure 15 — Figure 15: Summary Compression Prompt, adapted from ( Chang et al., 2024 ) .

Panel of Figure 16 — Figure 16: Book Outline Generation Prompt.

Panel of Figure 17 — Figure 17: Book Generation Start Prompt.

Panel of Figure 18 — Figure 18: Book Generation Incremental Prompt.

Explore similar work

Show Me How You Reason and I'll Tell You Who You Are: Reasoning Graphs for Robust LLM Authorship Attribution
Jul 16, 2026Zlata Kikteva, Artur Romazanov, Annette Hautli-Janisz +1Authorship Machine-Generated Text Detection
University of Passau, Germany · University of Dundee, United Kingdom
READER: Robust Evidence-based Authorship Decoding via Extracted Representations
Jun 9, 2026Jiaxu Liu, Sunnan Mu, Dong Huang +3Authorship Decoding
1National University of Singapore · 4Shanghai Artificial Intelligence Laboratory · 2Xidian University +1
One-shot Style Transfer LLM log-probabilities for Authorship Attribution and Verification
Oct 15, 2025Pablo Miralles-González, Javier Huertas-Tato, Alejandro Martín +1Authorship Stylometric
Department of Computer Systems Technical University of Madrid Madrid

Jul 16, 2026, cs.CLJ/K move · Enter open · S save

Show Me How You Reason and I'll Tell You Who You Are: Reasoning Graphs for Robust LLM Authorship Attribution

Zlata Kikteva, Artur Romazanov, Annette Hautli-Janisz, Ramon Ruiz-Dolz

University of Passau, Germany · University of Dundee, United Kingdom

Authorship Machine-Generated Text Detection LLM Reasoning Strategies Attacker Large Language Model Reasoning Graphs Reasoning Skills

Open paper AskPDF arXiv

Given the current trend to employ large language models (LLMs) in almost any imaginable context, LLM-generated text detection and authorship attribution have become a pressing issue. Prior work has primarily focused on surface-level linguistic features, an approach shown to be susceptible to paraphrasing and other obfuscation techniques. In this paper, we go beyond the linguistic surface, extracting and analysing reasoning structures in LLM-generated texts with the goal of capturing more complex signals of LLM authorship. We propose a graph neural network approach that leverages reasoning graphs extracted by an argument mining pipeline, demonstrating improved robustness and generalisation over a traditional Longformer baseline. Our approach outperforms the baseline by up to 27 percentage points under the obfuscation attacks such as paraphrasing and backtranslation, and 19 percentage points when evaluated on the texts generated by the unseen model versions, simulating real-world conditions in which new LLM versions are continuously released.

Abstract

Explore similar work

Show Me How You Reason and I'll Tell You Who You Are: Reasoning Graphs for Robust LLM Authorship Attribution

READER: Robust Evidence-based Authorship Decoding via Extracted Representations

One-shot Style Transfer LLM log-probabilities for Authorship Attribution and Verification