StoryScope: Investigating idiosyncrasies in AI fiction

Jenna Russell; Rishanth Rajendhran; Chau Minh Pham; Mohit Iyyer; John Wieting

cs.CLApr 3, 2026

StoryScope: Investigating idiosyncrasies in AI fiction

Authors: Jenna Russell, Rishanth Rajendhran, Chau Minh Pham, Mohit Iyyer, John Wieting

Organizations: University of Maryland, College Park · Google DeepMind

Abstract

As AI-generated fiction becomes increasingly prevalent, questions of authorship and originality are becoming central to how written work is evaluated. While most existing work in this space focuses on identifying surface-level signatures of AI writing, we ask instead whether AI-generated stories can be distinguished from human ones without relying on stylistic signals, focusing on discourse-level narrative choices such as character agency and chronological discontinuity. We propose StoryScope, a pipeline that automatically induces a fine-grained, interpretable feature space of discourse-level narrative features across 10 dimensions. We apply StoryScope to a parallel corpus of 10,272 writing prompts, each written by a human author and five LLMs, yielding 61,608 stories, each ~5,000 words, and 304 extracted features per story. Narrative features alone achieve 93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution, retaining over 97% of the performance of models that include stylistic cues. A compact set of 30 core narrative features captures much of this signal: AI stories over-explain themes and favor tidy, single-track plots while human stories frame protagonist' choices as more morally ambiguous and have increased temporal complexity. Per-model fingerprint features enable six-way attribution: for example, Claude produces notably flat event escalation, GPT over-indexes on dream sequences, and Gemini defaults to external character description. We find that AI-generated stories cluster in a shared region of narrative space, while human-authored stories exhibit greater diversity. More broadly, these results suggest that differences in underlying narrative construction, not just writing style, can be used to separate human-written original works from AI-generated fiction.

Figures & tables

Figure 1: Overview of the StoryScope pipeline. Stories are converted into structured templates, then compared across sources writing to the same prompt to induce discriminative narrative features, and finally featurized across the full corpus for downstream detection and authorship experiments. Story inspired by "Tiny and the Monster" ( Sturgeon, 1983 ) .

#	Feature	Question	Dim.	Type	Response Options
AI
1	Thematic Explicitness and Moralizing	How explicitly does the story articulate its themes or morals?	SIT	scale	1–5
2	Agency in Resolution $\rightarrow$ protagonist choice	Is resolution driven by protagonist’s choices or external events?	PLT	cat	protagonist_choice, mixed, external_fate
3	Narratorial Thematic Commentary $\rightarrow$ yes	Does the narrator explicitly comment on themes beyond characters’ perspectives?	SIT	binary	no, yes
Human
1	Intertextual Strategy Types $\rightarrow$ explicit named reference	What kinds of intertextual engagement does the story employ?	SIT	multi	explicit named, retelling, pastiche, myth/religion, self referential

Table 1: Examples of core features, see AI core features in Table 14 , human in Table 15 , and all 30 with human/AI means in Table 16 . NarraBench dimensions include situatedness (SIT), plot (PLT), revelation (REV) and temporal structure (TMP).

Narrative features (ours)
Method	Size	F1	AP
Narrative	257	93.2	.959
Core Only	30	84.8	.828
Core+FP	101	91.1	.934
Style ablations
Narr. + Style	304	96.0	.982

Table 2: Binary human vs. AI classification results. F1 = macro-F1 (%); AP = AUPRC.

Figure 2: Projection of narrative feature vectors onto the first two linear discriminant components. Human writing occupies a distinct region; the five AI models cluster together. Claude is the most distinct of the five AI models, Gemini and DeepSeek the nearest neighbors.

Narrative features (ours)
Method	Size	F1	Acc.
Narrative	257	68.4	68.4%
Core Only	30	46.5	46.8%
Core+FP	101	63.4	63.6%
Style ablations
Narr. + Style	304	77.3	77.3%

Table 3: Six-way authorship attribution on the test set. F1 = macro-F1 (%). As AI models converge, the attribution task proves much more difficult than the binary detection task.

Figure 3: Confusion matrix for authorship attribution (narrative model) as a percentage (%). Misclassifications concentrate among AI models, particularly DeepSeek–Gemini–Kimi.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Value
Prompts / human stories	10,272
Target words (mean / median)	6,242 / 5,000
Target words (IQR)	3,000–7,000
Human story words (mean / median)	6,403 / 5,035
Human story words (IQR)	2,930–7,650

Appendix

Table 4: Human-corpus summary. All stories extracted from Books3.

Model	$n$	Coverage	Mean words	Median words	Mean $\Delta$ words	Mean abs. % err.	Within 10%
Gemini 3 Flash	10,261	99.9%	3,155	2,967	-3,089	36.9%	19.4%
GPT-5.4	10,272	100.0%	6,651	6,515	409	40.2%	9.6%
Claude Sonnet 4.6	10,259	99.9%	6,817	6,187	575	16.6%	38.4%
DeepSeek V3.2	10,272	100.0%	2,946	2,693	-3,296	39.9%	12.8%
Kimi K2.5	10,272	100.0%	3,274	3,005	-2,968	35.2%	15.6%

Appendix

Table 5: Length and coverage statistics for AI-generated stories in the finalized six-model dataset. Word-target adherence is computed against the explicit “approximately

N

words” instruction embedded in each prompt;

\Delta

is signed model-word-count minus target-word-count.

Figure 4: Boxplots of story lengths across all stories in the finalized six-source dataset, shown separately for the human corpus and each model. Boxes show the interquartile range, center lines show medians, and whiskers show the non-outlier range.

Type	Count	Feature	Details
Categorical	124	Dominant narrative person ( Perspective )	Question: What is the primary grammatical person used by the main narrator across the story?
			Description: Identify the default pronoun/verb alignment in narrative (not dialogue): ‘I/we’, ‘you’, ‘he/she/they’. If no dominant usage, mark ‘mixed_no_dominant’.
			Answer choices: ‘first_person’; ‘second_person’; ‘third_person’; ‘mixed_no_dominant’.
Ordinal	59	Secondary character density ( Agents )	Question: How many distinct secondary/supporting characters appear in the story?
			Description: Include minor or functional characters; exclude major cast from AGENT_ID_001 . Map counts to bins.
			Answer choices: ‘0–1’; ‘2–3’; ‘4–7’; ‘8+’.

Appendix

Table 6: Distribution of feature types in the feature taxonomy. An example feature for each type, with the question, description, and answer given, to illustrate the type of features searched for.

Comparison	Encoded exact agreement (%)	Cohen’s $\kappa$
Annotator 1 vs model	91.67	0.91
Annotator 2 vs model	79.86	0.77
Mean human vs model	85.76	0.84
Human vs human	76.85	0.74

Appendix

Table 7: Human validation on 240 story-feature items across 12 stories.

Dimension only			Leave-one-dimension-out
Dimension	Binary	Six-way	Dimension	Binary	Six-way
Agents	80.2	46.0	Agents	92.0 ( $-$ 1.2)	66.3 ( $-$ 2.1)
Situatedness	77.3	41.3	Situatedness	92.3 ( $-$ 0.8)	66.1 ( $-$ 2.3)
Plot	74.5	40.5	Perspective	92.6 ( $-$ 0.6)	67.3 ( $-$ 1.2)
Perspective	71.3	32.3	Setting	92.8 ( $-$ 0.4)	66.8 ( $-$ 1.7)
Setting	70.6	38.4	Events	92.8 ( $-$ 0.4)	67.4 ( $-$ 1.0)

Appendix

Table 8: Per-dimension ablations of the narrative model, macro-F1 (%). Left: trained on one NarraBench dimension only. Right: trained on all narrative features except that dimension, with the change relative to the full narrative model in parentheses.

Model	Exact 13-gram	Near-verbatim
GPT-5.4	64 (0.63%)	7 (0.07%)
DeepSeek V3.2	58 (0.57%)	13 (0.13%)
Kimi K2.5	44 (0.43%)	7 (0.07%)
Gemini 3 Flash	119 (1.17%)	32 (0.32%)
Claude Sonnet 4.6	67 (0.66%)	14 (0.14%)

Appendix

Table 9: Memorization-risk rates by model.

Variant	Original test macro-F1	Low-risk test macro-F1	$\Delta$
Narrative + Style (binary)	96.00	96.06	+0.06
Narrative + Style (multiclass)	77.29	77.56	+0.27
Narrative (binary)	93.18	93.23	+0.05
Narrative (multiclass)	68.42	68.68	+0.26

Appendix

Table 10: Sensitivity of the final detectability results after dropping all high-risk prompts.

Model	Setting	Macro-F1 (%)	Human-F1	Bal. Acc.	AUPRC
Length only	Full test	55.9	0.330	0.597	0.261
Style Only	Full test	85.8	0.766	0.871	0.867
Style Only	Length-matched test	86.8	0.859	0.869	0.956
Narrative	Full test	93.2	0.886	0.932	0.959
Narrative	Length-matched test	93.2	0.929	0.932	0.990
Narrative + Style	Full test	96.0	0.933	0.956	0.982

Appendix

Table 11: Held-out final-test length-confound audit. “Length only” is a logistic regression trained on train+val story word counts alone. “Length-matched test” reuses the frozen final binary models but evaluates them on a 2,754-story subset (1,377 human, 1,377 AI) formed by stratifying AI stories into human-derived length deciles and sampling without replacement.

	Hum.	Cla.	GPT	Gem.	DS	Kimi
Narr. + Style	0.93	0.89	0.82	0.67	0.66	0.67
Narrative	0.89	0.77	0.73	0.60	0.57	0.55

Appendix

Table 12: Per-class F1 for six-way attribution. Human writing is easiest to classify, while Gemini, DeepSeek, and Kimi have the lowest performance.

Tail	Human	Claude	GPT	DeepSeek	Kimi	Gemini
Top 1%	42	5	5	11	6	14
Top 5%	180	33	34	58	64	45
Top 10%	340	66	71	127	135	88

Appendix

Table 13: Composition of the rarest test-set stories by source. Counts are out of 8,262 test stories (1,377 per source). Human stories are overrepresented in all tails relative to a

\frac{1}{6}

baseline, but AI stories are present throughout.

Figure 5: Per-story narrative rarity percentiles by author on the held-out test set. Solid lines show means; dashed lines show medians. Human stories are shifted toward higher rarity but all distributions overlap substantially.

#	Feature	Question	Dim	Type	Response Options
1	Thematic Explicitness and Moralizing	How explicitly does the story articulate its themes or morals?	SIT	scale	1–5
2	Dominant Emotional Expression $\rightarrow$ embodied	How are characters’ emotions most commonly conveyed?	AGENT	cat	explicit labels, embodied metaphors, behavioral cues, ambiguous
3	Thematic Unity	To what extent do subplots and flourishes serve a central thematic concern?	PLT	scale	1–5
4	Dominant Sensory Modalities $\rightarrow$ olfactory	Which sensory modalities does the story most frequently engage?	SET	multi	visual, auditory, olfactory, tactile, gustatory, kinesthetic
5	Character Introduction $\rightarrow$ external description	What narrative device primarily introduces the central character?	AGENT	cat	external_desc, in-action, in-dialogue, inner_thought, others_reports
6	Setting as Psychological Mirror	To what degree does physical environment mirror characters’ inner states?	SET	scale	1–5

Appendix

Table 14: Core AI-characterizing features (20). Dim = NarraBench dimension prefix; Type: scale = 1–5 Likert, cat = categorical, ord = ordinal, multi = multi-select, binary = yes/no. Arrow (

\rightarrow

) indicates the specific option value that is elevated for AI.

#	Feature	Question	Dim	Type	Response Options
1	Intertextual Strategy Types $\rightarrow$ explicit named reference	What kinds of intertextual engagement does the story employ?	SIT	multi	explicit named, retelling, pastiche, myth/religion, self referential
2	Frequency of Direct Reader Address	How often does the text directly address the reader?	PER	ord	never, occasional asides, frequent/structural
3	Reference Explicitness $\rightarrow$ balanced mix	Are intertextual gestures explicit or diffuse?	SIT	cat	none, explicit named, implicit echoes, balanced mix
4	Depth of Recontextualization After Surprise	How extensively does a revelation force reinterpretation of earlier scenes?	REV	scale	1 (none)–5 (complete re-reading)
5	Dialogue-to-Narration Proportion	What proportion of text is direct dialogue vs. narration?	PER	scale	1 (no dialogue)–5 (dialogue dominates)
6	Fourth-Wall Permeability	To what extent does the story break the boundary between story-world and reader?	SIT	ord	1 (no breaking)–4 (radical violations)

Appendix

Table 15: Core human-characterizing features (13), Arrow (

\rightarrow

) indicates the specific option value elevated for human authors. Abbreviations as in Table 14 .

Feature	Human	AI	Gap
AI-elevated: Thematic over-determination
Thematic Explicitness & Moralizing s	3.28	3.94	$-$ 0.65
Moral / Philosophical Weighting s	3.26	3.68	$-$ 0.42
Thematic Unity s	4.41	4.74	$-$ 0.33
Narratorial Thematic Commentary $\rightarrow$ yes	52%	77%	$-$ 25
Dialogue Function $\rightarrow$ philosophical debate	34%	59%	$-$ 25

Appendix

Table 16: Core features separating human from AI writing, grouped by theme. Features marked s are 1–5 Likert scales (values are means); o are ordinal scales (values are means over integer codes);

\rightarrow

indicates a specific categorical/binary option (values are prevalence %). Gap = Human

-

AI; negative gaps indicate AI-elevated features. AI column averages across all five models.

Author	#	Feature	Dim	SHAP	Uniq.
Human	1	Character introduction $\rightarrow$ in-dialogue	AGENT	0.110	21.4
	2	Breadth of focalization $\rightarrow$ single focal	PER	0.083	15.7
	3	Narrator address mode $\rightarrow$ no direct address	SIT	0.069	12.6
	4	Overall revelation pacing $\rightarrow$ back-loaded	REV	0.094	7.7
	5	Literary ambition $\rightarrow$ crossover genre	SIT	0.116	6.8
	+ 27 more (visibility of withholding, atmospheric techniques, subplot density, naming, twist placement, …)

Appendix

Table 17: Per-source fingerprint features, showing the top 5 (or all, if fewer) ranked by uniqueness ratio. SHAP = mean class SHAP importance; Uniq. = uniqueness ratio vs. next-best class.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 167 papers

Aug 2, 2026

CraftAlign: Feature-Grounded Evaluation and Revision Guidance for AI Stories

Large language models can now generate fluent and complete stories, yet many outputs still feel formulaic and unnatural because of cliches, over-explanation, linear causal progression, and stereotyped endings, an immediately recognizable AI flavor. Existing detection and evaluation methods often stop at source labels or holistic scores, while revision methods typically target predefined issues through localized edits, limiting their ability to support multiple plausible revision strategies or guide story-wide changes in information release, causal organization, and ending treatment. We introduce CraftAlign, a framework that aligns AI stories with the craft of human storytelling by both assessing Human/AI writing patterns and providing revision guidance. CraftAlign comprises two learned modules and an inference-time guidance pipeline. A feature estimator built on Qwen3.5-9B predicts 304 explicit writing features spanning style and narrative. A class-conditional energy model scores the resulting feature configuration against Human and AI writing patterns, conditioning on the original writing prompt when available. At inference time, CraftAlign applies schema-valid structured perturbations, selects changes that move the feature configuration toward the Human writing pattern, and converts them into natural-language guidance for a separate editor to rewrite the full story. Experiments show that CraftAlign accurately distinguishes Human and AI writing patterns and that its guidance outperforms revision baselines across editors and in a human study.

Yang Yang, Boyun Xu, Shaofeng Liang +5

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511400, China · Institute of Deep Perception Technology, JITRI, Wuxi 214000, China

AI-Generated Text Detection Language Model Generation Evaluation Generative AI Evaluation Narrative Generation

Ask PDF

Sep 14, 2026

SlopShape: Identifying AI-Generated Commercial Web Content

Word-level detectors identify unedited AI-generated text almost perfectly, but the literature documents their brittleness under rewording, and a word-level score neither characterizes a text nor identifies which AI model wrote it. We ask whether AI-generated text can be identified one level deeper, from structural signatures: how information is presented, in what order, with what evidence, and in what voice. We replicate StoryScope (Russell et al., 2026), which showed such patterns for AI-generated fiction, on commercial content: 2,250 pre-ChatGPT human blog posts from 268 company domains against 11,250 AI mirrors from five frontier models. A 203-feature instrument, applied by an LLM and validated in a human gold-annotation session (human-human kappa 0.939, human-model 0.951), detects AI posts from its 176 structural features alone at 97.0 macro-F1 on held-out companies, nearly unchanged (96.1) when every AI post is reworded by its own model. The signal characterizes and attributes: AI posts share a tidy, self-announcing shape, 68.6% are attributed to the correct source against a 16.7% chance rate, and human posts occupy rare structural configurations. All effects replicate StoryScope's, consistent in direction and at least as large in magnitude. We release pipeline, instrument, prompts, code, and aggregate artifacts.

Jochen Madler

Sitefire

AI-Generated Text Detection Language Model Fingerprinting Stylometry Authorship Attribution Robustness of Machine-Generated Text Detection

Ask PDF

Jun 21, 2026

CASPER in the Machine: Insights into Character Variety in LLM-Generated Stories

As LLM-generated text is increasingly used, especially in fictional domains, we explore how much LLM-generated stories differ from human-written stories. In this work, we focus on characters. We borrow definitions from narratology to analyze eight intricate dimensions of character, such as stylization and wholeness. These dimensions consider more than just basic characteristics. They assess how characters are portrayed within their stories. After automatically inferring categories of characters within both LLM and human-written stories, we compare and contrast these two sets of stories. We consider the following overarching questions: (1) Do LLMs and human-written stories have similar characters? and (2) Do LLMs generate stories with a variety of characters? Our analysis includes research questions that focus on stories generated by popular LLMs and recently published human-written stories. We describe a number of interesting similarities, differences and key takeaways.

Anneliese Brei, Abhisheik Sharma, Nicholas Sanaie +2

UNC Chapel Hill · Georgia Institute of Technology · University of Michigan

LLM Evaluation Diverse Text Generation Text Generation

Ask PDF

Abstract

AI-Generated Text Detection

Explore similar work

CraftAlign: Feature-Grounded Evaluation and Revision Guidance for AI Stories

SlopShape: Identifying AI-Generated Commercial Web Content

CASPER in the Machine: Insights into Character Variety in LLM-Generated Stories