What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces

Birger Moëll

cs.CLApr 7, 2026

What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces

Authors: Birger Moëll

Organizations: Department of Linguistics and Philology, Uppsala University, Uppsala, Sweden

Abstract

What makes writing "good" remains a persistent question in literary studies and computational linguistics. We present a two-study investigation of how reasoning-enabled LLMs evaluate literary quality. In Study 1, we construct a benchmark of 30 real texts spanning six quality tiers, from canonical literature to anonymous forum posts, and extract the model's implicit theory of quality from its reasoning traces. Across five DeepSeek replications, the model achieves 79.3% mean tier-classification accuracy. The traces reveal a consistent stated theory: the model values intentionality over correctness, prioritizing craft, depth, and distinctive voice. A familiarity experiment with style-matched but unrecognizable passages suggests that source recognition may inflate scores, although this is confounded by genuine quality differences between canonical originals and researcher-written pastiches. In Study 2, we probe this theory through systematic degradation of five canonical prose passages. We apply six manipulations - vocabulary simplification, rhythm flattening, imagery removal, voice genericization, structure simplification, and combined degradation - and reevaluate each version. Vocabulary simplification causes the smallest quality loss (0.41 +/- 0.46 points), far below structure (2.78) or voice (2.34) loss. Combined degradation is devastating (-5.64) but subadditive. An exploratory comparison with Qwen QwQ shows the same broad qualitative pattern. Together, these studies suggest that LLM judgments of writing quality are holistic, author-specific, and more sensitive to structural than lexical features, with implications for automated writing feedback and computational aesthetics.

Figures & tables

Tier	Excerpt (Source)
Literary	“124 was spiteful. Full of a baby’s venom. The women in the house knew it and so did the children…” —Morrison, Beloved
Prestige	“Frank Sinatra, holding a glass of bourbon in one hand and a cigarette in the other, stood in a dark corner of the bar…” —Talese, Esquire
News	“Federal Reserve officials signaled on Wednesday that interest rates would likely need to stay elevated for longer…” —Reuters style
Blog	“To do something well you have to like it. That idea is not exactly novel…” —Paul Graham
Social	“AITA for refusing to give my sister my wedding date?…” —Reddit
Chan	“>be me >wagecuck at fast food place >customer asks for burger with no pickles…” —4chan

Table 1: Real text examples from each quality tier.

Figure 1: Readability metrics by tier. News writing is most complex; literary prose is moderate. Complexity does not predict quality.

Figure 2: Confusion matrix. Literary and news achieve perfect recognition. Didion and Orwell are “promoted” from journalism to literary. Chan blurs with social media.

Tier	Texts	Recognized	Rate
Literary	5	4	80%
Prestige Journalism	5	2	40%
Quality News	5	0	0%
Blog	5	0	0%
Social Media	5	0	0%
Chan	5	0	0%

Table 2: Source recognition by tier. The model explicitly identifies the author or work in 6 of 30 reasoning traces, concentrated entirely in the top two tiers.

Style Source	Orig.	Style	$\Delta$	Orig. Tier	Style Tier
Morrison	9	9	0	Literary	Literary
García Márquez	10	8	$-2$	Literary	Prestige
Woolf	10	8	$-2$	Literary	Prestige
Ishiguro	9	7	$-2$	Literary	Prestige
Baldwin	8	8	0	Literary	Prestige
Mean	9.2	8.0	$\mathbf{-1.2}$

Table 3: Exploratory single-run familiarity comparison. “Orig.” = original canonical text score from the initial benchmark run; “Style” = style-matched unrecognizable text score from the initial familiarity run. Style-matched texts score 1.2 points lower on average and 4 of 5 drop a full tier in this snapshot. Note that the style-matched texts are researcher-written pastiches and genuinely lower quality than the originals; the score gap reflects both real quality differences and possible recognition effects.

Figure 3: Left: Dimension scores by tier. Right: Gap between high and low tiers. Craft (+5.2) and depth (+4.6) most distinguish quality; voice (+3.1) is least discriminating.

Version	Text (Morrison, Beloved )
Original	“124 was spiteful. Full of a baby’s venom. The women in the house knew it and so did the children…”
Vocabulary	“124 was mean. Full of a baby’s anger. The women in the house knew it…”
Voice	“The house at 124 was haunted. It had a malevolent presence. The women and children who lived there knew about it…”
Combined	“The house was bad. It had bad feelings. The women knew. The children knew too…”

Table 4: Example degradations of Morrison’s Beloved opening, isolating vocabulary, voice, and combined effects.

Figure 4: Average quality loss by degradation type across replications. Vocabulary simplification causes the smallest loss (0.41 points). Structure and voice cause the most single-feature damage. Combined degradation is catastrophic (5.6 points).

Figure 5: Author sensitivity heatmap (mean across replications). Woolf is devastated by voice genericization ( $-5.7$ ); Fitzgerald by structure simplification ( $-6.2$ ) and rhythm flattening ( $-5.4$ ). All authors show minimal sensitivity to vocabulary simplification.

Figure 6: Quality trajectories. All authors start at 9–10 but follow different degradation paths. All converge at 2–5 under combined degradation.

Degradation	Mean Loss	SD	95% CI	Cohen’s $d$	$p$
Vocabulary	0.41	0.46	[0.24, 0.59]	0.90	$3.1\times 10^{-4}$
Rhythm	1.99	2.01	[1.23, 2.78]	0.99	$4.8\times 10^{-7}$
Imagery	0.56	0.43	[0.39, 0.72]	1.29	$1.5\times 10^{-5}$
Voice	2.34	2.06	[1.59, 3.16]	1.14	$9.5\times 10^{-7}$
Structure	2.78	2.04	[2.04, 3.60]	1.36	$1.2\times 10^{-7}$
Combined	5.64	1.88	[4.87, 6.33]	3.00	$6.0\times 10^{-8}$

Table 5: Degradation effects across replications with DeepSeek R1. All degradation types produce statistically significant quality loss (

p<0.001

), but vocabulary loss (0.41) is an order of magnitude smaller than structure (2.78) or voice (2.34) loss.

Degradation	DeepSeek R1 Loss	QwQ Loss
Vocabulary	0.41	0.40
Rhythm	1.99	1.20
Imagery	0.56	0.60
Voice	2.34	4.50
Structure	2.78	3.20
Combined	5.64	3.80

Table 6: Cross-model comparison of degradation sensitivity. Both models show minimal vocabulary loss and substantial structure/voice loss. For QwQ, the Fitzgerald voice evaluation failed once with a connection error, so the QwQ voice mean is computed from the four available passages in that exploratory run.

Computational Literary Studies

Computational Literary Studies uses data analysis and language technologies to examine literary style, themes, narratives, influence, and change.

See all 49 papers

SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions
May 8, 2026Tianyu Wang, Nianjun ZhouComputational Literary Studies LLM Evaluation
Mercy University, Math & Computer Science Department · Dobbs Ferry, NY, USA · IBM T.J. Watson Research Center +1
Calibrated Surprise: An Information-Theoretic Account of Creative Quality
Apr 29, 2026Bo Zou, Chao XuCreativity Assessment Computational Creativity
The LALA Benchmark: Evaluating Literary and Linguistic Analysis in LLMs
Oct 25, 2026Michael Foster, Barbara McGillivray, Gard B. Jenset +7LLM Evaluation Language Model Post-Training
LALA Labs · King's College London · Ghent University +5

May 8, 2026J/K move · Enter open · S save

SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions

Tianyu Wang, Nianjun Zhou

Mercy University, Math & Computer Science Department · Dobbs Ferry, NY, USA · IBM T.J. Watson Research Center+1

Computational Literary Studies LLM Evaluation LLM-as-a-Judge Automated Evaluation

Open paper AskPDF arXiv

Evaluating literary quality requires assessing interpretive dimensions such as cultural representation, emotional depth, and philosophical sophistication that resist straightforward computational measurement. We introduce SAGE, a hierarchical evaluation framework that decomposes literary quality into ontology-grounded interpretive dimensions assessed through structured large language model evaluation with multi-round iterative reflection and independent validation. We validate the framework on 100 short stories (50 canonical works, 30 pulp fiction, 20 LLM-generated narratives) across three analytical layers (cultural, emotional-psychological, existential-philosophical) using dual-mode assessment. Across 600 evaluations, the framework achieves 98.8% score convergence and greater than 94% inter-rater agreement, with near-perfect mode invariance between content-based and metadata-based evaluation. Statistical analysis reveals a consistent genre hierarchy (Canonical > Pulp > LLM, all p<0.001) with layer-specific discrimination: cultural critique and philosophical depth exhibit very large effect sizes (Cohen's d>2.4), while emotional representation shows smaller gaps (d=1.68), suggesting that affective patterns are more learnable from training data than critical stance or philosophical depth. Cross-layer correlations (r=0.649-0.683) confirm the three dimensions capture empirically distinguishable quality facets. These findings demonstrate that theory-driven LLM evaluation can achieve measurement-grade reliability and support systematic identification of where current generative models fall short of human literary production, with direct implications for scalable automated evaluation of open-ended text generation.

Abstract

Computational Literary Studies

Explore similar work

SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions

Calibrated Surprise: An Information-Theoretic Account of Creative Quality

The LALA Benchmark: Evaluating Literary and Linguistic Analysis in LLMs