SAGE: A Hierarchical Framework for Evaluating Interpretive Literary Quality in Narratives

Tianyu Wang; Nianjun Zhou

cs.CLSep 6, 2026

SAGE: A Hierarchical Framework for Evaluating Interpretive Literary Quality in Narratives

Authors: Tianyu Wang, Nianjun Zhou

Organizations: Mercy University Math & Computer Science Department Dobbs Ferry, NY, USA · IBM T.J. Watson Research Center Yorktown Heights, NY, USA

Abstract

Assessing the literary quality of narratives requires evaluating interpretive dimensions (cultural representation, emotional depth, and philosophical engagement) that existing NLG metrics cannot measure. We introduce SAGE, a six-layer evaluation framework that separates rule-based assessment of observable textual properties from LLM-based evaluation of interpretive qualities drawn from cultural theory, affect theory, and existentialist philosophy. Each interpretive layer is assessed through multi-round iterative LLM evaluation with independent cross-validation, achieving measurement-grade reliability (98.8% convergence, >94% inter-rater agreement) stable across evaluator models. Validated on 600 evaluations across 100 short stories, our central finding is a systematic capability boundary: emotional-psychological representation approaches human levels, while cultural critique and philosophical depth exhibit approximately double the gap. LLM-generated narratives score below even commercial genre fiction on all three layers. We interpret this as a boundary between pattern-reproducible literary capacities learnable from training corpora and stance-requiring ones demanding cultural positioning and philosophical engagement that pattern matching alone cannot provide.

Figures & tables

Figure 1: SAGE architecture. Layers 1–3 assess textual properties through rule-based metrics; Layers 4–6 evaluate interpretive dimensions through a dual-track LLM pipeline (iterative evaluator + independent validator) in two modes.

L	Focus	Key Dimensions
Rule-Based (L1–L3)
1	Lexical	Vocabulary richness, syntactic complexity
2	Narrative	Entity coherence, emotional arcs
3	Thematic	Topic diversity, semantic networks
LLM-Based (L4–L6)
4	Cultural	IPD, CVP, CSP, CPC

Table 1: Six-layer hierarchical evaluation framework.

Work	Dims.	Values	Layer
“Hills Like White Eleph.”	AC / PI	H / L	L5
Mrs. Dalloway	AC / PI	H / H	L5
Heart of Darkness	IPD / CVP	H / L	L4
Things Fall Apart	IPD / CVP	H / H	L4
The Stranger	LP / MR	H / L	L6
Crime and Punishment	LP / MR	H / H	L6

Table 2: Selected exemplars demonstrating dimension non-redundancy across layers. H = High, M = Medium, L = Low. Dissociating pairs highlighted in bold.

Category	Stories	Evals.	Quality
Canonical	50	300	Academically recognized
Pulp Fiction	30	180	Commercial genre
LLM-Generated	20	120	Multi-model, stratified
Total	100	600	3 layers $\times$ 2 modes

Table 3: Evaluation corpus and matrix.

Metric	Value	Threshold
Evaluation success rate	100.0%	—
Convergence rate (R4 $\to$ R5)	98.8%	$\|\Delta\|<0.3$
Inter-rater agreement	$>$ 94%	MAD $<$ 0.5
Mode invariance (max $\|\Delta\|$ )	0.05	$p>0.05$

Table 4: System reliability metrics across 600 evaluations.

Figure 2: Genre comparison across three analytical layers. Error bars indicate standard deviations. All canonical vs. LLM-generated differences significant at $p<0.001$ .

Layer	Canon.	Pulp	LLM	$d$	$p$ (C vs L)
L4 Cultural	3.96 $\pm$ .41	3.83 $\pm$ .36	2.55 $\pm$ .62	2.68	$<$ 0.001
L5 Emotional	4.15 $\pm$ .31	4.04 $\pm$ .24	3.36 $\pm$ .59	1.68	$<$ 0.001
L6 Existential	3.95 $\pm$ .57	3.68 $\pm$ .56	2.59 $\pm$ .57	2.40	$<$ 0.001
Average	4.02	3.85	2.83	—	—

Table 5: Genre scores by layer (mean

\pm

SD) with Cohen’s

d

and

p

-values for Canonical vs. LLM pairwise comparison. Canon. vs. Pulp: all layers

p>0.05

(n.s.); Pulp vs. LLM: all layers

p<0.001

Layer Pair	Pearson $r$	Spearman $\rho$	$p$
L4 $\times$ L5	0.649	0.514	$<$ 0.001
L4 $\times$ L6	0.683	0.598	$<$ 0.001
L5 $\times$ L6	0.680	0.621	$<$ 0.001

Table 6: Cross-layer correlation analysis (

N

=200).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Form	IPD	CVP	CSP	CPC
Things Fall Apart (Achebe)	Novel	H	H	H	H
Heart of Darkness (Conrad)	Novel	H	L	M	M
The Great Gatsby (Fitzgerald)	Novel	L	L	H	L
Animal Farm (Orwell)	Novel	H	L	L	L
“Everyday Use” (Walker)	Short	H	H	H	H
“Lady with the Dog” (Chekhov)	Short	L	L	H	L

Appendix

Table 7: L4 exemplar matrix. H = High, M = Medium, L = Low.

Work	Form	AC	PI	EG	ENC
“Hills Like White Elephants” (Hemingway)	Short	H	L	L	H
“The Metamorphosis” (Kafka)	Short	M	L	L	H
“Runaway” (Munro)	Short	H	H	H	H
“Miss Brill” (Mansfield)	Short	M	H	H †	H
“Cathedral” (Carver)	Short	L $\to$ H	L	L	H
Mrs. Dalloway (Woolf)	Novel	H	H	H	H

Appendix

Table 8: L5 exemplar matrix. H = High, M = Medium, L = Low. † EG operates at narrative apparatus level, not character self-report.

Work	Form	LP	MR	HC	ME
The Stranger (Camus)	Novel	H	L	H	H
Crime and Punishment (Dostoevsky)	Novel	H	H	H	H
Germinal (Zola)	Novel	L	M	H	L
Middlemarch (Eliot)	Novel	L	H	H	M
Never Let Me Go (Ishiguro)	Novel	L ∗	M	H	H
“The Guest” (Camus)	Short	H	Susp. ‡	H	H

Appendix

Table 9: L6 exemplar matrix. H = High, M = Medium, L = Low. ∗ LP structurally inaccessible (clones possess no future). † ME collapses at death but constitutes the story’s subject throughout. ‡ MR suspended: narrative refuses moral adjudication.

R	Focus	Task
1	Initial	Extract layer-relevant content; score all 4 dimensions (1.0–5.0) with confidence and textual evidence
2	Bias check	L4: hallucination check. L5: projection bias (emotional norms). L6: Western framework imposition
3	Boundary	Verify no cross-layer contamination
4	Evidence	Adjust confidence (not scores) if evidence is sparse or ambiguous
5	Final	Confirm or adjust scores; convergence: $\|\Delta\|<0.3$ from R4 to R5

Appendix

Table 10: Five-round iterative evaluation protocol (all layers).

Score	Criteria
4.5–5.0	Exceptional: rich, multi-layered representation; extensive evidence; demonstrable mastery
4.0–4.5	Strong: substantial representation; clear evidence; depth and nuance
3.0–3.5	Moderate: some representation; adequate but not extensive evidence
2.0–2.5	Weak: minimal representation; surface-level; sparse evidence
1.0–1.5	Absent/Negligible: dimension effectively absent; no meaningful evidence

Appendix

Table 11: Scoring rubric applied uniformly across all layers.

Component	Design
System role	Expert analyst for the target layer; dimension definitions with theoretical citations
Core principles	(1) Evidence Sovereignty: all scores cite textual evidence. (2) Layer Boundary: evaluate only target-layer content. (3) Projection Awareness: no norm imposition. (4) Dimension Independence: score each dimension separately
Mode instruction	Content-limit: text only, no prior knowledge. Title-limit: scholarly consensus from title/author only
R1 output schema	JSON: extracted content, per-dimension score (1.0–5.0) / confidence (1–5) / reasoning / evidence strength, overall score
R2 bias check	L4 : hallucination check (fabricated cultural/historical claims). L5 : projection bias (Western emotional norms). L6 : framework imposition (Sartre/Camus onto non-Western texts; overlooking Buddhist/Confucian/Daoist depth)
R3 boundary	Verify no cross-layer contamination (e.g., L5 must not score cultural power or philosophical claims)

Appendix

Table 12: Prompt design: components and layer-specific instantiations.

Figure 3: Score convergence trajectories across five iterative rounds for each layer (L4–L6) and genre category. Scores stabilize by Rounds 3–4; canonical and pulp fiction consistently score higher than LLM-generated stories.

Figure 4: Effect size (Cohen’s $d$ ) for Canonical vs. LLM-generated stories across three layers. All values exceed the large-effect threshold ( $d=0.8$ , dashed line).

Layer	Pearson $r$	Spearman $\rho$	MAD	GPT $\mu$ / Claude $\mu$
L4 Cultural	0.812	0.799	0.705	3.46 / 2.87
L5 Emotional	0.636	0.751	0.622	3.83 / 3.39
L6 Existential	0.797	0.778	0.614	3.44 / 2.97
Overall	0.775	—	0.647	—

Appendix

Table 13: Cross-model consistency: GPT-5-mini vs. Claude Sonnet on 30-story subset (59 pairs per layer).

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions
May 8, 2026Tianyu Wang, Nianjun ZhouComputational Literary Studies LLM Evaluation
Mercy University, Math & Computer Science Department · Dobbs Ferry, NY, USA · IBM T.J. Watson Research Center +1
What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces
Apr 7, 2026Birger MoëllComputational Literary Studies LLM-as-a-Judge
Department of Linguistics and Philology, Uppsala University, Uppsala, Sweden
NARRA-Gym for Evaluating Interactive Narrative Agents
May 8, 2026Yue Huang, Yuchen Ma, Jiayi Ye +14Language Model Generation Evaluation Interactive Storytelling
University of Notre Dame · 2LMU Munich · 11Munich Center for Machine Learning +8

May 8, 2026J/K move · Enter open · S save

SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions

Tianyu Wang, Nianjun Zhou

Mercy University, Math & Computer Science Department · Dobbs Ferry, NY, USA · IBM T.J. Watson Research Center+1

Computational Literary Studies LLM Evaluation LLM-as-a-Judge Automated Evaluation

Open paper AskPDF arXiv

Evaluating literary quality requires assessing interpretive dimensions such as cultural representation, emotional depth, and philosophical sophistication that resist straightforward computational measurement. We introduce SAGE, a hierarchical evaluation framework that decomposes literary quality into ontology-grounded interpretive dimensions assessed through structured large language model evaluation with multi-round iterative reflection and independent validation. We validate the framework on 100 short stories (50 canonical works, 30 pulp fiction, 20 LLM-generated narratives) across three analytical layers (cultural, emotional-psychological, existential-philosophical) using dual-mode assessment. Across 600 evaluations, the framework achieves 98.8% score convergence and greater than 94% inter-rater agreement, with near-perfect mode invariance between content-based and metadata-based evaluation. Statistical analysis reveals a consistent genre hierarchy (Canonical > Pulp > LLM, all p<0.001) with layer-specific discrimination: cultural critique and philosophical depth exhibit very large effect sizes (Cohen's d>2.4), while emotional representation shows smaller gaps (d=1.68), suggesting that affective patterns are more learnable from training data than critical stance or philosophical depth. Cross-layer correlations (r=0.649-0.683) confirm the three dimensions capture empirically distinguishable quality facets. These findings demonstrate that theory-driven LLM evaluation can achieve measurement-grade reliability and support systematic identification of where current generative models fall short of human literary production, with direct implications for scalable automated evaluation of open-ended text generation.

Abstract

Language Model Generation Evaluation

Explore similar work

SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions

What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces

NARRA-Gym for Evaluating Interactive Narrative Agents